Merge pull request #96 from OSINT-TECHNOLOGIES/rolling

Stabilized v1.2

Author: OSINT-TECHNOLOGIES

.readthedocs.yaml

@@ -0,0 +1,16 @@
+version: 2
+
+build:
+  os: ubuntu-20.04
+  tools:
+    python: "3.10"
+  jobs:
+    pre_build:
+      - pip install mkdocs mkdocs-material
+
+python:
+  install:
+    - requirements: requirements.txt
+
+mkdocs:
+  configuration: docs/dpulse-docs/mkdocs.yml

README.md

@@ -1,7 +1,7 @@
 # DPULSE - Domain Public Data Collection Service 
 
 <p align="center">
-  <img src="https://github.com/OSINT-TECHNOLOGIES/dpulse/assets/77023667/fed8d368-7309-4aaa-a82e-2f0b277122de">
+  <img src="https://github.com/user-attachments/assets/e49c764a-5039-4a54-b002-6b625f0b371b">
 </p>
 
 <p align="center">
@@ -20,32 +20,27 @@
 
 > Attention! DPULSE is a research tool. It is not intended for criminal activities! Use DPULSE only on allowed domains and for legal purposes!
 
+> Please, before creating an issue or DMing developer about DPULSE, make sure that your problem or question is not covered with [DPULSE documentation](https://dpulse.readthedocs.io)
+
 # Repository map
 
 ## What to visit?
 
 | What do you want to see? | Link |
 | --- | --- |
-| What is DPULSE? | [See "About DPULSE" page](https://github.com/OSINT-TECHNOLOGIES/dpulse?tab=readme-ov-file#about-dpulse) |
-| Where I can find some demo and use-cases? | [See "Demo and use-cases" page](https://github.com/OSINT-TECHNOLOGIES/dpulse?tab=readme-ov-file#dpulse-demo-and-use-cases) |
-| I want to read project documentation | [See DPULSE wiki](https://github.com/OSINT-TECHNOLOGIES/dpulse/wiki/DPULSE-WIKI) |
+| I want to read project documentation | [See DPULSE Readthedocs Page](https://dpulse.readthedocs.io) |
+| I want to see developer's contacts | [See "Contact developer" page on Readthedocs](https://dpulse.readthedocs.io/en/latest/contact_dev/#) |
 | I want to see project roadmap and future development plans | [See DPULSE roadmap](https://github.com/users/OSINT-TECHNOLOGIES/projects/1) |
 
 ## What to download?
 
 | Your expectations | Version and link for you |
 | --- | --- |
 | I want to use only tested and stable version of DPULSE | [DPULSE stable ZIP archive](https://github.com/OSINT-TECHNOLOGIES/dpulse/archive/refs/heads/main.zip) |
-| I don't mind to use DPULSE with latest changes and you're OK with bugs and issues  | [DPULSE rolling ZIP archive](https://github.com/OSINT-TECHNOLOGIES/dpulse/archive/refs/heads/rolling.zip) |
+| I don't mind to use DPULSE with latest changes and I'm OK with bugs and issues  | [DPULSE rolling ZIP archive](https://github.com/OSINT-TECHNOLOGIES/dpulse/archive/refs/heads/rolling.zip) |
 | I want to use only one specific version of DPULSE  | [See DPULSE releases page](https://github.com/OSINT-TECHNOLOGIES/dpulse/releases) |
 | I want to see more detailed installation instructions | [See DPULSE installation guides](https://github.com/OSINT-TECHNOLOGIES/dpulse?tab=readme-ov-file#how-to-install-and-run-dpulse)
 
-## How can I contact the developer? 
-
-| Reasons to contact | Links & addresses |
-| --- | --- |
-| I want to talk with developer in person  | DM to osint.technologies@gmail.com |
-| I want to report some bug or issue, or maybe I have some good idea for developer  | [Make a new issue page](https://github.com/OSINT-TECHNOLOGIES/dpulse/issues/new/choose) |
 
 # About DPULSE
 
@@ -157,11 +152,11 @@ If you have problems with starting installer.sh, you should try to use `dos2unix
 
 ### You can start DPULSE and see the main menu on the screen using one of the recommended commands in DPULSE root folder. Don't forget to install all requirements before starting DPULSE
 
-![dpulse_start](https://github.com/user-attachments/assets/49cb476c-d9a5-4ff6-999f-afa0badf0e0d)
+![dpulse_start](https://github.com/user-attachments/assets/9ec0ab73-2206-4d38-bae6-e88656e17f95)
 
 ### After choosing first menu point, you will be able to enter target's URL and case comment, and then you will see scanning progress
 
-![dpulse_running](https://github.com/user-attachments/assets/9a3cd8c2-3281-4d58-b12e-91ced6dbbd91)
+![dpulse_bs](https://github.com/user-attachments/assets/b0ad7827-6dac-4f82-a369-4447a0e1c878)
 
 ### Finally, DPULSE will create report folder which contains case name (basically URL of target), date and time of scan. All report folders are contained in DPULSE root folder
 
@@ -194,6 +189,12 @@ If you have problems with starting installer.sh, you should try to use `dos2unix
 
 ### [by @UndeadSec](https://x.com/UndeadSec/status/1827692406797689032)
 
+### [by @akaclandestine](https://x.com/akaclandestine/status/1875980998418002151?t=Ac-fZ9oe2FLKkTTCClss8g&s=19)
+
+### [by @hdH4dg8](https://x.com/hdH4dg8/status/1876106586869104906?t=Awfas25ukXkblPRt4krSzA&s=19)
+
+### [by @wy88215534](https://x.com/wy88215534/status/1876522251828494794?t=cmwoNCDZv0kMEDMKokcVFA&s=19)
+
 ## Facebook mentions:
 
 ### [by Inteligência Cibernética](https://www.facebook.com/osintbrasil/posts/pfbid037ibycZcBWe2MjtV4HiWvRWxyKei8TJ5Ycfxai4TDNHXuwrYkDGuyjDsPow8WUNbyl)

datagather_modules/crawl_processor.py

@@ -103,6 +103,7 @@ def subdomains_gather(url, short_domain):
         print(Fore.RED + f"Cannot gather subdomains due to error. See journal for details" + Style.RESET_ALL)
         logging.error(f'SUBDOMAINS GATHERING: ERROR. REASON: {e}')
         pass
+        return ['No subdomains were found'], 0
 
 def sm_gather(url):
     response = requests.get(url)
@@ -173,7 +174,7 @@ def domains_reverse_research(subdomains, report_file_type):
         for subdomain in subdomains:
             subdomain_url = "http://" + subdomain + "/"
             subdomain_urls.append(subdomain_url)
-    except (socket.gaierror, requests.exceptions.SSLError, requests.exceptions.ConnectionError) as e:
+    except Exception as e:
         print(Fore.RED + "Some URL seems unreachable! DPULSE will continue to work, but the URL causing the error won't be included in report. See journal for details" + Style.RESET_ALL)
         logging.error(f'SUBDOMAINS URL FORMING: ERROR. REASON: {e}')
         pass
@@ -183,7 +184,7 @@ def domains_reverse_research(subdomains, report_file_type):
             subdomains_ip = ip_gather(subdomain)
             subdomain_ip.append(subdomains_ip)
             subdomain_ip = list(set(subdomain_ip))
-    except (socket.gaierror, requests.exceptions.SSLError, requests.exceptions.ConnectionError) as e:
+    except Exception as e:
         print(Fore.RED + "Some URL seems unreachable! DPULSE will continue to work, but the URL causing the error won't be included in report. See journal for details" + Style.RESET_ALL)
         logging.error(f'SUBDOMAINS IP GATHERING: ERROR. REASON: {e}')
         pass
@@ -194,7 +195,7 @@ def domains_reverse_research(subdomains, report_file_type):
             subdomain_mails.append(subdomain_mail)
             subdomain_social = sm_gather(subdomain_url)
             subdomain_socials.append(subdomain_social)
-    except (socket.gaierror, requests.exceptions.SSLError, requests.exceptions.ConnectionError) as e:
+    except Exception as e:
         print(Fore.RED + "Some URL seems unreachable! DPULSE will continue to work, but the URL causing the error won't be included in report. See journal for details" + Style.RESET_ALL)
         logging.error(f'SUBDOMAINS MAIL/SOCIALS GATHERING: ERROR. REASON: {e}')
         pass

datagather_modules/data_assembler.py

@@ -81,7 +81,12 @@ def data_gathering(self, short_domain, url, report_file_type, pagesearch_flag, k
         print(Fore.GREEN + 'Processing subdomain gathering' + Style.RESET_ALL)
         subdomains, subdomains_amount = cp.subdomains_gather(url, short_domain)
         print(Fore.GREEN + 'Processing social medias gathering' + Style.RESET_ALL)
-        social_medias = cp.sm_gather(url)
+        try:
+            social_medias = cp.sm_gather(url)
+        except:
+            print(Fore.RED + "Social medias were not gathered because of error" + Style.RESET_ALL)
+            social_medias = ['Social medias were not extracted because of error']
+            pass
         print(Fore.GREEN + 'Processing subdomain analysis' + Style.RESET_ALL)
         if report_file_type == 'xlsx':
             subdomain_urls, subdomain_mails, subdomain_ip, sd_socials = cp.domains_reverse_research(subdomains, report_file_type)
@@ -164,17 +169,8 @@ def data_gathering(self, short_domain, url, report_file_type, pagesearch_flag, k
                 api_scan_db.append('No')
                 pass
 
-            #data_array = [ip, res, mails, subdomains, subdomains_amount, social_medias, subdomain_mails, sd_socials,
-            #               subdomain_ip, issuer, subject, notBefore, notAfter, commonName, serialNumber, mx_records,
-            #               robots_txt_result, sitemap_xml_result, sitemap_links_status,
-            #               web_servers, cms, programming_languages, web_frameworks, analytics, javascript_frameworks, ports,
-            #               hostnames, cpes, tags, vulns, common_socials, total_socials, ps_emails_return,
-            #               accessible_subdomains, emails_amount, files_counter, cookies_counter, api_keys_counter,
-            #               website_elements_counter, exposed_passwords_counter, total_links_counter, accessed_links_counter, dorking_status, dorking_results, vt_cats,
-            #               vt_deturls, vt_detsamples, vt_undetsamples]
 
             cleaned_dorking = [item.strip() for item in dorking_results if item.strip()]
-            #print(cleaned_dorking)
 
             data_array = [ip, res, mails, subdomains, subdomains_amount, social_medias, subdomain_mails, sd_socials,
                           subdomain_ip, issuer, subject, notBefore, notAfter, commonName, serialNumber, mx_records,

datagather_modules/networking_processor.py

@@ -51,7 +51,7 @@ def get_ssl_certificate(short_domain, port=443):
         commonName = str(cert['issuer'][2][0][1]) + ', version: ' + str(cert['version'])
         serialNumber = cert['serialNumber']
         return issuer, subject, notBefore, notAfter, commonName, serialNumber
-    except (ssl.CertificateError, ssl.SSLError, socket.gaierror, ConnectionRefusedError) as e:
+    except Exception as e:
         print(Fore.RED + "Error while gathering info about SSL certificate. See journal for details")
         logging.error(f'SSL CERTIFICATE GATHERING: ERROR. REASON: {e}')
         issuer = subject = notBefore = notAfter = commonName = serialNumber = "No information about SSL certificate was gathered"

docs/__init__.py

@@ -0,0 +1 @@
+

docs/dpulse-docs/docs/api.md

@@ -0,0 +1,41 @@
+# Third-party API scan mode
+
+Currently DPULSE supports two third-party APIs: 
+
+* SecurityTrails API (securitytrails.com) for deep subdomains and DNS enumeration
+* VirusTotal API (virustotal.com) for brief domain information gathering
+
+## SecurityTrails API 
+
+SecurityTrails API is used to gather information about a specified domain. It retrieves various types of DNS records, subdomains, and other details. SecurityTrails API in DPULSE returns these details about target domain:
+
+* Alexa rank
+* Apex domain
+* Hostname
+* A/MX/NS/SOA/TXT records
+* All subdomains list
+* Alive (pingable) subdomains list
+
+## VirusTotal API
+
+VirusTotal API is used to interact with the VirusTotal service programmatically and analyze files and URLs using multiple antivirus engines and website scanners, providing insights into whether they are malicious. VirusTotal API in DPULSE returns these details about target domain:
+
+* Categories
+* Detected samples
+* Undetected samples
+* Detected URLs
+
+## API Keys database
+
+In order to ensure the functioning of API services individually for each DPULSE user, API keys storage database was created. Similar to report storage database, it is lightweight .db extension database with simple structure shown below: 
+
+![apistordb](https://github.com/user-attachments/assets/02233813-781e-4bf8-be7c-76ec7627be06)
+
+Since every API key is individual for each user, you can see fillers instead of actual keys when you start DPULSE for the first time, so until you replace filler with a real API key, you can't start using API in scans. You can enter your actual API keys using DPULSE CLI. You can see full process on the screenshot below:
+
+![apiproc](https://github.com/user-attachments/assets/effb27ab-dd4b-4470-b90c-34c6f9a43d8c)
+
+For the first time you will see red-colored API key field, which means that scan is not available with this API. After changing filler for actual key, you will see that color changed, which indicates that you can use your API key for scanning. Be advised that every free API service provided with some limitations (you can see them in DPULSE CLI for all supported API), so keep in mind that frequent usage of free API plans is not possible. 
+
+In case if you want to fully replace API keys, you can use reference API keys database. You can see menu point for this action on the screenshot above. This action will delete your actual API keys database, copy reference database and rename it. This action is very optional because you can change your API keys by just using first menu point in API Keys DB Manager (according to the screenshot above)
+

docs/dpulse-docs/docs/basic_scan.md

@@ -0,0 +1,18 @@
+# Basic scan
+
+Basic scan is default and the most basic scanning mode which starts after all preparation steps. It always starts first and you can't start scanning without it.
+ 
+## Basic scan results
+
+Basic scan returns some basic information open information about target domain, such as:
+
+1. WHOIS information which contains domain name, full URL, IP address, registrar info, creation and expiration dates, organization name and contact e-mails. 
+2. Subdomains list
+3. E-mail addresses list (gathered from subdomains)
+4. IP addresses list (gathered from subdomains)
+5. Social medias links, posts and profiles. Supported social medias are Facebook, Twitter (X.com), Instagram, Telegram, TikTok, LinkedIn, VKontakte, YouTube, Odnoklassniki, WeChat
+6. DNS and SSL information. DNS information contains name servers and MX addresses. SSL certificate information contains issuer, subject, creation and expiration dates, certificate name and serial number.
+7. Basic pre-pentest information such as possible vulnerabilities (CVEs), open ports and hostnames.
+8. Development and deployment services and frameworks information, such as CMS, web servers, used programming languages and web frameworks, analytics services, tags and so on. 
+9. Downloaded copies of sitemap.xml and robots.txt files from a domain
+     

docs/dpulse-docs/docs/contact_dev.md

@@ -0,0 +1,10 @@
+## Contact developer
+
+DPULSE developer will be glad to see your messages with feedback, questions and suggestions. Feel free to contact developer with most convenient ways for you:
+
+* OSINT-TECHNOLOGIES e-mail: osint.technologies@gmail.com
+* [Make an issue page on DPULSE GitHub repository](https://github.com/OSINT-TECHNOLOGIES/dpulse/issues/new/choose)
+
+
+
+

docs/dpulse-docs/docs/demo.md

@@ -0,0 +1,16 @@
+# DPULSE demos & use-cases
+
+On this page you can see some actual DPULSE demos and use-cases in some cybersecurity and OSINT scenarios
+
+## Demo №1. Starting DPULSE
+
+Yes, starting DPULSE is as simple as that. Don't forget to install all requirements before starting DPULSE
+
+![dpulse_start](https://github.com/user-attachments/assets/9ec0ab73-2206-4d38-bae6-e88656e17f95) 
+
+## Demo №2. DPULSE basic scan workflow
+
+Here you can see complete process of DPULSE basic scan from the beginning to the end. Remember that sometimes your scan might not be ideal and errors may appear. However, they will not interrupt scanning process, but will only affect the number and variety of results in the final report.
+
+![dpulse_bs](https://github.com/user-attachments/assets/b0ad7827-6dac-4f82-a369-4447a0e1c878)
+