Skip to content

Commit 4430dc2

Browse files
Stabilized v1.2.2
2 parents 283d8b5 + f7d2013 commit 4430dc2

File tree

17 files changed

+925
-459
lines changed

17 files changed

+925
-459
lines changed

README.md

Lines changed: 13 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -88,17 +88,7 @@ Since DPULSE repository is using Poetry* to manage dependencies, it is higly rec
8888

8989
_* Poetry is a tool for dependency management and packaging in Python. It can be simply installed everywhere using `pip install poetry` command, but more instructions you can find on [Poetry official documentation page](https://python-poetry.org/docs/#ci-recommendations)_
9090

91-
### <ins>First way (the simplest way)</ins>
92-
93-
Just download DPULSE using fast-access links at the top of the README:
94-
95-
![изображение](https://github.com/user-attachments/assets/bd1d9627-950b-40d4-91c4-6751476d7b65)
96-
97-
Then just unpack downloaded archive, open terminal in DPULSE root folder and use `pip install -r requirements.txt` command to install requirements. Then type `python dpulse.py` in terminal, and that's where program starts.
98-
99-
If `pip install -r requirements.txt` doesn't work, then just use `poetry install` command. After that, start DPULSE with `poetry run python dpulse.py`
100-
101-
### <ins>Second way (the most correct way)</ins>
91+
### <ins>First way (recommended way, using Poetry)</ins>
10292

10393
Use this set of commands to install DPULSE stable versions:
10494

@@ -118,6 +108,18 @@ Use this set of commands to install DPULSE rolling versions:
118108

119109
After installation, you simply start DPULSE using `poetry run python dpulse.py`
120110

111+
### <ins>Second way (recommended way, without using Poetry)</ins>
112+
113+
Just download DPULSE using fast-access links at the top of the README:
114+
115+
![image](https://github.com/user-attachments/assets/9ec2d2d7-706f-4385-9594-54e0cc72c695)
116+
117+
Decide what version you want to download and use basing on your expectations like stability, support, functionality and so on
118+
119+
Then just unpack downloaded archive, open terminal in DPULSE root folder and use `pip install -r requirements.txt` command to install requirements. Then type `python dpulse.py` in terminal, and that's where program starts.
120+
121+
If `pip install -r requirements.txt` doesn't work, then just use `poetry install` command. After that, start DPULSE with `poetry run python dpulse.py`
122+
121123
## _Other ways_
122124

123125
### <ins>Third way (using pip manager)</ins>

apis/api_hudsonrock.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -110,7 +110,7 @@ def format_section(title, data):
110110
formatted_output += f" {Fore.GREEN}| URL:{Style.RESET_ALL} {Fore.LIGHTCYAN_EX}{url_data.get('url', 'N/A')}{Style.RESET_ALL}"
111111
formatted_output += f" {Fore.GREEN}| Occurrence:{Style.RESET_ALL} {Fore.LIGHTCYAN_EX}{url_data.get('occurrence', 'N/A')}{Style.RESET_ALL}\n"
112112
else:
113-
formatted_output += f"{Fore.RED}No employee URLs available.{Style.RESET_ALL}\n"
113+
formatted_output += f"{Fore.RED}No employee URLs available{Style.RESET_ALL}\n"
114114
formatted_output += f"\n{Fore.GREEN}Sample Client URLs:{Style.RESET_ALL}\n"
115115
clients = data.get('data', {}).get('clients_urls', [])
116116
if clients:
@@ -119,7 +119,7 @@ def format_section(title, data):
119119
formatted_output += f" {Fore.GREEN}| URL:{Style.RESET_ALL} {Fore.LIGHTCYAN_EX}{url_data.get('url', 'N/A')}{Style.RESET_ALL}"
120120
formatted_output += f" {Fore.GREEN}| Occurrence:{Style.RESET_ALL} {Fore.LIGHTCYAN_EX}{url_data.get('occurrence', 'N/A')}{Style.RESET_ALL}\n"
121121
else:
122-
formatted_output += f"{Fore.LIGHTCYAN_EX}No client URLs available.{Style.RESET_ALL}\n"
122+
formatted_output += f"{Fore.RED}No client URLs available{Style.RESET_ALL}\n"
123123

124124
elif title == 'IP Data':
125125
formatted_output += f"{Fore.GREEN}Message:{Style.RESET_ALL} {Fore.LIGHTCYAN_EX}{data.get('message', 'No message available')}{Style.RESET_ALL}\n"

apis/api_securitytrails.py

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -110,7 +110,6 @@ def check_domain_securitytrails(domain, api_key):
110110
else:
111111
formatted_output += (f"{Fore.RED}Error while gathering subdomains: {subdomains_response.status_code}{Style.RESET_ALL}\n")
112112

113-
formatted_output += Fore.LIGHTBLUE_EX + "\n=== END OF SECURITYTRAILS API REPORT ===\n" + Style.RESET_ALL
114113
return formatted_output
115114

116115

apis/api_virustotal.py

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -55,7 +55,6 @@ def check_domain(domain, api_key):
5555
formatted_output += f"\n{Fore.GREEN}{category.title()} ({len(engines)}):{Style.RESET_ALL}\n"
5656
for engine in sorted(engines):
5757
formatted_output += f"{Fore.LIGHTCYAN_EX}- {engine}{Style.RESET_ALL}\n"
58-
formatted_output += f"\n{Fore.LIGHTBLUE_EX}=== END OF VIRUSTOTAL API REPORT ==={Style.RESET_ALL}\n"
5958
print(formatted_output)
6059
return formatted_output
6160
except Exception as e:

datagather_modules/data_assembler.py

Lines changed: 15 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -7,11 +7,11 @@
77
import crawl_processor as cp
88
import dorking_handler as dp
99
import networking_processor as np
10-
from pagesearch_main import normal_search
10+
from pagesearch_parsers import subdomains_parser
1111
from logs_processing import logging
1212
from api_virustotal import api_virustotal_check
1313
from api_securitytrails import api_securitytrails_check
14-
from api_hudsonrock import api_hudsonrock_check, api_hudsonrock_get
14+
from api_hudsonrock import api_hudsonrock_check
1515
from db_creator import get_dorking_query
1616
from screen_snapshotting import take_screenshot
1717
from config_processing import read_config
@@ -20,10 +20,8 @@
2020
try:
2121
import requests
2222
from datetime import datetime
23-
import jinja2
2423
import os
2524
from colorama import Fore, Style
26-
import webbrowser
2725
import sqlite3
2826
import configparser
2927
except ImportError as e:
@@ -78,31 +76,31 @@ def data_gathering(self, short_domain, url, report_file_type, pagesearch_flag, k
7876
casename, db_casename, db_creation_date, robots_filepath, sitemap_filepath, sitemap_links_filepath, report_file_type, report_folder, ctime, report_ctime = self.report_preprocessing(short_domain, report_file_type)
7977
logging.info(f'### THIS LOG PART FOR {casename} CASE, TIME: {ctime} STARTS HERE')
8078
print(Fore.GREEN + "Started scanning domain" + Style.RESET_ALL)
81-
print(Fore.GREEN + "Getting domain IP address" + Style.RESET_ALL)
79+
print(Fore.GREEN + "[1/11] Getting domain IP address" + Style.RESET_ALL)
8280
ip = cp.ip_gather(short_domain)
83-
print(Fore.GREEN + 'Gathering WHOIS information' + Style.RESET_ALL)
81+
print(Fore.GREEN + '[2/11] Gathering WHOIS information' + Style.RESET_ALL)
8482
res = cp.whois_gather(short_domain)
85-
print(Fore.GREEN + 'Processing e-mails gathering' + Style.RESET_ALL)
83+
print(Fore.GREEN + '[3/11] Processing e-mails gathering' + Style.RESET_ALL)
8684
mails = cp.contact_mail_gather(url)
87-
print(Fore.GREEN + 'Processing subdomain gathering' + Style.RESET_ALL)
85+
print(Fore.GREEN + '[4/11] Processing subdomain gathering' + Style.RESET_ALL)
8886
subdomains, subdomains_amount = cp.subdomains_gather(url, short_domain)
89-
print(Fore.GREEN + 'Processing social medias gathering' + Style.RESET_ALL)
87+
print(Fore.GREEN + '[5/11] Processing social medias gathering' + Style.RESET_ALL)
9088
try:
9189
social_medias = cp.sm_gather(url)
9290
except:
9391
print(Fore.RED + "Social medias were not gathered because of error" + Style.RESET_ALL)
9492
social_medias = ['Social medias were not extracted because of error']
9593
pass
96-
print(Fore.GREEN + 'Processing subdomain analysis' + Style.RESET_ALL)
94+
print(Fore.GREEN + '[6/11] Processing subdomain analysis' + Style.RESET_ALL)
9795
if report_file_type == 'xlsx':
9896
subdomain_urls, subdomain_mails, subdomain_ip, sd_socials = cp.domains_reverse_research(subdomains, report_file_type)
9997
elif report_file_type == 'html':
10098
subdomain_mails, sd_socials, subdomain_ip = cp.domains_reverse_research(subdomains, report_file_type)
101-
print(Fore.GREEN + 'Processing SSL certificate gathering' + Style.RESET_ALL)
99+
print(Fore.GREEN + '[7/11] Processing SSL certificate gathering' + Style.RESET_ALL)
102100
issuer, subject, notBefore, notAfter, commonName, serialNumber = np.get_ssl_certificate(short_domain)
103-
print(Fore.GREEN + 'Processing DNS records gathering' + Style.RESET_ALL)
101+
print(Fore.GREEN + '[8/11] Processing DNS records gathering' + Style.RESET_ALL)
104102
mx_records = np.get_dns_info(short_domain, report_file_type)
105-
print(Fore.GREEN + 'Extracting robots.txt and sitemap.xml' + Style.RESET_ALL)
103+
print(Fore.GREEN + '[9/11] Extracting robots.txt and sitemap.xml' + Style.RESET_ALL)
106104
robots_txt_result = np.get_robots_txt(short_domain, robots_filepath)
107105
sitemap_xml_result = np.get_sitemap_xml(short_domain, sitemap_filepath)
108106
if report_file_type == 'html':
@@ -114,9 +112,9 @@ def data_gathering(self, short_domain, url, report_file_type, pagesearch_flag, k
114112
sitemap_links_status = 'Sitemap links were not parsed'
115113
pass
116114

117-
print(Fore.GREEN + 'Gathering info about website technologies' + Style.RESET_ALL)
115+
print(Fore.GREEN + '[10/11] Gathering info about website technologies' + Style.RESET_ALL)
118116
web_servers, cms, programming_languages, web_frameworks, analytics, javascript_frameworks = np.get_technologies(url)
119-
print(Fore.GREEN + 'Processing Shodan InternetDB search' + Style.RESET_ALL)
117+
print(Fore.GREEN + '[11/11] Processing Shodan InternetDB search' + Style.RESET_ALL)
120118
ports, hostnames, cpes, tags, vulns = np.query_internetdb(ip, report_file_type)
121119
common_socials = {key: social_medias.get(key, []) + sd_socials.get(key, []) for key in set(social_medias) | set(sd_socials)}
122120
for key in common_socials:
@@ -128,19 +126,14 @@ def data_gathering(self, short_domain, url, report_file_type, pagesearch_flag, k
128126
if subdomains[0] != 'No subdomains were found':
129127
to_search_array = [subdomains, social_medias, sd_socials]
130128
print(Fore.LIGHTMAGENTA_EX + "\n[EXTENDED SCAN START: PAGESEARCH]\n" + Style.RESET_ALL)
131-
ps_emails_return, accessible_subdomains, emails_amount, files_counter, cookies_counter, api_keys_counter, website_elements_counter, exposed_passwords_counter, keywords_messages_list = normal_search(to_search_array, report_folder, keywords, keywords_flag)
129+
ps_emails_return, accessible_subdomains, emails_amount, files_counter, cookies_counter, api_keys_counter, website_elements_counter, exposed_passwords_counter, keywords_messages_list = subdomains_parser(to_search_array[0], report_folder, keywords, keywords_flag)
132130
total_links_counter = accessed_links_counter = "No results because PageSearch does not gather these categories"
133131
print(Fore.LIGHTMAGENTA_EX + "\n[EXTENDED SCAN END: PAGESEARCH]\n" + Style.RESET_ALL)
134132
else:
135133
print(Fore.RED + "Cant start PageSearch because no subdomains were detected")
136134
accessible_subdomains = files_counter = cookies_counter = api_keys_counter = website_elements_counter = exposed_passwords_counter = total_links_counter = accessed_links_counter = emails_amount = 'No results because no subdomains were found'
137135
ps_emails_return = ""
138136
pass
139-
#elif pagesearch_flag.lower() == 'si':
140-
#print(Fore.LIGHTMAGENTA_EX + "\n[EXTENDED SCAN START: PAGESEARCH SITEMAP INSPECTION]\n" + Style.RESET_ALL)
141-
#ps_emails_return, total_links_counter, accessed_links_counter, emails_amount = sitemap_inspection_search(report_folder)
142-
#accessible_subdomains = files_counter = cookies_counter = api_keys_counter = website_elements_counter = exposed_passwords_counter = "No results because Sitemap Inspection mode does not gather these categories"
143-
#print(Fore.LIGHTMAGENTA_EX + "\n[EXTENDED SCAN END: PAGESEARCH SITEMAP INSPECTION]\n" + Style.RESET_ALL)
144137
elif pagesearch_flag.lower() == 'n':
145138
ps_emails_return = ""
146139
accessible_subdomains = files_counter = cookies_counter = api_keys_counter = website_elements_counter = exposed_passwords_counter = total_links_counter = accessed_links_counter = emails_amount = "No results because user did not selected PageSearch for this scan"
@@ -213,7 +206,7 @@ def data_gathering(self, short_domain, url, report_file_type, pagesearch_flag, k
213206
if subdomains[0] != 'No subdomains were found':
214207
to_search_array = [subdomains, social_medias, sd_socials]
215208
print(Fore.LIGHTMAGENTA_EX + "\n[EXTENDED SCAN START: PAGESEARCH]\n" + Style.RESET_ALL)
216-
ps_emails_return, accessible_subdomains, emails_amount, files_counter, cookies_counter, api_keys_counter, website_elements_counter, exposed_passwords_counter, keywords_messages_list = normal_search(to_search_array, report_folder, keywords, keywords_flag)
209+
ps_emails_return, accessible_subdomains, emails_amount, files_counter, cookies_counter, api_keys_counter, website_elements_counter, exposed_passwords_counter, keywords_messages_list = subdomains_parser(to_search_array[0], report_folder, keywords, keywords_flag)
217210
total_links_counter = accessed_links_counter = "No results because PageSearch does not gather these categories"
218211
if len(keywords_messages_list) == 0:
219212
keywords_messages_list = ['No keywords were found']
@@ -224,11 +217,6 @@ def data_gathering(self, short_domain, url, report_file_type, pagesearch_flag, k
224217
accessible_subdomains = files_counter = cookies_counter = api_keys_counter = website_elements_counter = exposed_passwords_counter = total_links_counter = accessed_links_counter = emails_amount = 'No results because no subdomains were found'
225218
keywords_messages_list = ['No data was gathered because no subdomains were found']
226219
pass
227-
#elif pagesearch_flag.lower() == 'si':
228-
#print(Fore.LIGHTMAGENTA_EX + "\n[EXTENDED SCAN START: PAGESEARCH SITEMAP INSPECTION]\n" + Style.RESET_ALL)
229-
#ps_emails_return, total_links_counter, accessed_links_counter, emails_amount = sitemap_inspection_search(report_folder)
230-
#accessible_subdomains = files_counter = cookies_counter = api_keys_counter = website_elements_counter = exposed_passwords_counter = keywords_messages_list = "No results because Sitemap Inspection mode does not gather these categories"
231-
#print(Fore.LIGHTMAGENTA_EX + "\n[EXTENDED SCAN END: PAGESEARCH SITEMAP INSPECTION]\n" + Style.RESET_ALL)
232220
elif pagesearch_flag.lower() == 'n':
233221
accessible_subdomains = files_counter = cookies_counter = api_keys_counter = website_elements_counter = exposed_passwords_counter = total_links_counter = accessed_links_counter = emails_amount = keywords_messages_list = "No results because user did not selected PageSearch for this scan"
234222
ps_emails_return = ""

docs/dpulse-docs/docs/getting_started.md

Lines changed: 26 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -1,38 +1,44 @@
1+
## System requirements
2+
3+
DPULSE is a Python-based tool, so its basic requirement is installed Python. The most stable and tested versions are Python 3.10, 3.11 and 3.12. Other versions may decrease stability or even lead to problem with starting DPULSE because of used dependencies. It is also strongly recommended to have a high-speed and stable internet connection for stable work of such functions as Dorking scan and PageSearch which actively use web resources scraping. Also you need to install some Python packages using requirements.txt file or Poetry dependencies manager
4+
15
## Installing DPULSE
26

37
You can install DPULSE in several ways, use the way you like the most. But since DPULSE repository is using Poetry to manage dependencies, it is higly recommended to install and start DPULSE using Poetry, especially on Linux systems where a lot of Python packages which DPULSE requires are preinstalled. More information about Poetry you can find on [Poetry official documentation page](https://python-poetry.org/docs/#ci-recommendations)
48

5-
### Install and start DPULSE. Way №1
9+
### Install and start DPULSE. Way №1 (using Poetry)
610

7-
Just download DPULSE using fast-access links at the top of the README:
11+
Use this set of commands to install DPULSE stable versions:
812

9-
![image1](https://github.com/user-attachments/assets/bd1d9627-950b-40d4-91c4-6751476d7b65)
13+
```
14+
git clone https://github.com/OSINT-TECHNOLOGIES/dpulse
15+
cd dpulse
16+
poetry install
17+
```
1018

11-
Then just unpack downloaded archive, open terminal in DPULSE root folder and use `pip install -r requirements.txt` command to install requirements. Then type `python dpulse.py` in terminal, and that's where program starts.
19+
Use this set of commands to install DPULSE rolling versions:
1220

13-
If `pip install -r requirements.txt` doesn't work, then just use `poetry install` command. After that, start DPULSE with `poetry run python dpulse.py`
21+
```
22+
git clone --branch rolling --single-branch https://github.com/OSINT-TECHNOLOGIES/dpulse.git
23+
cd dpulse
24+
poetry install
25+
```
1426

15-
### Install and start DPULSE. Way №2
27+
After installation, you simply start DPULSE using `poetry run python dpulse.py`
1628

17-
Use this set of commands to install DPULSE stable versions:
29+
### Install and start DPULSE. Way №2 (without using Poetry)
1830

19-
```
20-
git clone https://github.com/OSINT-TECHNOLOGIES/dpulse
21-
cd dpulse
22-
poetry install
23-
```
31+
Just download DPULSE using fast-access links at the top of the README:
2432

25-
Use this set of commands to install DPULSE rolling versions:
33+
![image](https://github.com/user-attachments/assets/9ec2d2d7-706f-4385-9594-54e0cc72c695)
2634

27-
```
28-
git clone --branch rolling --single-branch https://github.com/OSINT-TECHNOLOGIES/dpulse.git
29-
cd dpulse
30-
poetry install
31-
```
35+
Decide what version you want to download and use basing on your expectations like stability, support, functionality and so on
3236

33-
After installation, you simply start DPULSE using `poetry run python dpulse.py`
37+
Then just unpack downloaded archive, open terminal in DPULSE root folder and use `pip install -r requirements.txt` command to install requirements. Then type `python dpulse.py` in terminal, and that's where program starts.
38+
39+
If `pip install -r requirements.txt` doesn't work, then just use `poetry install` command. After that, start DPULSE with `poetry run python dpulse.py`
3440

35-
### Install and start DPULSE. Way №3
41+
### Install and start DPULSE. Way №3 (using pip manager)
3642

3743
You also can install DPULSE using pip manager. It'll install DPULSE and necessery dependencies in one command: `pip install dpulse`. Then you just locate DPULSE root folder and type `python dpulse.py` to start program.
3844

0 commit comments

Comments
 (0)