Merge branch 'v1.7'

Author: ecstatic_nobel

InterestingFile/.placeholder renamed to Captures/.placeholder

@@ -2,17 +2,17 @@
 ##### Be the first to know on a need-to-know basis.  
 
 ### Description  
-With `aa_adhoc`, run through a list of URLs and recursively check sites for malicious files based on predefined file extensions.  
+With `aa_adhoc`, run through a list of URLs and check sites for malicious files based on predefined file extensions.  
 
 With `aa_certstream`, find out when a phishing kit has been staged on a domain. With this information, you can be amongst the first to:  
 - Know  
 - Block  
 - Report  
 - Analyze  
 
-With `aa_urlscan`, easily search [urlscan.io](https://urlscan.io/) and recursively check sites for malicious files based on predefined file extensions.  
+With `aa_urlscan`, easily search [urlscan.io](https://urlscan.io/) and check sites for malicious files based on predefined file extensions.  
 
-With `aa_whoisds`, download a list of newly registered domains from [WHOIS Domain Search](https://whoisds.com/newly-registered-domains), score the domains, and search for signs of phishing activity.  
+With `aa_whoisds`, download a list of newly registered domains from [WHOIS Domain Search](https://whoisds.com/newly-registered-domains), score the domains, and search for signs of malicious activity.  
 
 ### Prerequisites  
 - Ubuntu 18.04+ (should work on other Linux distros)  
@@ -32,24 +32,22 @@ With `aa_whoisds`, download a list of newly registered domains from [WHOIS Domai
 **aa_adhoc**  
 The following command will:  
 - Make requests to the domains retrieved from a file  
-- Recursively download the site when an open directory hosting a file with the desired file extension  
+- Download files from the site when an open directory is found hosting a file with the desired file extension  
 
-3 positional arguments needed:  
+1 positional arguments needed:  
 - **Input File**     : Path to the file containing URLs  
-- **File Extension** : 7z, apk, bat, bz, bz2, crypt, dll, doc, docx, exe, gz, hta, iso, jar, json, lnk, ppt, ps1, py, rar, sfx, sh, tar, vb, vbs, xld, xls, xlsx, zip  
 
 Optional arguments:  
-- **--file-dir**      : Directory to use for interesting files detected (default: ./InterestingFiles/)  
-- **--kit-dir**       : Directory to use for phishing kits detected (default: ./KitJackinSeason/)  
-- **--level**         : Recursion depth (default=1, infinite=0)  
-- **--quiet**         : Don't show wget output  
-- **--threads**       : Numbers of threads to spawn  
-- **--timeout**       : Set time to wait for a connection  
-- **--tor**           : Download files via the Tor network  
-- **--very-verbose**  : Show error messages  
+- **--directory**    : Download data to CAP_DIR (default: ./Captures)  
+- **--level**        : Recursion depth (default=1, infinite=0)  
+- **--quiet**        : Don't show wget output  
+- **--threads**      : Numbers of threads to spawn  
+- **--timeout**      : Set the connection timeout to TIMEOUT  
+- **--tor**          : Download files via the Tor network  
+- **--very-verbose** : Show error messages  
 
 ```bash  
-python aa_adhoc.py <INPUT_FILE> <FILE_EXTENSION> [--file-dir] [--kit-dir] [--level] [--quiet] [--threads] [--timeout] [--tor] [--very-verbose]  
+python aa_adhoc.py <INPUT_FILE> [--directory] [--level] [--quiet] [--threads] [--timeout] [--tor] [--very-verbose]  
 ```  
 
 **aa_certstream**  
@@ -61,20 +59,19 @@ The following command will:
 
 Optional arguments:  
 - **--dns-twist**    : Check the twisted keywords found in dns_twisted.yaml  
-- **--file-dir**     : Directory to use for interesting files detected (default: ./InterestingFiles/)  
-- **--kit-dir**      : Directory to use for phishing kits detected (default: ./KitJackinSeason/)  
+- **--directory**    : Download data to CAP_DIR (default: ./Captures)  
 - **--level**        : Recursion depth (default=1, infinite=0)  
 - **--log-nc**       : File to store domains that have not been checked  
 - **--quiet**        : Don't show wget output  
 - **--score**        : Minimum score to trigger a session (Default: 75)  
 - **--threads**      : Numbers of threads to spawn  
-- **--timeout**      : Set time to wait for a connection  
+- **--timeout**      : Set the connection timeout to TIMEOUT  
 - **--tor**          : Download files via the Tor network  
 - **--verbose**      : Show domains being scored  
 - **--very-verbose** : Show error messages  
 
 ```bash  
-python aa_certstream.py [--dns-twist] [--file-dir] [--kit-dir] [--level] [--log-nc] [--quiet] [--score] [--threads] [--timeout] [--tor] [--verbose] [--very-verbose]  
+python aa_certstream.py [--dns-twist] [--directory] [--level] [--log-nc] [--quiet] [--score] [--threads] [--timeout] [--tor] [--verbose] [--very-verbose]  
 ```  
 
 **aa_urlscan**  
@@ -88,17 +85,16 @@ The following command will:
 - **File Extension** : 7z, apk, bat, bz, bz2, crypt, dll, doc, docx, exe, gz, hta, iso, jar, json, lnk, ppt, ps1, py, rar, sfx, sh, tar, vb, vbs, xld, xls, xlsx, zip  
 
 Optional arguments:  
-- **--file-dir**      : Directory to use for interesting files detected (default: ./InterestingFiles/)  
-- **--kit-dir**       : Directory to use for phishing kits detected (default: ./KitJackinSeason/)  
-- **--level**         : Recursion depth (default=1, infinite=0)  
-- **--quiet**         : Don't show wget output  
-- **--threads**       : Numbers of threads to spawn  
-- **--timeout**       : Set time to wait for a connection  
-- **--tor**           : Download files via the Tor network  
-- **--very-verbose**  : Show error messages  
+- **--directory**    : Download data to CAP_DIR (default: ./Captures)  
+- **--level**        : Recursion depth (default=1, infinite=0)  
+- **--quiet**        : Don't show wget output  
+- **--threads**      : Numbers of threads to spawn  
+- **--timeout**      : Set the connection timeout to TIMEOUT  
+- **--tor**          : Download files via the Tor network  
+- **--very-verbose** : Show error messages  
 
 ```bash  
-python aa_urlscan.py <QUERY_TYPE> <DELTA> <FILE_EXTENSION> [--file-dir] [--kit-dir] [--level] [--quiet] [--threads] [--timeout] [--tor] [--very-verbose]  
+python aa_urlscan.py <QUERY_TYPE> <DELTA> <FILE_EXTENSION> [[--directory] [--level] [--quiet] [--threads] [--timeout] [--tor] [--very-verbose]  
 ```  
 **Note**: If the path is a file, it will be automatically downloaded.  
 
@@ -113,20 +109,19 @@ python aa_urlscan.py <QUERY_TYPE> <DELTA> <FILE_EXTENSION> [--file-dir] [--kit-d
 
 Optional arguments:  
 - **--dns-twist**    : Check the twisted keywords found in dns_twisted.yaml  
-- **--file-dir**     : Directory to use for interesting files detected (default: ./InterestingFiles/)  
-- **--kit-dir**      : Directory to use for phishing kits detected (default: ./KitJackinSeason/)  
+- **--directory**    : Download data to CAP_DIR (default: ./Captures)  
 - **--level**        : Recursion depth (default=1, infinite=0)  
 - **--log-nc**       : File to store domains that have not been checked  
 - **--quiet**        : Don't show wget output  
 - **--score**        : Minimum score to trigger a session (Default: 75)  
 - **--threads**      : Numbers of threads to spawn  
-- **--timeout**      : Set time to wait for a connection  
+- **--timeout**      : Set the connection timeout to TIMEOUT  
 - **--tor**          : Download files via the Tor network  
 - **--verbose**      : Show domains being scored  
 - **--very-verbose** : Show error messages  
 
 ```bash  
-python aa_whoisds.py <DELTA> [--dns-twist] [--file-dir] [--kit-dir] [--level] [--log-nc] [--quiet] [--score] [--threads] [--timeout] [--tor] [--verbose] [--very-verbose]  
+python aa_whoisds.py <DELTA> [--dns-twist] [--directory] [--level] [--log-nc] [--quiet] [--score] [--threads] [--timeout] [--tor] [--verbose] [--very-verbose]  
 ```  
 
 ### Things to know  
@@ -141,6 +136,11 @@ python aa_whoisds.py <DELTA> [--dns-twist] [--file-dir] [--kit-dir] [--level] [-
   - **Triggered**: a domain was found with the minimum score specified  
   - **Download**: checks passed and a download was started  
   - **Complete**: download complete or the site canceled it prematurely  
+  - **Directory**: the output directory is unavailable  
+- If the keywords in `config.yaml` have been modified and `--dns-twist` is going to be used, regenerate `dns_twisted.yaml` by running the following command:  
+    ```bash
+    bash dnstwist.sh PATH_TO_DNSTWIST_SCRIPT
+    ```
 - Using the `--dns-twist` flag will default to a minimum of 20 threads  
 
 Please fork, create merge requests, and help make this better.  

KitJackinSeason/.placeholder

@@ -4,23 +4,21 @@
 - Make requests to the domains retrieved from a file
 - Recursively download the site when an open directory hosting a file with the desired file extension
 
-3 positional arguments needed:
+1 positional arguments needed:
 - Input File : Path to the file containing URLs
-- File Extension : 7z, apk, bat, bz, bz2, crypt, dll, doc, docx, exe, gz, hta, iso, jar, json, lnk, ppt, ps1, py, rar, sfx, sh, tar, vb, vbs, xld, xls, xlsx, zip
 
 Optional arguments:
-- --file-dir     : Directory to use for interesting files detected (default: ./InterestingFiles/)
-- --kit-dir      : Directory to use for phishing kits detected (default: ./KitJackinSeason/)
+- --directory    : Save data to CAP_DIR (default: ./Captures/)
 - --level        : Recursion depth (default=1, infinite=0)
 - --quiet        : Don't show wget output
 - --threads      : Numbers of threads to spawn
-- --timeout      : Set time to wait for a connection
+- --timeout      : Set the connection timeout to TIMEOUT
 - --tor          : Download files via the Tor network
 - --very-verbose : Show error messages
 
 Usage:
 ```
-python aa_adhoc.py <INPUT_FILE> <FILE_EXTENSION> [--file-dir] [--kit-dir] [--level] [--quiet] [--threads] [--timeout] [--tor] [--very-verbose]
+python aa_adhoc.py <INPUT_FILE> [--directory] [--level] [--quiet] [--threads] [--timeout] [--tor] [--very-verbose]
 ```
 
 Debugger: open("/tmp/aa.txt", "a").write("{}: <MSG>\n".format(<VAR>))
@@ -36,23 +34,14 @@
 parser.add_argument(metavar="input file",
                     dest="input_file",
                     help="Path to the file containing URLs")
-parser.add_argument(metavar="file extension",
-                    dest="ext",
-                    choices=["7z", "apk", "bat", "bz", "bz2", "crypt", "dll", "doc", "docx", "exe", "gz", "hta", "iso", "jar", "json", "lnk", "ppt", "ps1", "py", "rar", "sfx", "sh", "tar", "vb", "vbs", "xld", "xls", "xlsx", "zip"],
-                    help="7z, apk, bat, bz, bz2, crypt, dll, doc, docx, exe, gz, hta, iso, jar, json, lnk, ppt, ps1, py, rar, sfx, sh, tar, vb, vbs, xld, xls, xlsx, zip")
-parser.add_argument("--file-dir",
-                    dest="file_dir",
-                    default="./InterestingFile/",
+parser.add_argument("--directory",
+                    dest="cap_dir",
+                    default="./Captures/",
                     required=False,
-                    help="Directory to use for interesting files detected (default: ./InterestingFiles)")
-parser.add_argument("--kit-dir",
-                    dest="kit_dir",
-                    default="./KitJackinSeason/",
-                    required=False,
-                    help="Directory to use for phishing kits detected (default: ./KitJackinSeason)")
+                    help="Download data to CAP_DIR (default: ./Captures)")
 parser.add_argument("--level",
                     dest="level",
-                    default=0,
+                    default=1,
                     required=False,
                     type=str,
                     help="Directory depth (default=1, infinite=0")
@@ -72,7 +61,7 @@
                     default=30,
                     required=False,
                     type=int,
-                    help="Set time to wait for a connection")
+                    help="Set the connection timeout to TIMEOUT")
 parser.add_argument("--tor",
                     dest="tor",
                     action="store_true",
@@ -83,11 +72,8 @@
                     action="store_true",
                     required=False,
                     help="Show error messages")
-args = parser.parse_args()
-uagent = "Mozilla/5.0 (Windows NT 6.3; Trident/7.0; rv:11.0) like Gecko"
-
 # Fix directory names
-args = commons.fix_directory(args)
+args = commons.fix_directory(parser.parse_args())
 
 def main():
     """ """
@@ -96,36 +82,28 @@ def main():
 
     # Print start messages
     commons.show_summary(args)
-    commons.show_networking(args, uagent)
+    commons.show_networking(args) # globals: proxies, torsocks
 
-    # Read suspicious.yaml
-    suspicious = commons.read_suspicious(args)
+    # Read config.yaml
+    commons.read_config(args) # globals: config
 
     # Recompile exclusions
-    commons.recompile_exclusions()
-
-    # Build dict of extensions
-    extensions = {}
-    extensions.update(suspicious["archives"])
-    extensions.update(suspicious["files"])
-
-    # Read file containing URLs
-    urls = commons.read_file(args.input_file)
+    commons.recompile_exclusions() # globals: exclusions
 
     # Create queues
-    recursion_queue = commons.create_queue("recursion_queue")
+    url_queue = commons.create_queue("url_queue")
 
     # Create threads
-    commons.RecursiveQueueManager(args, recursion_queue, uagent, extensions)
+    commons.UrlQueueManager(args, url_queue)
+
+    # Read file containing URLs
+    urls = commons.read_file(args.input_file)
 
     # Process URLs
     for url in urls:
-        if not (url.startswith("http://") or url.startswith("https://")):
-            continue
-
-        recursion_queue.put(url)
+        url_queue.put(url)
 
-    recursion_queue.join()
+    url_queue.join()
     return
 
 if __name__ == "__main__":

README.md

@@ -8,21 +8,20 @@
 
 Optional arguments:
 - --dns-twist    : Check the twisted keywords found in dns_twisted.yaml
-- --file-dir     : Directory to use for interesting files detected (default: ./InterestingFiles/)
-- --kit-dir      : Directory to use for phishing kits detected (default: ./KitJackinSeason/)
+- --directory    : Save data to CAP_DIR (default: ./Captures/)
 - --level        : Recursion depth (default=1, infinite=0)
 - --log-nc       : File to store domains that have not been checked
 - --quiet        : Don't show wget output
 - --score        : Minimum score to trigger a session (Default: 75)
 - --threads      : Numbers of threads to spawn
-- --timeout      : Set time to wait for a connection
+- --timeout      : Set the connection timeout to TIMEOUT
 - --tor          : Download files via the Tor network
 - --verbose      : Show domains being scored
 - --very-verbose : Show error messages
 
 Usage:
 ```
-python aa_certstream.py [--dns-twist] [--file-dir] [--kit-dir] [--level] [--log-nc] [--quiet] [--score] [--threads] [--timeout] [--tor] [--verbose] [--very-verbose]
+python aa_certstream.py [--dns-twist] [--directory] [--level] [--log-nc] [--quiet] [--score] [--threads] [--timeout] [--tor] [--verbose] [--very-verbose]
 ```
 
 Debugger: open("/tmp/aa.txt", "a").write("{}: <MSG>\n".format(<VAR>))
@@ -50,16 +49,11 @@
                     action="store_true",
                     required=False,
                     help="Check the twisted keywords found in dns_twisted.yaml")
-parser.add_argument("--file-dir",
-                    dest="file_dir",
-                    default="./InterestingFile/",
+parser.add_argument("--directory",
+                    dest="cap_dir",
+                    default="./Captures/",
                     required=False,
-                    help="Directory to use for interesting files detected (default: ./InterestingFiles/)")
-parser.add_argument("--kit-dir",
-                    dest="kit_dir",
-                    default="./KitJackinSeason/",
-                    required=False,
-                    help="Directory to use for phishing kits detected (default: ./KitJackinSeason/)")
+                    help="Download data to CAP_DIR (default: ./Captures)")
 parser.add_argument("--level",
                     dest="level",
                     default=1,
@@ -93,7 +87,7 @@
                     default=30,
                     required=False,
                     type=int,
-                    help="Set time to wait for a connection")
+                    help="Set the connection timeout to TIMEOUT")
 parser.add_argument("--tor",
                     dest="tor",
                     action="store_true",
@@ -109,16 +103,13 @@
                     action="store_true",
                     required=False,
                     help="Show error messages")
-args   = parser.parse_args()
-uagent = "Mozilla/5.0 (Windows NT 6.3; Trident/7.0; rv:11.0) like Gecko"
+# Fix directory names
+args = commons.fix_directory(parser.parse_args())
 
 # Set threads to a minimum of 20 if using --dns-twist
 if args.dns_twist and args.threads < 20:
     args.threads = 20
 
-# Fix directory names
-args = commons.fix_directory(args)
-
 tqdm.tqdm.monitor_interval = 0
 
 def callback(message, context):
@@ -145,7 +136,7 @@ def callback(message, context):
 
         pbar.update(1)
 
-        score = commons.score_domain(suspicious, domain.lower(), args)
+        score = commons.score_domain(config, domain.lower(), args)
 
         if "Let's Encrypt" in message["data"]["chain"][0]["subject"]["aggregated"]:
             score += 10
@@ -215,27 +206,27 @@ def on_open(instance):
 def main():
     """ """
     global exclusions
-    global suspicious
+    global config
     global url_queue
 
     # Check if output directories exist
     commons.check_path(args)
 
     # Print start messages
     commons.show_summary(args)
-    commons.show_networking(args, uagent)
+    commons.show_networking(args) # globals: proxies, torsocks
 
-    # Read suspicious.yaml
-    suspicious = commons.read_suspicious(args)
+    # Read config.yaml
+    config = commons.read_config(args) # globals: config
 
     # Recompile exclusions
-    exclusions = commons.recompile_exclusions()
+    exclusions = commons.recompile_exclusions() # globals: exclusions
 
     # Create queues
     url_queue = commons.create_queue("url_queue")
 
     # Create threads
-    commons.UrlQueueManager(args, url_queue, uagent)
+    commons.UrlQueueManager(args, url_queue)
 
     # Listen for events via Certstream
     print(colored("Connecting to Certstream...\n", "yellow", attrs=["bold"]))