cleaned up code blocks; scans conform to 1 template for sections

* initial work on sphinx docs; much left to do

* first pass at docs complete; still has some warts

* added requirements for readthedocs

* added requirements for readthedocs

* added requirements for readthedocs

* cleaned up code blocks; scans conform to 1 template for sections

* trying to fix code blocks not rendering on readthedocs

* trying to fix code blocks not rendering on readthedocs

* trying to fix code blocks not rendering on readthedocs

* trying to fix code blocks not rendering on readthedocs

* trying to fix code blocks not rendering on readthedocs

* trying to fix code blocks not rendering on readthedocs

docs/conf.py

@@ -12,6 +12,7 @@
 #
 import os
 import sys
+import sphinx_rtd_theme
 
 sys.path.insert(0, os.path.abspath(".."))
 
@@ -58,7 +59,10 @@
 # The theme to use for HTML and HTML Help pages.  See the documentation for
 # a list of builtin themes.
 #
+pygments_style = "sphinx"
+
 html_theme = "sphinx_rtd_theme"
+html_theme_path = [sphinx_rtd_theme.get_html_theme_path()]
 
 
 # Add any paths that contain custom static files (such as style sheets) here,

docs/modifications/new_wrapper.rst

@@ -4,17 +4,17 @@ Creating a New Wrapper Scan
 If for whatever reason you want something other than FullScan, the process for defining a new scan is relatively simple.
 The ``HTBScan`` is a good example.
 
-1. Define your new class, inheriting from :class:`luigi.WrapperTask` and use the ``inherits`` decorator to include any scan you want to utilize
+1. Define your new class, inheriting from **luigi.WrapperTask** and use the ``inherits`` decorator to include any scan you want to utilize
 
-.. code-block::
+.. code-block:: python
 
     @inherits(SearchsploitScan, AquatoneScan, GobusterScan, WebanalyzeScan)
     class HTBScan(luigi.WrapperTask):
         ...
 
 2. Include all parameters needed by any of the scans passed to ``inherits``
 
-.. code-block::
+.. code-block:: python
 
     def requires(self):
         """ HTBScan is a wrapper, as such it requires any Tasks that it wraps. """
@@ -36,7 +36,7 @@ The ``HTBScan`` is a good example.
 
 3. ``yield`` from each scan, keeping in mind that some of the parameters won't be universal (i.e. need to be removed/added)
 
-.. code-block::
+.. code-block:: python
 
     def requires(self):
         """ HTBScan is a wrapper, as such it requires any Tasks that it wraps. """

docs/overview/installation.rst

@@ -13,7 +13,7 @@ Manual Steps
 
 First, the manual steps to get cmd2 installed in a virtual environment are as follows (and shown below)
 
-.. code-block::
+.. code-block:: console
 
     apt install pipenv
     git clone https://github.com/epi052/recon-pipeline.git
@@ -52,12 +52,12 @@ for the auto installer to function:
 - derivative of debian (some tools are installed using apt)
 
 The alternatives would be to manually install each tool or to modify the distro-specific portions of the commands
-laid out in ``recon.__init__``.  For example, on Fedora, you could change the package manager from ``apt-get`` to
+laid out in ``recon.__init__.py``.  For example, on Fedora, you could change the package manager from ``apt-get`` to
 ``dnf`` and remove any ``apt-get`` specific options.
 
-Example from ``recon-pipeline/recon/__init__.py``
+Example from ``recon.__init__.py``
 
-.. code-block::
+.. code-block:: python
 
     "pipenv": {
         "installed": False,
@@ -67,16 +67,10 @@ Example from ``recon-pipeline/recon/__init__.py``
 
 would become
 
-.. code-block::
+.. code-block:: python
 
     "pipenv": {
         "installed": False,
         "dependencies": None,
         "commands": ["sudo dnf install -y pipenv"],
     },
-
-
-
-
-
-

docs/overview/scope.rst

@@ -7,7 +7,7 @@ The pipeline expects a file that describes the target's scope to be provided as
 ``--target-file`` option. The target file can consist of domains, ip addresses, and ip ranges, one per line.  Ip
 addresses and ip ranges can be mixed/matched, but domains cannot.
 
-.. code-block::
+.. code-block:: console
 
     tesla.com
     tesla.cn
@@ -17,7 +17,7 @@ addresses and ip ranges can be mixed/matched, but domains cannot.
 Some bug bounty scopes have expressly verboten subdomains and/or top-level domains, for that there is the
 ``--exempt-list`` option. The exempt list follows the same rules as the target file.
 
-.. code-block::
+.. code-block:: console
 
     shop.eu.teslamotors.com
     energysupport.tesla.com

recon/amass.py

@@ -10,18 +10,30 @@
 
 @inherits(TargetList)
 class AmassScan(ExternalProgramTask):
-    """ Run amass scan to perform subdomain enumeration of given domain(s).
+    """ Run ``amass`` scan to perform subdomain enumeration of given domain(s).
 
-    Expects TARGET_FILE.domains file to be a text file with one top-level domain per line.
+    Note:
+        Expects **TARGET_FILE.domains** file to be a text file with one top-level domain per line.
 
-    Commands are similar to the following
+    Install:
+        .. code-block:: console
 
-    amass enum -ip -brute -active -min-for-recursive 3 -df tesla -json amass.tesla.json
+            sudo apt-get install -y -q amass
+
+    Basic Example:
+        .. code-block:: console
+
+            amass enum -ip -brute -active -min-for-recursive 3 -df tesla -json amass.tesla.json
+
+    Luigi Example:
+        .. code-block:: console
+
+            PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.amass AmassScan --target-file tesla
 
     Args:
         exempt_list: Path to a file providing blacklisted subdomains, one per line.
-        target_file: specifies the file on disk containing a list of ips or domains *--* Required by upstream Task
-        results_dir: specifes the directory on disk to which all Task results are written *--* Required by upstream Task
+        target_file: specifies the file on disk containing a list of ips or domains *Required by upstream Task*
+        results_dir: specifes the directory on disk to which all Task results are written *Required by upstream Task*
     """
 
     exempt_list = luigi.Parameter(default="")
@@ -83,9 +95,9 @@ class ParseAmassOutput(luigi.Task):
     """ Read amass JSON results and create categorized entries into ip|subdomain files.
 
     Args:
-        target_file: specifies the file on disk containing a list of ips or domains *--* Required by upstream Task
-        exempt_list: Path to a file providing blacklisted subdomains, one per line. *--* Optional for upstream Task
-        results_dir: specifes the directory on disk to which all Task results are written *--* Required by upstream Task
+        target_file: specifies the file on disk containing a list of ips or domains *Required by upstream Task*
+        exempt_list: Path to a file providing blacklisted subdomains, one per line. *Optional by upstream Task*
+        results_dir: specifes the directory on disk to which all Task results are written *Required by upstream Task*
     """
 
     def requires(self):

recon/masscan.py

@@ -14,25 +14,37 @@
 
 @inherits(TargetList, ParseAmassOutput)
 class MasscanScan(luigi.Task):
-    """ Run masscan against a target specified via the TargetList Task.
+    """ Run ``masscan`` against a target specified via the TargetList Task.
 
-    Masscan commands are structured like the example below.  When specified, --top_ports is processed and
-    then ultimately passed to --ports.
+    Note:
+        When specified, ``--top_ports`` is processed and then ultimately passed to ``--ports``.
 
-    masscan -v --open-only --banners --rate 1000 -e tun0 -oJ masscan.tesla.json --ports 80,443,22,21 -iL tesla.ips
+    Install:
+        .. code-block:: console
 
-    The corresponding luigi command is shown below.
+            git clone https://github.com/robertdavidgraham/masscan /tmp/masscan
+            make -s -j -C /tmp/masscan
+            sudo mv /tmp/masscan/bin/masscan /usr/local/bin/masscan
+            rm -rf /tmp/masscan
 
-    PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.masscan Masscan --target-file tesla --ports 80,443,22,21
+    Basic Example:
+        .. code-block:: console
+
+            masscan -v --open-only --banners --rate 1000 -e tun0 -oJ masscan.tesla.json --ports 80,443,22,21 -iL tesla.ips
+
+    Luigi Example:
+        .. code-block:: console
+
+            PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.masscan Masscan --target-file tesla --ports 80,443,22,21
 
     Args:
         rate: desired rate for transmitting packets (packets per second)
         interface: use the named raw network interface, such as "eth0"
         top_ports: Scan top N most popular ports
         ports: specifies the port(s) to be scanned
-        target_file: specifies the file on disk containing a list of ips or domains *--* Required by upstream Task
-        exempt_list: Path to a file providing blacklisted subdomains, one per line. *--* Optional for upstream Task
-        results_dir: specifies the directory on disk to which all Task results are written *--* Optional for upstream Task
+        target_file: specifies the file on disk containing a list of ips or domains *Required by upstream Task*
+        results_dir: specifes the directory on disk to which all Task results are written *Required by upstream Task*
+        exempt_list: Path to a file providing blacklisted subdomains, one per line. *Optional by upstream Task*
     """
 
     rate = luigi.Parameter(default=defaults.get("masscan-rate", ""))
@@ -114,12 +126,12 @@ class ParseMasscanOutput(luigi.Task):
     """ Read masscan JSON results and create a pickled dictionary of pertinent information for processing.
 
     Args:
-        top_ports: Scan top N most popular ports *--* Required by upstream Task
-        ports: specifies the port(s) to be scanned *--* Required by upstream Task
-        interface: use the named raw network interface, such as "eth0" *--* Required by upstream Task
-        rate: desired rate for transmitting packets (packets per second) *--* Required by upstream Task
-        target_file: specifies the file on disk containing a list of ips or domains *--* Required by upstream Task
-        results_dir: specifes the directory on disk to which all Task results are written *--* Required by upstream Task
+        top_ports: Scan top N most popular ports *Required by upstream Task*
+        ports: specifies the port(s) to be scanned *Required by upstream Task*
+        interface: use the named raw network interface, such as "eth0" *Required by upstream Task*
+        rate: desired rate for transmitting packets (packets per second) *Required by upstream Task*
+        target_file: specifies the file on disk containing a list of ips or domains *Required by upstream Task*
+        results_dir: specifes the directory on disk to which all Task results are written *Required by upstream Task*
     """
 
     def requires(self):

recon/nmap.py

@@ -13,24 +13,30 @@
 
 @inherits(ParseMasscanOutput)
 class ThreadedNmapScan(luigi.Task):
-    """ Run nmap against specific targets and ports gained from the ParseMasscanOutput Task.
+    """ Run ``nmap`` against specific targets and ports gained from the ParseMasscanOutput Task.
 
-    nmap commands are structured like the example below.
+    Install:
+        ``nmap`` is already on your system if you're using kali.  If you're not using kali, refer to your own
+        distributions instructions for installing ``nmap``.
 
-    nmap --open -sT -sC -T 4 -sV -Pn -p 43,25,21,53,22 -oA htb-targets-nmap-results/nmap.10.10.10.155-tcp 10.10.10.155
+    Basic Example:
+        .. code-block:: console
 
-    The corresponding luigi command is shown below.
+            nmap --open -sT -sC -T 4 -sV -Pn -p 43,25,21,53,22 -oA htb-targets-nmap-results/nmap.10.10.10.155-tcp 10.10.10.155
 
-    PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.nmap ThreadedNmap --target-file htb-targets --top-ports 5000
+    Luigi Example:
+        .. code-block:: console
+
+            PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.nmap ThreadedNmap --target-file htb-targets --top-ports 5000
 
     Args:
         threads: number of threads for parallel nmap command execution
-        rate: desired rate for transmitting packets (packets per second) *--* Required by upstream Task
-        interface: use the named raw network interface, such as "eth0" *--* Required by upstream Task
-        top_ports: Scan top N most popular ports *--* Required by upstream Task
-        ports: specifies the port(s) to be scanned *--* Required by upstream Task
-        target_file: specifies the file on disk containing a list of ips or domains *--* Required by upstream Task
-        results_dir: specifes the directory on disk to which all Task results are written *--* Required by upstream Task
+        rate: desired rate for transmitting packets (packets per second) *Required by upstream Task*
+        interface: use the named raw network interface, such as "eth0" *Required by upstream Task*
+        top_ports: Scan top N most popular ports *Required by upstream Task*
+        ports: specifies the port(s) to be scanned *Required by upstream Task*
+        target_file: specifies the file on disk containing a list of ips or domains *Required by upstream Task*
+        results_dir: specifes the directory on disk to which all Task results are written *Required by upstream Task*
     """
 
     threads = luigi.Parameter(default=defaults.get("threads", ""))
@@ -127,24 +133,30 @@ def run(self):
 
 @inherits(ThreadedNmapScan)
 class SearchsploitScan(luigi.Task):
-    """ Run searchcploit against each nmap*.xml file in the TARGET-nmap-results directory and write results to disk.
+    """ Run ``searchcploit`` against each ``nmap*.xml`` file in the **TARGET-nmap-results** directory and write results to disk.
+
+    Install:
+        ``searchcploit`` is already on your system if you're using kali.  If you're not using kali, refer to your own
+        distributions instructions for installing ``searchcploit``.
 
-    searchsploit commands are structured like the example below.
+    Basic Example:
+        .. code-block:: console
 
-    searchsploit --nmap htb-targets-nmap-results/nmap.10.10.10.155-tcp.xml
+            searchsploit --nmap htb-targets-nmap-results/nmap.10.10.10.155-tcp.xml
 
-    The corresponding luigi command is shown below.
+    Luigi Example:
+        .. code-block:: console
 
-    PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.nmap Searchsploit --target-file htb-targets --top-ports 5000
+            PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.nmap Searchsploit --target-file htb-targets --top-ports 5000
 
     Args:
-        threads: number of threads for parallel nmap command execution *--* Required by upstream Task
-        rate: desired rate for transmitting packets (packets per second) *--* Required by upstream Task
-        interface: use the named raw network interface, such as "eth0" *--* Required by upstream Task
-        top_ports: Scan top N most popular ports *--* Required by upstream Task
-        ports: specifies the port(s) to be scanned *--* Required by upstream Task
-        target_file: specifies the file on disk containing a list of ips or domains *--* Required by upstream Task
-        results_dir: specifies the directory on disk to which all Task results are written *--* Required by upstream Task
+        threads: number of threads for parallel nmap command execution *Required by upstream Task*
+        rate: desired rate for transmitting packets (packets per second) *Required by upstream Task*
+        interface: use the named raw network interface, such as "eth0" *Required by upstream Task*
+        top_ports: Scan top N most popular ports *Required by upstream Task*
+        ports: specifies the port(s) to be scanned *Required by upstream Task*
+        target_file: specifies the file on disk containing a list of ips or domains *Required by upstream Task*
+        results_dir: specifies the directory on disk to which all Task results are written *Required by upstream Task*
     """
 
     def requires(self):

recon/web/aquatone.py

@@ -12,24 +12,35 @@
 class AquatoneScan(luigi.Task):
     """ Screenshot all web targets and generate HTML report.
 
-    aquatone commands are structured like the example below.
+    Install:
+        .. code-block:: console
 
-    cat webtargets.tesla.txt | /opt/aquatone -scan-timeout 900 -threads 20
+            mkdir /tmp/aquatone
+            wget -q https://github.com/michenriksen/aquatone/releases/download/v1.7.0/aquatone_linux_amd64_1.7.0.zip -O /tmp/aquatone/aquatone.zip
+            unzip /tmp/aquatone/aquatone.zip -d /tmp/aquatone
+            sudo mv /tmp/aquatone/aquatone /usr/local/bin/aquatone
+            rm -rf /tmp/aquatone
 
-    An example of the corresponding luigi command is shown below.
+    Basic Example:
+        ``aquatone`` commands are structured like the example below.
 
-    PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.web.aquatone AquatoneScan --target-file tesla --top-ports 1000
+        ``cat webtargets.tesla.txt | /opt/aquatone -scan-timeout 900 -threads 20``
+
+    Luigi Example:
+        .. code-block:: python
+
+            PYTHONPATH=$(pwd) luigi --local-scheduler --module recon.web.aquatone AquatoneScan --target-file tesla --top-ports 1000
 
     Args:
         threads: number of threads for parallel aquatone command execution
         scan_timeout: timeout in miliseconds for aquatone port scans
-        exempt_list: Path to a file providing blacklisted subdomains, one per line. *--* Optional for upstream Task
-        top_ports: Scan top N most popular ports *--* Required by upstream Task
-        ports: specifies the port(s) to be scanned *--* Required by upstream Task
-        interface: use the named raw network interface, such as "eth0" *--* Required by upstream Task
-        rate: desired rate for transmitting packets (packets per second) *--* Required by upstream Task
-        target_file: specifies the file on disk containing a list of ips or domains *--* Required by upstream Task
-        results_dir: specifes the directory on disk to which all Task results are written *--* Required by upstream Task
+        exempt_list: Path to a file providing blacklisted subdomains, one per line. *Optional by upstream Task*
+        top_ports: Scan top N most popular ports *Required by upstream Task*
+        ports: specifies the port(s) to be scanned *Required by upstream Task*
+        interface: use the named raw network interface, such as "eth0" *Required by upstream Task*
+        rate: desired rate for transmitting packets (packets per second) *Required by upstream Task*
+        target_file: specifies the file on disk containing a list of ips or domains *Required by upstream Task*
+        results_dir: specifes the directory on disk to which all Task results are written *Required by upstream Task*
     """
 
     threads = luigi.Parameter(default=defaults.get("threads", ""))