added prime module

Author: lopes

README.md

@@ -1,14 +1,18 @@
 # netbox-scanner
-A scanner util for [NetBox](https://netbox.readthedocs.io/en/stable/), because certain networks can be updated automagically.  netbox-scanner aims to create, update, and delete hosts (`/32`) in NetBox, either discovered after network scans and synchronized from other sources.
+A scanner util for [NetBox](https://netbox.readthedocs.io/en/stable/), because certain networks can be updated *automagically*.  netbox-scanner aims to create, update, and delete hosts (`/32`) in NetBox, either discovered by network scans and synchronized from other sources.
+
+> I know it's no more a scanner since version 2, because the focus now is to synchronize different databases to NetBox, so a better name would be `netbox-sync`.
 
 
 ## Installation
 netbox-scanner is compatible with **Python 3.7+**, and can be installed like this:
 
-    $ wget https://github.com/forkd/netbox-scanner/archive/master.zip
-    $ unzip netbox-scanner-master.zip -d netbox-scanner
-    $ cd netbox-scanner
-    $ pip install -r requirements.txt
+```bash
+$ wget https://github.com/lopes/netbox-scanner/archive/master.zip
+$ unzip netbox-scanner-master.zip -d netbox-scanner
+$ cd netbox-scanner
+$ pip install -r requirements.txt
+```
 
 After installation, use the `netbox-scanner.conf` file as an example to create your own and put this file in `/opt/netbox` or prepend its name with a dot and put it in your home directory --`~/.netbox-scanner.conf`.  Keep reading to learn more about configuration.
 
@@ -18,6 +22,8 @@ netbox-scanner reads a user-defined source to discover IP addresses and descript
 
 It is important to note that if netbox-scanner cannot define the description for a given host, then it will insert the string defined in the `unknown` parameter.  Users can change those names at their own will.
 
+For NetBox access, this script uses [pynetbox](https://github.com/digitalocean/pynetbox) --worth saying that was tested under NetBox v2.6.7.
+
 ### Garbage Collection
 If the user marked the `cleanup` option to `yes`, then netbox-scanner will run a garbage collector after the synchronization finishes.  Basically, it will get all IP addresses recorded in NetBox under the same tag.  Then, both lists will be compared: the one just retrieved from NetBox and the list that was synced.  Hosts in the first list that don't appear in the second list will be deleted.
 
@@ -27,7 +33,7 @@ Users can interact with netbox-scanner by command line and configuration file.
 
 The configuration file (`netbox-scanner.conf`) is where netbox-scanner looks for details such as authentication data and path to files.  This file can be stored on the user's home directory or on `/opt/netbox`, but if you choose the first option, it must be a hidden file --`.netbox-scanner.conf`.
 
-    Remember that netbox-scanner will always look for this file at home directory, then at `/opt/netbox`, in this order.  The first occurrence will be considered.
+> Remember that netbox-scanner will always look for this file at home directory, then at `/opt/netbox`, in this order.  The first occurrence will be considered.
 
 
 ## Modules
@@ -39,14 +45,37 @@ Since version 2.0, netbox-scanner is based on modules.  This way, this program i
 
 
 ## Nmap Module
-Performing the scans is beyond netbox-scanner features, so you must run Nmap and save the output as an XML file using the `-oX` parameter.  Since this file can grow really fast, you can scan each network and save it as a single XML file.  You just have to assure that all files are under the same directory before running the script --see `samples/nmap.sh` for an example.
+Performing the scans is beyond netbox-scanner features, so you must run [Nmap](https://nmap.org/) and save the output as an XML file using the `-oX` parameter.  Since this file can grow really fast, you can scan each network and save it as a single XML file.  You just have to assure that all files are under the same directory before running the script --see `samples/nmap.sh` for an example.
 
-To properly setup this module, you must inform the path to the directory where the XML files reside, define a tag to insert to discovered hosts, and decide if clean up will take place.
+To properly setup this module, you must inform the path to the directory where the XML files reside, define a tag to insert to discovered hosts, and decide if clean up will take place.  Tested on Nmap v7.80.
 
 
 ## Prime Module
-To be written.
+This script accesses [Prime](https://www.cisco.com/c/en/us/products/cloud-systems-management/prime-infrastructure/index.html) through RESTful API and all routines are implemented here.  Users only have to point to Prime's API, which looks like `https://prime.domain/webacs/api/v4/`, inform valid credentials allowed to use the API, and fill the other variables, just like in Nmap.
+
+It is important to note everything was tested on Cisco Prime v3.4.0, using API v4.  It was noticed that when trying to retrieve access points data, Prime requires more privileges, so you must explicitly inform that you wish this by using `Prime().run(access_points=True)`.
+
+
+## Tests
+Some basic tests are implemented under `tests`.  This directory comes with a shell script to run all the tests at once, but before running it, remember to setup some environment variables required by them, such as `NETBOX_ADDRESS` and `NMAP_PATH`.
+
+Here's the list of all variables:
+
+```bash
+NETBOX_ADDRESS
+NETBOX_TOKEN 
+
+NMAP_PATH
+
+PRIME_ADDRESS
+PRIME_USER
+PRIME_PASS
+```
+
+
+## New Modules
+New modules should be implemented as a new file with the name of the module, for instance `nbs/netxms.py`.  In this case, a class `NetXMS` should be created in this file with a method `run`.  Finally, in `netbox-scanner.py`, a function `cmd_netxms` should be created to execute the just created class, and another option should be created both in the argument parsing section and in the if structure inside the main block.
 
 
 ## License
-`netbox-scanner` is licensed under an MIT license --read `LICENSE` file for more information.
+netbox-scanner is licensed under an MIT license --read `LICENSE` file for more information.

nbs/__init__.py

@@ -5,9 +5,8 @@
 
 class NetBoxScanner(object):
 
-    def __init__(self, address, token, tls_verify, hosts, tag, cleanup):
+    def __init__(self, address, token, tls_verify, tag, cleanup):
         self.netbox = api(address, token, ssl_verify=tls_verify)
-        self.hosts = hosts
         self.tag = tag
         self.cleanup = cleanup
         self.stats = {
@@ -56,29 +55,29 @@ def sync_host(self, host):
 
         return True
 
-    def garbage_collector(self):
+    def garbage_collector(self, hosts):
         '''Removes records from NetBox not found in last sync'''
         nbhosts = self.netbox.ipam.ip_addresses.filter(tag=self.tag)
         for nbhost in nbhosts:
             nbh = str(nbhost).split('/')[0]
-            if not any(nbh == addr[0] for addr in self.hosts):
+            if not any(nbh == addr[0] for addr in hosts):
                 nbhost.delete()
-                logging.info(f'deleted: {nbhost[0]}')
+                logging.info(f'deleted: {nbhost}')
                 self.stats['deleted'] += 1
 
-    def sync(self):
-        '''Synchronizes self.hosts to NetBox
-        Returns synching statistics
+    def sync(self, hosts):
+        '''Syncs hosts to NetBox
+        hosts: list of tuples like [(addr,description),...]
         '''
         for s in self.stats:
             self.stats[s] = 0
 
-        logging.info('started: {} hosts'.format(len(self.hosts)))
-        for host in self.hosts:
+        logging.info('started: {} hosts'.format(len(hosts)))
+        for host in hosts:
             self.sync_host(host)
 
         if self.cleanup:
-            self.garbage_collector()
+            self.garbage_collector(hosts)
 
         logging.info('finished: .{} +{} ~{} -{} !{}'.format(
             self.stats['unchanged'],

nbs/prime.md

@@ -0,0 +1,24 @@
+# Prime
+This is the [Cisco Prime](https://www.cisco.com/c/en/us/products/cloud-systems-management/prime-infrastructure/index.html) API wrapper.  Prime is self-explanatory and it's highly recommended to read it before start using any endpoints.  You can access this documentation page at API's root address (see below).
+
+Prerequisites:
+
+* Tested under Prime 3.4 and API v4
+* URL for API (usually `https://prime.corp/webacs/api/v4`)
+* Credentials for API access
+
+To start using this wrapper, you must create a user in Prime with NBI permissions, according to the resources you need to access.  Read `?id=authentication-doc` under Prime's own documentation to learn more about such privileges.
+
+Remember that Prime's resources are case sensitive, so programmers must be aware when requesting something.  Then, reading the documentation is pretty important.  At this point, this wrapper only accepts reading requests, but it should be improved in the near future.
+
+
+## Basic Usage
+
+```python
+>>> from corsair.cisco.prime import Api
+>>> prime = Api('https://prime.corp/webacs/api/v4', 'cors', 'Strong_P4$$w0rd!')
+>>> prime.op.read('aaa/tacacsPlusServer')
+>>> prime.data.read('Devices')
+>>> prime.data.read('Devices', full='true')
+>>> prime.data.read('AccessPoints', firstResult=350, full='true')
+```

nbs/prime.py

@@ -0,0 +1,135 @@
+import urllib.request
+
+from urllib.parse import urlencode, urlsplit
+from json import loads
+from socket import timeout
+from ssl import _create_unverified_context
+from base64 import b64encode
+
+
+TIMEOUT = 20
+
+
+def gen_auth(username, password):
+    'Generate basic authorization using username and password'
+    return b64encode(f'{username}:{password}'.encode('utf-8')).decode()
+
+def make_url(base_url, endpoint, resource):
+    'Corsair creates URLs using this method'
+    base_url = urlsplit(base_url)
+    path = base_url.path + f'/{endpoint}/{resource}'
+    path = path.replace('//', '/')
+    path = path[:-1] if path.endswith('/') else path
+    return base_url._replace(path=path).geturl()
+
+
+class Api(object):
+    def __init__(self, base_url, username, password, tls_verify=True):
+        self.base_url = base_url if base_url[-1] != '/' else base_url[:-1]
+        self.auth = gen_auth(username, password)
+        self.tls_verify = tls_verify
+        self.credentials = (self.base_url, self.auth, self.tls_verify)
+
+        self.data = Endpoint(self.credentials, 'data')
+        self.op = Endpoint(self.credentials, 'op')
+
+
+class Endpoint(object):
+    def __init__(self, credentials, endpoint):
+        self.base_url = credentials[0]
+        self.endpoint = endpoint
+        self.resource = ''
+        self.auth = credentials[1]
+        self.tls_verify = credentials[2]
+    
+    def read(self, _resource, **filters):
+        self.resource = f'{_resource}.json'  # will only deal with JSON outputs
+        first_result = 0 if 'firstResult' not in filters else filters['firstResult']
+        max_results = 1000 if 'maxResults' not in filters else filters['maxResults']
+        filters.update({'firstResult':first_result, 'maxResults':max_results})
+        req = Request(make_url(self.base_url, self.endpoint, self.resource), 
+            self.auth, self.tls_verify)
+        try:
+            res = req.get(**filters)
+        except timeout:
+            raise Exception('Operation timedout')
+        return loads(res.read())  # test for possible Prime errors
+
+
+class Request(object):
+    def __init__(self, url, auth, tls_verify):
+        self.url = url
+        self.auth = auth
+        self.timeout = TIMEOUT
+        self.context = None if tls_verify else _create_unverified_context()
+        self.headers = {
+            'Content-Type': 'application/json',
+            'Authorization': f'Basic {self.auth}'
+        }
+
+    def get(self, **filters):
+        url = f'{self.url}?{self.dotted_filters(**filters)}' if filters else self.url
+        req = urllib.request.Request(url, headers=self.headers, method='GET')
+        return urllib.request.urlopen(req, timeout=self.timeout, context=self.context)
+    
+    def dotted_filters(self, **filters):
+        'Prime filters start with a dot'
+        if not filters:
+            return ''
+        else:
+            return f'.{urlencode(filters).replace("&", "&.")}'
+
+
+class Prime(object):
+
+    def __init__(self, address, username, password, tls_verify, unknown):
+        self.prime = Api(address, username, password, tls_verify)
+        self.unknown = unknown
+        self.hosts = list()
+    
+    def run(self, access_points=False):
+        '''Extracts devices from Cisco Prime
+        access_points: if set to True, will try to get APs data
+        Returns False for no errors or True if errors occurred
+        '''
+        errors = False
+        devices = self.get_devices('Devices')
+        for device in devices:
+            try:
+                self.hosts.append((
+                    device['devicesDTO']['ipAddress'],
+                    device['devicesDTO']['deviceName']
+                ))
+            except KeyError:
+                errors = True
+        
+        if access_points:
+            aps = self.get_devices('AccessPoints')
+            for ap in aps:
+                try:
+                    self.hosts.append((
+                        ap['accessPointsDTO']['ipAddress']['address'],
+                        ap['accessPointsDTO']['model']
+                    ))
+                except KeyError:
+                    errors = True
+        return errors
+        
+    def get_devices(self, resource):
+        'This function is used to support run()'
+        raw = list()
+        res = self.prime.data.read(resource, full='true')
+        count = res['queryResponse']['@count']
+        last = res['queryResponse']['@last']
+        raw.extend(res['queryResponse']['entity'])
+        while last < count - 1:
+            first_result = last + 1
+            last += 1000
+            res = self.prime.data.read(
+                resource, 
+                full='true', 
+                firstResult=first_result
+            )
+            raw.extend(res['queryResponse']['entity'])
+        return raw
+

netbox-scanner.conf

@@ -11,9 +11,10 @@ tag      = nmap
 cleanup  = yes
 
 [PRIME]
-address  = https://prime.domain
-username = 
-password = 
-unknown  = autodiscovered:netbox-scanner
-tag      = prime
-cleanup  = yes
+address    = https://prime.domain/webacs/api/v4
+username   = 
+password   = 
+tls_verify = no
+unknown    = autodiscovered:netbox-scanner
+tag        = prime
+cleanup    = yes

netbox-scanner.py

@@ -11,6 +11,7 @@
 
 from nbs import NetBoxScanner
 from nbs.nmap import Nmap
+from nbs.prime import Prime
 
 
 local_config = expanduser('~/.netbox-scanner.conf')
@@ -49,24 +50,37 @@
 disable_warnings(InsecureRequestWarning)
 
 
-def cmd_nmap():  # nmap handler
+def cmd_nmap(s):  # nmap handler
     h = Nmap(nmap['path'], nmap['unknown'])
     h.run()
-    scan = NetBoxScanner(
+    s.sync(h.hosts)
+
+def cmd_prime(s):  # prime handler
+    h = Prime(
+        prime['address'],
+        prime['username'],
+        prime['password'],
+        prime.getboolean('tls_verify'), 
+        prime['unknown']
+    )
+    h.run()  # set access_point=True to process APs
+    s.sync(h.hosts)
+
+
+if __name__ == '__main__':
+    scanner = NetBoxScanner(
         netbox['address'],
         netbox['token'],
         netbox.getboolean('tls_verify'), 
-        h.hosts, 
         nmap['tag'], 
         nmap.getboolean('cleanup')
     )
-    scan.sync()
-
-def cmd_prime():  # prime handler
-    pass
 
+    if args.command == 'nmap':
+        cmd_nmap(scanner)
+    elif args.command == 'prime':
+        scanner.tag = prime['tag']
+        scanner.cleanup = prime.getboolean('cleanup')
+        cmd_prime(scanner)
 
-if __name__ == '__main__':
-    if args.command == 'nmap': cmd_nmap()
-    elif args.command == 'prime': cmd_prime()
     exit(0)

tests/run_tests.sh

@@ -10,3 +10,4 @@
 
 python -m unittest tests.test_netbox
 python -m unittest tests.test_nmap
+python -m unittest tests.test_prime

tests/test_netbox.py

@@ -8,9 +8,9 @@ def test_api(self):
         address = environ.get('NETBOX_ADDRESS')
         token = environ.get('NETBOX_TOKEN')
 
-        netbox = NetBoxScanner(address, token, False, [], 'test', False)
+        netbox = NetBoxScanner(address, token, False, 'test', False)
         self.assertIsInstance(netbox, NetBoxScanner)
-        self.assertEqual(netbox.sync(), True)
+        self.assertEqual(netbox.sync([]), True)
 
 
 if __name__ == '__main__':