v1.6.0

## v1.6.0 ||| |---|---| |Date|2022-08 |Kind| BUGFIX release |Author|[email protected] - **Features** - Support for Akamai Guardicore Segmentation (experimental) - Available feeds: netlog, incident, agent, system Please ensure to [update your .edgerc](./AKAMAI_API_CREDENTIALS.md#guardicore-api-integration) file for GC usage - Support for Akamai Linode Cloud (experimental) - Available feed: audit Please ensure to [update your .edgerc](AKAMAI_API_CREDENTIALS.md#linode-api-credentials) file for GC usage - **Minor improvements** - ULS [Install Script](COMMAND_LINE_USAGE.md#automated-installation) allows fully working ULS installation via a single script - ULS [Updater](COMMAND_LINE_USAGE.md#automated-update) helps to maintain a proper updated version of ULS + Modules - Amended [Command Line Usage](COMMAND_LINE_USAGE.md) documentation on how to use the installer / updater - bumped python container (docker) to version 3.10.7 - bumped ETP-CLI version to 0.3.9 which should massively reduce the fetch lag - Added [documentation](./HIGH_AVAILABILITY.md) to explain high availability options for ULS
akamai · Sep 15, 2022 · 24b1594 · 24b1594
1 parent 96c31cf
commit 24b1594
Show file tree

Hide file tree

Showing 18 changed files with 1,061 additions and 81 deletions.
diff --git a/Dockerfile b/Dockerfile
@@ -1,4 +1,4 @@
-FROM            python:3.10.5-slim-bullseye
+FROM            python:3.10.7-slim-bullseye
 LABEL 	        MAINTAINER="Mike Schiessl - [email protected]"
 LABEL	        APP_LONG="Akamai Universal Log Streamer"
 LABEL           APP_SHORT="ULS"
@@ -10,9 +10,11 @@ ARG             HOMEDIR="/opt/akamai-uls"
 ARG             ULS_DIR="$HOMEDIR/uls"
 ARG             EXT_DIR="$ULS_DIR/ext"
 
-ARG             ETP_CLI_VERSION="0.3.8"
+ARG             ETP_CLI_VERSION="0.3.9"
 ARG             EAA_CLI_VERSION="0.5.0.2"
 ARG             MFA_CLI_VERSION="0.0.9"
+ARG             GC_CLI_VERSION="dev"
+ARG             LINODE_CLI_VERSION="dev"
 
 # ENV VARS
 ENV             ULS_DIR=$ULS_DIR
@@ -63,6 +65,16 @@ ENV             MFA-CLI_VERSION=$MFA_CLI_VERSION
 RUN             git clone --depth 1 -b "${MFA_CLI_VERSION}" --single-branch https://github.com/akamai/cli-mfa.git ${EXT_DIR}/cli-mfa && \
                 pip3 install -r ${EXT_DIR}/cli-mfa/requirements.txt
 
+## GC CLI
+ENV             GC_CLI_VERSION=$GC_CLI_VERSION
+RUN             git clone --depth 1 -b "${GC_CLI_VERSION}" --single-branch https://github.com/MikeSchiessl/gc-logs.git ${EXT_DIR}/cli-gc && \
+                pip3 install -r ${EXT_DIR}/cli-gc/bin/requirements.txt
+
+## LINODE CLI
+ENV             LINODE_CLI_VERSION=$LINODE_CLI_VERSION
+RUN             git clone --depth 1 -b "${LINODE_CLI_VERSION}" --single-branch https://github.com/MikeSchiessl/ln-logs.git ${EXT_DIR}/cli-linode && \
+                pip3 install -r ${EXT_DIR}/cli-linode/bin/requirements.txt
+
 # ENTRYPOINTS / CMD
 VOLUME          ["${ULS_DIR}/var"]
 ENTRYPOINT      ["/usr/local/bin/python3","-u","bin/uls.py"]

diff --git a/README.md b/README.md
@@ -6,21 +6,18 @@ The Unified Log Streamer (ULS) is designed to simplify SIEM integrations for Aka
 - [Enterprise Application Access (EAA)](https://www.akamai.com/us/en/products/security/enterprise-application-access.jsp) 
 - [Enterprise Threat Protector (ETP)](https://www.akamai.com/us/en/products/security/enterprise-threat-protector.jsp)
 - [Akamai MFA (MFA)](https://www.akamai.com/us/en/products/security/akamai-mfa.jsp)
+- [Guardicore Micro Segmentation](https://www.akamai.com/lp/guardicore)
 
 Thanks to its modular design, ULS allows the connection of many SIEM solutions out-of-the-box.  
 ULS can send data into any SIEM that supports either file, TCP, UDP or HTTP ingestion.  
 
 It can be run directly as Python code, as a provided Docker container,  through `docker compose` scripts or through helm within kubernetes.
 
-
-
 ![ULS docker compose usage](docs/images/uls_docker-compose_complex_example.png)
 
-
-## Table of contents
+## Table of contents<!-- omit in toc -->
 - [Akamai Unified Log Streamer (ULS)](#akamai-unified-log-streamer-uls)
   - [Introduction](#introduction)
-  - [Table of contents](#table-of-contents)
   - [Key Features](#key-features)
   - [Documentation](#documentation)
     - [Generic Requirements](#generic-requirements)
@@ -35,7 +32,7 @@ It can be run directly as Python code, as a provided Docker container,  through
 
 ## Key Features
 
-- Supported Inputs (Secure Enterprise Access Products)
+- Supported Inputs
     - [Enterprise Application Access (EAA)](https://www.akamai.com/us/en/products/security/enterprise-application-access.jsp)
       - [ACCESS](docs/LOG_OVERVIEW.md#access-logs-access) 
       - [ADMIN](docs/LOG_OVERVIEW.md#admin-logs-admin)
@@ -48,6 +45,13 @@ It can be run directly as Python code, as a provided Docker container,  through
       - [PROXY](docs/LOG_OVERVIEW.md#proxy)
     - [Akamai Phish-proof Multi Factor Authenticator (AKAMAI-MFA)](https://www.akamai.com/us/en/products/security/akamai-mfa.jsp)
       - [EVENT](docs/LOG_OVERVIEW.md#authentication-logs-auth)
+    - [Akamai Guardicore Segmentation](https://www.akamai.com/lp/guardicore) (experimental)
+      - [NETLOG](docs/LOG_OVERVIEW.md#netlog)
+      - [INCIDENT](docs/LOG_OVERVIEW.md#incident)
+      - AGENT
+      - SYSTEM
+    - [Linode](https://www.linode.com/) (experimental)
+      - AUDIT
 
 
 - Supported data outputs
@@ -122,6 +126,7 @@ In parallel, all new versions within the "main" branch will also be available on
 Contributions to this software can be provided via [Pull Requests](https://docs.github.com/en/github/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/about-pull-requests) and will get merged after successful review. 
 
 ## Changelog
+
 Find a full Changelog of all added Features and fixed Bugs here:  
 [ULS - Changelog](./docs/CHANGELOG.md)
 
@@ -130,10 +135,11 @@ Find a full Changelog of all added Features and fixed Bugs here:
 Akamai ULS is provided "as-is". It is not supported by Akamai Support. Akamai is neither liable for the function nor for any caused problems that come along with the usage or caused by this tool. Please refer to the [LICENSE](./LICENSE) document for more information.
 
 To report an issue, feature request or bug, please open a new issue into the [GitHub Issues page](https://github.com/akamai/uls/issues).
-This software is released under the "Apache License". Please read the [frequently asked questions](docs/FAQ.md) and visit the [debugging instructions](./docs/DEBUGGING.md) before opening a bug request.  
+This software is released under the "Apache License". Please read the [frequently asked questions](docs/FAQ.md) and visit the [troubleshooting and debugging instructions](./docs/DEBUGGING.md) before opening a ticket.  
 
 [Pull requests](#development) to improve the code or enhance the functionality are welcome.
 
 ## LINKS / REFERENCES
+
 [The ULS Project on GitHub](https://github.com/akamai/uls)  
 [The ULS Project on Dockerhub](https://hub.docker.com/r/akamai/uls)
diff --git a/bin/config/global_config.py b/bin/config/global_config.py
@@ -1,7 +1,7 @@
 #!/usr/bin/env python3
 
 # Common global variables / constants
-__version__ = "1.5.1"
+__version__ = "1.6.0"
 __tool_name_long__ = "Akamai Unified Log Streamer"
 __tool_name_short__ = "ULS"
 
@@ -31,8 +31,16 @@
 bin_mfa_cli = "ext/cli-mfa/bin/akamai-mfa"      # Path to the MFA CLI Executable
 mfa_cli_feeds = ['EVENT']              # Available MFA CLI feeds
 
+    # Guardicore
+bin_gc_cli = "ext/cli-gc/bin/akamai-gc"                                         # Path to the GC CLI Executable
+gc_cli_feeds = ['NETLOG', 'INCIDENT', 'AGENT', 'SYSTEM']                       # Available GC CLI feeds
+
+    # LINODE
+bin_linode_cli = "ext/cli-linode/bin/akamai-linode"         # Path to the LINODE CLI Executable
+linode_cli_feeds = ['AUDIT']                        # Available LINODE CLI feeds
+
     # INPUT Choices
-input_choices = ['EAA', 'ETP', 'MFA']           # Available input types
+input_choices = ['EAA', 'ETP', 'MFA', 'GC', 'LINODE']           # Available input types
 input_format_choices = ['JSON', 'TEXT']         # Available input format choices (need to be supported by cli)
 
     # OUTPUT Choices
@@ -85,6 +93,8 @@
 edgerc_openapi = ["host", "client_token", "client_secret", "access_token"]          # required fields for OPENAPI
 edgerc_eaa_legacy = ["eaa_api_host", "eaa_api_key", "eaa_api_secret"]               # required for EAA - Legacy
 edgerc_mfa = ["mfa_integration_id", "mfa_signing_key"]                              # Required for MFA
+edgerc_gc = ["gc_username", "gc_password", "gc_hostname"]                           # Required for Guardicore
+edgerc_linode = ["linode_hostname", "linode_token"]                                             # Required for Linode
 edgerc_documentation_url = "https://github.com/akamai/uls/blob/main/docs/AKAMAI_API_CREDENTIALS.md"
 edgerc_mock_file = "ext/edgerc"                  # Required for display the version if no edgercfile was given
 

diff --git a/bin/modules/UlsInputCli.py b/bin/modules/UlsInputCli.py
@@ -256,6 +256,67 @@ def proc_create(self):
                                   self._uls_useragent(self.product, "rawcmd") +\
                                   shlex.split(self.rawcmd)
 
+            # Guardicore config
+            elif self.product == "GC":
+                product_path = self.root_path + "/" + uls_config.bin_gc_cli
+                product_feeds = uls_config.gc_cli_feeds
+                if not self.cliformat == "JSON":
+                    aka_log.log.warning(f"{self.name} - Selected LOG Format ({self.cliformat}) "
+                                        f"not available for {product_path}, continuing with JSON.")
+                if not self.rawcmd:
+                    UlsTools.uls_check_edgerc(self.credentials_file,
+                                              self.credentials_file_section,
+                                              uls_config.edgerc_gc)
+                    my_feed = self._feed_selector(self.feed, product_feeds)
+                    cli_command = [self.bin_python, '-u', product_path, 'events', my_feed.lower(), '-f']
+                    cli_command[3:3] = self._uls_useragent(self.product, my_feed)
+                    cli_command[3:3] = edgegrid_auth
+                    cli_command[3:3] = self._prep_proxy(self.inproxy)
+
+                    # Append End and Starttime
+                    if self.endtime:
+                        # We need to remove "-f" from the end of the cli cmd if we work with endtime
+                        cli_command = cli_command[:-1]
+                        cli_command.extend(self._prep_start_endtime('--end', self.endtime))
+                    if self.starttime:
+                        cli_command.extend(self._prep_start_endtime('--start', self.starttime))
+
+                else:
+                    cli_command = [self.bin_python, product_path] +\
+                                  self._uls_useragent(self.product, "rawcmd") +\
+                                  shlex.split(self.rawcmd)
+
+            # LINODE config
+            elif self.product == "LINODE":
+                product_path = self.root_path + "/" + uls_config.bin_linode_cli
+                product_feeds = uls_config.linode_cli_feeds
+                if not self.cliformat == "JSON":
+                    aka_log.log.warning(f"{self.name} - Selected LOG Format ({self.cliformat}) "
+                                        f"not available for {product_path}, continuing with JSON.")
+                if not self.rawcmd:
+                    UlsTools.uls_check_edgerc(self.credentials_file,
+                                              self.credentials_file_section,
+                                              uls_config.edgerc_linode)
+                    my_feed = self._feed_selector(self.feed, product_feeds)
+                    cli_command = [self.bin_python, '-u', product_path, 'events', my_feed.lower(), '-f']
+                    cli_command[3:3] = self._uls_useragent(self.product, my_feed)
+                    cli_command[3:3] = edgegrid_auth
+                    cli_command[3:3] = self._prep_proxy(self.inproxy)
+
+                    # Append End and Starttime
+                    if self.endtime:
+                        # We need to remove "-f" from the end of the cli cmd if we work with endtime
+                        cli_command = cli_command[:-1]
+                        cli_command.extend(self._prep_start_endtime('--end', self.endtime))
+                    if self.starttime:
+                        cli_command.extend(self._prep_start_endtime('--start', self.starttime))
+
+                else:
+                    cli_command = [self.bin_python, product_path] +\
+                                  self._uls_useragent(self.product, "rawcmd") +\
+                                  shlex.split(self.rawcmd)
+
+
             # Everything else (undefined)
             else:
                 aka_log.log.critical(f" {self.name} - No valid product selected "

diff --git a/bin/modules/UlsTools.py b/bin/modules/UlsTools.py
@@ -50,6 +50,8 @@ def _check_cli_installed(cli_bin):
     _check_cli_installed(root_path + "/" + uls_config.bin_eaa_cli)
     _check_cli_installed(root_path + "/" + uls_config.bin_etp_cli)
     _check_cli_installed(root_path + "/" + uls_config.bin_mfa_cli)
+    _check_cli_installed(root_path + "/" + uls_config.bin_gc_cli)
+    _check_cli_installed(root_path + "/" + uls_config.bin_linode_cli)
 
 
 def uls_version(root_path):
@@ -60,9 +62,14 @@ def uls_version(root_path):
     my_edgerc_mock_file = root_path + "/" + uls_config.edgerc_mock_file
     def _get_cli_version(cli_bin, edgerc_mock_file):
         try:
-            version_proc = subprocess.Popen([uls_config.bin_python, cli_bin, "--edgerc", edgerc_mock_file, "version"],
+            if "gc" in cli_bin or 'linode' in cli_bin:
+               version_proc = subprocess.Popen([uls_config.bin_python, cli_bin, "--edgerc", edgerc_mock_file, "--version"],
                                             stdout=subprocess.PIPE,
                                             stderr=subprocess.PIPE)
+            else:
+                version_proc = subprocess.Popen([uls_config.bin_python, cli_bin, "--edgerc", edgerc_mock_file, "version"],
+                    stdout=subprocess.PIPE,
+                    stderr=subprocess.PIPE)
             my_cli_version = version_proc.communicate()[0].decode().strip('\n')
             version_proc.terminate()
             if my_cli_version:
@@ -84,7 +91,9 @@ def _get_cli_version(cli_bin, edgerc_mock_file):
           f"ULS Version\t\t{uls_config.__version__}\n\n"
           f"EAA Version\t\t{_get_cli_version(root_path + '/' + uls_config.bin_eaa_cli, my_edgerc_mock_file)}\n"
           f"ETP Version\t\t{_get_cli_version(root_path + '/' + uls_config.bin_etp_cli, my_edgerc_mock_file)}\n"
-          f"MFA Version\t\t{_get_cli_version(root_path + '/' + uls_config.bin_mfa_cli, my_edgerc_mock_file)}\n\n"
+          f"MFA Version\t\t{_get_cli_version(root_path + '/' + uls_config.bin_mfa_cli, my_edgerc_mock_file)}\n"
+          f"GC Version\t\t{_get_cli_version(root_path + '/' + uls_config.bin_gc_cli, my_edgerc_mock_file)}\n"
+          f"LINODE Version\t\t{_get_cli_version(root_path + '/' + uls_config.bin_linode_cli, my_edgerc_mock_file)}\n\n"
           f"OS Plattform\t\t{platform.platform()}\n"
           f"OS Version\t\t{platform.release()}\n"
           f"Python Version\t\t{sys.version_info.major}.{sys.version_info.minor}.{sys.version_info.micro}\n"

diff --git a/bin/requirements.txt b/bin/requirements.txt
@@ -1,2 +1,3 @@
 jmespath>=0.10.0
-requests>=2.25.1
+requests>=2.25.1
+pytz>=2021.1
diff --git a/docs/AKAMAI_API_CREDENTIALS.md b/docs/AKAMAI_API_CREDENTIALS.md
@@ -13,24 +13,25 @@ This document describes how to create Akamai API credentials and configure them
     - [ETP {OPEN} API Reporting](#etp-open-api-reporting)
   - [Akamai MFA](#akamai-mfa)
     - [MFA Integration for logging](#mfa-integration-for-logging)
+  - [Guardicore](#guardicore)
+    - [Guardicore API Integration](#guardicore-api-integration)
+  - [Linode](#linode)
+    - [Linode API Token](#linode-api-credentials)
 - [Advanced .edgerc usage](#advanced-edgerc-usage)
   - [Multiple customer contracts](#multiple-customer-contracts)
   - [Partner & employee enhancement](#partner--employee-enhancement)
   - [ETP API EVENT Filters](#etp-api-event-filters)
 
 ## Feeds / API overview
 
-|Product long name|Acronym|Feed|API|
-|---|---|---|---|
-|Enterprise Application Access|EAA|ACCESS|[EAA Legacy API](#eaa-legacy-api-for-access-and-admin-audit-feeds)|
-|Enterprise Application Access|EAA|ADMIN|[EAA Legacy API](#eaa-legacy-api-for-access-and-admin-audit-feeds)|
-|Enterprise Application Access|EAA|HEALTH|[{OPEN} API / Enterprise Application Access](#eaa-open-api-for-connector-health-feed)|
-|Enterprise Threat Protector|ETP|THREAT|[{OPEN} API / ETP Report](#etp-open-api-reporting)|
-|Enterprise Threat Protector|ETP|AUP|[{OPEN} API / ETP Report](#etp-open-api-reporting)|
-|Enterprise Threat Protector|ETP|DNS|[{OPEN} API / ETP Report](#etp-open-api-reporting)|
-|Enterprise Threat Protector|ETP|PROXY|[{OPEN} API / ETP Report](#etp-open-api-reporting)|
-|Akamai MFA|MFA|AUTH|[MFA Integration](#mfa-integration-for-logging)|
-|Akamai MFA|MFA|POLICY|[MFA Integration](#mfa-integration-for-logging)|
+|Product long name|Acronym| Feed(s)                         | API                                                                                   |
+|---|---|---------------------------------|---------------------------------------------------------------------------------------|
+|Enterprise Application Access|EAA| ACCESS, ADMIN                   | [EAA Legacy API](#eaa-legacy-api-for-access-and-admin-audit-feeds)                    |
+|Enterprise Application Access|EAA| HEALTH                          | [{OPEN} API / Enterprise Application Access](#eaa-open-api-for-connector-health-feed) |
+|Enterprise Threat Protector|ETP| THREAT, AUP, DNS, PROXY         | [{OPEN} API / ETP Report](#etp-open-api-reporting)                                    |
+|Akamai MFA|MFA| EVENTS                          | [MFA Integration](#mfa-integration-for-logging)                                       |
+|Guardicore|GC| NETLOG, INCIDENT, AGENT, SYSTEM | [Guardicore API Integration](#guardicore-api-integration)                             |
+|Linode|LN| AUDIT                           | [Linode API Credentials](#linode-api-credentials) |                       
 
 ## Setting up API credentials for ULS
 
@@ -137,6 +138,44 @@ mfa_integration_id = app_xxxxxxxxxxxxxxxxxxxxx
 mfa_signing_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 ```
 
+### Guardicore
+
+#### Guardicore API Integration
+Guardicore is using the portal users for API access. Therefore it is recommended to create a "read only" (= GUEST role) user within Centra Administration.
+- Go to Administration
+- Select "Users" in the left navigation tree
+- Click the "Create User" button
+- Enter a username and a password, select "Guest" as permission scheme
+- Confirm by clicking the SAVE button
+- Now logout and login with the newly created user and follow tha password change procedure
+- Note down your guardicore Adminsitration (=API) url without https
+- Add/replace/amend the following section to your `.edgerc` file and replace the data accordingly:
+
+```INI
+[default]
+; Guardicore integration credentials
+gc_hostname = your_hostname.guardicore.com          # Do not prepend https://
+gc_username = XXXXXXXXXXXX
+gc_password = XXXXXXXXXXXXX
+```
+
+### Linode
+#### Linode API Credentials
+- Login into your linode cloud console
+- Click on your user name on the top right
+- Select API Tokens
+- Create a personal Access Token
+- Enter a Label and select all privleges to "READ ONLY"
+- Set expiry to your needs
+- Confirm by clicking the CREATE TOKEN button
+- Copy the token provided in the next field
+- Add/replace/amend the following section to your `.edgerc` file and replace the data accordingly:
+```INI
+[default]
+; Guardicore integration credentials
+linode_hostname = your_hostname.guardicore.com          # Do not prepend https://
+linode_token = XXXXXXXXXXXX
+```
 ## Advanced .edgerc usage
 
 ### Multiple customer contracts