Malcolm v25.06.0

Malcolm v25.06.0 includes a some new and oft-requested features, bug fixes, and component version bumps.

v25.05.0...v25.06.0

NOTE: As this Malcolm release enables the OpenSearch Security Plugin as described below, even inter-container access to OpenSearch must now be authenticated when using Malcolm's embedded OpenSearch instance. To accomplish this, an internal-use-only account and password is used for connecting to OpenSearch by Malcolm's other components as needed. This credential (saved in .opensearch.primary.curlrc in the Malcolm installation directory) needs to be generated before Malcolm starts up the first time after upgrading. To do so, please run ./scripts/auth_setup and select (Re)generate internal passwords for local primary OpenSearch instance. This credential is only used internally for OpenSearch and cannot be used to remotely access Malcolm.

• ✨ Features and enhancements
  • This release adds role-based access control (RBAC) to Malcolm (#460).
    • Malcolm's RBAC feature is based on Keycloak realm roles and is implemented in to layers:
      1. Whenever possible, Malcolm's backend Keycloak realm roles are mapped to the roles/groups/permissions features provided by the components that make up Malcolm, for example:
         • OpenSearch (provided by the OpenSearch Security plugin)
           • Configuring the Security backend
           • Modifying the YAML files
           • Predefined roles
         • Arkime
           • Note: As per @awick, until Arkime v6.0.0 is out, not all of the Arkime permissions can be set on roles. For now, then, Malcolm's Arkime roles are going to be handled purely based on URI path in the NGINX stuff as described below.
         • NetBox
           • Authentication & Permissions
           • Object-Based Permissions
           • Mastering NetBox User Access with Permission Constraints
      2. For other Malcolm components that don't implement their own permission management systems, Malcolm handles the enforcement roles based on request URIs in its NGINX proxy layer.
    • This is an optional feature. RBAC is only available when the authentication method is keycloak or keycloak_remote. With other authentication methods such as HTTP basic or LDAP, or when RBAC is disabled, all Malcolm users effectively have administrator privileges.
    • Because the OpenSearch Security Plugin requires TLS even internally, Malcolm's internal connections to the embedded OpenSearch instance, when used, are now all performed over HTTPS. However, this is all handled internally and should not behave or appear different to the user than it did in previous versions.
    • See the role-based access control documentation for more information on this feature.
  • Malcolm's embedded KeyCloak instance now automatically creates and configures the default client by ID, if specified in ./config/keycloak.env.
  • Allow user to specify subnet filters for NetBox autopopulation (#634)
    • This feature is especially useful for excluding dynamic address ranges such as those used by DHCP, which should generally not trigger autopopulation in NetBox. Since these addresses can change frequently and aren't tied to specific devices, including them could result in inaccurate or noisy inventory data. By fine-tuning which private subnets are included or excluded, users can ensure that only meaningful, typically static assignments are autopopulated.
  • Expose init arguments for Arkime's db.pl and also use them for Malcolm's creation of its own index templates (#692)
  • Extend Zeek's intel.log with additional fields using corelight/ExtendIntel (part 1) (#502)
    • This integrates the corelight/ExtendIntel plugin into Malcolm internally but does not significantly change how Malcolm presents intel.log to the user. Further work to do so will be continued in #695.
  • Some internal tweaks to the PCAP processing pipeline that are going to be leveraged by the Malcolm-Helm project (idaholab#630)
  • Handle a fix in the ICSNPP OPCUA-Binary plugin that adds a new sec_token_id field (cisagov/icsnpp-opcua-binary#101)
  • Moved the configuration for Zeek's use of the zeek-kafka plugin to its own file (kafka.zeek) to make it easier to override in Docker using a volume bind mount or in K8s using a configMap.
  • Changed some internal objects used for NetBox enrichment caching from Ruby's Concurrent::Hash to Concurrent::Map for better performance
  • Minor improvements to the icons, shortcuts, and convenience bash functions in the ISO-installed Malcolm desktop environment
  • NGINX now generates a robots.txt file to avoid web crawlers
• ✅ Component version updates
  • Alpine base Docker image to v3.22.0
  • Arkime to v5.7.0
  • capa to v9.2.1
  • flask-cors Python library to v6.0.0 to address CVE-2024-6839, CVE-2024-6844, and CVE-2024-6866
  • OpenSearch and OpenSearch Dashboards to v3.0.0
    • What To Expect
    • Announcing OpenSearch 3.0
    • Performance Improvements
  • opensearch-py Python library to v3.0.0
  • osd_transform_vis Dashboards visualization library to v3.0.0
  • requests Python library to v2.32.4 to address CVE-2024-47081
  • YARA to v4.5.3
  • Zeek to v7.2.1
• 🐛 Bug fixes
  • NetBox autodiscovery no longer populating host name from DNS, DHCP, NTLM (regression, #699)
  • documentation served at /readme is trying to pull fonts from use.fontawesome.com (#694)
  • support fractional gigabytes correctly when generating Arkime's config.ini setting maxFileSizeG from PCAP_ROTATE_MEGABYTES
  • Improved logstash filters that calculate unique hashes used as document IDs for Zeek and Suricata logs to better prevent duplicate logs from being written to the document store
• 📄 Configuration changes (in environment variables in ./config/) for Malcolm and in control_vars.conf for Hedgehog Linux. The Malcolm control script (e.g., ./scripts/status, ./scripts/start, etc.) should take care of creating new variables and migrating existing ones as needed based on the rules in ./config/env-var-actions.yml.
  • arkime.env's OPENSEARCH_MAX_SHARDS_PER_NODE has been moved to opensearch.env and renamed to CLUSTER_MAX_SHARDS_PER_NODE
  • auth-common.env's NGINX_LDAP_TLS_… variables have been moved to nginx.env
  • [auth-common.env](https://github.com/cisagov/Malcolm/b...

Malcolm v25.05.0

Malcolm v25.05.0 adds support for the Emerson ROC Plus protocol (including a Zeek analyzer and corresponding dashboard), component updates, and bug fixes.

v25.04.1...v25.05.0

• ✨ Features and enhancements
  • Added support for ROC Plus (#661)
  • Make Zeek metrics port configurable (thanks to @divinehawk) (cherry-picked from #668)
  • Improve ability to upload PCAP files via cURL
  • Minor UI improvements to desktop environment for Malcolm and Hedgehg Linux ISO-installed instances
• ✅ Component version updates
  • Fluent Bit to v4.0.2
  • yq v4.45.4
  • Zeek v7.2.0
• 🐛 Bug fixes
  • race condition in suricata offline container between pcap processing and suricata socket (#667)
  • NetBox autopopulation not working with prefixes correctly (#670) (regression)
  • ensure Arkime's queryExtraIndices config.ini setting is only set when Zeek/Suricata logs are using a different index pattern
  • set number_of_replicas cluster setting to 0 for embedded single-node OpenSearch instance to avoid yellow state
• 📄 Configuration changes (in environment variables in ./config/) for Malcolm and in control_vars.conf for Hedgehog Linux
  • added ZEEK_METRICS_PORT (default blank, meaning use the default port) in zeek.env and control_vars.conf for #668
  • added ZEEK_DISABLE_ICS_ROC_PLUS (default blank, meaning not disabled) in zeek.env and control_vars.conf for #661
• 🧹 Code and project maintenance
  • minor slides and documentation updates
  • Replace AWS Fargate documentation with AWS EKS Auto documentation

Malcolm is a powerful, easily deployable network 🖧 traffic analysis tool suite for network security monitoring 🕵🏻‍♀️.

Malcolm operates as a cluster of containers 📦, isolated sandboxes which each serve a dedicated function of the system. This makes Malcolm deployable with frameworks like Docker 🐋, Podman 🦭, and Kubernetes ⎈. Check out the Quick Start guide for examples on how to get up and running.

Alternatively, dedicated official ISO installer images 💿 for Malcolm and Hedgehog Linux 🦔 can be downloaded from Malcolm's releases page on GitHub. Due to limits on individual files in GitHub releases, these ISO files have been split 🪓 into 2GB chunks and can be reassembled with scripts provided for both Bash 🐧 (release_cleaver.sh) and PowerShell 🪟 (release_cleaver.ps1). See Downloading Malcolm - Installer ISOs for instructions.

As always, join us on the Malcolm discussions board 💬 to engage with the community, or pop some corn 🍿 and watch a video 📼.

Malcolm v25.04.1

Malcolm v25.04.1 contains only one change: updating Arkime to v5.6.4 which mitigates newly-discovered remote code execution (RCE) vulnerabilities.

v25.04.0...v25.04.1

• ✅ Component version updates
  • Arkime v5.6.4 to resolve RCE vulnerabilities, as described below in the #announcements channel on the Arkime slack:
    • possible to bypass forced expressions for some API calls
    • direct access to OpenSearch/Elasticsearch could be used to create session documents that hang viewer or have viewer execute code
    • since Arkime 5.1.0 any arkimeUser user could create OpenSearch/Elasticsearch documents in any index that viewer had access to

Malcolm is a powerful, easily deployable network 🖧 traffic analysis tool suite for network security monitoring 🕵🏻‍♀️.

Malcolm operates as a cluster of containers 📦, isolated sandboxes which each serve a dedicated function of the system. This makes Malcolm deployable with frameworks like Docker 🐋, Podman 🦭, and Kubernetes ⎈. Check out the Quick Start guide for examples on how to get up and running.

Alternatively, dedicated official ISO installer images 💿 for Malcolm and Hedgehog Linux 🦔 can be downloaded from Malcolm's releases page on GitHub. Due to limits on individual files in GitHub releases, these ISO files have been split 🪓 into 2GB chunks and can be reassembled with scripts provided for both Bash 🐧 (release_cleaver.sh) and PowerShell 🪟 (release_cleaver.ps1). See Downloading Malcolm - Installer ISOs for instructions.

As always, join us on the Malcolm discussions board 💬 to engage with the community, or pop some corn 🍿 and watch a video 📼.

Malcolm v25.04.0

Malcolm v25.04.0 contains new features and improvements, component version updates, bug fixes, and other great stuff.

v25.03.1...v25.04.0

• ✨ Features and enhancements

  • add option to use external NetBox instance (#597)

  • add -q/--quiet option for start/restart (#656)

  • handle non-HTTPS arkime case (#629)

  • lots of improvements to control.py and install.py for Kubernetes deployment

    • improved start/stop/wipe control script behavior
    • allow providing resource requests in manifests via YML file and command-line argument

    ...
    Kubernetes:
      -n, --namespace <string>
                            Kubernetes namespace
      --skip-persistent-volume-checks [SKIPPERVOLCHECKS]
                            Skip checks for PersistentVolumes/PersistentVolumeClaims in manifests (only for "start" operation with Kubernetes)
      --no-capture-pods [NOCAPTUREPODSSTART]
                            Do not deploy pods for traffic live capture/analysis (only for "start" operation with Kubernetes)
      --no-capabilities [NOCAPABILITIES]
                            Do not specify modifications to container capabilities (only for "start" operation with Kubernetes)
      --inject-resources [INJECTRESOURCES]
                            Inject container resources from kubernetes-container-resources.yml (only for "start" operation with Kubernetes)
      --image-source <string>
                            Source for container images (e.g., "ghcr.io/idaholab/malcolm"; only for "start" operation with Kubernetes)
      --image-tag <string>  Tag for container images (e.g., "25.04.0"; only for "start" operation with Kubernetes)
      --delete-namespace [DELETENAMESPACE]
                            Delete Kubernetes namespace (only for "wipe" operation with Kubernetes)
    ...
    

  • improvements to Malcolm's vanilla Kubernetes manifests

    • lowered the amount of storage for the persistent volumes in the AWS EFS example
    • replaced name label with app label for deployments in accordance with best practices

  • improve links on landing page for NetBox and auth to accurately reflect what Malcolm is using

  • added more smarts to the NGINX startup script to dynamically set up upstreams that may or may not exist based on enabled or disabled Malcolm features

  • fixed a minor issue in the script setting up Zeek intelligence updates where it would remove its own lockfile

• ✅ Component version updates
  • Alpine Linux v3.21
  • Arkime v5.6.3
  • Keycloak v26.2
  • NetBox v4.2.8
  • netbox-initializers v4.2.0
  • netbox-topology v4.2.1
  • Fluent Bit to v4.0.1
• 🐛 Bug fixes
  • API tokens created in NetBox still require authentication through NGINX reverse proxy (#383)
  • adjust Logstash health check so K8s liveness probe doesn't kill it (#630)
  • be more resilient in zeekctl status checks in zeekdeploy.sh (#652)
  • in deployments with multiple zeek-live containers, each container's restarting causes the others to restart zeek (#651)
• 📄 Configuration changes (in environment variables in ./config/) for Malcolm and in control_vars.conf for Hedgehog Linux
  • replaced NETBOX_DISABLED with NETBOX_MODE in netbox-common.env for #597
  • added NETBOX_URL in netbox-common.env for #597
  • added NETBOX_TOKEN in netbox-secret.env for #597
  • removed unused NETBOX_CRON variable from netbox-common.env
  • added LOGIN_REQUIRED, LOGIN_PERSISTENCE, and ISOLATED_DEPLOYMENT in netbox.env
  • added MALCOLM_NETWORK_INDEX_DEFAULT_PIPELINE, MALCOLM_NETWORK_INDEX_LIFECYCLE_NAME, MALCOLM_NETWORK_INDEX_LIFECYCLE_ROLLOVER_ALIAS, MALCOLM_OTHER_INDEX_DEFAULT_PIPELINE, MALCOLM_OTHER_INDEX_LIFECYCLE_NAME, MALCOLM_OTHER_INDEX_LIFECYCLE_ROLLOVER_ALIAS in opensearch.env for #642; these are used to support customizations in the index templates, primarily for when using a remote Elasticsearch instance as the backing document store
  • added EXTRACTED_FILE_ENABLE_VTOT in zeek.env rather than just relying on the presence of VTOT_API2_KEY in zeek-secret.env
• 🧹 Code and project maintenance
  • various minor documenation improvements
  • improvements to build and appliance packaging scripts (#640)
  • document customizing Malcolm with an additional output pipeline (#643)
  • overhaul "deploying Malcolm on AWS" documentation (#655)
  • integrate customizations from Malcolm-Helm as options in vanilla Malcolm (part 1) (#642)
  • put in version pinning for Python packages (#644)
  • remove redundant storage of URLs in documents as artifact of NetBox enrichment
  • removed references to AWS client access and secret keys from packer_vars.json.example and documentation for building AWS AMIs (for security, these variables are now passed in via environment variables on the command line in the examples)

Malcolm is a powerful, easily deployable network 🖧 traffic analysis tool suite for network security monitoring 🕵🏻‍♀️.

Malcolm operates as a cluster of containers 📦, isolated sandboxes which each serve a dedicated function of the system. This makes Malcolm deployable with frameworks like Docker 🐋, Podman 🦭, and Kubernetes ⎈. Check out the Quick Start guide for examples on how to get up and running.

Alternatively, dedicated official ISO installer images 💿 for Malcolm and Hedgehog Linux 🦔 can be downloaded from Malcolm's releases page on GitHub. Due to limits on individual files in GitHub releases, these ISO files have been split 🪓 into 2GB chunks and can be reassembled with scripts provided for both Bash 🐧 (release_cleaver.sh) and PowerShell 🪟 (release_cleaver.ps1). See Downloading Malcolm - Installer ISOs for instructions.

As always, join us on the Malcolm discussions board 💬 to engage with the community, or pop some corn 🍿 and watch a video 📼.

Malcolm v25.03.1

This has been a busy month for Malcolm! We pushed hard to get v25.03.0 out earlier this month, as it contained pretty much just the Keycloak integration one of our partners (and major funding sources) was waiting for. Rather than wait until April for the other stuff that would have gone into the regular end-of-the-month release, I decided to pull those items into this smaller release just a week and a half after the last one.

Malcolm v25.03.1 contains a few enhancements, bug fixes, and several component version updates, including one that addresses a CVE that may affect Hedgehog Linux Kiosk mode and Malcolm's API container.

v25.03.0...v25.03.1

NOTE: If you have not already upgraded to v25.03.0, read the notes for v25.02.0 and v25.03.0 and follow the Read Before Upgrading instructions on those releases.

• ✨ Features and enhancements
  • Incorporate new S7comm device identification log, s7comm_known_devices.log (#622)
  • Display current PCAP, Zeek, and Suricata capture results in Hedgehog Linux Kiosk mode (#566)
  • Keycloak authentication: configurable group or role membership restrictions for login (#633) (see Requiring user groups and realm roles)
  • Mark newly-discovered and uninventoried devices in logs during NetBox enrichment (#573)
  • Added "Apply recommended system tweaks automatically without asking for confirmation?" question to install.py to allow the user to accept changes to sysctl.conf, grub kernel parameters, etc., without having to answer "yes" to each one.
• ✅ Component version updates
  • Arkime to v5.6.2
  • evtx to v0.9.0
  • Fluent Bit to v3.2.10
  • gunicorn to v23.0.0 to address CVE-2024-6827, "Gunicorn HTTP Request/Response Smuggling vulnerability"
  • Zeek to v7.1.1
• 🐛 Bug fixes
  • Fix install.py error when answering yes to "Pull Malcolm images?" with podman (#604)
  • Order of user-provided tags from PCAP upload interface not preserved (#624)
• 📄 Configuration changes (in environment variables in ./config/) for Malcolm and in control_vars.conf for Hedgehog Linux
  • added NGINX_REQUIRE_GROUP and NGINX_REQUIRE_ROLE to auth-common.env to support Requiring user groups and realm roles for Keycloak authentication
• 🧹 Code and project maintenance
  • Ensure Malcolm's NetBox configuration Python scripts are baked into the image in addition to bind-mounting them in docker-compose.yml at runtime.

Malcolm is a powerful, easily deployable network 🖧 traffic analysis tool suite for network security monitoring 🕵🏻‍♀️.

Malcolm operates as a cluster of containers 📦, isolated sandboxes which each serve a dedicated function of the system. This makes Malcolm deployable with frameworks like Docker 🐋, Podman 🦭, and Kubernetes ⎈. Check out the Quick Start guide for examples on how to get up and running.

Alternatively, dedicated official ISO installer images 💿 for Malcolm and Hedgehog Linux 🦔 can be downloaded from Malcolm's releases page on GitHub. Due to limits on individual files in GitHub releases, these ISO files have been split 🪓 into 2GB chunks and can be reassembled with scripts provided for both Bash 🐧 (release_cleaver.sh) and PowerShell 🪟 (release_cleaver.ps1). See Downloading Malcolm - Installer ISOs for instructions.

As always, join us on the Malcolm discussions board 💬 to engage with the community, or pop some corn 🍿 and watch a video 📼.

Malcolm v25.03.0

Malcolm v25.03.0 adds authentication via Keycloak and includes a few component version updates.

v25.02.0...v25.03.0

Read Before Upgrading

• As described below, a number of changes were made to environment variables in this release. The Malcolm control script should automatically migrate environment variables between Malcolm versions (e.g., moving environment variables from one .env file to another, removing deprecated/unused environment variables from .env files, etc.) as these actions are specified in config/env-var-actions.yml. However, these actions should be taking when migrating from a previous version of Malcolm to v25.03.0:
  • Before upgrading, while Malcolm is up, execute ./scripts/netbox-backup to backup the NetBox database and save the resulting .gz file(s) in case something goes wrong with the migration of the location of the PostgreSQL database or the environment variables associated with it. Should this happen, ./scripts/netbox-restore could be executed afterwards to restore the contents of the NetBox database.
  • If you have not already upgraded to v25.02.0, read the notes for that release and manually update the redis-related environment variables as described there.
  • Once updating to v25.03.0, but before starting Malcolm, run ./scripts/status to automatically migrate the other environment variables as described above.

Release Notes

• ✨ Features and enhancements
  • Support authentication via Keycloak (#459)
    • In addition to local account management and LDAP authentication, Malcolm can now utilize Keycloak, an identity and access management (IAM) tool, to provide a more robust authentication and authorization experience, including single sign-on (SSO) functionality.
    • Malcolm can connect to an existing Keycloak server or it can use its own embedded Keycloak instance.
    • While this feature has been developed and tested with Keycloak in mind, the lua-resty-openidc library used to implement the OpenID connection functionality may work with other OpenID providers as well. If you find this does work, let us know on the discussions board; if not, please log an issue with details.
    • This feature will pave the way for fine-grained access controls to be implemented in a future Malcolm version.
    • To support this feature, the postgres container has been decoupled from NetBox and now runs independent of that service. This is similar to what was done with the redis container in v25.02.0.
    • To support this feature, the vanilla NGINX web server used internally has been replaced with OpenResty, a version of NGINX extended with Lua.
    • New functionality was added to the authentication setup tool.
    • Refer to the new documentation on this feature for details, including a known limitation when using this authentication method with Hedgehog Linux.
  • Change to ./wipe command behavior
    • Prior to this release, running ./wipe also cleared the contents of the directory of the PostgreSQL database containing the NetBox inventory. PostgreSQL is now used to store both the NetBox inventory and the embedded Keycloak instance data. For this reason, and because it was probably not users' intention to blow away their network inventory with ./wipe, that script no longer deletes this data.
• ✅ Component version updates
  • OpenSearch and OpenSearch Dashboards to v2.19.1
  • Jinja2 to v3.1.6 to fix "Jinja2 vulnerable to sandbox breakout through attr filter selecting format method" vulnerability (CVE-2025-27516)
  • Fluent Bit to v3.2.8
  • Capa to v9.1.0
• 📄 Configuration changes (in environment variables in ./config/) for Malcolm and in control_vars.conf for Hedgehog Linux
  • the following are all to support authentication via Keycloak (#459)
    • renamed NGINX_BASIC_AUTH with NGINX_AUTH_MODE in auth-commmon.env; the new code handling this variable should be backwards-compatible with the previously-accepted values
    • added keycloak.env
    • renamed nginx-postgres to postgres.env and completely overhauled the variables in that file
    • added several new environment variables to nginx.env (see the comments in that file for details)
    • removed NETBOX_POSTGRES_DISABLED from in netbox-common.env

Malcolm is a powerful, easily deployable network 🖧 traffic analysis tool suite for network security monitoring 🕵🏻‍♀️.

Malcolm operates as a cluster of containers 📦, isolated sandboxes which each serve a dedicated function of the system. This makes Malcolm deployable with frameworks like Docker 🐋, Podman 🦭, and Kubernetes ⎈. Check out the Quick Start guide for examples on how to get up and running.

Alternatively, dedicated official ISO installer images 💿 for Malcolm and Hedgehog Linux 🦔 can be downloaded from Malcolm's releases page on GitHub. Due to limits on individual files in GitHub releases, these ISO files have been split 🪓 into 2GB chunks and can be reassembled with scripts provided for both Bash 🐧 (release_cleaver.sh) and PowerShell 🪟 (release_cleaver.ps1). See Downloading Malcolm - Installer ISOs for instructions.

As always, join us on the Malcolm discussions board 💬 to engage with the community, or pop some corn 🍿 and watch a video 📼.

Malcolm v25.02.0

Malcolm v25.02.0 contains some major performance improvements, a few smaller new features and enhancements, several component version updates, bug fixes, and documentation updates.

v25.01.0...v25.02.0

NOTE: As a result of some of the changes to environment variables made for decoupling Redis from NetBox (#580), environment variables from previous version may cause NetBox to fail to connect to Redis which prevents successful startup. To fix this, you should perform the following steps once prior to starting Malcolm:

1. Stop Malcolm (./scripts/stop)
2. Change the values for REDIS_CACHE_HOST and REDIS_HOST, removing the netbox- prefix from the values, so that they look like REDIS_HOST=redis and REDIS_CACHE_HOST=redis-cache, respectively.
   • These values were found in netbox.env in previous versions, but are found in redis.env in this release.
   • Alternatively, you may remove the lines for REDIS_HOST and REDIS_CACHE_HOST completely and they will be restored with correct values the next time the control script is run.
3. Run ./scripts/status which will check the .env files and restore the missing values if you removed them.
4. Start Malcolm (./scripts/start)

• ✨ Features and enhancements
  • performance improvements for NetBox enrichment (#547)
    • NetBox enrichment and autopopulation is now approximately 4x faster than it was before (depending on resources)
  • performance improvements for Suricata's processing of uploaded PCAP files (#457)
    • Suricata's processing of large sets of uploaded PCAP files is now approximately 18x faster faster than it was before (depending on resources)
  • add validate_local_site_policy.sh script for validating Zeek local site policy (#598)
  • include corelight/zeek-long-connections plugin to log long connections (#585)
    • new zeek.conn.long field is available to indicate long connections
    • Connections dashboard updated to include this new field
    • see notes below on environment variable additions for configuring this plugin
  • standardize container health checks into scripts for all containers (#491)
    • added container health checks for containers that did not previously have them (live capture containers)
  • significant work-in-progress towards support for Sigma rules via OpenSearch Security Analytics (still incomplete due to some blocking issues upstream, see #475 for details)
    • changed normalization of Windows event log records (evtx) to more closely match Winlogbeat fields which are closer to what the Sigma rules for Windows events use, and updated corresponding Windows Event Logs dashboard
  • dnp3_control.log now includes clear_bit field to indicate if control code clear bit is set or unset
  • improved shared-object-creation.sh's cURL commands so that import failures for OpenSearch/Elasticsearch shared objects are printed to the debug logs rather than being redirected to /dev/null
• ✅ Component version updates
  • Arkime to v5.6.1
  • capa to v9.0.0
  • Fluent Bit to v3.2.7
  • OpenSearch and OpenSearch Dashboards to v2.19.0
    • includes workaround for [BUG] HTTP REST API hangs with Accept-Encoding zstd (opensearch-project/OpenSearch#17339) by disabling http.compression for internal connections to the OpenSearch API
• 🐛 Bug fixes
  • ANSI color codes from croc displayed in ssl-client-transmit (#559)
  • clear screen after auth_setup when using Dialog mode (#574)
  • warn and prompt user before changing NetBox database passwords out from underneath existing database (#565)
  • UFW software firewall for Malcolm ISO should automatically open ports for syslog (#560)
    • removed default port allowances (e.g., 5044/tcp, 9200/tcp, etc.) so that they could be set dynamically as part of configuration
• 📄 Configuration changes (in environment variables in ./config/) for Malcolm and in control_vars.conf for Hedgehog Linux
  • added functionality to control script to migrate environment variables between malcolm versions (e.g., moving environment variables from one .env file to another, removing deprecated/unused environment variables from .env files, etc.) by specifying these actions in config/env-var-actions.yml
  • removed CONNECTION_SECONDS_SEVERITY_THRESHOLD from lookup-common.env in favor of the new variables in zeek.env outlined below
  • increased defaults for NETBOX_CACHE_SIZE (to 10000) and NETBOX_CACHE_TTL (to 300 seconds) in netbox-common.env in relation to #547
  • some standardization of NetBox-related environment variables to reduce duplication across different .env files
    • DB_HOST and DB_NAME were moved from netbox.env to netbox-postgres.env
    • DB_PASSWORD and DB_USER were moved from netbox-secret.env to netbox-postgres.env
    • REDIS_CACHE_HOST and REDIS_HOST were moved from netbox.env to redis.env in relation to #580
    • REDIS_CACHE_PASSWORD and REDIS_PASSWORD were moved from netbox-secret.env to redis.env
    • removed netbox-redis-cache.env in relation to #580
    • renamed netbox-redis.env to redis.env in relation to #580
    • added MALCOLM_NETWORK_INDEX_ALIAS and MALCOLM_OTHER_INDEX_ALIAS to opensearch.env
      • these variables define the names for index aliases used in the index templates that are necessary for OpenSearch security analitics detectors
    • added the following variables to zeek.env for configuring the Zeek long connections plugin
      • ZEEK_LONG_CONN_REPEAT_LAST_DURATION (default true)
      • ZEEK_LONG_CONN_DO_NOTICE (default true)
      • ZEEK_LONG_CONN_DURATIONS (default 600,1800,3600,43200,86400)
• 🧹 Code and project maintenance
  • use arm-hosted runners for GitHub build actions for arm64 images (#557)
  • decouple redis from netbox (#580)
  • document standards for supply chain and code provenance checking (#555)
  • document incorporating new Suricata rules (and removing old ones) without restarting the Suricata containers (#589)
  • updates to documentation for Docker-based installation exampl...

Malcolm v25.01.0

Malcolm v25.01.0 contains quite a few UI/UX improvements; new parsers; a bevy of component version updates including to Arkime, Zeek, NetBox; and several bug fixes.

v24.12.0...v25.01.0

• ✨ Features and enhancements
  • integrate Omron FINS parser and added corresponding dashboard (#554)
  • integrate PostgreSQL parser (added in Zeek v7.1.0) and added corresponding dashboard (#553)
  • normalize Winlogbeat with Fluent Bit's winlog/winevtlog event and evtx event schemas (#356)
    • Winlogbeat seems to parse more fields from Windows events than Fluent Bit's winevtlog or winlog do, so users forwarding Windows event logs to Malcolm using Fluent Bit may want to evaluate Winlogbeat as an alternative.
  • support syslog ingestion over UDP and/or TCP (#354)
  • clicking field values in Dashboards tables will now pivot to Arkime or NetBox (#551)
  • add navigation pane to all non-network dashboards (#543)
• ✅ Component version updates
  • Arkime to v5.6.0
  • beats to v8.17.0
  • elasticsearch-dsl Python library to v8.17.1
  • Jinja Python library to v3.1.5 (security fix release)
  • Logstash to v8.17.0
  • NetBox to v4.1.11
  • osd_transform_vis (Dashboards visualization library) to v2.18.0
  • yq to v4.45.1
  • Zeek to v7.1.0 (#553)
• 🐛 Bug fixes
  • Extracted File Downloads interface not working with some filenames (#524)
  • user-defined custom field formats for index patterns are overwritten (#542)
  • port numbers should not be shown with commas in Dashboards (#540)
  • pivoting between Arkime and Dashboards doesn't work when Malcolm is behind a reverse proxy (e.g., traefik) (#552)
  • opensearch.keystore not created when running in Hedgehog run profile (#533)
  • ensure all conn.log entries are tagged ics for OT protocols (#541)
• 📄 Configuration changes (in environment variables in ./config/) for Malcolm and in control_vars.conf for Hedgehog Linux
  • The following variables in ./config/filebeat.env configure Malcolm's ability to accept syslog messages:
    • FILEBEAT_SYSLOG_TCP_LISTEN and FILEBEAT_SYSLOG_UDP_LISTEN - if set to true, Malcolm will accept syslog messages over TCP and/or UDP, respectively
    • FILEBEAT_SYSLOG_TCP_PORT and FILEBEAT_SYSLOG_UDP_PORT - the port on which Malcolm will accept syslog messages over TCP and/or UDP, respectively
    • FILEBEAT_SYSLOG_TCP_FORMAT and FILEBEAT_SYSLOG_UDP_FORMAT - one of auto, rfc3164, or rfc5424, to specify the allowed format for syslog messages over TCP and/or UDP, respectively (default auto)
    • FILEBEAT_SYSLOG_TCP_MAX_MESSAGE_SIZE and FILEBEAT_SYSLOG_UDP_MAX_MESSAGE_SIZE - defines the maximum message size of the message received over TCP and/or UDP, respectively (default: 10KiB for UDP, 20MiB for TCP)
    • FILEBEAT_SYSLOG_TCP_MAX_CONNECTIONS - specifies the maximum current number of TCP connections for syslog messages
    • FILEBEAT_SYSLOG_TCP_SSL - if set to true, syslog messages over TCP will require the use of TLS. When ./scripts/auth_setup is run, self-signed certificates are generated which may be used by remote log forwarders. Located in Malcolm's ./filebeat/certs/ directory, the certificate authority and client certificate and key files should be copied to the host on which the forwarder is running and used when defining its settings for connecting to Malcolm.
  • The following variables in ./config/zeek.env for Malcolm and control_vars.conf for Hedgehog Linux pertain to the new Omron FINS protocol parser:
    • ZEEK_DISABLE_ICS_OMRON_FINS - if set to true, the Omron FINS parser will be disabled
    • ZEEK_OMRON_FINS_DETAILED - if set to true, a verbose Omron FINS details log (omron_fins_detail.log) will be created
• 🧹 Code and project maintenance
  • Changed ⓒ year to 2025

Malcolm is a powerful, easily deployable network 🖧 traffic analysis tool suite for network security monitoring 🕵🏻‍♀️.

Malcolm operates as a cluster of containers 📦, isolated sandboxes which each serve a dedicated function of the system. This makes Malcolm deployable with frameworks like Docker 🐋, Podman 🦭, and Kubernetes ⎈. Check out the Quick Start guide for examples on how to get up and running.

Alternatively, dedicated official ISO installer images 💿 for Malcolm and Hedgehog Linux 🦔 can be downloaded from Malcolm's releases page on GitHub. Due to limits on individual files in GitHub releases, these ISO files have been split 🪓 into 2GB chunks and can be reassembled with scripts provided for both Bash 🐧 (release_cleaver.sh) and PowerShell 🪟 (release_cleaver.ps1). See Downloading Malcolm - Installer ISOs for instructions.

As always, join us on the Malcolm discussions board 💬 to engage with the community, or pop some corn 🍿 and watch a video 📼.

Malcolm v24.12.0

Malcolm v24.12.0 contains several improvements to the Malcolm configuration script, the Malcolm user interface, and the Malcolm API, as well as component version updates and bug fixes. This release also corresponds with the release of malcolm-test (#486), a Malcolm system testing framework.

v24.11.0...v24.12.0

• Features and enhancements
  • Creation of a Malcolm systems testing framework (#486)
  • Added a number of Zeek packages to detect various CVEs
  • Improvements to the Indices, Ready, and Document Ingest Statistics APIs
  • Use new arkime tag-hiding feature to hide netbox tag from UI (#495)
  • Provide configuration script options for pulling from threat intel feeds (#532)
  • Prompt during configuration whether to enable capture statistics (#504)
  • Add additional EVTX fields to index template (#525) and minor improvements to normalization
  • Add simple readiness indicator to upload page (#528)
  • Add option to upload page to disable NetBox enrichment for the currently-uploaded batch of PCAPs
  • Expose more of the Logstash API passthrough to the Malcolm API
• Component version updates
  • Arkime to v5.5.1
  • capa to v8.0.1
  • elasticearch-dsl Python library to v8.17.0
  • elasticsearch Python library to v8.17.0
  • Fluent Bit to v3.2.2
  • NetBox to v4.1.8 (major update from the v4.0.x series, see #496)
  • opensearch-py Python library to v2.8.0
  • yq to v4.44.6
  • Zeek to v7.0.5 (security and bugfix release)
• Bug fixes
  • Zeek DNS records don't open correctly in Arkime sessions (#509)
  • Mandiant threat intel source doesn't get split correctly when using JSON zeek log format (#494)
  • Set indices.query.bool.max_clause_count to 8192 to reflect maximum number of fields
  • Increase Java stack size (-Xss) for Logstash from 1536k to 2048k
  • Minor fixes for parsing Zeek intel.log (some fields not named correctly with Zeek JSON-formatted logs)
  • Fixes to some Zeek dns.log parsing conflicts between ECS's DNS fields and what the Arkime schema is expecting
  • Fixed setting the Signature event severity tags
• Code and project maintenance
  • Replaced hard-coded Malcolm version number in documentation markdown files with variable-based replacer populated during generation
  • Documentation and screenshot updates

Official ISO installer images for Malcolm and Hedgehog Linux can be downloaded from Malcolm's releases page on GitHub. Due to limits on individual files in GitHub releases, these ISO files have been split into 2GB chunks and can be reassembled with scripts provided for both Bash (release_cleaver.sh) and PowerShell (release_cleaver.ps1). See Downloading Malcolm - Installer ISOs for instructions.

Malcolm v24.11.0

Malcolm v24.11.0 contains a new threat intelligence feed integration, a few new API calls, other minor improvements, bug fixes, and component version updates.

v24.10.1...v24.11.0

• Features and enhancements
  • Added dashboard-export to the list of Malcolm APIs (#401)
  • Added ingest-stats to the list of Malcolm APIs (#488)
  • Added support for pulling from the Mandiant Threat Intelligence service to feed the Zeek intelligence framework as used by Malcolm's and Hedgehog Linux's Zeek processes. The integration uses the google/mandiant-ti-client library for Python. (#358)
  • Improved normalization of Zeek's intel.log to the ECS's threat fields
  • Improved the Zeek Intel dashboard
  • Improved the health/liveness probe for the Logstash container
  • Changed behavior of Malcolm's non-live Zeek container (responsible for processing uploaded PCAPs) so that it becomes available to process data even before an intelligence feed pull is finished
  • Implemented paging for extracted files download dialog (#361)
  • Implemented support for sending Zeek logs to Kafka using the SeisoLLC/zeek-kafka plugin (#357)
  • Added the NetBox HealthCheck plugin as a default NetBox plugin
  • Updated the Malcolm services readiness status API to use the new LogStash health report API and the NetBox HealthCheck plugin as the basis for reporting the state of LogStash and NetBox, respectively.
  • Added parsing for the new OPCUA-Binary write subscription service log
• Component version updates
  • Arkime to v5.5.0
  • Beats to v8.16.0
  • elasticsearch Python library to v8.16.0
  • elasticsearch-dsl Python library to v8.16.0
  • evtx to v0.8.4
  • LogStash to v8.16.0
  • OpenSearch and OpenSearch Dashboard to v2.18.0
  • watchdog Python library to v6.0.0
  • werkzeug Python library to v3.0.6 to address CVE-2024-49767 and CVE-2024-49766
• Bug fixes
  • Fixed an issue with the ./scripts/configure script not prompting to regenerate the internal NetBox passwords when it should have
  • Fixed errors when running malcolm_appliance_packager.sh on macOS (#492, thanks @robrui)
• Configuration changes (in environment variables in ./config/) for Malcolm and in control_vars.conf for Hedgehog Linux
  • Malcolm
    • ZEEK_KAFKA_ENABLED, ZEEK_KAFKA_BROKERS, and ZEEK_KAFKA_TOPIC have been added to ./config/zeek.env, which can be used to enable Zeek's sending of its logs to Kafka (#357)
    • ZEEK_DISABLE_DETECT_ROUTERS (default value: true) has been added to ./config/zeek.env which controls an experimental Zeek script for detecting the presence of routers (logging them to known_routers.log) in a network based on packet TTL; it is recommended to leave this set to true as this script is not yet ready for general production use
    • ZEEK_INTEL_REFRESH_ON_STARTUP has been renamed from ZEEK_INTEL_REFRESH_ON_ENTRYPOINT in ./config/zeek.env to more accurately reflect the purpose of the variable
  • Hedgehog Linux
    • ZEEK_KAFKA_ENABLED, ZEEK_KAFKA_BROKERS, and ZEEK_KAFKA_TOPIC have been added to control_vars.conf for the same purpose as described above
    • ZEEK_DISABLE_DETECT_ROUTERS (default value: true) has been added to control_vars.conf for the same purpose as described above
    • ZEEK_INTEL_REFRESH_ON_STARTUP has been renamed from ZEEK_INTEL_REFRESH_ON_ENTRYPOINT in control_vars.conf to more accurately reflect the purpose of the variable
• Code and project maintenance
  • All open issues and the project board have been migrated from the Idaho National Lab fork to the upstream CISA fork. The repos will continue to be kept in sync going forward. (#350)

Official ISO installer images for Malcolm and Hedgehog Linux can be downloaded from Malcolm's releases page on GitHub. Due to limits on individual files in GitHub releases, these ISO files have been split into 2GB chunks and can be reassembled with scripts provided for both Bash (release_cleaver.sh) and PowerShell (release_cleaver.ps1). See Downloading Malcolm - Installer ISOs for instructions.