Malcolm v23.10.0

Malcolm v23.10.0 is a feature release.

v23.09.0...v23.10.0

• Features and enhancements
  • support both OpenSearch and Elasticsearch output (#258)
  • "capture-only" Malcolm configuration (AKA "dockerized Hedgehog") (#254)
  • don't run kiosk mode on Hedgehog first boot (#263)
  • let Arkime check its own database to see if it needs to be upgraded
  • allow specifying Arkime password hash secret for Viewer clusters
  • documentation improvements
  • minor updates to slide decks
  • allow specifying ports for EtherNet/IP parser via environment variable
• Component version updates
  • Arkime to v4.6.0
  • supercronic to v0.2.27
  • beats to v8.10.4
  • Logstash to v8.10.4
  • NetBox to v3.6.4
  • Fluent Bit to v2.1.10
• Bug fixes
  • set "autorestart" to true for all started services (#267)
  • changed toolchain for building Zeek and Zeek plugins to clang/libc++ to address some build issues with Spicy plugins using GCC
  • ensure Arkime is started before creating OpenSearch artifacts
  • ensure Arkime and OpenSearch artifacts are populated before starting LogStash
  • don't log "0.0" temperatures from Fluent Bit thermal forwarders

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/.

Malcolm v23.09.0

Malcolm v23.09.0 is a release containing enhancements and bug fixes.

v23.08.1...v23.09.0

• Features and enhancements
  • enable/disable Zeek's ICS parsers via environment variable (#256)
  • fully automated configuration and installation (#237) via command-line arguments
  • improvements to several dashboards
  • improvements to field normalization for BACnet and Modbus
  • improvements to the install.py and control.py scripts
• Component version updates
  • ICSNPP parsers to latest releases
  • Arkime to v4.5.0
  • Beats to v8.10.0
  • capa to v6.1.0
  • Fluent Bit to v2.1.9
  • Logstash to v8.10.0
  • NetBox to v3.6.1 (see also v3.6.0 release notes)
  • opensearch-py to v2.3.1
  • Zeek to v6.0.1 (see also v6.0.0 release notes)
• Bug fixes
  • filtering in Arkime sessions view returned zero rows for some reason (#212)
  • Hedgehog - logrotate service not starting (#243)
  • Documentation issue (#245)
  • Error with configure-interfaces.py on both new server images (23.08.1) when setting ntp to 0.pool.ntp.org (#247)
  • installer script not loading prepackaged tarball correctly (#257)
  • logs inserted before template gets created cause field conflicts (#261)

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/.

Malcolm v23.08.1

Malcolm v23.08.1 is a patch release fixing a regression in Hedgehog Linux which would cause disks to not be detected and used for artifact storage.

v23.08.0...v23.08.1

• Bug fixes
  • sensor-capture-disk-config.py not detecting disks correctly (#239)

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/.

Malcolm v23.08.0

Malcolm v23.08.0 is a minor release with a few improvements, bug fixes and component updates.

EDIT: I've discovered a regression in the Hedgehog Linux startup script that formats drives to make them available for artifact capture. I'm investigating now. If this affects you, you might want to avoid this release until I put out a patch.

v23.07.1...v23.08.0

• Features and enhancements

  • Rewrote the Network Traffic Artifact Upload interface and backend, replacing the defunct jQuery-File-Upload with FilePond. This was mainly due to jQuery-File-Upload no longer receiving security fixes and having some known vulnerabilities. see #235
  • Use netbox-initializers plugin, adding the ability to drop YAML files for various NetBox obects to be preloaded at startup. see #228
  • handle changes to ICSNPP parsers with source_ip/destination_ip fields (#233 and #226)

• Bug fixes

  • Fixed extracting Malcolm version during ISO build
  • Workaround for wireshark no longer publishing raw manuf (OUI) list (#230)
  • Remove news feed from default NetBox dashboard (as it would try to reach out to the web for RSS updates)

• Component version updates

  • Rebased Docker and ISO images to Debian 12 (bookworm)
  • live-build tool for building ISO images to debian/1%20230131
  • Arkime to v4.4.0
  • supercronic to v0.2.26
  • FileBeat to v8.9.0
  • LogStash to v8.9.0 (#234)
  • NetBox to v3.5.7
  • PostgreSQL (used by NetBox) to v15
  • opensearch-py to v2.3.0
  • PHP (as used by Upload interface) to v8.2
  • Fluent Bit to v2.1.8
  • certifi to v2023.7.22 (#229)

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/.

Malcolm v23.07.1

Malcolm v23.07.1 is a patch release fixing a single bug.

v23.07.0...v23.07.1

• Bug fixes
  • Fix issue parsing modbus.log (#225)

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/.

Malcolm v23.07.0 [see EDIT at the top of the release notes]

EDIT - A bug in how Modbus traffic was parsed was discovered shortly after this release. A v23.07.1 release will be put out in the next day or so, you may want to wait for that.

Malcolm v23.07.0 is a feature release with a number of improvements, bux fixes and component updates.

v23.05.1...v23.07.0

• New features

  • scan docker images built via GitHub actions for vulnerabilities using Trivy (#218)
  • document building and deplolying Malcolm with an AWS AMI image (#205)
  • handle Arkime field actions (#200)
  • kubernetes: document how to get running on Amazon EKS (#194)
  • Populate NetBox inventory via passively-gathered network traffic metadata (basic functionality, work in progress) (#135)

• Enhancements

  • use .tar.xz instead of .tar.gz for packaging Malcolm docker images for better compression (and smaller ISO file size)
  • Malcolm documentation edits (#204)
  • add option to enable SSH via password in hedgehog's configure-interfaces.py script (#158)
  • updated "Network Traffic Analysis with Malcolm" slides
  • use an init container in Kubernetes container startup to ensure necessary directories get created under PersistentVolume objects before startup
  • improvements to identifying source of third-party logs sent via fluent bit
  • don't do unnecessary clone of Zeek plugins, just install using URL
  • parse bacnet_device_control.log produced by the icsnpp-bacnet parser for Zeek

• Bug fixes

  • maxlogins value includes tmux sessions, can lock user out of SSH (#214)
  • curl rc file for connecting to external OpenSearch without auth enabled causes logstash startup to fail (#209)
  • failure to parse some suricata alerts due to integer type which should be indexed as long (#206)
  • netbox-restore doesn't work in Kubernetes (#202)
  • PCAP File with no - in pcapng Fails to Upload (cisagov#265)
  • disable NetBox telemetry

• Component version updates

  • Alpine (docker container image base) to v3.18.0
  • Arkime to v4.3.2
  • capa to v6.0.0
  • filebeat to v8.8.2
  • NetBox to v3.5.4
  • OpenSearch and OpenSearch Dashboards to v2.8.0
  • Supercronic to v0.2.25
  • YARA to v4.3.2
  • Zeek to v5.2.2

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/.

Malcolm v23.05.1

Malcolm v23.05.1 is a minor release with a few component version updates and bug fixes, particularly to fix an issue with install.py where the ownership of .env files in the config directory may get incorrectly set to root rather than the unprivileged user.

v23.05.0...v23.05.1

• Enhancements and bug fixes

  • install.py can create .env files 0:0 ownership instead of unprivileged user ownership (cisagov#253)
  • both zeek and zeek-live containers are trying to pull intel feeds on startup (#196)
  • Make sure a few Arkime fields (http.xff*) get created in the index template with the right field types to avoid aggregation query issues
  • Tweaks to convenience scripts (malcolmmonitor and sensormonitor) in ISO-installed Malcolm and Hedgehog Linux environments
  • Added some .service files for the ISO-installed version of Malcolm to be able to feed itself resource statistics via Fluent Bit
  • Documentation updates

• Component version updates

  • Arkime to v4.3.1
  • OpenSearch and OpenSearch Dashboards to v2.7.0
  • NetBox to v3.5.1
  • Beats to v8.7.1

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/.

Malcolm v23.05.0

Malcolm v23.05.0 is a major release with new features, enhancements, component version updates and bug fixes.

IMPORTANT NOTE: Malcolm v23.05.0 has completely changed the way it manages its settings: rather than using environment variables found at the top of the docker-compose.yml file, it uses environment variables in .env files inside of the config directory. The locations of a number of configuration files have also changed. It's not recommended to update to Malcolm v23.05.0 from a previous version of Malcolm. Instead, shut down Malcolm, rename your old Malcolm installation directory to something else, and reconfigure Malcolm using ./scripts/configure and ./scripts/auth_setup.

v23.04.0...v23.05.0

• New features

  • integrate ICSNPP-Synchrophasor parser (#190)
  • End-to-end Malcolm and Hedgehog Linux ISO Installation document (#181)
  • support Malcolm deployment with Kubernetes (#149)
    • see Deploying Malcolm with Kubernetes
    • This could be considered a "beta" release for Malcolm deployment with Kubernetes, as there is still some work to be done in this area. Please let us know what issues or suggestions you have via the issue tracker or via email to [email protected].
    • contributing issues:
      • inotify issue (#168)
      • htadmin/nginx and htpasswd (#169)
      • opensearch (#170)
      • uploading large PCAP files (#171)
      • script consolidation (#172)
      • documentation (#173)
      • user-defined persistent volumes (#174)
      • opensearch keystore (#176)
      • expose other TCP services (#183)
      • provide with filebeat access to nginx access and error logs (#186)
      • use Secrets for some environment variables instead of ConfigMaps (#189)

• Enhancements and fixes

  • remove name-map-ui container (#165) in favor of using NetBox for asset identification
  • Python script refactoring, consolidation and cleanup
  • standardization of Docker container entrypoints
  • create ./scripts/configure alias for ./scripts/install.py --configure

• Component version updates

  • Arkime to v4.3.0
  • Capa to v5.1.0
  • Fluent Bit to v2.1.2
  • NetBox to v3.5.0
  • NGINX to v1.22.1
  • Supercronic to v0.2.24
  • Suricata to v6.0.10
  • Yara to v4.3.0
  • Zeek to v5.2.1

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/.

Malcolm v23.04.0

Malcolm v23.04.0 is a release with enhancements, component version updates and bug fixes.

IMPORTANT NOTE: In March 2023 Docker Inc. announced its decision to sunset the "Docker Free Team" plan, which prompted us to decide to migrate away from Docker Hub to the Github Container Registry or "ghcr" (see #163). Due to public backlash, Docker Inc. reversed its decision. However, the Malcolm project will continue with the decision to use GHCR beginning with this release (Malcolm v23.04.0) and moving forward. If you're updating an existing instance of Malcolm, it's recommended that you back up your docker-compose.yml and docker-compose-standalone.yml files, replace them with the ones from this release and re-run ./scripts/install.py --configure to ensure that you're pointing at the latest images (this is actually always good practice when moving to a new release of Malcolm).

v23.03.0...v23.04.0

• Enhancements

  • autostart install.py --configure on Malcolm ISO first boot (#157)
  • clarify information about auth_setup's use of external OpenSearch connections (#160)
  • migrate away from DockerHub container registry (#163)
  • give easier option for transferring SSL client files from Malcolm to forwarder (#177)
    • added tx-rx-secure.sh script as wrapper around croc automatically creating and using a local-only relay

• Component version updates

  • Zeek v5.2.0 (#161)
  • fluent bit v2.0.10
  • NetBox v3.4.7

• Fixes

  • XFCE4's "save session on exit" causes conflict with Hedgehog kiosk mode if firefox instance is started upon session restore (#164)
  • docker-compose move from go-yaml/v3 breaks Malcolm's docker-compose YAML files (#178, docker/compose#10411)
  • increase index.mapping.nested_fields.limit in opensearch index template (#180)

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/.

Malcolm v23.03.0

Malcolm v23.03.0 is a release with enhancements, component version updates and bug fixes.

v23.02.0...v23.03.0

• Enhancements

  • Replace Zeek's misc/scan.zeek with ncsa/bro-simple-scan
  • terminate start and restart scripts once Malcolm has started properly (cisagov#240 and cisagov#241, thanks @Njinx)
  • minor usability improvements for ISO-installed Malcolm and Hedgehog (#155)
    • Added a "Configure Malcolm" menu item (under the "Internet" GTK menu with the other Malcolm stuff) and launcher on the top panel of icons in Malcolm. This runs ./scripts/install.py --configure in full screen. May look at starting this automatically on first boot in the future. (Malcolm)
    • Added Malcolm shortcut to gtk-3.0/bookmarks so it shows up in Thunar sidebar (Malcolm)
    • Added /opt/sensor/sensor_ctl shortcut to gtk-3.0/bookmarks so it shows up in Thunar sidebar (Hedgehog)
    • Have tilix from launcher panel start in /opt/sensor/sensor_ctl (Hedgehog)
  • minor tweaks to defaults for install.py --configure (enable offline-capable file scanners by default)
  • interrupt NetBox startup import script when netbox-restore is run
  • added NetBox restore logic to reset_and_auto_populate.sh script (used mostly for demos and presentations)

• Component version updates

  • Arkime to v4.2.0
  • OpenSearch and OpenSearch Dashboards to 2.6.0
  • Logstash from v8.4.0 to v8.6.1
  • Beats to v8.6.2
  • Zeek to v5.0.7
  • OpenSearch-Py to v2.2.0 (and remove opensearch-dsl which is now part of opensearch-py)
  • Supercronic to v0.2.2
  • Capa to v5.0.0
  • Fluent Bit to v2.0.9
  • Version updates to various Python package dependencies

• Fixes

  • last few seconds' Zeek logs prior to log rotation may be lost (#151)
  • in ISO-packaged Malcolm installation scripts directory, symlink netbox-backup and netbox-restore to control.py
  • improve opensearchpy connect/health check logig in pcap_watcher.py in pcap-monitor container

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/.