[24.10] - backport changes to snort3, vectorscan, and gperftools #27667
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Changelog: https://github.com/snort3/snort3/releases/tag/3.6.2.0 % snort --version ,,_ -*> Snort++ <*- o" )~ Version 3.6.2.0 '''' By Martin Roesch & The Snort Team http://snort.org/contact#team Copyright (C) 2014-2024 Cisco and/or its affiliates. All rights reserved. Copyright (C) 1998-2013 Sourcefire, Inc., et al. Using DAQ version 3.0.18 Using Hyperscan version 5.4.2 2025-01-28 Using libpcap version 1.10.5 (with TPACKET_V3) Using LuaJIT version 2.1.0-beta3 Using LZMA version 5.6.2 Using OpenSSL 3.0.15 3 Sep 2024 Using PCRE2 version 10.42 2022-12-11 Using ZLIB version 1.3.1 Signed-off-by: John Audia <[email protected]>
Changelog: https://github.com/snort3/snort3/releases/tag/3.6.3.0 Signed-off-by: John Audia <[email protected]>
Changelog: https://github.com/snort3/snort3/releases/tag/3.7.0.0 Signed-off-by: John Audia <[email protected]>
Changelog: https://github.com/snort3/snort3/releases/tag/3.7.1.0 Signed-off-by: John Audia <[email protected]>
This patch is wrongly rebased and applied twice as the same change might be possible and it does not break anything. Since that, the patch is still being refreshed and included in this repository. No need as the patch is already included in the snort3 repository: snort3/snort3@70b811c Drop it once for all. :-) Fixes: 65f6fee ("snort3: update to 3.1.84.0") Signed-off-by: Josef Schlehofer <[email protected]>
Changelog: https://github.com/snort3/snort3/releases/tag/3.8.1.0 ,,_ -*> Snort++ <*- o" )~ Version 3.8.1.0 '''' By Martin Roesch & The Snort Team http://snort.org/contact#team Copyright (C) 2014-2025 Cisco and/or its affiliates. All rights reserved. Copyright (C) 1998-2013 Sourcefire, Inc., et al. Using DAQ version 3.0.19 Using Hyperscan version 5.4.2 2025-05-27 Using libpcap version 1.10.5 (with TPACKET_V3) Using LuaJIT version 2.1.0-beta3 Using LZMA version 5.6.2 Using OpenSSL 3.5.0 8 Apr 2025 Using PCRE2 version 10.42 2022-12-11 Using ZLIB version 1.3.1 Build system: x86/64 Build-tested: x86/64 Run-tested: x86/64 Signed-off-by: John Audia <[email protected]>
hangelog: https://github.com/snort3/snort3/releases/tag/3.9.1.0 % # snort --version ,,_ -*> Snort++ <*- o" )~ Version 3.9.1.0 '''' By Martin Roesch & The Snort Team http://snort.org/contact#team Copyright (C) 2014-2025 Cisco and/or its affiliates. All rights reserved. Copyright (C) 1998-2013 Sourcefire, Inc., et al. Using DAQ version 3.0.20 Using Hyperscan version 5.4.2 2025-06-30 Using libpcap version 1.10.5 (with TPACKET_V3) Using LuaJIT version 2.1.0-beta3 Using LZMA version 5.6.2 Using OpenSSL 3.5.0 8 Apr 2025 Using PCRE2 version 10.42 2022-12-11 Using ZLIB version 1.3.1 Build system: x86/64 Build-tested: x86/64 Run-tested: x86/64 Signed-off-by: John Audia <[email protected]>
Add conditional to disable gperftools-runtime depends for powerpc and mips due to inability to compile introduced with 7345b73 Co-authored-by: Josef Schlehofer <[email protected]> Signed-off-by: John Audia <[email protected]>
Changelog: https://github.com/gperftools/gperftools/releases/tag/gperftools-2.17 Add patch to fix build of some ARM toolchains Build system: x86/64 Build-tested: x86/64-glibc Run-tested: x86/64-glibc Signed-off-by: John Audia <[email protected]>
Changelog: https://github.com/snort3/snort3/releases/tag/3.9.3.0 Build system: x86/64 Build-tested: x86/64-glibc Run-tested: x86/64-glibc (Intel N150 based box) Signed-off-by: John Audia <[email protected]>
Bump to latest release and removed upstreamed patch. Resulting libs give a fully functional snort3 which was built against this new version. Build system: x86/64 Build-tested: x86/64-glibc Run-tested: x86/64-glibc (Intel N150 based box) Signed-off-by: John Audia <[email protected]>
The logic in e57cc98 was flawed causing gperftools-runtime to fail to get detected when building resulting in: ... ninja: Entering directory `/scratch/union/build_dir/target-x86_64_glibc/snort3-3.9.1.0' ninja: error: '/scratch/union/staging_dir/target-x86_64_glibc/usr/lib/libtcmalloc.so', needed by 'src/snort', missing and no known rule to make it make[2]: *** [Makefile:161: /scratch/union/build_dir/target-x86_64_glibc/snort3-3.9.1.0/.built] Error 1 It was missed due testing in build root that already had gperftools-runtime built only discovered when building from a clean build root. This commit fixes this flaw. Test: cat <<EOF > .config CONFIG_TARGET_x86=y CONFIG_TARGET_x86_64=y CONFIG_TARGET_x86_64_DEVICE_generic=y CONFIG_PACKAGE_snort3=y EOF make defconfig grep gperftools-run .config CONFIG_PACKAGE_gperftools-runtime=y cat <<EOF > .config CONFIG_TARGET_qoriq=y CONFIG_TARGET_qoriq_generic=y CONFIG_TARGET_qoriq_generic_DEVICE_watchguard_firebox-m300=y CONFIG_PACKAGE_snort3=y EOF make defconfig grep gperftools-run .config Signed-off-by: John Audia <[email protected]>
Changelog: https://github.com/snort3/snort3/releases/tag/3.9.5.0 Build system: x86/64 Build-tested: x86/64-glibc Run-tested: x86/64-glibc Signed-off-by: John Audia <[email protected]>
Use upstream tarballs for source rather than using git. If we ever need to build from git we can cherry pick and make a patch. This gives a cleaner Makefile and faster build. Signed-off-by: John Audia <[email protected]>
Simplification of Makefile: remove line splits to increase readability. Signed-off-by: John Audia <[email protected]>
Simplification of Makefile: replace complex sed calls with a patch to improve readability. This commit also renames an existing patch. Signed-off-by: John Audia <[email protected]>
Vectorscan is fork of Hyperscan, a high-performance multiple regex matching library. It follows the regular expression syntax of the commonly-used libpcre library, but is a standalone library with its own C API. Currently ARM NEON/ASIMD and Power VSX are 100% functional. ARM SVE2 support is in ongoing with access to hardware now. More platforms will follow in the future. The performance difference of snort3 compiled against this is sizable for aarch64 confirmed on two different SoCs: Test SoC #1 flogic/glinet_gl-mt6000 IDS mode: Download speed wo/ vectorscan: 91.2 ±0.21 Mbit/s (n=3) Download speed using vectorscan: 331.0 ±27.34 Mbit/s (n=3) Gain of 3.6x IPS mode: Download speed wo/ vectorscan: 30.0 ±0.06 Mbit/s (n=3) Download speed using vectorscan: 52.9 ±0.78 Mbit/s (n=3) Gain of 1.8x Notes: * Data generated on snapshot build on 12-Apr-2024 using kernel 6.6.26, snort 3.1.84.0, vectorscan 5.4.11. * Speedtest script hitting the same server. * Snort rules file of was 37,917 lines/22 MB. * In all cases, single core CPU saturation occurred which speaks to the efficiency gains supplied by vectorscan. Test Soc #2 bcm2712/RPi5B IPS mode: Download speed wo/ vectorscan: 164.3 ±0.64 Mbit/s (n=3) Download speed using vectorscan: 232.8 ±0.26 Mbit/s (n=3) Gain of 1.4x Notes: * Data generated on snapshot build on 13-Apr-2024 using kernel 6.1.86, snort 3.1.84.0, vectorscan 5.4.11. * Google fiber speedtest (https://fiber.google.com/speedtest/) hitting the same server. * Snort rules contained 39,801 rules/22 MB. * In all cases, single core CPU saturation occurred which speaks to the efficiency gains supplied by vectorscan. Build system: x86/64 Build-tested: flogic/glinet_gl-mt6000, bcm2712/RPi5B, x86/64-glibc Run-tested: flogic/glinet_gl-mt6000, bcm2712/RPi5B, x86/64-glibc (Intel N150 based box) Co-authored-by: Tianling Shen <[email protected]> Co-authored-by: Jeffery To <[email protected]> Signed-off-by: John Audia <[email protected]>
Drop 100-remove-HAVE_HS_COMPILE_LIT-to-work-around-upstream-b.patch as it was only needed to fix the build against hyperscan. Vectorscan builds fine without it. Signed-off-by: John Audia <[email protected]>
Remove hyperscan since Intel announced a proprietary/closed source license beginning with hyperscan 5.5[1,2] and a general lock of support for the 5.4.x branch which has not seen a commit since 19-Apr-2023[3]. 1. https://networkbuilders.intel.com/docs/networkbuilders/accelerate-snort-performance-with-hyperscan-and-intel-xeon-processors-on-public-clouds-1680176363.pdf 2. https://www.phoronix.com/news/Intel-Hyperscan-Now-Proprietary (and references therein) 3. intel/hyperscan@bc3b191 Signed-off-by: John Audia <[email protected]>
* Replacement of hyperscan-runtime reference with vectorscan-runtime * Added support for all aarch64 targets which I believe is exhaustive For x86 and x86/64, I found that vectorscan is truly a drop-in replacement for hyperscan as assessed by speedtests with snort3 running on my Intel N150 PC. CPU load during the test with each condition was nearly saturating on a single core for both cases on a symmetrical Gbps line. Using: https://www.waveform.com/tools/bufferbloat in IPS mode: Download speed w/ hyperscan: 950-960 Mbit/s (n=2) Download speed w/ vectorscan: 942-960 Mbit/s (n=2) Using: https://www.speedtest.net in IPS mode: Download speed w/ hyperscan: 996-1002 Mbit/s (n=2) Download speed w/ vectorscan: 993-988 Mbit/s (n=2) Build system: x86/64 Build-tested: x86/64-glibc Run-tested: x86/64-glibc (Intel N150 based box running snort3) Signed-off-by: John Audia <[email protected]>
When snort is run with the --version option, it advertises components' versions in the output. Add a patch to modify the output to clearly show vectorscan is in use. Signed-off-by: John Audia <[email protected]>
There is no reason to have custom specific DEPENDS_COMMON, I dropped it and added it to DEPENDS. Simplified, easier to read and understand. Signed-off-by: Josef Schlehofer <[email protected]>
It should be working for mips*, so enable it and let's see. :-) In the past, there were some issues related to mips, when the package was added, but these days, it appears that these issues are gone. More details about those issues could be found in the GitHub pull request when gperftools was added. Reference is in the Fixes tag. Fixes: c1b4e80 ("gperftools: add new package") Signed-off-by: Josef Schlehofer <[email protected]>
Make libunwind support optional depending on package availability. Previously, gperftools unconditionally enabled libunwind as mandatory dependency, which led to build failures on architectures where libunwind is not provided. Signed-off-by: Josef Schlehofer <[email protected]>
1. Enabled hyperscan/vectorscan together with adding dependency only for x86_64 and aarch64. 2. Disabled tmalloc (from gperftools package) for powerpc and mips. By doing this refactor, snort3 is going to be available for more OpenWrt devices (as it was in the past) as currently it was compiled only for x86_x64 and aarch64 by mistake. Fixes: 257e2fc ("snort3: fix logic in gpertools-runtime depends") Signed-off-by: Josef Schlehofer <[email protected]>
The libtirpc package is only needed when building with musl, as glibc
includes the required RPC functionality. This change makes libtirpc a
conditional dependency and adjusts the build flags accordingly.
Building with x86_64-glibc:
...
Feature options:
DAQ Modules: Dynamic
libatomic: User-specified
Hyperscan: ON
ICONV: ON
Libunwind: OFF
LZMA: ON
RPC DB: Built-in
SafeC: OFF
TCMalloc: ON
JEMalloc: OFF
UUID: ON
NUMA: OFF
LibML: OFF
...
Building with aarch64_cortex-a76_musl:
...
Feature options:
DAQ Modules: Dynamic
libatomic: User-specified
Hyperscan: ON
ICONV: ON
Libunwind: OFF
LZMA: ON
RPC DB: TIRPC
SafeC: OFF
TCMalloc: ON
JEMalloc: OFF
UUID: ON
NUMA: OFF
LibML: OFF
...
Build system: x86/64
Build-tested: x86/64-glibc, bcm27flogic/xiaomi_redmi-router-ax6000-ubootmod (for musl)
Run-tested: x86/64-glibc
Signed-off-by: John Audia <[email protected]>
This simplifies checks enabling/disabling features, if packages are present instead of having checks for specific architectures. TCMALLOC_LIBRARIES is removed as it's auto-detected, unlike vectorscan which requires explicit HS_INCLUDE_DIRS. Fixes: 126364e ("snort3: refactor architecture-specific dependencies and CMake options") Signed-off-by: Josef Schlehofer <[email protected]>
This fixes version detection issues when other packages (like snort3) try to find the tcmalloc library using CMake's find_package(). Without the headers in the staging directory, CMake cannot read the version information from tcmalloc.h, resulting in empty version strings. Fixes: Found TCMalloc: /builder/staging_dir/target-x86_64_musl/usr/lib/libtcmalloc.so (found version "") Signed-off-by: Josef Schlehofer <[email protected]>
The gperftools-headers package installed headers to the target device, but headers are only needed during the build process (via Build/InstallDev). - Remove gperftools-headers package - Rename gperftools-runtime to gperftools - Add ABI_VERSION for tracking ABI changes While looking for e.g. on repology, there is only gperftools package [1] and not gperftools-runtime and gperftools-headers. [1] https://repology.org/project/gperftools/versions Signed-off-by: Josef Schlehofer <[email protected]>
The vectorscan-headers package installed headers to the target device, but headers are only needed during the build process (via Build/InstallDev). - Rename vectorscan-runtime to vectorscan to simplify things - Add ABI_VERSION:=5 to track library soname versioning Signed-off-by: Josef Schlehofer <[email protected]>
The gperftools and vectorscan packages have been simplified by removing their -runtime and -headers splits. Update snort3 to use the new package names. Signed-off-by: Josef Schlehofer <[email protected]>
Since vectorscan-runtime was dropped in the following commit, need to replace references to it with just vectorscan in order to compile snort3 against it: openwrt@8a3c7a6 Without this change, even having CONFIG_PACKAGE_vectorscan=y in the .config will result in a failure to compile against it, e.g: ... Feature options: DAQ Modules: Dynamic libatomic: User-specified Hyperscan: OFF ... Signed-off-by: John Audia <[email protected]>
Release notes: https://github.com/snort3/snort3/releases/tag/3.9.6.0 % snort --version ,,_ -*> Snort++ <*- o" )~ Version 3.9.6.0 '''' By Martin Roesch & The Snort Team http://snort.org/contact#team Copyright (C) 2014-2025 Cisco and/or its affiliates. All rights reserved. Copyright (C) 1998-2013 Sourcefire, Inc., et al. Using DAQ version 3.0.21 Using Vectorscan version 5.4.12 2025-10-06 Using libpcap version 1.10.5 (with TPACKET_V3) Using LuaJIT version 2.1.0-beta3 Using LZMA version 5.8.1 Using OpenSSL 3.6.0 1 Oct 2025 Using PCRE2 version 10.46 2025-08-27 Using ZLIB version 1.3.1 Build system: x86/64 Build-tested: x86/64-glibc Run-tested: x86/64-glibc Signed-off-by: John Audia <[email protected]>
Add a comment to the package description to inform users that the build system will not automatically pick gperftools-runtime and vectorscan- runtime when building from source. References to performance benefits of using them: c1b4e80 b6b2d1e Signed-off-by: John Audia <[email protected]>
Running as a dedicated user is better from both a security and an isolation perspective than running as root. Signed-off-by: John Audia <[email protected]>
graysky2
force-pushed
the
openwrt-24.10-snort3-update
branch
from
1ee86d5 to
4060488
Compare
Contributor
Author
|
Not sure what is causing the failure to compile... this is my first major back port. |
Member
|
This PR contains a lot of commits, which are not necessary. I can see that snort3 in OpenWrt 24.10 is at version 3.9.5.0. I think your branch is too old. You should rebase it on top of openwrt-24.10 branch and force push it. |
4 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
📦 Package Details
Maintainer: me
(You can find this by checking the history of the package
Makefile.)Description:
This PR is a backport of recent changes to
net/snort3/libs/gperftoolslibs/vectorscan. It also includes removinglibs/hyperscanand brings openwrt-24.10 in parity with main.🧪 Run Testing Details
I have not run tested on 24.10
✅ Formalities
If your PR contains a patch:
git am(e.g., subject line, commit description, etc.)
We must try to upstream patches to reduce maintenance burden.