Fix bytes rendering and improve NBI data handling in HTML reports #215
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit addresses issue mandiant#181 where bytes data was being displayed as `b'...'` literals in HTML reports, and implements several related improvements to Network-Based Indicators (NBI) collection and display. Changes: 1. Improve HTTP POST body handling: - Detect printable vs. binary POST data - Display printable data as decoded text - Generate hexdump for binary content (showing first 16 lines) 2. Consolidate shared utilities: - Renamed `listeners/ListenerBase.py` in `listeners/utils.py` since it contains utility functions - Moved `hexdump_table()` from `listeners/RawListener.py` to `listeners/utils.py` - Updated all listeners file (__init__, HTTP, Raw, FTP, SMTP, POP, TFTP) to import from utils 3. Fix HTML report template: - Allowed hexdump display for both "Data Hexdump" (from TCP listerner) and "Request Body (Hexdump)" (from HTTP listener) - Rewrote JavaScript `copyNbiData()` function that did not parse correctly when special character (like \n) were present 4. Clean up SSL utilities: - Removed unused `ListenerBase` import from `ssl_utils/__init__.py` Resolves: mandiant#181
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
Contributor
|
Hi @LucaPalumbo Thank you for the PR! I will review and merge this when I get a chance which is probably going to be early January. |
Author
|
Thanks for the update! Sounds good. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fix HTML byte display issue and improve NBI handling (Closes #181)
This PR fixes issue #181, where byte data was displayed in HTML reports as Python byte literals (b'...').
It also includes improvements to Network-Based Indicators (NBI) processing, display formatting, and shared listener utilities.
Summary of Changes
1. Improved HTTP POST body handling
Detects whether POST data is printable text or binary.
Printable data is now decoded and displayed as text.
Binary data is shown as a truncated hexdump (first 16 lines, consistent with RawListener).
2. Consolidated shared utilities
ListenerBase.pywas not a true base listener but contained only utility functions.Renamed
listeners/ListenerBase.py→listeners/utils.pyto better reflect its purpose.Moved
hexdump_table()fromRawListener.pytoutils.pyso it can be shared across listeners.Updated all listener modules (__init__.py, HTTP, Raw, FTP, SMTP, POP, TFTP) to import from
utils.py.Removed an unused ListenerBase import from
ssl_utils/__init__.py.3. HTML report fixes and improvements
Enabled hexdump display for both Data Hexdump (from RawListener) and Request Body (Hexdump) (from HTTPListener).
Rewrote the JavaScript function
copyNbiData(), which previously failed when special character \n was present.Screenshots
New behavior:
If raw bytes are present, a hexdump is shown.
Otherwise, the content is displayed as a decoded string.
Previous behavior:
Raw bytes caused a Python Traceback because an invalid UTF-8 decoding was attempted.
Feedback
I am open to comments or suggestions!
Let me know if you would prefer these changes split into separate PRs or adjusted in any way.
Resolves
Closes #181