Logging enhancements #69
Conversation
ProxyListener - Initialize dport to none if diverter.getOriginalDestPort does not return a destination port. Feature enhancements - Added ability to send logs to Splunk using HEC - Modified the listeners to use the same logger instance - Modified listeners to log key session information using json format Known issues - logger.name is not unique to each listener, which makes following the logs difficult - Logging json formatted logs to streamhandler/filehandler is not very useful.
- Extended logging capabilities to include remote syslog and Splunk loggers. - Implemented remote logger feature to ListenerBase - Splunk loggers are configure with a filter to drop all non-json messages - Log messages for remote loggers are created using Common Information Model when applicable - Improved logging consistency across most listeners - Each listener instance is created with its own stream and remote logger (if configured) - Added log and log_cmd to TLS_FakeFTPHander Bugfix - Fixed SSL configuration of SMTP listener ToDo - Improve banner and server configuration for SMTP listener, similar to FTPListener
…enerBase. - Added ability to specify more than one remote loggers - Added json_only flag to offer the ability to avoid logging JSON to syslog.
…ger = [1|0] Fixed type in ProxyListener. Renamed log_mesage to log_message.
…ged if remote logger is not set.
… and too much noise.
…sage under high load.
…ilter out JSON. Additionally, addressed the issue of excluding json output in listener stream handler.
…gEnhancements # Conflicts: # fakenet/listeners/ListenerBase.py
…pecification to 1|0 from true|false.
|
Hi @vavarachen, thank you for your contribution, I am impressed by the depth here and I perceive this to be of significant value. I've wanted to come up with a more structured kind of logging for quite a while. I'll evaluate this as a candidate for that. Unfortunately, we had to prioritize refactoring FakeNet-NG to make it tenable to continue maintaining and adding features to FakeNet-NG on multiple operating systems and this has created some conflicts. I'm sensitive to how frustrating that can be. This is on us for not telegraphing what our plans were for the codebase. All that aside, depending on what I find in code review, would you be available this month to help me resolve the conflicts? I also have a few questions if we're going to move forward:
At a glance, I see one thing that might need revision:
Other notes: I'm going to take a look at the code you've put forth. Pending that, please let me know if you're available to help hit it home. |
|
@strictlymike @vavarachen I have attempted to use this code, and have ran into multiple problems. I think there is a significant issue with splunk_http_handler in that it may take some time to complete the HTTP POST (requests is not asynchronous). In even running the simple test from that project, the code loops as logging tries to move on before the emit is complete. I think the most valuable thing here is to modify the type of log output (like JSON). I would steer away from introducing a direct integration with Splunk. Rather let the Splunk admin/user find a way to get syslog or log file HEC'd into Splunk. We just need good log output. |
|
Andrew,
Yes, the old splunk_http_handler code had some performance issues. Did you
by any chance test using the latest splunk_hec_handler? Regardless, I
would agree that simply outputting the log message in JSON might be a
better solution. If you are integrating my old code, can you please
include the destination IP (dest) field in the output? I regret not
including it because we had a situation where I was wanting to add a
secondary interfaces and forward sinkhole traffic to it from a commercial
network gear to it. My hope was to assign the events a different level
of confidence based on which interface it arrived on.
Thank you for your efforts!
V
…On Wed, Nov 14, 2018 at 1:01 AM Andrew Stanton ***@***.***> wrote:
@strictlymike <https://github.com/strictlymike> @vavarachen
<https://github.com/vavarachen> I have attempted to use this code, and
have ran into multiple problems. I think there is a significant issue with
splunk_http_handler in that it may take some time to complete the HTTP POST
(requests is not asynchronous). In even running the simple test from that
project, the code loops as logging tries to move on before the emit is
complete. I think the most valuable thing here is to modify the type of log
output (like JSON). I would steer away from introducing a direct
integration with Splunk. Rather let the Splunk admin/user find a way to get
syslog or log file HEC'd into Splunk. We just need good log output.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#69 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AJuaty0-5ImXo8nhfTOzKRDz8rpbh8meks5uu7_jgaJpZM4SLQtY>
.
|
I use fakenet-ng as a sinkhole and one of our primary needs is to be able to monitor traffic hitting the sinkhole and capture useful details such as HTTP headers, requested resources, credentials etc.
P.S I tried to minimize drastic changes to the upstream code.
1. Implemented remote logging capabilities.
2. HTTPListener/BITSListener
Logging implemented by overloading BaseHTTPRequestHander.log_message.
Sample get request logged in json format
3. FTPListener
Logging implemented by overloading FTPHandler.log_cmd function (Log commands and responses in a standardized format.).
We can reconstruct the entire transaction (
index=hec source="listeners.FTPListener*" | transaction src, src_port) using user,src and src_port. For example, the transaction below show the user login, download a file, download a file and then quit.3. ProxyListener/RawListener
Logging implemented by extending Threated[TCP|UDP]RequestHandler classes. In the case of RawListener, log message preserves the hex output.
4. SMTPListener/TFTPListener
Logging implemented by extending handle() method of Threaded[TCP|UDP]RequestHander classes.
5. ListenerBase
Remote logger functions are implemented in the ListenerBase, which has been imported into all listeners. I had to add the 'remotelogger_config' to 'listener_config' in order to avoid drastic changes.
To Do
** Known Issues**