Peek is a simple streaming pre-processor and enrichment tool for structured logs.
Peek was written to be central data normalization engine for Frankenstack, Crossed Swords Cyber Defense Exercise Yellow team feedback system. Originally developed during 2018 Frankencoding hackathon, as a lightweight alternative to general purpose log processing tools (e.g., LogStash, Rsyslog) and custom (nasty) Python scripts used in prior exercise iterations. Each message is enriched using inventory information from targets to simplify event correlation, no normalize addressing (i.e. IPv6 short format vs long), etc.
Current version is designed to consume events from one Kafka cluster and produce transformed messages to another. This might seem odd, but keep in mind that tool is designed in the confines of exercise environment. Some information cannot be exposed to the players, such as host names that might reveal insights into the game network layout and scenarios. Thus, original and processed messages are kept separate on cluster level. Nevertheless, messages could easily be fed back into original cluster if so configured or if output cluster configuration is omitted.