New Docker installation - logstash container constantly restarting

[netgun]

Just new to pfelk and anything to do with ELK in general. I did a fresh installation of Ubuntu 20.04, installed the pfelk with docker (via docker-compose) and have not been able to get logs showing in the dashboard. I noticed that the container running logstash restarts constantly. Doing a docker container logs show this error:

ERROR] 2021-12-13 23:14:11.545 [[pfelk]-pipeline-manager] javapipeline - Pipeline error {:pipeline_id=>"pfelk", :exception=>#<Grok::PatternError: pattern %{OPENVPN_RAW} not defined>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/jls-grok-0.11.5/lib/grok-pure.rb:123:in block in compile'", "org/jruby/RubyKernel.java:1442:in loop'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/jls-grok-0.11.5/lib/grok-pure.rb:93:in compile'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-grok-4.4.0/lib/logstash/filters/grok.rb:282:in block in register'", "org/jruby/RubyArray.java:1820:in each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-grok-4.4.0/lib/logstash/filters/grok.rb:276:in block in register'", "org/jruby/RubyHash.java:1415:in each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-grok-4.4.0/lib/logstash/filters/grok.rb:271:in register'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:75:in register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:228:in block in register_plugins'", "org/jruby/RubyArray.java:1820:in each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:227:in register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:586:in maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:240:in start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:185:in run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:137:in block in start'"], "pipeline.sources"=>["/etc/pfelk/conf.d/01-inputs.conf", "/etc/pfelk/conf.d/02-types.conf", "/etc/pfelk/conf.d/03-filter.conf", "/etc/pfelk/conf.d/05-apps.conf", "/etc/pfelk/conf.d/20-interfaces.conf", "/etc/pfelk/conf.d/30-geoip.conf", "/etc/pfelk/conf.d/37-enhanced_user_agent.conf", "/etc/pfelk/conf.d/38-enhanced_url.conf", "/etc/pfelk/conf.d/45-cleanup.conf", "/etc/pfelk/conf.d/50-outputs.conf"], :thread=>"#<Thread:0x654cc20e@/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:125 run>"}

I am not sure how best to troubleshooting this so any suggestions greatly appreaciated.

[a3ilson]

This sounds indicative of the virtual memory not being properly set.

Try this prior to running docker-compose

sudo sysctl -w vm.max_map_count=262144

[netgun]

I am reasonably certain that I did increase the memory as above. LogStash seems to be a bit fussy in the container so I built a non containerized version and it does work! I might retry the docker build later on but for now, I am up and running! Nice work BTW - Thanks

[a3ilson]

Agreed! It may be the particular setup and although there may be enough ram I find an Elastic stack works best with more ram. My current setup is 48GB and running with this docker-compose.yml