Filebeat keeps restarting

[alleniverson33]

Hi

malcolm23.08.1 k8s

filebeat pod

supervisorctl status

unix:///tmp/supervisor.sock no such file

[mmguero]

If you're really running Malcolm 23.08.1, that was released nearly a year and a half ago and would be difficult for me to help you debug. For what it's worth, in the latest version of Malcolm it looks like this for me:

root@filebeat-deployment-557569cd5b-qmjls:/usr/share/filebeat# ps auxf
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root        1980  0.1  0.0   4728  4236 pts/1    Ss   14:26   0:00 /bin/bash
root        1988  0.0  0.0   7904  4032 pts/1    R+   14:26   0:00  \_ ps auxf
root           1  0.0  0.0   2700  1092 pts/0    Ss   14:19   0:00 /usr/bin/tini -- /usr/local/bin/docker-uid-gid-setup.sh /usr/local/bin/service_check_passthrough.sh -s filebeat /usr/local/bin/supervisord -c /etc/supervisord.conf -u root -n
root           7  0.0  0.0   4980  4048 pts/0    S+   14:19   0:00 /bin/bash /usr/local/bin/docker-uid-gid-setup.sh /usr/local/bin/service_check_passthrough.sh -s filebeat /usr/local/bin/supervisord -c /etc/supervisord.conf -u root -n
root        1105  0.0  0.0   4340  2860 pts/0    S+   14:19   0:00  \_ su -s /bin/bash -p root
root        1107  0.0  0.0   4892  3676 pts/0    S+   14:19   0:00      \_ bash
root        1110  0.1  0.0  37352 31852 pts/0    S+   14:19   0:00          \_ /usr/bin/python3 /usr/local/bin/supervisord -c /etc/supervisord.conf -u root -n
filebeat    1115  0.0  0.0 1235216 16820 pts/0   Sl   14:19   0:00              \_ /usr/local/bin/supercronic -json /etc/crontab
filebeat    1116  0.0  0.0   4892  3836 pts/0    S    14:19   0:00              \_ /bin/bash /usr/local/bin/filebeat.sh -h /usr/share/filebeat-logs -c /usr/share/filebeat-logs -d /usr/share/filebeat-logs/data -f /usr/share/filebeat-logs/filebeat-logs.yml
filebeat    1140  0.1  0.1 2459036 81476 pts/0   Sl   14:19   0:00              |   \_ filebeat -e --strict.perms=false --path.home /usr/share/filebeat-logs --path.config /usr/share/filebeat-logs --path.data /usr/share/filebeat-logs/data -c /usr/share/filebeat-logs/filebeat
filebeat    1117  0.0  0.0   4892  3724 pts/0    S    14:19   0:00              \_ /bin/bash /usr/local/bin/filebeat.sh -h /usr/share/filebeat-nginx -c /usr/share/filebeat-nginx -d /usr/share/filebeat-nginx/data -f /usr/share/filebeat-nginx/filebeat-nginx.yml
filebeat    1862  0.1  0.1 2533280 78496 pts/0   Sl   14:23   0:00              |   \_ filebeat -e --strict.perms=false --path.home /usr/share/filebeat-nginx --path.config /usr/share/filebeat-nginx --path.data /usr/share/filebeat-nginx/data -c /usr/share/filebeat-nginx/file
filebeat    1118  0.0  0.0   4892  3792 pts/0    S    14:19   0:00              \_ /bin/bash /usr/local/bin/filebeat.sh -h /usr/share/filebeat-tcp -c /usr/share/filebeat-tcp -d /usr/share/filebeat-tcp/data -f /usr/share/filebeat-tcp/filebeat-tcp.yml
filebeat    1863  0.0  0.1 2460060 74744 pts/0   Sl   14:23   0:00              |   \_ filebeat -e --strict.perms=false --path.home /usr/share/filebeat-tcp --path.config /usr/share/filebeat-tcp --path.data /usr/share/filebeat-tcp/data -c /usr/share/filebeat-tcp/filebeat.C5C
root        1119  0.1  0.0 194704 24140 pts/0    Sl   14:19   0:00              \_ python3 /usr/local/bin/filebeat-watch-zeeklogs-uploads-folder.py --start-sleep 30 --polling true --closed-sec 10 --in /zeek//upload --out /zeek/ --uid 1000 --gid 1000
root@filebeat-deployment-557569cd5b-qmjls:/usr/share/filebeat# supervisorctl status
cron                             RUNNING   pid 1115, uptime 0:06:25
filebeat                         RUNNING   pid 1116, uptime 0:06:25
filebeat-nginx                   RUNNING   pid 1117, uptime 0:06:25
filebeat-syslog-tcp              STOPPED   Not started
filebeat-syslog-udp              STOPPED   Not started
filebeat-tcp                     RUNNING   pid 1118, uptime 0:06:25
watch-upload                     RUNNING   pid 1119, uptime 0:06:25

Is there any more information you can give me, any logs about what is failing?

[alleniverson33]

At present, it is suspected to be an issue with the k8s cluster resources, as deleting a pod allows Filebeat to start

Observing the filebeat log now, there are a large number of Permission denied. Is this a normal phenomenon

{"channel":"stderr","iteration":24,"job.command":"/usr/local/bin/filebeat-clean-zeeklogs-processed-folder.py","job.position":1,"job.schedule":"*/5 * * * *","level":"info","msg":"Cannot stat file /proc/771/fd/4: Permission denied","time":"2025-02-26T00:52:45Z"}

[mmguero]

Hmmm, as far as the filebeat permission denied error goes my guess is it has to do with a check in that script calling fuser to check to see if files are in use by other processes in the system, but in more recent versions of Malcolm at least I think we're sending those STDERR to /dev/null now so I don't think it'll be an issue once you update.

[alleniverson33]

Thank you for your answer