can't collect KUB stats, plugin can't be used

[hchybz]

Can't figure how to make kub plugin correctly used. Does not appear in documentation, but present in plugin directory

Help please

root@k8s-master:~/sos# grep -v ^# sos.conf
[global]
verbose = 3
batch = yes

[report]
enable-plugins = kubernetes

[collect]

[clean]

[plugin_options]

root@k8s-master:~/sos# kubectl get no
NAME          STATUS   ROLES           AGE   VERSION
k8s-master    Ready    control-plane   14h   v1.28.1
k8s-worker1   Ready    <none>          14h   v1.28.1
k8s-worker2   Ready    <none>          14h   v1.28.1

root@k8s-master:~/sos# sos report --enable-plugins kubernetes -vvv --batch 

WARNING: Failed to load 'magic' module version >= 0.4.20 which sos aims to use for detecting binary files. A less effective method will be used. It is recommended to install proper python3-magic package with the module.

sosreport (version 4.6.1)

set sysroot to '/' (default)
Network devices not enumerated by nmcli. Will attempt to manually compile list of devices.
[sos.report:setup] executing 'sos report --enable-plugins kubernetes -vvv --batch'
[sos.report:setup] using 'none' preset defaults ()
[sos.report:setup] effective options now: --batch --enable-plugins kubernetes --tmp-dir /tmp -vvv
a non-existing plugin (kubernetes) was specified in the command line.
root@k8s-master:~/sos# 

[jcastill]

Some questions to move the diagnostics of the issue, until our maintainers can comment:

• What do you get when you run:

  sos report --list-plugins

• Can you run the command with --debug? I.e. simplyfing the command:

  sos report -o kubernetes -vvv --batch --debug

• Where's sos installed? I.e. if you run 'which sos' what do you get?

[hchybz]

Thx for your very quick answer.. can't find what I'm doing wrong

root@k8s-master:~# dpkg -l| grep sos
root@k8s-master:~# sos 
root@k8s-master:~# apt install gh
root@k8s-master:~# export GH_TOKEN=foobar...
root@k8s-master:~# gh repo clone sosreport/sos
Cloning into 'sos'...
remote: Enumerating objects: 33876, done.
remote: Counting objects: 100% (9054/9054), done.
remote: Compressing objects: 100% (3611/3611), done.
remote: Total 33876 (delta 5633), reused 5966 (delta 5432), pack-reused 24822
Receiving objects: 100% (33876/33876), 19.11 MiB | 4.39 MiB/s, done.
Resolving deltas: 100% (24852/24852), done.

root@k8s-master:~/sos# grep -v ^# /root/sos/sos.conf 
[global]
verbose = 3
batch = yes

[report]
enable-plugins = kubernetes

[collect]
[clean]
[plugin_options]


root@k8s-master:~/sos# ./bin/sos report --config /root/sos/sos.conf -l
WARNING: Failed to load 'magic' module version >= 0.4.20 which sos aims to use for detecting binary files. A less effective method will be used. It is recommended to install proper python3-magic package with the module.
sosreport (version 4.6.1)

set sysroot to '/' (default)
Network devices not enumerated by nmcli. Will attempt to manually compile list of devices.
[sos.report:setup] executing 'sos report --config /root/sos/sos.conf -l'
[sos.report:setup] using 'none' preset defaults ()
[sos.report:setup] effective options now: --batch --config-file /root/sos/sos.conf --enable-plugins kubernetes --list-plugins -vvv 
a non-existing plugin (kubernetes) was specified in the command line.

root@k8s-master:~/sos# ls -al sos/report/plugins/kubernetes.py 
-rw-r--r-- 1 root root 8206 Jan 18 12:44 sos/report/plugins/kubernetes.py

what you've asked

root@k8s-master:~/sos# /root/sos/bin/sos report -o kubernetes -vvv --batch --debug

WARNING: Failed to load 'magic' module version >= 0.4.20 which sos aims
to use for detecting binary files. A less effective method will be used.
It is recommended to install proper python3-magic package with the
module.

WARNING: Unable to read configuration file /etc/sos/sos.conf : No such file or directory

sosreport (version 4.6.1)

set sysroot to '/' (default)
Network devices not enumerated by nmcli. Will attempt to manually compile list of devices.
[sos.report:setup] executing 'sos report -o kubernetes -vvv --batch --debug'
[sos.report:setup] using 'none' preset defaults ()
[sos.report:setup] effective options now: --batch --debug --only-plugins kubernetes -vvv
a non-existing plugin (kubernetes) was specified in the command line.

sos has been installed manually (deb 12 still have 4.0.2).. so not in system path

root@k8s-master:~/sos# sos report --list-plugins
-bash: sos: command not found
root@k8s-master:~/sos# which sos
root@k8s-master:~/sos# 

kub does not appear in plugins, neither active , nor inactive .. strange

root@k8s-master:~/sos# bin/sos report --list-plugins | grep -i kub
WARNING: Failed to load 'magic' module version >= 0.4.20 which sos aims...

[jcastill]

It could be that's running somehow sos 4.0.2, but it's strange that it failed 'bin/sos report --list-plugins'.
Just to be on the safe side, can you install the magic module version asked in deb 12?
I'll try to launch a vm and reproduce as well.

[hchybz]

sure.. 4.0.2 has been uninstalled (my first commands) to be sure .. I'm testing the magic module..

Here is what you've asked

root@k8s-master:~/sos# apt install python3-magic
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
  python3-magic
0 upgraded, 1 newly installed, 0 to remove and 3 not upgraded.
Need to get 14.5 kB of archives.
After this operation, 57.3 kB of additional disk space will be used.
Get:1 file:/etc/apt/mirrors/debian.list Mirrorlist [30 B]
Get:2 https://deb.debian.org/debian bookworm/main amd64 python3-magic all 2:0.4.26-3 [14.5 kB]
Fetched 14.5 kB in 0s (118 kB/s)        
Selecting previously unselected package python3-magic.
(Reading database ... 55909 files and directories currently installed.)
Preparing to unpack .../python3-magic_2%3a0.4.26-3_all.deb ...
Unpacking python3-magic (2:0.4.26-3) ...
Setting up python3-magic (2:0.4.26-3) ...


root@k8s-master:~/sos# /root/sos/bin/sos report -o kubernetes --config-file /root/sos/sos.conf -vvv --batch --debug
sosreport (version 4.6.1)

set sysroot to '/' (default)
Network devices not enumerated by nmcli. Will attempt to manually compile list of devices.
[sos.report:setup] executing 'sos report -o kubernetes --config-file /root/sos/sos.conf -vvv --batch --debug'
[sos.report:setup] using 'none' preset defaults ()
[sos.report:setup] effective options now: --batch --config-file /root/sos/sos.conf --debug --only-plugins kubernetes -vvv
a non-existing plugin (kubernetes) was specified in the command line.

[jcastill]

I think I've found the problem and a possible solution - Add DebianPlugin like this:

diff --git a/sos/report/plugins/kubernetes.py b/sos/report/plugins/kubernetes.py
index 5cd416c5..e15418ce 100644
--- a/sos/report/plugins/kubernetes.py
+++ b/sos/report/plugins/kubernetes.py
@@ -9,7 +9,7 @@

-from sos.report.plugins import Plugin, RedHatPlugin, UbuntuPlugin, PluginOpt
+from sos.report.plugins import Plugin, RedHatPlugin, DebianPlugin, UbuntuPlugin, PluginOpt
from fnmatch import translate
import re

@@ -205,7 +205,7 @@ class RedHatKubernetes(Kubernetes, RedHatPlugin):
super(RedHatKubernetes, self).setup()

-class UbuntuKubernetes(Kubernetes, UbuntuPlugin):
+class UbuntuKubernetes(Kubernetes, UbuntuPlugin, DebianPlugin):

packages = ('kubernetes',)
files = (

I'm sending a PR now

[hchybz]

Hi José.

Thx for the fix. tested on same server (git pull from sos directoty).. it produces some logs but I still have questions :

• sos report -l does not show kubernetes as an inactive plugin , but show that some plugin options are avail . why ?
• i browsed all logs produced by /root/sos/bin/sos report -o kubernetes -vvv --batch --debug and there no much interesting informations except logs . I would expect to find the output (txt or json) of some classical "kubectl get XXX -o wide" commands .

moreover some of the expected commands are described in the kubernetes.py plugin .

how shall I get those infos ?

root@k8s-master:~# kubectl get no -o wide -o json
root@k8s-master:~# kubectl get po -A -o wide -o json
root@k8s-master:~# kubectl get namespace -o json
root@k8s-master:/var# kubectl get pods -A -o wide -o json
...

Thx !

[jcastill]

Hi Hervé,

sos report -l does not show kubernetes as an inactive plugin , but show that some plugin
options are avail . why ?

I'm not sure I understand. Can you paste the output you get with that command? This is what I get in my Debian cluster:

$ bin/sos report -l
sosreport (version 4.6.1)

The following plugins are currently enabled:

[...]
kubernetes Kubernetes container orchestration platform
[...]
The following plugin options are available:
[...]
kubernetes.all off collect all namespace output separately
kubernetes.describe off collect describe output of all resources
kubernetes.podlogs off capture stdout/stderr logs from pods
kubernetes.podlogs-filter only collect logs from pods matching this pattern
[...]

Which is the expected output, afaik.

i browsed all logs produced by /root/sos/bin/sos report -o kubernetes -vvv --batch --debug and there
no much interesting informations except logs . I would expect to find the output (txt or json) of some
classical "kubectl get XXX -o wide" commands .
moreover some of the expected commands are described in the kubernetes.py plugin .

What do you have in the file sos_logs/sos.log? That should tell you what the plugin tried to capture and if it failed or not. In my cluster, I get this:

2024-01-24 10:11:07,606 INFO: [plugin:kubernetes] added cmd output 'kubectl --kubeconfig=/etc/kubernetes/admin.conf get namespaces'
2024-01-24 10:11:07,607 INFO: [plugin:kubernetes] added cmd output 'kubectl --kubeconfig=/etc/kubernetes/admin.conf get projects'
2024-01-24 10:11:07,607 INFO: [plugin:kubernetes] added cmd output 'kubectl --kubeconfig=/etc/kubernetes/admin.conf get pvs'
2024-01-24 10:11:57,644 INFO: [plugin:kubernetes] added cmd output 'kubectl --kubeconfig=/etc/kubernetes/admin.conf get --raw /metrics'
2024-01-24 10:11:57,644 INFO: [plugin:kubernetes] added cmd output 'kubectl --kubeconfig=/etc/kubernetes/admin.conf get --all-namespaces=true deployments'
2024-01-24 10:11:57,644 INFO: [plugin:kubernetes] added cmd output 'kubectl --kubeconfig=/etc/kubernetes/admin.conf get --all-namespaces=true ingresses'
2024-01-24 10:11:57,644 INFO: [plugin:kubernetes] added cmd output 'kubectl --kubeconfig=/etc/kubernetes/admin.conf get --all-namespaces=true limitranges'
2024-01-24 10:11:57,644 INFO: [plugin:kubernetes] added cmd output 'kubectl --kubeconfig=/etc/kubernetes/admin.conf get --all-namespaces=true pods'
2024-01-24 10:11:57,644 INFO: [plugin:kubernetes] added cmd output 'kubectl --kubeconfig=/etc/kubernetes/admin.conf get --all-namespaces=true policies'
2024-01-24 10:11:57,644 INFO: [plugin:kubernetes] added cmd output 'kubectl --kubeconfig=/etc/kubernetes/admin.conf get --all-namespaces=true pvc'
2024-01-24 10:11:57,644 INFO: [plugin:kubernetes] added cmd output 'kubectl --kubeconfig=/etc/kubernetes/admin.conf get --all-namespaces=true rc'
2024-01-24 10:11:57,644 INFO: [plugin:kubernetes] added cmd output 'kubectl --kubeconfig=/etc/kubernetes/admin.conf get --all-namespaces=true resourcequotas'
2024-01-24 10:11:57,644 INFO: [plugin:kubernetes] added cmd output 'kubectl --kubeconfig=/etc/kubernetes/admin.conf get --all-namespaces=true routes'
2024-01-24 10:11:57,644 INFO: [plugin:kubernetes] added cmd output 'kubectl --kubeconfig=/etc/kubernetes/admin.conf get --all-namespaces=true services'
2024-01-24 10:12:08,492 INFO: [plugin:kubernetes] collecting output of 'kubectl --kubeconfig=/etc/kubernetes/admin.conf get namespaces'
2024-01-24 10:12:58,534 INFO: [plugin:kubernetes] collecting output of 'kubectl --kubeconfig=/etc/kubernetes/admin.conf get projects'
2024-01-24 10:13:48,576 INFO: [plugin:kubernetes] collecting output of 'kubectl --kubeconfig=/etc/kubernetes/admin.conf get pvs'
2024-01-24 10:14:38,621 INFO: [plugin:kubernetes] collecting output of 'kubectl --kubeconfig=/etc/kubernetes/admin.conf get --raw /metrics'
2024-01-24 10:14:48,658 INFO: [plugin:kubernetes] collecting output of 'kubectl --kubeconfig=/etc/kubernetes/admin.conf get --all-namespaces=true deployments'
2024-01-24 10:15:38,701 INFO: [plugin:kubernetes] collecting output of 'kubectl --kubeconfig=/etc/kubernetes/admin.conf get --all-namespaces=true ingresses'
2024-01-24 10:16:28,740 INFO: [plugin:kubernetes] collecting output of 'kubectl --kubeconfig=/etc/kubernetes/admin.conf get --all-namespaces=true limitranges'
2024-01-24 10:17:18,778 INFO: [plugin:kubernetes] collecting output of 'kubectl --kubeconfig=/etc/kubernetes/admin.conf get --all-namespaces=true pods'
2024-01-24 10:18:08,829 INFO: [plugin:kubernetes] collecting output of 'kubectl --kubeconfig=/etc/kubernetes/admin.conf get --all-namespaces=true policies'
2024-01-24 10:18:58,872 INFO: [plugin:kubernetes] collecting output of 'kubectl --kubeconfig=/etc/kubernetes/admin.conf get --all-namespaces=true pvc'
2024-01-24 10:19:48,923 INFO: [plugin:kubernetes] collecting output of 'kubectl --kubeconfig=/etc/kubernetes/admin.conf get --all-namespaces=true rc'
2024-01-24 10:20:38,966 INFO: [plugin:kubernetes] collecting output of 'kubectl --kubeconfig=/etc/kubernetes/admin.conf get --all-namespaces=true resourcequotas'
2024-01-24 10:21:29,009 INFO: [plugin:kubernetes] collecting output of 'kubectl --kubeconfig=/etc/kubernetes/admin.conf get --all-namespaces=true routes'

Do you have something similar in your sos.log?

[hchybz]

Hi José

Seems that all options are not active...

root@k8s-master:~/sos# bin/sos report  -l |grep kub
 kubernetes           Kubernetes container orchestration platform
 kubernetes.all            off             collect all namespace output separately
 kubernetes.describe       off             collect describe output of all resources
 kubernetes.podlogs        off             capture stdout/stderr logs from pods
 kubernetes.podlogs-filter                 only collect logs from pods matching this pattern

[jcastill]

Hi Hervé,

That's expected - the options are off by default:

    option_list = [
        PluginOpt('all', default=False,
                  desc='collect all namespace output separately'),
        PluginOpt('describe', default=False,
                  desc='collect describe output of all resources'),
        PluginOpt('podlogs', default=False,
                  desc='capture stdout/stderr logs from pods'),
        PluginOpt('podlogs-filter', default='', val_type=str,
                  desc='only collect logs from pods matching this pattern')
    ]

If you want to enable them, you have to do it explicitly on the command line. I.e., with something like this:

sos report -o kubernetes -k kubernetes.all=True,kubernetes.describe=True,kubernetes.podlogs=True

Basically:

-k PLUGNAME.PLUGOPT[=VALUE], --plugin-option=PLUGNAME.PLUGOPT[=VALUE]

Try that and see if that works for your system.

[hchybz]

root@k8s-worker2:~/sos# bin/sos report -o kubernetes -k kubernetes.all=True,kubernetes.describe=True,kubernetes.podlogs=True --config-file=/root/sos/sos.conf --batch -vvv
root@k8s-worker2:~/sos# cd /var/tmp/
root@k8s-worker2:/var/tmp# tar xvf sosreport-k8s-worker2-2024-01-26-cxnmopp.tar.xz
sosreport-k8s-worker2-2024-01-26-cxnmopp/version.txt
sosreport-k8s-worker2-2024-01-26-cxnmopp/sos_reports/
sosreport-k8s-worker2-2024-01-26-cxnmopp/sos_reports/manifest.json
sosreport-k8s-worker2-2024-01-26-cxnmopp/sos_reports/sos.html
sosreport-k8s-worker2-2024-01-26-cxnmopp/sos_reports/sos.json
sosreport-k8s-worker2-2024-01-26-cxnmopp/sos_reports/sos.txt
sosreport-k8s-worker2-2024-01-26-cxnmopp/sos_logs/
sosreport-k8s-worker2-2024-01-26-cxnmopp/sos_logs/sos.log
sosreport-k8s-worker2-2024-01-26-cxnmopp/sos_logs/ui.log
sosreport-k8s-worker2-2024-01-26-cxnmopp/
sosreport-k8s-worker2-2024-01-26-cxnmopp/environment
sosreport-k8s-worker2-2024-01-26-cxnmopp/etc/
sosreport-k8s-worker2-2024-01-26-cxnmopp/etc/kubernetes/
sosreport-k8s-worker2-2024-01-26-cxnmopp/etc/kubernetes/kubelet.conf
sosreport-k8s-worker2-2024-01-26-cxnmopp/etc/kubernetes/manifests/
sosreport-k8s-worker2-2024-01-26-cxnmopp/etc/kubernetes/pki/
sosreport-k8s-worker2-2024-01-26-cxnmopp/etc/kubernetes/pki/ca.crt
sosreport-k8s-worker2-2024-01-26-cxnmopp/sos_commands/
sosreport-k8s-worker2-2024-01-26-cxnmopp/sos_commands/kubernetes/
sosreport-k8s-worker2-2024-01-26-cxnmopp/sos_commands/kubernetes/journalctl_--no-pager_--unit_cdk.master.auth-webhook
sosreport-k8s-worker2-2024-01-26-cxnmopp/sos_commands/kubernetes/journalctl_--no-pager_--unit_kube-apiserver
sosreport-k8s-worker2-2024-01-26-cxnmopp/sos_commands/kubernetes/journalctl_--no-pager_--unit_kube-controller-manager
sosreport-k8s-worker2-2024-01-26-cxnmopp/sos_commands/kubernetes/journalctl_--no-pager_--unit_kube-proxy
sosreport-k8s-worker2-2024-01-26-cxnmopp/sos_commands/kubernetes/journalctl_--no-pager_--unit_kube-scheduler
sosreport-k8s-worker2-2024-01-26-cxnmopp/sos_commands/kubernetes/journalctl_--no-pager_--unit_kubelet
sosreport-k8s-worker2-2024-01-26-cxnmopp/sos_commands/kubernetes/journalctl_--no-pager_--unit_snap.kube-apiserver.daemon
sosreport-k8s-worker2-2024-01-26-cxnmopp/sos_commands/kubernetes/journalctl_--no-pager_--unit_snap.kube-controller-manager.daemon
sosreport-k8s-worker2-2024-01-26-cxnmopp/sos_commands/kubernetes/journalctl_--no-pager_--unit_snap.kube-proxy.daemon
sosreport-k8s-worker2-2024-01-26-cxnmopp/sos_commands/kubernetes/journalctl_--no-pager_--unit_snap.kube-scheduler.daemon
sosreport-k8s-worker2-2024-01-26-cxnmopp/sos_commands/kubernetes/journalctl_--no-pager_--unit_snap.kubelet.daemon
sosreport-k8s-worker2-2024-01-26-cxnmopp/sos_strings/
sosreport-k8s-worker2-2024-01-26-cxnmopp/sos_strings/kubernetes/
sosreport-k8s-worker2-2024-01-26-cxnmopp/sos_strings/kubernetes/journalctl_--no-pager_--unit_kubelet.tailed

almost nothing

root@k8s-worker2:/var/tmp# ls -al sosreport-k8s-worker2-2024-01-26-cxnmopp/sos_commands/kubernetes
total 52
drwx------ 2 root root 4096 Jan 26 10:39 .
drwxr-xr-x 3 root root 4096 Jan 26 10:38 ..
-rw-r--r-- 1 root root 17 Jan 26 10:38 journalctl_--no-pager_--unit_cdk.master.auth-webhook
-rw-r--r-- 1 root root 17 Jan 26 10:39 journalctl_--no-pager_--unit_kube-apiserver
-rw-r--r-- 1 root root 17 Jan 26 10:39 journalctl_--no-pager_--unit_kube-controller-manager
-rw-r--r-- 1 root root 17 Jan 26 10:39 journalctl_--no-pager_--unit_kube-proxy
-rw-r--r-- 1 root root 17 Jan 26 10:39 journalctl_--no-pager_--unit_kube-scheduler
lrwxrwxrwx 1 root root 72 Jan 26 10:39 journalctl_--no-pager_--unit_kubelet -> ../../sos_strings/kubernetes/journalctl_--no-pager_--unit_kubelet.tailed
-rw-r--r-- 1 root root 17 Jan 26 10:39 journalctl_--no-pager_--unit_snap.kube-apiserver.daemon
-rw-r--r-- 1 root root 17 Jan 26 10:39 journalctl_--no-pager_--unit_snap.kube-controller-manager.daemon
-rw-r--r-- 1 root root 17 Jan 26 10:39 journalctl_--no-pager_--unit_snap.kube-proxy.daemon
-rw-r--r-- 1 root root 17 Jan 26 10:39 journalctl_--no-pager_--unit_snap.kube-scheduler.daemon
-rw-r--r-- 1 root root 17 Jan 26 10:39 journalctl_--no-pager_--unit_snap.kubelet.daemon
root@k8s-worker2:/var/tmp# cat sosreport-k8s-worker2-2024-01-26-cxnmopp/sos_commands/kubernetes/journalctl_--no-pager_--unit_kube-apiserver
-- No entries --

[hchybz]

no news ?

[jcastill]

Hi Hervé,

Apologies, I missed your last note. Let me try some tests in my Debian cluster and I'll update tomorrow.