failed to enable bpfLsmEnforcer

[greenhandatsjtu]

I tried to enable bpfLsmEnforcer with this command:

helm install varmor varmor-0.5.4.tgz  --namespace varmor --create-namespace --set image.registry="elkeid-cn-beijing.cr.volces.com" --set bpfLsmEnforcer.enabled=true 

However, I found vamor-agent failed to start:

kubectl get pod -n varmor

NAME                              READY   STATUS             RESTARTS   AGE
varmor-agent-5prn9                0/1     CrashLoopBackOff   8          16m
varmor-agent-d78l4                0/1     CrashLoopBackOff   8          16m
varmor-agent-xq6sf                0/1     CrashLoopBackOff   8          16m
varmor-manager-599f6fd885-5frk5   1/1     Running            0          16m
varmor-manager-599f6fd885-jqmp7   1/1     Running            0          16m
varmor-manager-599f6fd885-p8z4b   1/1     Running            0          16m

This is the error log:

kubectl describe pod -n varmor varmor-agent-5prn9

Events:
  Type     Reason     Age                  From                 Message
  ----     ------     ----                 ----                 -------
  Normal   Scheduled  <unknown>            default-scheduler    Successfully assigned varmor/varmor-agent-5prn9 to k8s-master
  Warning  Failed     16m (x4 over 17m)    kubelet, k8s-master  Error: failed to start container "agent": Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: unable to apply apparmor profile: apparmor failed to apply profile: open /proc/self/attr/apparmor/exec: read-only file system: unknown
  Normal   Pulling    15m (x5 over 17m)    kubelet, k8s-master  Pulling image "elkeid-cn-beijing.cr.volces.com/varmor/varmor:v0.5.4"
  Normal   Pulled     15m (x5 over 17m)    kubelet, k8s-master  Successfully pulled image "elkeid-cn-beijing.cr.volces.com/varmor/varmor:v0.5.4"
  Normal   Created    15m (x5 over 17m)    kubelet, k8s-master  Created container agent
  Warning  BackOff    2m9s (x66 over 16m)  kubelet, k8s-master  Back-off restarting failed container

It seems runc failed to open apparmor profile?

open /proc/self/attr/apparmor/exec: read-only file system: unknown

[Danny-Wei]

vArmor will enable the AppArmor enforcer by default. If your system does not support the AppArmor LSM, it may lead to startup failure. You can use the flag --set appArmorLsmEnforcer.enabled=false to manually disable the AppArmor enforcer.

[greenhandatsjtu]

Hi @Danny-Wei , thanks for your answer! However, I think my system supports AppArmor LSM, as I've tried default installation with AppArmor enforcer enabled, everything is OK, so I don't know why :(

helm install varmor varmor-0.5.4.tgz  --namespace varmor --create-namespace --set image.registry="elkeid-cn-beijing.cr.volces.com"

NAME                              READY   STATUS    RESTARTS   AGE
varmor-agent-4pnqh                1/1     Running   0          7s
varmor-agent-f27j4                1/1     Running   0          7s
varmor-agent-nnckt                1/1     Running   0          7s
varmor-manager-599f6fd885-8vsdz   1/1     Running   0          7s
varmor-manager-599f6fd885-pjn8z   1/1     Running   0          7s
varmor-manager-599f6fd885-tw2hl   1/1     Running   0          7s

[Danny-Wei]

Can you provide the logs of agent after enabling the BPF enforcer?

[greenhandatsjtu]

Hi @Danny-Wei , sorry there's no logs of varmor-agent, I think it's because the container of these agents even can't be created.

[Danny-Wei]

It might be because the kubelet is attempting to restart the agent container, making it difficult to retrieve logs for the deleted container. You can try quickly retrieving logs with this command: kubectl logs -n varmor $(kubectl get Pods -n varmor | grep varmor-agent | head -n 1 | awk '{print $1}')

Please also provide the system and kernel version information for the investigation.

[greenhandatsjtu]

Hi @Danny-Wei , thanks for your advice
I've tried the command you suggested, but still got nothing :(

Here is K8s version:

Here is Linux version:

[Danny-Wei]

Please provide the output of the following commands:

kubectl describe Pods -n varmor $(kubectl get Pods -n varmor | grep varmor-agent | head -n 1 | awk '{print $1}')
containerd --version
runc --version
cat /proc/cmdline
aa-status

You can reinstall apparmor and reboot to see if it resolves this issue.

apt install apparmor
reboot

[greenhandatsjtu]

Hi @Danny-Wei , sorry for my delayed response, here are the output of the commands:

kubectl describe Pods -n varmor $(kubectl get Pods -n varmor | grep varmor-agent | head -n 1 | awk '{print $1}')

Name:         varmor-agent-5q98c
Namespace:    varmor
Priority:     0
Node:         k8s-master/10.10.21.172
Start Time:   Fri, 08 Dec 2023 18:40:43 +0800
Labels:       app.kubernetes.io/component=varmor-agent
              app.kubernetes.io/name=varmor
              controller-revision-hash=799f46d74
              pod-template-generation=1
Annotations:  cni.projectcalico.org/podIP: 10.244.235.239/32
Status:       Running
IP:           10.244.235.239
IPs:
  IP:           10.244.235.239
Controlled By:  DaemonSet/varmor-agent
Containers:
  agent:
    Container ID:  docker://531a2a42064278f368e4980741a783f76c956ca02557aee6b6353d1fa93bc3f7
    Image:         elkeid-cn-beijing.cr.volces.com/varmor/varmor:v0.5.4
    Image ID:      docker-pullable://elkeid-cn-beijing.cr.volces.com/varmor/varmor@sha256:42a2c4971af65fe75306ae6ce8e1d4ae11b687421d9df668d81a73c0a87b606b
    Port:          <none>
    Host Port:     <none>
    Command:
      /varmor/vArmor
      --agent
    Args:
      --enableBpfEnforcer
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       ContainerCannotRun
      Message:      failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: unable to apply apparmor profile: apparmor failed to apply profile: open /proc/self/attr/apparmor/exec: read-only file system: unknown
      Exit Code:    128
      Started:      Thu, 14 Dec 2023 17:29:14 +0800
      Finished:     Thu, 14 Dec 2023 17:29:14 +0800
    Ready:          False
    Restart Count:  1679
    Limits:
      cpu:     200m
      memory:  200Mi
    Requests:
      cpu:        100m
      memory:     100Mi
    Environment:  <none>
    Mounts:
      /etc/apparmor.d from apparmor-dir (rw)
      /proc from procfs (ro)
      /run/containerd/ from containerd (rw)
      /sys/kernel/btf/vmlinux from btf (rw)
      /sys/kernel/security from securityfs (rw)
      /sys/module/apparmor from apparmor (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from varmor-agent-token-9xnv8 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  securityfs:
    Type:          HostPath (bare host directory volume)
    Path:          /sys/kernel/security
    HostPathType:  Directory
  apparmor:
    Type:          HostPath (bare host directory volume)
    Path:          /sys/module/apparmor
    HostPathType:  Directory
  apparmor-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/varmor/apparmor.d
    HostPathType:  DirectoryOrCreate
  containerd:
    Type:          HostPath (bare host directory volume)
    Path:          /run/containerd/
    HostPathType:  Directory
  procfs:
    Type:          HostPath (bare host directory volume)
    Path:          /proc
    HostPathType:  Directory
  btf:
    Type:          HostPath (bare host directory volume)
    Path:          /sys/kernel/btf/vmlinux
    HostPathType:  File
  varmor-agent-token-9xnv8:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  varmor-agent-token-9xnv8
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     :NoSchedule
                 node.kubernetes.io/disk-pressure:NoSchedule
                 node.kubernetes.io/memory-pressure:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute
                 node.kubernetes.io/pid-pressure:NoSchedule
                 node.kubernetes.io/unreachable:NoExecute
                 node.kubernetes.io/unschedulable:NoSchedule
Events:
  Type     Reason   Age                      From                 Message
  ----     ------   ----                     ----                 -------
  Warning  BackOff  90s (x38568 over 5d22h)  kubelet, k8s-master  Back-off restarting failed container

---

containerd --version

containerd containerd.io 1.6.14 9ba4b250366a5ddde94bb7c9d1def331423aa323

---

runc --version

runc version 1.1.4
commit: v1.1.4-0-g5fd4c4d
spec: 1.0.2-dev
go: go1.18.9
libseccomp: 2.5.1

---

I alse tried to reinstall apparmor and reboot, the issue is still not solved with the same error

Events:
  Type     Reason     Age                    From                Message
  ----     ------     ----                   ----                -------
  Normal   Scheduled  <unknown>              default-scheduler   Successfully assigned varmor/varmor-agent-fvh2v to k8s-node1
  Warning  Failed     2m48s (x4 over 3m29s)  kubelet, k8s-node1  Error: failed to start container "agent": Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: unable to apply apparmor profile: apparmor failed to apply profile: open /proc/self/attr/apparmor/exec: read-only file system: unknown
  Warning  BackOff    2m13s (x6 over 3m13s)  kubelet, k8s-node1  Back-off restarting failed container
  Normal   Pulling    2m (x5 over 3m30s)     kubelet, k8s-node1  Pulling image "elkeid-cn-beijing.cr.volces.com/varmor/varmor:v0.5.4"
  Normal   Pulled     119s (x5 over 3m29s)   kubelet, k8s-node1  Successfully pulled image "elkeid-cn-beijing.cr.volces.com/varmor/varmor:v0.5.4"
  Normal   Created    119s (x5 over 3m29s)   kubelet, k8s-node1  Created container agent

[Danny-Wei]

Hi, thanks for your response. We haven't adapted vArmor for a k8s + Docker environment for now. Could you install vArmor in a k8s + containerd environment? We will then create an environment with k8s + Docker to investigate the root cause of this issue.

[greenhandatsjtu]

Hi @Danny-Wei thanks for your advice! I will try to install vArmor in containerd env

[zer0-1s]

Is it that the kernel version does not support CAP_BPF, or that vArmor does not support Kubernetes + Docker? @Danny-Wei