Sidecar containers in 0.23 are not working

[sahilsk]

Brief summary

Hi,
In 0.0.22 sidecar containers were introduced.

I tried to use them so that my runners sidecar containers can send generated reports to s3 bucket .

But to my surprise they are not working.

k6-operator version or image

0.0.23

Helm chart version (if applicable)

helm-k6-operator-3.16.0

TestRun / PrivateLoadZone YAML

---
apiVersion: k6.io/v1alpha1
kind: TestRun
metadata:
  name: ad2f79e1-testrun
  namespace: ops
spec:
  parallelism: 4
  script:
    configMap:
      name: ad2f79e1-loadscript
      file: script.js
  arguments: --vus 200 --duration 300s --max-redirects 10 --batch 20 --batch-per-host 6 --user-agent Teload_Grafana_K6/1.2.1 --tag testid=ad2f79e1-579c-43ec-b173-61bcafc4fdb0 --tag teamname=unknown --include-system-env-vars --summary-mode compact --out experimental-prometheus-rw --out dashboard=export=/data/ad2f79e1.html&period=2s --out json=/data/ad2f79e1.json
  runner:
    initContainers:
      - name: foo-sidecar
        image: busybox:1.28
        restartPolicy: Always
        command: ['sh','-c','while true; do ls -alh /data/; sleep 1; done']
        workingDir: /data
        volumeMounts:
          - mountPath: /data
            name: k6-provision-location
    metadata:
      annotations:
        sidecar.istio.io/inject: "false"
        testid: ad2f79e1-579c-43ec-b173-61bcafc4fdb0
        created_by: N/A
        team: N/A
        slack_channel: N/A
    resources:
      requests:
        cpu: 500m
        memory: 128Mi
      limits:
        cpu: 1
        memory: 512Mi
    volumes:
      - emptyDir: {}
        name: k6-provision-location
    volumeMounts:
      - mountPath: /data
        name: k6-provision-location
    envFrom:
    - configMapRef:
        name: ad2f79e1-envs
    image: 036476006320.dkr.ecr.us-west-1.amazonaws.com/external-dockerhub/grafana/k6-extended:v2025.08.22-main10-latest-amd64

Other environment details (if applicable)

No response

Steps to reproduce the problem

I am not seeing sidecar containers spawned under runner pods. I am seeing them under initializer pods and they are not sidecar. Initializer pods stuck waiting for sidecar container to finish which will never happen because my sidecar containers are supposed to run in parallel with runnders until they exit.

Expected behaviour

There should be foo-sidecar container spawned when runner pods when initializer pods done spawning runner pods

Actual behaviour

foo-sidecar containers are started insdie -initializer pods and that too as init containers. They are not sidecar containers and spawned under wrong pods (initializers instead of runnners)

[yorugac]

Hi @sahilsk, that's weird. Just in case, have you checked the version of Kubernetes and the state of sidecar feature?

Also, about this:

Initializer pods stuck waiting for sidecar container to finish

How do you see that they're stuck because of the sidecar containers? What is in the logs of both initializer and of k6-operator?

[sahilsk]

Thanks @yorugac for the reply.
I don't see any logs because initializer pod is waiting for the initcontainer to finish. (i've a while loop in init container so they won't finish).

I do see the of the initiailizer and it ssays "restartpolicy: Never". But my question is why the init pods are int

k6 manager logs

│ 2025-08-29T18:35:21Z    INFO    controllers.TestRun    Reconcile(); stage = initialization    {"namespace": "ops", "name": "ad2f79e1-testru │
│ 2025-08-29T18:35:21Z    INFO    controllers.TestRun    Waiting for initializing pod to finish    {"namespace": "ops", "name": "ad2f79e1-tes │
│ 2025-08-29T18:35:21Z    INFO    controllers.TestRun    Creation of initializer pod takes too long: your configuration might be off. Check i │
│

Initializer pod spec

apiVersion: v1
kind: Pod
metadata:
  annotations:
    created_by: N/A
    kubernetes.io/limit-ranger: 'LimitRanger plugin set: cpu, memory request for init
      container foo-sidecar'
    sidecar.istio.io/inject: "false"
    slack_channel: N/A
    team: N/A
    testid: ad2f79e1-579c-43ec-b173-61bcafc4fdb0
  creationTimestamp: "2025-08-29T18:26:26Z"
  finalizers:
  - batch.kubernetes.io/job-tracking
  generateName: ad2f79e1-testrun-initializer-
  labels:
    1keyes.net/environment: stg
    1keyes.net/instance: us1
    1keyes.net/partition: commercial
    app: k6
    batch.kubernetes.io/controller-uid: 3fca08e4-fbf5-4300-9550-f449745839c6
    batch.kubernetes.io/job-name: ad2f79e1-testrun-initializer
    controller-uid: 3fca08e4-fbf5-4300-9550-f449745839c6
    job-name: ad2f79e1-testrun-initializer
    k6_cr: ad2f79e1-testrun
  name: ad2f79e1-testrun-initializer-v8ggj
  namespace: ops
  ownerReferences:
  - apiVersion: batch/v1
    blockOwnerDeletion: true
    controller: true
    kind: Job
    name: ad2f79e1-testrun-initializer
    uid: 3fca08e4-fbf5-4300-9550-f449745839c6
  resourceVersion: "5962552427"
  uid: f8b690ad-c490-47d0-87ac-e3a34855034b
spec:
  automountServiceAccountToken: true
  containers:
  - command:
    - sh
    - -c
    - mkdir -p $(dirname /tmp/script.js.archived.tar) && k6 archive /test/script.js
      -O /tmp/script.js.archived.tar --vus 200 --duration 30s --max-redirects 10 --batch
      20 --batch-per-host 6 --user-agent Teload_Grafana_K6/1.2.1 --tag testid=ad2f79e1-579c-43ec-b173-61bcafc4fdb0
      --tag teamname=unknown --include-system-env-vars --summary-mode compact 2> /tmp/k6logs
      && k6 inspect --execution-requirements /tmp/script.js.archived.tar 2> /tmp/k6logs
      ; ! cat /tmp/k6logs | grep 'level=error'
    env:
    - name: POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
...
    - name: AWS_USE_FIPS_ENDPOINT
      value: "false"
    envFrom:
    - configMapRef:
        name: ad2f79e1-envs
    image: 036476006320.dkr.ecr.us-west-1.amazonaws.com/external-dockerhub/grafana/k6-extended:v2025.08.22-main10-latest-amd64
    imagePullPolicy: IfNotPresent
    name: k6
    ports:
    - containerPort: 6565
      protocol: TCP
    resources:
      limits:
        cpu: "1"
        memory: 512Mi
      requests:
        cpu: 500m
        memory: 128Mi
    securityContext: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /test
      name: k6-test-volume
      readOnly: true
    - mountPath: /data
      name: k6-provision-location
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-hj9dm
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  initContainers:
  - command:
    - sh
    - -c
    - while true; do ls -alh /data/; sleep 1; done
    env:
...
    image: busybox:1.28
    imagePullPolicy: IfNotPresent
    name: foo-sidecar
    resources:
      requests:
        cpu: "0"
        memory: "0"
    securityContext: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /test
      name: k6-test-volume
      readOnly: true
    - mountPath: /data
      name: k6-provision-location
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-hj9dm
      readOnly: true
    workingDir: /data
  nodeName: ip-10-57-238-173.us-west-1.compute.internal
  nodeSelector:
    asg: default
    kubernetes.io/arch: amd64
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Never
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - configMap:
      defaultMode: 420
      name: ad2f79e1-loadscript
    name: k6-test-volume
  - emptyDir: {}
    name: k6-provision-location
  - name: kube-api-access-hj9dm
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2025-08-29T18:26:31Z"
    status: "True"
    type: PodReadyToStartContainers
  - lastProbeTime: null
    lastTransitionTime: "2025-08-29T18:26:26Z"
    message: 'containers with incomplete status: [foo-sidecar]'
    reason: ContainersNotInitialized
    status: "False"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2025-08-29T18:26:26Z"
    message: 'containers with unready status: [k6]'
    reason: ContainersNotReady
    status: "False"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2025-08-29T18:26:26Z"
    message: 'containers with unready status: [k6]'
    reason: ContainersNotReady
    status: "False"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2025-08-29T18:26:26Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - image: 036476006320.dkr.ecr.us-west-1.amazonaws.com/external-dockerhub/grafana/k6-extended:v2025.08.22-main10-latest-amd64
    imageID: ""
    lastState: {}
    name: k6
    ready: false
    restartCount: 0
    started: false
    state:
      waiting:
        reason: PodInitializing
    volumeMounts:
    - mountPath: /test
      name: k6-test-volume
      readOnly: true
      recursiveReadOnly: Disabled
    - mountPath: /data
      name: k6-provision-location
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-hj9dm
      readOnly: true
      recursiveReadOnly: Disabled
  hostIP: 10.57.238.173
  hostIPs:
  - ip: 10.57.238.173
  initContainerStatuses:
  - containerID: containerd://8a0e48d69a8deb4aada5e526178a50481b70f98b65c496e8937d22943b185a39
    image: docker.io/library/busybox:1.28
    imageID: docker.io/library/busybox@sha256:141c253bc4c3fd0a201d32dc1f493bcf3fff003b6df416dea4f41046e0f37d47
    lastState: {}
    name: foo-sidecar
    ready: false
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2025-08-29T18:26:30Z"
    volumeMounts:
    - mountPath: /test
      name: k6-test-volume
      readOnly: true
      recursiveReadOnly: Disabled
    - mountPath: /data
      name: k6-provision-location
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-hj9dm
      readOnly: true
      recursiveReadOnly: Disabled
  phase: Pending
  podIP: 10.57.252.112
  podIPs:
  - ip: 10.57.252.112
  qosClass: Burstable
  startTime: "2025-08-29T18:26:26Z"

Kubernetes version: 1.32

EKS and sidecars enabled by default, otherwise i should have gotten error like restartpolicy: Always invalid. isn't it?

 kubectl version
Client Version: v1.33.3
Kustomize Version: v5.6.0
Server Version: v1.32.6-eks-931bdca

No logs

I don't see any logs because initializer pod is waiting for the initcontainer to finish. (i've a while loop in init container so they won't finish).

❯ k logs ad2f79e1-testrun-initializer-v8ggj
Defaulted container "k6" out of: k6, foo-sidecar (init)
Error from server (BadRequest): container "k6" in pod "ad2f79e1-testrun-initializer-v8ggj" is waiting to start: PodInitializing

The initpod is started inside initializer

Testrun file

Full testrun manifest is pasted in the issue above.

[yorugac]

Hi @sahilsk,
Sorry for the delay! I've just tried this TestRun:

apiVersion: k6.io/v1alpha1
kind: TestRun
metadata:
  name: ttt
spec:
  parallelism: 4
  script:
    configMap:
      name: crocodile-stress-test
      file: test.js
  runner:
    initContainers:
      - name: foo-sidecar
        image: busybox:1.28
        restartPolicy: Always
        command: ["sh", "-c", "while true; do ls -alh /data/; sleep 1; done"]
        workingDir: /data
        volumeMounts:
          - mountPath: /data
            name: k6-provision-location
    resources:
      requests:
        cpu: 500m
        memory: 128Mi
      limits:
        cpu: 1
        memory: 512Mi
    volumes:
      - emptyDir: {}
        name: k6-provision-location
    volumeMounts:
      - mountPath: /data
        name: k6-provision-location

With the v1.33.1 kind cluster and latest k6-operator, and it works successfully. Container with busybox runs in parallel to the main test: it logs the volume's folder in a loop and is terminated with error after the test is finished (because of the infinite loop I imagine).

So I'm not clear how to repeat this case ATM 😕 I've simplified your TestRun a bit, but the removed parts shouldn't be impacting the behaviour of containers.

Looking at your initializer pod, it does seem as if it's waiting for the init container... OTOH, if init container is running, why are there no logs from it? There should the output of ls command in the logs of the foo-sidecar container. Did the init container actually start, I wonder? Could you also check what kubectl describe says for your initializer pod pod?

[sahilsk]

@yorugac Thank you for checking it.

Init contianer did start and it should list the files output by the testrun pods (same volume mounted) which in case never got to start. So, no output is expected behaviour.

Let me retry this all and come back to you. I'll use your configuration only this time

[stytchiz]

By the way, I also notice my initializer pod has restartPolicy: Never. This seems normal though as a job is spawning the pod.

    command: ["sh", "-c", "while true; do ls -alh /data/; sleep 1; done"]

If I had to guess, since this command never has a chance to exit, initializer pod gets stuck at the init phase (hence blocking the runner/starter pods).

Off the top of my head, this can be quickly mitigated by overriding the initializer spec to not run that command. But @yorugac can advise on the best way forward here.

[yorugac]

@stytchiz, this is expected: the k6 container will have restartPolicy: Never because that's how it's defined by k6-operator. You can't override that. But init container will have what you define in TestRun and with this example it should be restartPolicy: Always.

Actually, now that I'm re-checking pod in the comment here, it's not clear what the restart policy for init container is... I imagine the line got lost during copy-pasting, so worth re-checking. For completeness, here's how the initializer spec looks for me with the above TestRun:

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: "2025-09-05T06:41:50Z"
  generateName: ttt-initializer-
  generation: 1
  labels:
    app: k6
    batch.kubernetes.io/controller-uid: 8e4f7f24-37fc-4df9-b4a7-7811845fb36d
    batch.kubernetes.io/job-name: ttt-initializer
    controller-uid: 8e4f7f24-37fc-4df9-b4a7-7811845fb36d
    job-name: ttt-initializer
    k6_cr: ttt
  name: ttt-initializer-rlvqf
  namespace: default
  ownerReferences:
  - apiVersion: batch/v1
    blockOwnerDeletion: true
    controller: true
    kind: Job
    name: ttt-initializer
    uid: 8e4f7f24-37fc-4df9-b4a7-7811845fb36d
  resourceVersion: "722"
  uid: 4fda59d2-0542-462b-b03c-5c5d07916f11
spec:
  automountServiceAccountToken: true
  containers:
  - command:
    - sh
    - -c
    - mkdir -p $(dirname /tmp/test.js.archived.tar) && k6 archive /test/test.js -O
      /tmp/test.js.archived.tar  2> /tmp/k6logs && k6 inspect --execution-requirements
      /tmp/test.js.archived.tar 2> /tmp/k6logs ; ! cat /tmp/k6logs | grep 'level=error'
    image: grafana/k6:latest
    imagePullPolicy: Always
    name: k6
    ports:
    - containerPort: 6565
      protocol: TCP
    resources:
      limits:
        cpu: "1"
        memory: 512Mi
      requests:
        cpu: 500m
        memory: 128Mi
    securityContext: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /test
      name: k6-test-volume
      readOnly: true
    - mountPath: /data
      name: k6-provision-location
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-fsg8g
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  initContainers:
  - command:
    - sh
    - -c
    - while true; do ls -alh /data/; sleep 1; done
    image: busybox:1.28
    imagePullPolicy: IfNotPresent
    name: foo-sidecar
    resources: {}
    restartPolicy: Always
    securityContext: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /test
      name: k6-test-volume
      readOnly: true
    - mountPath: /data
      name: k6-provision-location
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-fsg8g
      readOnly: true
    workingDir: /data
  nodeName: kind-control-plane
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Never
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - configMap:
      defaultMode: 420
      name: crocodile-stress-test
    name: k6-test-volume
  - emptyDir: {}
    name: k6-provision-location
  - name: kube-api-access-fsg8g
    ...
status:
 ...
  phase: Succeeded
  podIP: 10.244.0.6
  podIPs:
  - ip: 10.244.0.6
  qosClass: Burstable
  startTime: "2025-09-05T06:41:50Z"

As you can see, there are two diff values for restartPolicy for two containers. For the main container it's Never, for init container it's Always.

[sahilsk]

I've this as well restartPolicy: Always as indicated in the issue body message. In fact, this is the definition of sidecar containers as well (:

[yorugac]

@sahilsk, do you mean that you don't have two diff values of restartPolicy for the main container and for init container? Your main message does not contain the description of the Pod spec and the pod spec in the comment does not contain full description of the init container.