kubectl logs returns "Forbidden" on k3s with AWX Operator – cannot retrieve container logs

[nakayama1869]

Please confirm the following

• I agree to follow this project's code of conduct.
• I have checked the current issues for duplicates.
• I understand that the AWX Operator is open source software provided for free and that I might not receive a timely response.

Bug Summary

I’m running a single-node k3s cluster (on Amazon Linux 2023) and have deployed the AWX Operator. However, whenever I try to retrieve logs from the awx-operator-controller-manager deployment, I get a Forbidden error, like this:

$ kubectl logs -n awx deployments/awx-operator-controller-manager -c awx-manager
Error from server: Get "https://26.56.xxx.xxx:10250/containerLogs/awx/awx-operator-controller-manager-687b856498-xxxxx/awx-manager": Forbidden

Environment

OS: Amazon Linux 2023
k3s version: v1.31.5+k3s1 (for example)
Single-node cluster (control-plane + worker on the same instance)
AWX Operator installed and running (Pods are in Running status)
Node INTERNAL-IP is 26.56.xxx.xxx

Troubleshooting Steps Taken

NO_PROXY configuration
    curl -k https://26.56.xxx.xxx:10250/ now returns 404, so we’re no longer blocked by a corporate proxy or firewall (previously we saw a 403 from the proxy).
    So the network path to port 10250 is open.

RBAC
    Confirmed I can get pods/log as the maintain user (kubectl auth can-i get pods/log -n awx --as=maintain → yes).
    Also tried running kubectl logs with cluster-admin credentials via sudo KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl logs ..., still Forbidden.

Kubelet Configuration
    Attempted to set anonymous-auth=false and authentication-token-webhook=true in /etc/rancher/k3s/config.yaml and via systemd drop-in (--kubelet-arg=...).
    Restarted k3s multiple times, but the setting might not actually take effect; the logs remain Forbidden.

kubeconfig Permission
    Overcame permission denied by adjusting file perms or using sudo, so that’s no longer an issue.
    But the Forbidden error persists.

k3s logs (journalctl -u k3s)
    Whenever the Forbidden error occurs, the k3s log only shows:

    E... status.go:71] "Unhandled Error" err="apiserver received an error: ...Get \"https://26.56.xxx.xxx:10250/...\": Forbidden"

    No clear indication about whether anonymous-auth or token webhook is actually enabled.

Question / Request for Help

Despite configuring NO_PROXY, confirming RBAC privileges, and updating the Kubelet settings (authentication-token-webhook=true), the kubectl logs request remains Forbidden.
I’d prefer not to fully reinstall k3s if possible. Are there any additional debug steps or configurations I should check?
If anyone has encountered a similar issue with k3s returning Forbidden for logs, please let me know how you resolved it.

I’m happy to provide more info if needed. Thanks in advance for your help!

AWX Operator version

2.19.1

AWX version

latest

Kubernetes platform

kubernetes

Kubernetes/Platform version

latest

Modifications

no

Steps to reproduce

curl -sfL https://get.k3s.io | sh -s - --write-kubeconfig-mode 644 &&
curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh" | bash &&
mv kustomize /usr/local/bin/

●install AWS Operator
cd /kustomize
vi kustomization.yaml
kustomize build . | kubectl apply -f -
kubectl get pods -n awx
kubectl config set-context --current --namespace=awx
vi kustomization.yaml
vi awx-demo.yaml
kustomize build . | kubectl apply -f -
kubectl get pods -l "app.kubernetes.io/managed-by=awx-operator" -n awx

Expected results

Actual results

Additional information

No response

Operator Logs

No response