[Q&A] undefined method `have_capability?' for nil"

[krish0596]

What is a problem?

I am getting this error
2025-11-13 09:34:22 +0000 [error]: fluent/log.rb:414:error: unexpected error error_class=NoMethodError error="undefined method have_capability?' for nil"`
when trying to load a containerized fluentd image with added capablities through daemon set, not able to resolve this issue

Describe the configuration of Fluentd

FROM oraclelinux:8-slim

USER root
WORKDIR /fluentd

ARG RUBY_VERSION=3.3

ENV LANG=en_US.UTF-8

ENV PATH /fluentd/vendor/bundle/ruby/${RUBY_VERSION}/bin:$PATH
ENV GEM_PATH /fluentd/vendor/bundle/ruby/${RUBY_VERSION}:$GEM_PATH
ENV GEM_HOME /fluentd/vendor/bundle/ruby/${RUBY_VERSION}
# skip runtime bundler installation
ENV FLUENTD_DISABLE_BUNDLER_INJECTION 1

COPY Gemfile /fluentd/

# Install ruby, ruby-libs along with rubygems and bundler.
RUN microdnf -y module enable ruby:${RUBY_VERSION} \
# Install ruby and ruby-libs, disabling week dependencies
 && microdnf -y install --setopt=install_weak_deps=0 --nodocs ruby ruby-libs rubygems \
 && gem install bundler \
# Install development dependent packages for gems native installation
 && microdnf --enablerepo ol8_codeready_builder -y install --nodocs gcc make ruby-devel gcc-c++ libffi-devel bzip2 libyaml-devel libcap which mysql-devel libcap-ng libcap-ng-devel pkg-config \
 && bundle config silence_root_warning true \
 && bundle config --local path /fluentd/vendor/bundle \
 && bundle config --global jobs 9 \
 && bundle install --gemfile=/fluentd/Gemfile \
 && FLUENTD_BIN=$(readlink -f /fluentd/vendor/bundle/ruby/*/bin/fluentd) \
 && echo "Resolved Fluentd binary: $FLUENTD_BIN" \
 && setcap cap_dac_override+ep "$FLUENTD_BIN" \
# Install tini, init for containers (from EPEL repo)
 && microdnf -y install --setopt=install_weak_deps=0 --nodocs oracle-epel-release-el8 \
 && microdnf -y install --nodocs tini \
 && groupadd -r -g 2000 fluentd \
 && useradd -u 2000 -r -g fluentd -d /fluentd -s /sbin/nologin -c "Fluentd User" fluentd \
 && microdnf clean all \
 && rm -rf /var/cache/dnf /tmp/* /var/tmp/*  \
 && mkdir -p /fluentd/etc /fluentd/plugins /fluentd/log /fluentd/state \
 && touch /fluentd/etc/disable.conf \
 && chown -R fluentd:fluentd /fluentd

#COPY fluent.conf /fluentd/etc
COPY entrypoint.sh /fluentd/entrypoint.sh
RUN ["chmod", "+x", "/fluentd/entrypoint.sh"]

ENV FLUENTD_CONF="/fluentd/etc/fluent.conf"
ENV FLUENTD_OPT=""

USER fluentd:fluentd

ENTRYPOINT ["tini", "--", "/fluentd/entrypoint.sh"]

with the Gemfile being

source "https://rubygems.org"

gem "oj"
gem "json"
gem "capng_c"
gem "fluentd", "1.19.0"
gem "fluent-plugin-sql"
gem "fluent-plugin-opensearch"
gem "mysql2"
gem "rexml"

and entrypoint.sh
bundle exec fluentd -c ${FLUENTD_CONF} -p /fluentd/plugins --gemfile /fluentd/Gemfile -vv

Describe the logs of Fluentd

>  k logs test-fluentd-daemonset-cap-nbug-b8q98 -n kube-system                                                                                  ─╯
> 2025-11-13 09:34:22 +0000 [info]: fluent/log.rb:372:info: init supervisor logger path=nil rotate_age=nil rotate_size=nil
> 2025-11-13 09:34:22 +0000 [info]: fluent/log.rb:372:info: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
> 2025-11-13 09:34:22 +0000 [info]: fluent/log.rb:372:info: inaccessible include directory was specified path="/etc/fluent/conf.d"
> 2025-11-13 09:34:22 +0000 [info]: fluent/log.rb:372:info: gem 'fluentd' version '1.19.0'
> 2025-11-13 09:34:22 +0000 [info]: fluent/log.rb:372:info: gem 'fluent-plugin-opensearch' version '1.1.5'
> 2025-11-13 09:34:22 +0000 [info]: fluent/log.rb:372:info: gem 'fluent-plugin-sql' version '2.3.1'
> 2025-11-13 09:34:22 +0000 [trace]: fluent/log.rb:329:trace: registered output plugin 'stdout'
> 2025-11-13 09:34:22 +0000 [trace]: fluent/log.rb:329:trace: registered metrics plugin 'local'
> 2025-11-13 09:34:22 +0000 [trace]: fluent/log.rb:329:trace: registered buffer plugin 'memory'
> 2025-11-13 09:34:22 +0000 [trace]: fluent/log.rb:329:trace: registered formatter plugin 'stdout'
> 2025-11-13 09:34:22 +0000 [trace]: fluent/log.rb:329:trace: registered formatter plugin 'json'
> 2025-11-13 09:34:22 +0000 [trace]: fluent/log.rb:329:trace: registered parser plugin 'regexp'
> 2025-11-13 09:34:22 +0000 [trace]: fluent/log.rb:329:trace: registered parser plugin 'multiline'
> 2025-11-13 09:34:22 +0000 [trace]: fluent/log.rb:329:trace: registered input plugin 'tail'
> 2025-11-13 09:34:22 +0000 [error]: fluent/log.rb:414:error: unexpected error error_class=NoMethodError error="undefined method `have_capability?' for nil"
>   2025-11-13 09:34:22 +0000 [debug]: command/fluentd.rb:362:<top (required)>: /fluentd/vendor/bundle/ruby/3.3.0/gems/fluentd-1.19.0/lib/fluent/plugin/in_tail.rb:317:in `have_read_capability?'
>   2025-11-13 09:34:22 +0000 [debug]: command/fluentd.rb:362:<top (required)>: /fluentd/vendor/bundle/ruby/3.3.0/gems/fluentd-1.19.0/lib/fluent/plugin/in_tail.rb:356:in `block (2 levels) in expand_paths_raw'
>   2025-11-13 09:34:22 +0000 [debug]: command/fluentd.rb:362:<top (required)>: /fluentd/vendor/bundle/ruby/3.3.0/gems/fluentd-1.19.0/lib/fluent/plugin/in_tail.rb:353:in `select'
>   2025-11-13 09:34:22 +0000 [debug]: command/fluentd.rb:362:<top (required)>: /fluentd/vendor/bundle/ruby/3.3.0/gems/fluentd-1.19.0/lib/fluent/plugin/in_tail.rb:353:in `block in expand_paths_raw'
>   2025-11-13 09:34:22 +0000 [debug]: command/fluentd.rb:362:<top (required)>: /fluentd/vendor/bundle/ruby/3.3.0/gems/fluentd-1.19.0/lib/fluent/plugin/in_tail.rb:346:in `each'
>   2025-11-13 09:34:22 +0000 [debug]: command/fluentd.rb:362:<top (required)>: /fluentd/vendor/bundle/ruby/3.3.0/gems/fluentd-1.19.0/lib/fluent/plugin/in_tail.rb:346:in `expand_paths_raw'
>   2025-11-13 09:34:22 +0000 [debug]: command/fluentd.rb:362:<top (required)>: /fluentd/vendor/bundle/ruby/3.3.0/gems/fluentd-1.19.0/lib/fluent/plugin/in_tail.rb:221:in `check_dir_permission'
>   2025-11-13 09:34:22 +0000 [debug]: command/fluentd.rb:362:<top (required)>: /fluentd/vendor/bundle/ruby/3.3.0/gems/fluentd-1.19.0/lib/fluent/plugin/in_tail.rb:168:in `configure'
>   2025-11-13 09:34:22 +0000 [debug]: command/fluentd.rb:362:<top (required)>: /fluentd/vendor/

and this was the daemonSet manifest

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: test-fluentd-daemonset-cap-nbug
  namespace: kube-system
  labels:
    app: test-fluentd-daemonset-ca
    version: v1
spec:
  selector:
    matchLabels:
      app: test-fluentd-daemonset-ca
      version: v1
  template:
    metadata:
      labels:
        app: test-fluentd-daemonset-ca
        version: v1
    spec:
      terminationGracePeriodSeconds: 30
      containers:
        - name: test-fluentd-daemonset-c1
          image: localhost/myfluentuser7-nc:latest
          securityContext:
            capabilities:
              add:
                - CAP_DAC_OVERRIDE
                - CAP_DAC_READ_SEARCH
          imagePullPolicy: Never
          resources:
            limits:
              memory: 2Gi
            requests:
              cpu: 100m
              memory: 1Gi
          volumeMounts:
            # Mount Kubernetes logs
            - name: varlog
              mountPath: /var/log
              readOnly: true

            # Optional: pod-specific log folders (depends on setup)
            - name: dockercontainerlogdirectory
              mountPath: /var/log/pods
              readOnly: true

            - name: dockercontainerdatadirectory
              mountPath: /u01/data/docker/containers
              readOnly: true

            - name: varlogcontainers
              mountPath: /var/log/containers
              readOnly: true

              # Mount fluentd config (from ConfigMap)
            - name: fluentdconfigdir
              mountPath: /fluentd/etc/fluent.conf
              subPath: fluent.conf
              readOnly: true

            # Mount a writable directory for Fluentd state/position files
            - name: fluentdstate
              mountPath: /fluentd/state

      volumes:
        - name: varlog
          hostPath:
            path: /var/log
        - name: dockercontainerlogdirectory
          hostPath:
            path: /var/log/pods
        - name: dockercontainerdatadirectory
          hostPath:
            path: /u01/data/docker/containers
        - name: varlogcontainers
          hostPath:
            path: /var/log/containers
        - name: fluentdconfigdir
          configMap:
            name: fluentd-config
        - name: fluentdstate
          emptyDir: {}

the fluent.conf was loaded via a config map and uses tail

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd-config
  namespace: kube-system
data:
  fluent.conf: |
    <source>
      @type tail
      path /var/log/containers/*.log
      exclude_path ["/var/log/containers/test-fluentd*"]
      pos_file /fluentd/state/containers.pos
      tag kubernetes.*
      read_from_head false
    </source>

    <match **>
      @type stdout
    </match>

Environment

RUBY_VERSION=3.3

[krish0596]

@Watson1978 Tagging you for help 🙏

[Watson1978]

Seems it is a bug in #4865.
I think you can avoid the error by prevent to use Linux Capability feature.

[Watson1978]

I filed an issue at #5152

[Watson1978]

Hmm, @capability instance variable will be available at
https://github.com/fluent/fluentd/blob/08f8c24f849a3f7533ae2bee09f760360506d24b/lib/fluent/plugin/in_tail.rb#L198C1-L198C61

However, the instance variable has been used at

fluentd/lib/fluent/plugin/in_tail.rb

Line 168 in 08f8c24

check_dir_permission unless Fluent.windows?

before the instance variable is available

• check_dir_permission -> expand_paths_raw -> have_read_capability?