Amazon Linux 2 - fluent-bit.service fails to start with error "Dependency failed for Fluent Bit."

[charltonstanley]

Bug Report

Describe the bug
#9845 was implemented to address a race condition, however this will fail for any instances that require the use of version 2 of AWS's Instance Metadata Service. IMDSv2 requires retrieving a token via a PUT and then passing that token when performing your GET.
See the documentation here: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-service.html#instance-metadata-retrieval-examples

To Reproduce

1. Install fluent-bit 4.0.0 on an Amazon Linux 2 instance that requires IMDSv2
2. observe that the fluent-bit service won't even start.

Log output from the new setservice.service dependency that was added in #9845.

[ec2-user@REDACTED ~]$ sudo service sethostname status
Redirecting to /bin/systemctl status sethostname.service
● sethostname.service - Set Hostname Workaround coreos/bugs#1272
   Loaded: loaded (/usr/lib/systemd/system/sethostname.service; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Tue 2025-04-08 17:55:02 CDT; 1min 16s ago
  Process: 5306 ExecStart=/bin/sh -c /usr/bin/hostnamectl set-hostname $(curl -s http://169.254.169.254/latest/meta-data/hostname) (code=exited, status=1/FAILURE)
 Main PID: 5306 (code=exited, status=1/FAILURE)

Apr 08 17:55:02 REDACTED.ec2.internal systemd[1]: Starting Set Hostname Workaround coreos/bugs#1272...
Apr 08 17:55:02 REDACTED.ec2.internal sh[5306]: Invalid number of arguments.
Apr 08 17:55:02 REDACTED.ec2.internal systemd[1]: sethostname.service: main process exited, code=exited, status=1/FAILURE
Apr 08 17:55:02 REDACTED.ec2.internal systemd[1]: Failed to start Set Hostname Workaround coreos/bugs#1272.
Apr 08 17:55:02 REDACTED.ec2.internal systemd[1]: Unit sethostname.service entered failed state.
Apr 08 17:55:02 REDACTED.ec2.internal systemd[1]: sethostname.service failed.

Expected behavior
Fluent-bit service starts

Screenshots

Screenshot of relevant EC2 Instance setting

Your Environment

• Version used: 4.0.0
• Configuration:
• Environment name and version (e.g. Kubernetes? What version?):
• Server type and version: n/a
• Operating System and version: Amazon Linux 2
• Filters and plugins: n/a

Additional context

We are unable to upgrade to v4.0.0 on all of the instances in our AWS organization. We are smaller organization, but I know that we can't be the only AWS customers that, for security reasons, require IMDSv2 to be used over IMDSv1.

Security benefits of IMDSv2: https://aws.amazon.com/blogs/security/defense-in-depth-open-firewalls-reverse-proxies-ssrf-vulnerabilities-ec2-instance-metadata-service/

[cpandya-we]

@charltonstanley
Replace your /lib/systemd/system/sethostname.service file with the contents below:

[Unit]
Description=Set Hostname Workaround coreos/bugs#1272 with EC2 IMDSv2 support
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
RemainAfterExit=yes

ExecStartPre=/bin/bash -c 'curl -sX PUT "http://169.254.169.254/latest/api/token" \
  -H "X-aws-ec2-metadata-token-ttl-seconds: 21600" > /run/imds_token'

ExecStartPre=/bin/bash -c 'curl -s "http://169.254.169.254/latest/meta-data/hostname" \
  -H "X-aws-ec2-metadata-token: $(cat /run/imds_token)" > /run/ec2_hostname'

ExecStart=/usr/bin/hostnamectl set-hostname "$(cat /run/ec2_hostname)"

ExecStartPost=/bin/bash -c 'rm -f /run/imds_token /run/ec2_hostname'

[Install]
WantedBy=multi-user.target

Then run the below two commands to pickup the new sethostname service

sudo systemctl daemon-reload
sudo systemctl start sethostname.service

[charltonstanley]

@cpandya-we Thanks for this. I had to change ExecStart from

ExecStart=/usr/bin/hostnamectl set-hostname "$(cat /run/ec2_hostname)"

to

ExecStart=/bin/bash -c '/usr/bin/hostnamectl set-hostname "$(cat /run/ec2_hostname)"'

In order for the "$(cat /run/ec2_hostname)" part of the command to work.

[rasturic]

This broke for us also.

• removed combined command, not tested -

However, we've been running fluent-bit for years on AL2 without needing this additional service or encountering this race condition. I'm tempted to just remove the dependency from our fluent-bit unit file.

[gonfva-flextrade]

Unless I'm missing something, another alternative is to use ec2-metadata directly

# cat /usr/lib/systemd/system/sethostname.service
[Unit]
Description=Set Hostname Workaround coreos/bugs#1272
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/sh -c "/usr/bin/hostnamectl set-hostname $(ec2-metadata -h|cut -d' ' -f2)"

[Install]
WantedBy=multi-user.target

[charltonstanley]

Unless I'm missing something, another alternative is to use ec2-metadata directly

# cat /usr/lib/systemd/system/sethostname.service
[Unit]
Description=Set Hostname Workaround coreos/bugs#1272
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/sh -c "/usr/bin/hostnamectl set-hostname $(ec2-metadata -h|cut -d' ' -f2)"

[Install]
WantedBy=multi-user.target

Wow! I had no idea that this was a utility we could use. This is pretty neat. However, to use it instead of curling IMDS directly would assume that the package ec2-utils is installed by default on any AL2 deployment. Do we know if this is the case?