Index out of bounds in brupop-agent in agent/src/apiclient.rs:237:89

[Broders]

Image I'm using: 1.6.0
Bottlerocket OS I'm using: bottlerocket-aws-k8s-1.32-x86_64-v1.39.0-0968c061

Issue:

Hello, the brupop-agent crashes with a panic in agent/src/apiclient.rs:237:89 due to an index out of bounds: the len is 1 but the index is 1 error.

thread 'main' panicked at agent/src/apiclient.rs:237:89:
index out of bounds: the len is 1 but the index is 1

I couldn´t figure out why this error occured, it worked for weeks until now. Bottlerocket is new to me.

if I enable DEBUG via values.yml I get these before the container dies:

2025-08-07T08:43:40.282815Z DEBUG agent::apiclient::api: Invoking apiclient: ["raw", "-u", "/os"]
at agent/src/apiclient.rs:267
in agent::apiclient::api::invoke_apiclient with args: ["raw", "-u", "/os"]
in agent::apiclient::api::get_os_info
in agent::agentclient::shadow_status_with_refreshed_system_matadata with shadow_error_info: ShadowErrorInfo { crash_count: 0, state_transition_failure_timestamp: None }
in agent::agentclient::initialize_metadata_shadow
in agent::agentclient::initialize_shadow_if_not_initialized
in agent::agentclient::run
2025-08-07T08:43:40.284426Z DEBUG hyper::proto::h1::io: parsed 7 headers
at /src/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/hyper-0.14.32/src/proto/h1/io.rs:211
2025-08-07T08:43:40.284439Z DEBUG hyper::proto::h1::conn: incoming body is chunked encoding
at /src/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/hyper-0.14.32/src/proto/h1/conn.rs:224
thread 'main' panicked at agent/src/apiclient.rs:237:89:
index out of bounds: the len is 1 but the index is 1
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace

I aktivated rust_backtrace

stack backtrace:
0: rust_begin_unwind
1: core::panicking::panic_fmt
2: core::panicking::panic_bounds_check
3: agent::apiclient::api::extract_status_code_from_error
4: <tokio_retry::future::RetryIf<I,A,C> as core::future::future::Future>::poll
5: agent::apiclient::api::invoke_apiclient::{{closure}}::{{closure}}
6: <tracing::instrument::Instrumented as core::future::future::Future>::poll
7: agent::apiclient::api::invoke_apiclient::{{closure}}
8: agent::apiclient::api::get_os_info::{{closure}}::{{closure}}
9: <tracing::instrument::Instrumented as core::future::future::Future>::poll
10: agent::apiclient::get_os_info::{{closure}}
11: agent::agentclient::BrupopAgent::shadow_status_with_refreshed_system_matadata::{{closure}}::{{closure}}
12: <tracing::instrument::Instrumented as core::future::future::Future>::poll
13: agent::agentclient::BrupopAgent::shadow_status_with_refreshed_system_matadata::{{closure}}
14: <tracing::instrument::Instrumented as core::future::future::Future>::poll
15: <tokio_retry::future::RetryIf<I,A,C> as core::future::future::Future>::poll
16: agent::agentclient::BrupopAgent::initialize_shadow_if_not_initialized::{{closure}}::{{closure}}
17: <tracing::instrument::Instrumented as core::future::future::Future>::poll
18: agent::agentclient::BrupopAgent::run::{{closure}}::{{closure}}::{{closure}}
19: <tracing::instrument::Instrumented as core::future::future::Future>::poll
20: <core::future::poll_fn::PollFn as core::future::future::Future>::poll
21: tokio::runtime::park::CachedParkThread::block_on
22: tokio::runtime::context::runtime::enter_runtime
23: tokio::runtime::runtime::Runtime::block_on
24: agent::main
note: Some details are omitted, run with RUST_BACKTRACE=full for a verbose backtrace.

I tried:

• redeployment via ArgoCD

[jpculp]

@Broders, thank you for reaching out! Apologies for the delayed response. It looks like somehow you managed to trigger an error that didn't contain a Status code. That might have been a bold assumption on our part that a Status would always exist. We'd like to try to reproduce this on our end. I see you were running bottlerocket-aws-k8s-1.32-x86_64-v1.39.0-0968c061. Did you customize your user data at all or are you more or less running default settings?

[MPritsch]

@jpculp Thanks for your response, I'm a colleague of broders.
Our EKS Nodes are created by the terraform-aws-modules/eks/aws module. I'm not sure if they specify userData explicitely, but we use following options:

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 20.26"

  # EKS Managed Node Group(s)
  eks_managed_node_group_defaults = {
    platform                        = "bottlerocket"
    ami_type                        = "BOTTLEROCKET_x86_64"
    use_name_prefix                 = true
    iam_role_use_name_prefix        = false
    launch_template_use_name_prefix = false
    iam_role_additional_policies = {
      "ssm" = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
    }
  }

  eks_managed_node_groups = {
    "${local.cluster_name}-cluster-worker" = {
      instance_types = ["m6a.xlarge"]
      min_size       = 0
      max_size       = local.private_workers
      desired_size   = local.private_workers
      labels = {
        subnet                                       = "private"
        "bottlerocket.aws/updater-interface-version" = "2.0.0"
      }
      vpc_security_group_ids = [
        ...
      ]
      # vm.max_map_count is needed by elasticsearch
      bootstrap_extra_args = <<-EOT
[settings.kernel.sysctl]
"vm.max_map_count" = "262144"
EOT
    }

    "${local.cluster_name}-public-worker" = {
      instance_types = ["t3a.large"]
      min_size       = 0
      max_size       = local.public_workers
      desired_size   = local.public_workers
      labels = {
        subnet                                       = "public"
        "bottlerocket.aws/updater-interface-version" = "2.0.0"
      }

      block_device_mappings = {
        xvda = {
          device_name = "/dev/xvdb"
          ebs = {
            volume_size           = 40
            volume_type           = "gp3"
            iops                  = 3000
            throughput            = 125
            encrypted             = true
            delete_on_termination = true
          }
        }
      }
      vpc_security_group_ids = [
      	...
      ]
      # ebs volume is in av eu-central-1b
      subnet_ids = [data.aws_subnet.public-b.id]
      # user.max_user_namespaces is needed by buildkitd
      bootstrap_extra_args = <<-EOT
[settings.kernel.sysctl]
"user.max_user_namespaces" = "16384"
EOT
    }
  }
  ...
}

[MPritsch]

I just checked on the instance, we do not use custom user-data:

$ apiclient get settings.user-data
{}

[MPritsch]

@jpculp We solved the particular Error of this Thread. Turns out, since we build all our own images, we used the wrong user. The User of the container needs to be "root" to access the apiclient cli from bottlerocket. But we did something like this:

ARG VERSION
FROM public.ecr.aws/bottlerocket/bottlerocket-update-operator:$VERSION
USER 65532:65532

This is 100% an error on us. However the error message broders posted did not help identify this. I only found this by looking in the brupop code, discovering "apiclient" calls and going onto the machine myself. Only then I got the same errors when using the default user, since he is not permitted to use OS commands

brupop now works on public nodes. We still looking up why it is not working on private nodes, but I think it might be related to missing VPC-Endpoints

[maherthomsi]

Hey @MPritsch, glad to hear you were able to resolve your issue!

As for the second part of your message, the main reason you aren't able to utilize BRUPOP on your private nodes would be because your nodes don't have access to our https://updates.bottlerocket.aws endpoint. In order to resolve this, please take a look at our README.

On your private nodes you would have to add internet access to https://updates.bottlerocket.aws by doing the following:

1. Create Internet Gateway and attach it to your VPC
2. Create a public subnet in one availability zone with appropriate CIDR block
3. Allocate an Elastic IP for the NAT Gateway
4. Create NAT Gateway in the public subnet using the Elastic IP
5. Create public route table with route to Internet Gateway (0.0.0.0/0 → IGW)
6. Associate public subnet with the public route table
7. Wait for NAT Gateway to become available
8. Add NAT Gateway routes to all private subnet route tables (0.0.0.0/0 → NAT Gateway)

Verification

• Connect to your private Bottlerocket instance via SSM
• Run apiclient update check - should now show available updates (direct API call that BRUPOP will be doing)
• Your nodes remain in private subnets but can reach the internet for updates

[MPritsch]

@maherthomsi Hey, thanks for the response. Regarding the fix: We wanted to use this tool for nodes in public and private networks. The private networks should not have access to the internet, including NAT-Gateways. It is a security concern for our customer.

There is a solution mentioned in the README, by providing a TUF-Repo Proxy which proxies the metadata required by brupop. This requires more implementation time than we have available at our customer, which means we will instead update our bottlerocket-instances manually or by using terraform. Here is the issue that would address implementing a proxy in this repo, rather than requiring a DIY implementation: #337

The private network issue is off topic to the original problem of this issue. The main issue was the confusing error message when using a user with insufficient permissions

[maherthomsi]

Hi @MPritsch,

The error message that was propagated when accessing apiclient with a user that has insufficient permissions looks to be a bug that occurs when triggering an error that does not contain a status code. We will be providing a fix for the index out of bounds: the len is 1 but the index is 1 exception.

I will follow up with the private network issue in #337.