BRUPOP - Agent, APIservers & Controller all working but no updates

[jimmy-ceridian]

Hi, I'm new to BRUPOP and i have installed the BRUPOP in my cluster and below is my information. However, after installed and configured the Scheduler CRON, nothing is happening to my Nodes. All my Nodes have the label of "UPDATER-INTERFACE-VERSION=2.0.0" and allow Egress of ALL TRAFFIC to 0.0.0.0/0. Can anyone help as I have no clue of what have gone wrong with my setup. Thanks a lot.

@yeazelm here's my new thread and required logs. Thanks

[jimmy-ceridian]

Hi, anybody can help to shed some light? Thanks

[yeazelm]

Hello @jimmy-ceridian, I see from your agent logs that at the time, you were on 1.42.0 and 1.42.0 was detected as the latest version. Which variant are you using, do you have any special configuration in your nodes for updates? If you log into one of the hosts and run apiclient update apply --check does it fetch a new version? From what I'm seeing in the logs, brupop is being told by the node its on the latest version.

[yeazelm]

It looks like our waves might be in an odd spot for 1.43.0 where they were set too far out into the future. If your node isn't on an early enough seed, you might not be getting 1.44.0. For example, k8s 1.29 has waves of for 1.43.0:

{
  "variant": "aws-k8s-1.29",
  "arch": "x86_64",
  "version": "1.43.0",
  "max_version": "1.44.0",
  "waves": {
    "0": "2025-08-28T18:00:00Z",
    "20": "2025-08-28T21:00:00Z",
    "102": "2025-08-29T17:00:00Z",
    "307": "2025-08-30T17:00:00Z",
    "819": "2025-09-01T17:00:00Z",
    "1228": "2025-09-02T17:00:00Z",
    "1843": "2025-09-03T17:00:00Z"
  },
  "images": {
    "boot": "bottlerocket-aws-k8s-1.29-x86_64-1.43.0-c71bcbe9-boot.ext4.lz4",
    "root": "bottlerocket-aws-k8s-1.29-x86_64-1.43.0-c71bcbe9-root.ext4.lz4",
    "hash": "bottlerocket-aws-k8s-1.29-x86_64-1.43.0-c71bcbe9-root.verity.lz4"
  }
}

and 1.44.0 has waves of:

{
  "variant": "aws-k8s-1.29",
  "arch": "x86_64",
  "version": "1.44.0",
  "max_version": "1.44.0",
  "waves": {
    "0": "2025-08-11T18:00:00Z",
    "20": "2025-08-11T21:00:00Z",
    "102": "2025-08-12T17:00:00Z",
    "307": "2025-08-13T17:00:00Z",
    "819": "2025-08-15T17:00:00Z",
    "1228": "2025-08-16T17:00:00Z",
    "1843": "2025-08-17T17:00:00Z"
  },
  "images": {
    "boot": "bottlerocket-aws-k8s-1.29-x86_64-1.44.0-244cd3a5-boot.ext4.lz4",
    "root": "bottlerocket-aws-k8s-1.29-x86_64-1.44.0-244cd3a5-root.ext4.lz4",
    "hash": "bottlerocket-aws-k8s-1.29-x86_64-1.44.0-244cd3a5-root.verity.lz4"
  }
}

My nodes happened to have early enough seeds to take the update, but your nodes might just now be getting the update.

You can confirm your seed with apiclient get settings.updates.seed, this will determine which wave your node is in. You can read more about this on bottlerocket-os/bottlerocket#3755 for how this works.

[jimmy-ceridian]

Hello @jimmy-ceridian, I see from your agent logs that at the time, you were on 1.42.0 and 1.42.0 was detected as the latest version. Which variant are you using, do you have any special configuration in your nodes for updates? If you log into one of the hosts and run apiclient update apply --check does it fetch a new version? From what I'm seeing in the logs, brupop is being told by the node its on the latest version.

Hi @yeazelm, thanks for looking into my issue. I do not have any special configuration in my BottleRocket as I also restricted the access into BottleRocket instances.

[jimmy-ceridian]

Is this error causing anything?
"The bottlerocket-update-operator is not aware of any BottlerocketShadow objects. Is the bottlerocket-shadow CRD installed?"

[yeazelm]

Yes, that could be the problem. It seems strange to get that error, but here is the full error in the code:

bottlerocket-update-operator/controller/src/controller.rs

Line 284 in c0d2f62

"Nothing to do: The bottlerocket-update-operator is not aware of any BottlerocketShadow objects. \

It should be seeing them if the agents are running.

[jimmy-ceridian]

@yeazelm thanks for your reply. So, if no BRS (if self.all_brss().is_empty()) only this error will be threw but funny thing is my BRS all are running. I reckon below shows that BRS is running in all my nodes but in Idle condition. How or Where should I troubleshoot from here? Thanks

[yeazelm]

It looks like your nodes have updated to 1.44.0. There is no new version available so it looks like its working now.

[jimmy-ceridian]

yeah, i did a manual update few days ago and hoping it will catch the next update wave. let's see how it works and will update again here. thanks @yeazelm

[jimmy-ceridian]

@yeazelm sorry to disturb. BottleRocket still not updating to 1.45, really got no idea what is wrong as I'm too new to K8s.

[yeazelm]

Hello @jimmy-ceridian, udates are rolled out in waves, the waves have just begun for 1.45.0, the nodes will update over the next few days as they hit "their wave".

The waves for 1.45.0 are set for this week:

cat manifest.json| jq '.updates | .[] | select(.version=="1.45.0")'
{
  "variant": "aws-k8s-1.32",
  "arch": "x86_64",
  "version": "1.45.0",
  "max_version": "1.45.0",
  "waves": {
    "0": "2025-08-25T18:00:00Z",
    "20": "2025-08-25T21:00:00Z",
    "102": "2025-08-26T17:00:00Z",
    "307": "2025-08-27T17:00:00Z",
    "819": "2025-08-29T17:00:00Z",
    "1228": "2025-08-30T17:00:00Z",
    "1843": "2025-08-31T17:00:00Z"
  },
  "images": {
    "boot": "bottlerocket-aws-k8s-1.32-x86_64-1.45.0-2f4223e5-boot.ext4.lz4",
    "root": "bottlerocket-aws-k8s-1.32-x86_64-1.45.0-2f4223e5-root.ext4.lz4",
    "hash": "bottlerocket-aws-k8s-1.32-x86_64-1.45.0-2f4223e5-root.verity.lz4"
  }
}

[jimmy-ceridian]

Hi @yeazelm thanks for your info. will wait and see if mine got updated. thanks