How to trouble shoot image

[sujit-jadhav]

I have created Rocky 10 image, and it is created successfully, but it is not booting on the bare metal system.

[root@demo ~]# s3cmd ls -Hr s3://boot-images | grep compute/base
2025-08-18 11:19 909M s3://boot-images/compute/base/rocky10.0-compute-base-rocky10
2025-08-18 11:19 68M s3://boot-images/efi-images/compute/base/initramfs-6.12.0-55.25.1.el10_0.x86_64.img
2025-08-18 11:19 15M s3://boot-images/efi-images/compute/base/vmlinuz-6.12.0-55.25.1.el10_0.x86_64
[root@demo ~]#

[root@demo boot]# cat boot-compute.yaml
kernel: 'http://172.16.0.254:9000/boot-images/efi-images/compute/base/vmlinuz-6.12.0-55.25.1.el10_0.x86_64'
initrd: 'http://172.16.0.254:9000/boot-images/efi-images/compute/base/initramfs-6.12.0-55.25.1.el10_0.x86_64.img'
params: 'nomodeset ro root=live:http://172.16.0.254:9000/boot-images/compute/base/rocky10.0-compute-base-rocky10 ip=dhcp overlayroot=tmpfs overlayroot_cfgdisk=disabled apparmor=0 selinux=0 console=ttyS0,115200 ip6=off cloud-init=enabled ds=nocloud-net;s=http://172.16.0.254:8081/cloud-init'
macs:

• c4:cb:e1:cb:25:0e
• d0:94:66:90:cb:5e
  [root@demo boot]#

[root@demo ~]# ochami bss dumpstate | jq
{
"Components": [
{
"Enabled": true,
"EndpointEnabled": true,
"FQDN": "",
"ID": "x1000c1s7b0n0",
"MAC": [
"c4:cb:e1:cb:25:0e",
"c4:cb:e1:cb:25:0f",
"02:00:00:91:31:b3",
"c4:cb:e1:cb:25:0e"
],
"NID": 1,
"Role": "Compute",
"Type": "Node"
},
{
"Enabled": true,
"EndpointEnabled": true,
"FQDN": "",
"ID": "x1000c2s7b0n0",
"MAC": [
"d0:94:66:90:cb:5e",
"d0:94:66:90:cb:5e"
],
"NID": 2,
"Role": "Compute",
"Type": "Node"
}
],
"Params": [
{
"cloud-init": {
"meta-data": null,
"phone-home": {
"fqdn": "",
"hostname": "",
"instance_id": "",
"pub_key_dsa": "",
"pub_key_ecdsa": "",
"pub_key_rsa": ""
},
"user-data": null
},
"initrd": "http://172.16.0.254:9000/boot-images/efi-images/compute/base/initramfs-6.12.0-55.25.1.el10_0.x86_64.img",
"kernel": "http://172.16.0.254:9000/boot-images/efi-images/compute/base/vmlinuz-6.12.0-55.25.1.el10_0.x86_64",
"macs": [
"d0:94:66:90:cb:5e",
"c4:cb:e1:cb:25:0e"
],
"params": "nomodeset ro root=live:http://172.16.0.254:9000/boot-images/compute/base/rocky10.0-compute-base-rocky10 ip=dhcp overlayroot=tmpfs overlayroot_cfgdisk=disabled apparmor=0 selinux=0 console=ttyS0,115200 ip6=off cloud-init=enabled ds=nocloud-net;s=http://172.16.0.254:8081/cloud-init"
}
]
}
[root@demo ~]#

Booting starts but it gets stuck,
Please check the screen shot.

[alexlovelltroy]

There's good news and bad news here. I'll see what I can do to point you in the right direction.

I'm not familiar with the console you've got there, but if the tux images are coming from the kernel, you've got good news!

DHCP and BSS are working and you're getting a valid kernel over http. And, it is even loading and starting to boot.

The bad news here is that you've gotten past the OpenCHAMI and image-builder bits and are now troubleshooting kernel parameters and image contents which are both fully under your control.

What options do you have?

1. You can check the contents of your system image by loading it as a container and looking around. docker run -it --rm <image-name> /bin/bash
2. You can try different kernel parameters to activate the serial console differently
3. You can try a different initrd that includes troubleshooting tools instead of the RHEL compiled version. There are even initrd options that never try to mount the rootfs.

On the other hand, if the console screenshot doesn't represent the kernel from BSS, you may need to back up and review the logs from bss/coredhcp to ensure that DHCP is working correctly and that your node is getting a valid PXE script. Further, you can look at the transfer logs on your http server to ensure that the node is downloading the kernel/initrd/rootfs.

[sujit-jadhav]

Is there a way in OpenCHAMI where we can get the status of the node, whether they are booted successfully or booting or failed?

[alexlovelltroy]

The OpenCHAMI logs can tell you a lot about the state of the node. In a normal boot, you can find the following things in container logs:

1. The inital DHCP request/response and TFTP of the iPXE binary are all part of the coredhcp logs
2. Once the iPXE binary is loaded, it requests a boot script from bss which logs the request as well
3. The boot script references three URLs. In your example they are all http urls. These are all served by whatever http daemon you're using. If it is minio, as we use in the tutorial, you can get the access log(s) from that container.
4. Once a node has booted, if it is using cloud-init, the cloud-init container will log each of those requests as well.
5. Finally, if you configure phone-home in your cloud-init configuration, the node will reach back out to the cloud-init server to announce that it is complete. This will also be in the cloud-init access logs.

Because OpenCHAMI is agnostic of node operating system, there is no daemon or service introduced by OpenCHAMI in the node image that can be used for detailed troubleshooting of the booted node. You are free to add your own node-health-checks, ansible plays, or slurm status checks which will allow you to have a deeper understanding of your node through the workload manager.