Getting Local Constellation Running on CentOS Stream 9 - Failing to Start

[thedude257954]

Issue: Can't Start Local Constellation Cluster on CentOS Stream 9 Machine

Description
Following the steps given here to setup a local constellation cluster on an INTEL(R) XEON(R) GOLD 6542Y running CentOS Stream 9

Failure occurs while running the “constellation mini up” command and when attempting to build the cluster with qemu. The failure occurs on the “constellation apply -y” command.

Environment
-Constellation version: v2.24.0

• Kubectl version: v1.30.12
• For qemu build the config was made via “constellation config generate qemu”

Troubleshooting Paths Taken

Following Local Install Troubleshooting Steps
Followed the provided steps on the local install guide that ensures iptables are configured correctly

VM Creation

Ubuntu 22.04 VM:
Followed the steps here on the VM and the Constellation cluster was setup correctly and the test workload could be applied

CentOS Stream 9 VM:
Followed the same steps on the Constellation local install guide and it failed with the same error output as seen below. This seems to be only a CentOS Stream 9 problem, since it doesn't work on both a base install of CentOS Stream 9 and a VM install of the same OS.

Virtual Networks
The user being used to setup the constellation cluster did not have access to the default virtual network that virsh creates. This was thought to be an issue but after access was given the error still occurred.

Monitored Virtual Bridge and Networks that Constellation Creates
tcpdump was used to see the traffic on the bridge and networks but will suddenly stop and soon after constellation will error out with the same failure message.

Looking into /var/log/messages
It was found that the iscsi.service was off and constellation was throwing errors while running. The service was turned on but the error was still gained after starting the service. Another error was seen from the docker container that is started during cluster creation with the message “Container failed to exit within 0s of signal 15 - using the force”. The same image was started outside of the cluster creation process and the container will not stay running after it completes its task so this error seems to make some sense.

Libvirt Missing Network Daemon Socket
A network daemon socket was thought to be causing an error in the libvirtd service. The following missing pieces were installed

• libvirt-daemon
• libvirt-daemon-driver-qemu
• libvirt-daemon-driver-network
• libvirt-daemon-config-network
• libvirt-client
• qemu-kvm dnsmasq virt-install

Once installed they were all enabled and started. This did not fix the issue and the same error was gained when the command to start constellation was executed again.

Issue Output

Mini up Failure Screenshot

An error occurred: terraform apply: terraform apply: exit status 1

Error: couldn't retrieve IP address of domain id: f00ba33c-8ec5-4ed1-b67e-244871a4c661. Please check following:
1) is the domain running properly?
2) has the network interface an IP address?
3) Networking issues on your libvirt setup?
 4) is DHCP enabled on this Domain's network?
5) if you use bridge network, the domain should have the pkg qemu-agent installed
IMPORTANT: This error is not a terraform libvirt-provider error, but an error caused by your KVM/libvirt infrastructure configuration/setup
 context deadline exceeded

  with module.node_group["worker_default"].libvirt_domain.instance_group[0],
  on modules/instance_group/main.tf line 13, in resource "libvirt_domain" "instance_group":
  13: resource "libvirt_domain" "instance_group" {


Error: couldn't retrieve IP address of domain id: eb77fe54-cae1-4418-bb11-eb1aabe068ea. Please check following:
1) is the domain running properly?
2) has the network interface an IP address?
3) Networking issues on your libvirt setup?
 4) is DHCP enabled on this Domain's network?
5) if you use bridge network, the domain should have the pkg qemu-agent installed
IMPORTANT: This error is not a terraform libvirt-provider error, but an error caused by your KVM/libvirt infrastructure configuration/setup
 context deadline exceeded

  with module.node_group["control_plane_default"].libvirt_domain.instance_group[0],
  on modules/instance_group/main.tf line 13, in resource "libvirt_domain" "instance_group":
  13: resource "libvirt_domain" "instance_group" {


Attempting to roll back.

QEMU Failure Screenshot

An error occurred: terraform apply: terraform apply: exit status 1

Error: couldn't retrieve IP address of domain id: 82f24cbd-76b7-489e-ae1f-75470abb193a. Please check following:
1) is the domain running properly?
2) has the network interface an IP address?
3) Networking issues on your libvirt setup?
 4) is DHCP enabled on this Domain's network?
5) if you use bridge network, the domain should have the pkg qemu-agent installed
IMPORTANT: This error is not a terraform libvirt-provider error, but an error caused by your KVM/libvirt infrastructure configuration/setup
 context deadline exceeded

  with module.node_group["control_plane_default"].libvirt_domain.instance_group[1],
  on modules/instance_group/main.tf line 13, in resource "libvirt_domain" "instance_group":
  13: resource "libvirt_domain" "instance_group" {


Error: couldn't retrieve IP address of domain id: 2f07931e-4138-4dfa-b03d-984f3b17528c. Please check following:
1) is the domain running properly?
2) has the network interface an IP address?
3) Networking issues on your libvirt setup?
 4) is DHCP enabled on this Domain's network?
5) if you use bridge network, the domain should have the pkg qemu-agent installed
IMPORTANT: This error is not a terraform libvirt-provider error, but an error caused by your KVM/libvirt infrastructure configuration/setup
 context deadline exceeded

  with module.node_group["control_plane_default"].libvirt_domain.instance_group[2],
  on modules/instance_group/main.tf line 13, in resource "libvirt_domain" "instance_group":
  13: resource "libvirt_domain" "instance_group" {


Error: couldn't retrieve IP address of domain id: 526bc70d-d366-4fa1-8c7b-06c7ce171a51. Please check following:
1) is the domain running properly?
2) has the network interface an IP address?
3) Networking issues on your libvirt setup?
 4) is DHCP enabled on this Domain's network?
5) if you use bridge network, the domain should have the pkg qemu-agent installed
IMPORTANT: This error is not a terraform libvirt-provider error, but an error caused by your KVM/libvirt infrastructure configuration/setup
 context deadline exceeded

  with module.node_group["worker_default"].libvirt_domain.instance_group[0],
  on modules/instance_group/main.tf line 13, in resource "libvirt_domain" "instance_group":
  13: resource "libvirt_domain" "instance_group" {


Error: couldn't retrieve IP address of domain id: 84cf35c7-8f42-406a-b868-c6824b4c76c0. Please check following:
1) is the domain running properly?
2) has the network interface an IP address?
3) Networking issues on your libvirt setup?
 4) is DHCP enabled on this Domain's network?
5) if you use bridge network, the domain should have the pkg qemu-agent installed
IMPORTANT: This error is not a terraform libvirt-provider error, but an error caused by your KVM/libvirt infrastructure configuration/setup
 context deadline exceeded

  with module.node_group["control_plane_default"].libvirt_domain.instance_group[0],
  on modules/instance_group/main.tf line 13, in resource "libvirt_domain" "instance_group":
  13: resource "libvirt_domain" "instance_group" {

[msanft]

Hey there,

Unfortunately, we're not able to test on every distribution in CI, and thus only recommend Ubuntu {22,24}.04 to be used with MiniConstellation. Regarding your error, have you tried to perform the troubleshooting steps provided by the Terraform provider?

1. is the domain running properly?
2. has the network interface an IP address?
3. Networking issues on your libvirt setup?
   3. a) is DHCP enabled on this Domain's network?
4. if you use bridge network, the domain should have the pkg qemu-agent installed

Best,
Moritz