SSL issues with the System vm running on Vmware80u3 and Cloudstack 4.20.1

[kiranchavala]

problem

Currently, the systemvms using the template (https://download.cloudstack.org/systemvm/4.20/systemvmtemplate-4.20.1-x86_64-vmware.ova ) and running on vmware 8u03 and Cloudstack 4.20.1 are not connecting to the management server, the agent state is not up

versions

Cloudstack: 4.20.1
Vmware: 8u03
systemvm template :
https://download.cloudstack.org/systemvm/4.20/systemvmtemplate-4.20.1-x86_64-vmware.ova

The steps to reproduce the bug

1. Create a clousstack (4.20.1) env with vmware 8u03

2. Observe the systemvm state , the agent state will not be up

3. Check the logs of the systemvm

logs

2025-06-23T06:50:39,496 INFO  [utils.nio.NioClient] (Agent-Handler-2:[]) Connected to 10.0.35.27:8250
2025-06-23T06:50:39,497 INFO  [utils.nio.Link] (Agent-Handler-2:[]) Conf file found: /usr/local/cloud/systemvm/conf/agent.properties
2025-06-23T06:50:39,754 ERROR [utils.nio.Link] (Agent-Handler-2:[]) SSL error caught during wrap data: No trusted certificate found, for local address=/10.0.43.213:44648, remote address=/10.0.35.27:8250.
2025-06-23T06:50:39,757 INFO  [utils.nio.NioClient] (Agent-Handler-2:[]) SSL: Handshake done
2025-06-23T06:50:39,805 INFO  [cloud.agent.Agent] (Agent-Handler-2:[]) Lost connection to host: 10.0.35.27. Attempting reconnection while we still have 0 commands in progress.
2025-06-23T06:50:39,810 INFO  [utils.nio.NioClient] (Agent-Handler-2:[]) NioClient connection closed
2025-06-23T06:50:39,813 ERROR [utils.nio.Link] (Agent-Handler-2:[]) SSL error caught during wrap data: No trusted certificate found, for local address=/10.0.43.213:44640, remote address=/10.0.35.27:8250.
2025-06-23T06:50:39,814 INFO  [utils.nio.NioClient] (Agent-Handler-2:[]) SSL: Handshake done
2025-06-23T06:50:39,822 WARN  [cloud.agent.Agent] (Agent-Handler-1:[]) Unable to send request to /10.0.35.27:8250 due to 'null', request: null
2025-06-23T06:50:39,809 ERROR [utils.nio.NioClient] (Agent-Handler-2:[]) IOException while connecting to 10.0.35.27:8250 java.nio.channels.ClosedChannelException
        at java.base/sun.nio.ch.SocketChannelImpl.ensureOpenAndConnected(SocketChannelImpl.java:215)
        at java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:403)
        at com.cloud.utils.nio.Link.doHandshakeUnwrap(Link.java:487)
        at com.cloud.utils.nio.Link.doHandshake(Link.java:627)
        at com.cloud.utils.nio.NioClient.init(NioClient.java:74)
        at com.cloud.utils.nio.NioConnection.start(NioConnection.java:112)
        at com.cloud.agent.Agent.reconnect(Agent.java:655)
        at com.cloud.agent.Agent$ServerHandler.doTask(Agent.java:1233)
        at com.cloud.utils.nio.Task.call(Task.java:83)
        at com.cloud.utils.nio.Task.call(Task.java:29)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)

2025-06-23T06:50:39,851 ERROR [utils.nio.NioClient] (Agent-Handler-2:[]) Unable to initialize the threads. java.nio.channels.ClosedChannelException

What to do about it?

There should be no ssl related errors when using the systemvm template

https://download.cloudstack.org/systemvm/4.20/systemvmtemplate-4.20.1-x86_64-vmware.ova

As a workaround, execute the following command on the systemvm's to reimport cloud.ca.crt into cloud.jks  -trustcacerts is removed from the keytool command, so  cacerts will not be checked when import the ca cert

KS_FILE=/usr/local/cloud/systemvm/conf/cloud.jks KS_PASS=$(grep keystore.passphrase /usr/local/cloud/systemvm/conf/agent.properties |cut -d "=" -f2) keytool -import -noprompt -storepass "$KS_PASS" -alias "cloudca.1" -file "/usr/local/cloud/systemvm/conf/cloud.ca.crt" -keystore "$KS_FILE"

[weizhouapache]

@sureshanaparti @kiranchavala @vladimirpetrov

I checked on one of the testing environments where CPVM is running well and SSVM is not.

I destroyed the SSVM multiple times, sometimes the new SSVM is good, sometimes not.

the environment has two primary storage pools, I checked the pool_id of the volumes of each system vm

when the volume is allocated on pool with id=2, SSVM worked.
if the volume is on pool id=1, Agent is no Up

I think there is still an issue when copy template to primary storage.

[leduyquy]

Hi @weizhouapache . I have same issue when upgrade Cloudstack from 4.20.0 to 4.20.1. System VM running but agent state not UP

2025-06-25T08:50:30,394 ERROR [utils.nio.Link] (Agent-Handler-2:[]) SSL error caught during wrap data: No trusted certificate found, for local address=/172.29.0.49:47538, remote address=/172.29.0.1:8250. 2025-06-25T08:50:30,409 INFO [utils.nio.NioClient] (Agent-Handler-2:[]) SSL: Handshake done 2025-06-25T08:50:30,393 ERROR [utils.nio.Link] (Agent-Handler-2:[]) SSL error caught during wrap data: No trusted certificate found, for local address=/172.29.0.49:47560, remote address=/172.29.0.1:8250. 2025-06-25T08:50:30,409 INFO [utils.nio.NioClient] (Agent-Handler-2:[]) SSL: Handshake done 2025-06-25T08:50:30,408 ERROR [utils.nio.Link] (Agent-Handler-2:[]) SSL error caught during wrap data: No trusted certificate found, for local address=/172.29.0.49:47542, remote address=/172.29.0.1:8250. 2025-06-25T08:50:30,410 INFO [utils.nio.NioClient] (Agent-Handler-2:[]) SSL: Handshake done 2025-06-25T08:50:30,400 INFO [cloud.agent.Agent] (Agent-Handler-2:[]) Attempted to re-connect to the server, but received an unexpected exception, trying again... com.cloud.utils.exception.NioConnectionException
after i did it @kiranchavala guide then agent is UP again. I running Cloudstack with vCenter 8.0.0

With KVM I don't have the above problem.

[weizhouapache]

Update:

I rebuilt the systemvm template, the error seems to be fixed with the new systemvm template.

cc @sureshanaparti @kiranchavala @vladimirpetrov @leduyquy

[DaanHoogland]

@weizhouapache , did you upload that to download.cloudstack.org?

[weizhouapache]

@weizhouapache , did you upload that to download.cloudstack.org?

no .. unless it is fully tested (not only with vmware 80u3, but also with other hypervisors and OS versions)

[weizhouapache]

Update:

I rebuilt the systemvm template, the error seems to be fixed with the new systemvm template.

cc @sureshanaparti @kiranchavala @vladimirpetrov @leduyquy

well, just checked the trillian test (still in progress), CKS is broken with the new systemvm template
need more testing