zos systems not connecting to jenkins

[sxa]

Earlier in the week it was noticed that the zos systems in the test CI were offline. Attempts to ssh to the machines were failing. While that appears to have been a temporary problem, the agents were still not connecting back in. Analysis of the longs showed the following exception:

INFO: Protocol JNLP4-connect encountered an unexpected exception
java.util.concurrent.ExecutionException: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Connection closed before acknowledgement sent
	at org.jenkinsci.remoting.util.SettableFuture.get(SettableFuture.java:223)
	at hudson.remoting.Engine.innerRun(Engine.java:799)
	at hudson.remoting.Engine.run(Engine.java:544)
Caused by: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Connection closed before acknowledgement sent
	at org.jenkinsci.remoting.protocol.impl.AckFilterLayer.onRecvClosed(AckFilterLayer.java:280)
	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:825)
	at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:155)
	at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer.access$1500(BIONetworkLayer.java:51)
	at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer$Reader.run(BIONetworkLayer.java:257)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:126)
	at hudson.remoting.Engine$1$$Lambda$8/000000006D78A0B0.run(Unknown Source)
	at java.lang.Thread.run(Thread.java:820)

[sxa]

Today I have downloaded a version of IBM Semeru Certified Edition 11 onto one of the machines and started up the agent with that - it appears to have solved the problem. It seems likely that this happened as a result of the upgrade to jenkins last week. Further analysis work will be required to do a "proper" install of java 11 and switch in my start_J11.sh which points at my extracted java 11 to be the default start.sh

[richardlau]

FWIW we've never had the ability to do "proper" installs of Java on the marist hosted z/OS VMs as we are not system admins there. I suspect the version of Java 8 that is the system Java on those VMs is really old (basically whatever was imaged on there when we were migrated from z/OS 2.2 to z/OS 2.4 VMs) -- I do not know if perhaps a later Java 8 update would have also worked.

In any case going to Java 11 (or later) would be the logical thing to do in light of #3030. Also the IBM team responsible for the z/OS port of Node.js are intending to migrate us from the marist hosted z/OS VMs to z/OS VMs in IBM Cloud (#3009) so it might not be so terrible to put in place a temporary solution on the marist VMs until the migration happens.

[github-actions]

This issue is stale because it has been open many days with no activity. It will be closed soon unless the stale label is removed or a comment is made.