🧑‍🌾 test verb list and dump connection refused

[Crola1702]

Operating System:

Ubuntu 24.04 and RHEL

ROS version or commit hash:

rolling

RMW implementation (if applicable):

All

RMW Configuration (if applicable):

No response

Client library (if applicable):

No response

'ros2 doctor --report' output

No response

Steps to reproduce issue

1. Run a build in nightly_linux_release.
2. See test regressions https://ci.ros2.org/view/nightly/job/nightly_linux_release/3442/

Expected behavior

Tests should not fail

Actual behavior

Test are failing

Additional information

Reference builds:

• https://ci.ros2.org/view/nightly/job/nightly_linux_release/3442/
• https://ci.ros2.org/job/nightly_linux_coverage/27505
• https://ci.ros2.org/job/nightly_linux-aarch64_release/3079/

Log output:

=====================================================================================================================================
FAIL: test_verb_dump.TestVerbDump.test_verb_dump[rmw_zenoh_cpp]
-------------------------------------------------------------------------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jenkins-agent/workspace/nightly_linux_release/ws/src/ros2/ros2cli/ros2param/test/test_verb_dump.py", line 158, in setUp
    services = node.get_service_names_and_types_by_node(TEST_NODE, TEST_NAMESPACE)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/xmlrpc/client.py", line 1122, in __call__
    return self.__send(self.__name, args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/xmlrpc/client.py", line 1461, in __request
    response = self.__transport.request(
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/xmlrpc/client.py", line 1166, in request
    return self.single_request(host, handler, request_body, verbose)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/xmlrpc/client.py", line 1178, in single_request
    http_conn = self.send_request(host, handler, request_body, verbose)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/xmlrpc/client.py", line 1291, in send_request
    self.send_content(connection, request_body)
  File "/usr/lib/python3.12/xmlrpc/client.py", line 1321, in send_content
    connection.endheaders(request_body)
  File "/usr/lib/python3.12/http/client.py", line 1331, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.12/http/client.py", line 1091, in _send_output
    self.send(msg)
  File "/usr/lib/python3.12/http/client.py", line 1035, in send
    self.connect()
  File "/usr/lib/python3.12/http/client.py", line 1001, in connect
    self.sock = self._create_connection(
                ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/socket.py", line 852, in create_connection
    raise exceptions[0]
  File "/usr/lib/python3.12/socket.py", line 837, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused


=====================================================================================================================================
FAIL: test_verb_dump.TestVerbDump.test_verb_dump_invalid_node[rmw_zenoh_cpp]
-------------------------------------------------------------------------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jenkins-agent/workspace/nightly_linux_release/ws/src/ros2/ros2cli/ros2param/test/test_verb_dump.py", line 174, in setUp
    self.fail(f'CLI daemon failed to find test node after {TEST_TIMEOUT} seconds')
AssertionError: CLI daemon failed to find test node after 20.0 seconds


=====================================================================================================================================
FAIL: test_verb_dump.TestVerbDump.test_verb_dump[rmw_connextdds]
-------------------------------------------------------------------------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jenkins-agent/workspace/nightly_linux_release/ws/src/ros2/ros2cli/ros2param/test/test_verb_dump.py", line 174, in setUp
    self.fail(f'CLI daemon failed to find test node after {TEST_TIMEOUT} seconds')
AssertionError: CLI daemon failed to find test node after 20.0 seconds


=====================================================================================================================================
FAIL: test_verb_dump.TestVerbDump.test_verb_dump_invalid_node[rmw_connextdds]
-------------------------------------------------------------------------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jenkins-agent/workspace/nightly_linux_release/ws/src/ros2/ros2cli/ros2param/test/test_verb_dump.py", line 174, in setUp
    self.fail(f'CLI daemon failed to find test node after {TEST_TIMEOUT} seconds')
AssertionError: CLI daemon failed to find test node after 20.0 seconds

This test has a flaky behavior:

job_name	last_fail	first_fail	build_count	failure_count	failure_percentage
nightly_linux_humble_coverage	2025-04-23	2025-04-17	13	7	53.85
nightly_linux_coverage	2025-04-23	2025-04-17	13	4	30.77
nightly_linux-rhel_release	2025-04-23	2025-04-08	14	9	64.29
nightly_linux-rhel_debug	2025-04-23	2025-04-09	15	11	73.33
nightly_linux-aarch64_repeated	2025-04-23	2025-04-15	16	7	43.75
nightly_linux-aarch64_release	2025-04-23	2025-04-16	12	6	50.0
nightly_linux_debug	2025-04-22	2025-04-15	14	6	42.86
nightly_linux-rhel_repeated	2025-04-22	2025-04-08	14	9	64.29
Rci__nightly-connext_ubuntu_noble_amd64	2025-04-22	2025-04-18	5	5	100.0
nightly_linux_release	2025-04-20	2025-04-15	12	6	50.0
nightly_linux-aarch64_debug	2025-04-20	2025-04-17	13	3	23.08
nightly_linux_repeated	2025-04-19	2025-04-18	12	2	16.67

[fujitatomoya]

I cannot reproduce this issue with rolling source build / ubuntu noble. I do not get to the root cause of this issue, so the following is just a note.

https://ci.ros2.org/view/nightly/job/nightly_linux_release/3442/testReport/ros2param.ros2param.test/test_verb_list/test_verb_list/ (https://ci.ros2.org/view/nightly/job/nightly_linux_release/3442/testReport/junit/ros2param.ros2param.test/test_verb_dump/test_verb_dump/ looks almost same.)

• This test starts the parameter node test fixture and restart the ros2daemon before each test.
• This test uses NodeStrategy, it is likely that it uses DaemonNode since ros2daemon is restarted in the test. But if something happens to ros2daemon, it falls back to DirectNode. (NodeStrategy checks corresponding XMLRPC connection to the daemon every time by using getattr.)
• ConnectionRefusedError: [Errno 111] Connection refused exception indicates that something happened to the daemon when it is calling get_service_names_and_types_by_node, but test does not catch the exception. I think we can catch this ConnectionRefusedError in the setUp, that gives the retry to fall back to use DirectNode to keep the test running.
• timeout of setUp indicates that maybe the parameter node test fixture is not running anymore?

[fujitatomoya]

This test uses NodeStrategy, it is likely that it uses DaemonNode since ros2daemon is restarted in the test. But if something happens to ros2daemon, it falls back to DirectNode. (NodeStrategy checks corresponding XMLRPC connection to the daemon every time by using getattr.)

#1014 only mitigates the failure, but addresses the root cause of this issue, which is test fixture parameter node is not started as expected.

@Crola1702 can we see if the nightly builds becomes more stable for a bit?

[Crola1702]

@fujitatomoya the flaky ratio is almost the same in the last 2 weeks:

job_name	last_fail	first_fail	build_count	failure_count	failure_percentage
nightly_linux_release	2025-05-19	2025-05-06	15	6	40.0
nightly_linux_coverage	2025-05-19	2025-05-07	11	7	63.64
nightly_linux-rhel_release	2025-05-19	2025-05-04	15	6	40.0
nightly_linux-aarch64_repeated	2025-05-19	2025-05-04	13	13	100.0
nightly_linux-aarch64_release	2025-05-19	2025-05-06	15	8	53.33
Rci__nightly-connext_ubuntu_noble_amd64	2025-05-19	2025-05-04	10	10	100.0
nightly_linux_debug	2025-05-18	2025-05-05	15	9	60.0
nightly_linux-rhel_debug	2025-05-18	2025-05-07	14	8	57.14
Rci__nightly-release_ubuntu_noble_amd64	2025-05-18	2025-05-06	5	5	100.0
nightly_linux_repeated	2025-05-17	2025-05-04	12	6	50.0
Rci__nightly-debug_ubuntu_noble_amd64	2025-05-17	2025-05-06	4	4	100.0
Kci__nightly-connext_ubuntu_noble_amd64	2025-05-16	2025-05-13	2	2	100.0
nightly_linux-rhel_repeated	2025-05-14	2025-05-04	13	6	46.15
nightly_linux-aarch64_debug	2025-05-14	2025-05-10	15	3	20.0
Kci__nightly-debug_ubuntu_noble_amd64	2025-05-13	2025-05-13	1	1	100.0

[Crola1702]

So, this is happening through all linux:

Reference builds:

• https://ci.ros2.org/job/nightly_linux_release/3496
• https://ci.ros2.org/job/nightly_linux_debug/3446
• https://ci.ros2.org/job/nightly_linux-rhel_repeated/2306
• https://ci.ros2.org/job/nightly_linux-rhel_debug/2310
• https://ci.ros2.org/job/nightly_linux-aarch64_repeated/3181
• https://ci.ros2.org/job/nightly_linux_repeated/3859/
• https://ci.ros2.org/job/nightly_linux-rhel_release/2289/
• https://ci.ros2.org/job/nightly_linux-aarch64_release/3132/
• https://ci.ros2.org/job/nightly_linux-aarch64_debug/3158/
• https://build.ros2.org/job/Rci__nightly-connext_ubuntu_noble_amd64/416/
• https://build.ros2.org/job/Kci__nightly-debug_ubuntu_noble_amd64/8/
• https://build.ros2.org/job/Rci__nightly-debug_ubuntu_noble_amd64/407/
• https://build.ros2.org/job/Kci__nightly-connext_ubuntu_noble_amd64/6/
• https://build.ros2.org/job/Kci__nightly-release_ubuntu_noble_amd64/5/
• https://build.ros2.org/job/Rci__nightly-release_ubuntu_noble_amd64/416/

[fujitatomoya]

these references include many unrelated failures instead of test_verb_list and test_verb_dump.

here i picked up several related failures from test_verb_list and test_verb_dump.

https://ci.ros2.org/job/nightly_linux_release/3496/testReport/junit/ros2param.ros2param.test/test_verb_dump/test_verb_dump/

can not find the test node that is supposed to be running for the test.

Traceback (most recent call last):
  File "/home/jenkins-agent/workspace/nightly_linux_release/ws/src/ros2/ros2cli/ros2param/test/test_verb_dump.py", line 176, in setUp
    self.fail(f'CLI daemon failed to find test node after {TEST_TIMEOUT} seconds')
AssertionError: CLI daemon failed to find test node after 20.0 seconds

https://ci.ros2.org/job/nightly_linux-rhel_repeated/2306/testReport/junit/ros2param.ros2param.test/test_verb_dump/test_verb_dump/
https://ci.ros2.org/job/nightly_linux-rhel_repeated/2306/testReport/junit/ros2param.ros2param.test/test_verb_list/test_verb_list/

Exception: Launch stopped before the active tests finished.

https://ci.ros2.org/job/nightly_linux-aarch64_repeated/3181/testReport/junit/ros2param.ros2param.test/test_verb_load/test_verb_load/

dump command failed to execute.

Traceback (most recent call last):
  File "/home/jenkins-agent/workspace/nightly_linux-aarch64_repeated/ws/src/ros2/ros2cli/ros2param/test/test_verb_load.py", line 374, in test_verb_load_wildcard
    assert param_dump_command.exit_code == launch_testing.asserts.EXIT_OK
AssertionError: assert 1 == 0
 +  where 1 = <launch_testing.tools.process.ProcessProxy object at 0xf0ade383bf50>.exit_code
 +  and   0 = <module 'launch_testing.asserts' from '/home/jenkins-agent/workspace/nightly_linux-aarch64_repeated/ws/install/launch_testing/lib/python3.12/site-packages/launch_testing/asserts/__init__.py'>.EXIT_OK
 +    where <module 'launch_testing.asserts' from '/home/jenkins-agent/workspace/nightly_linux-aarch64_repeated/ws/install/launch_testing/lib/python3.12/site-packages/launch_testing/asserts/__init__.py'> = launch_testing.asserts

i think these failures are different from the connection issue originally reported here.

@cottsay mentioned that different RMW tests are executing in the single process space, if i am correct? this flaky tests could be related? btw, those flaky tests are recently happening all the sudden or they have been there for a long time? just want to know if there is any trigger to make those flaky tests during kilted development...

[Crola1702]

@cottsay do you think this can be solved by adding rmw_test_fixture?