Slow downloading of logs

[as51340]

Hi,

Downloading logs take significantly more time in my test than actual test running. Do you have some ideas how can I solve this problem? Logs are very important for me because of debugging...

This is the output example from jepsen.log:

2024-09-12 02:20:57,283{GMT}	INFO	[jepsen test runner] jepsen.core: Run complete, writing
2024-09-12 02:20:58,177{GMT}	INFO	[jepsen test runner] jepsen.core: Snarfing log files
2024-09-12 02:20:58,187{GMT}	INFO	[jepsen node n3] jepsen.core: downloading /opt/memgraph/memgraph.log to memgraph.log
2024-09-12 02:20:58,187{GMT}	INFO	[jepsen node n4] jepsen.core: downloading /opt/memgraph/memgraph.log to memgraph.log
2024-09-12 02:20:58,187{GMT}	INFO	[jepsen node n5] jepsen.core: downloading /opt/memgraph/memgraph.log to memgraph.log
2024-09-12 02:20:58,188{GMT}	INFO	[jepsen node n1] jepsen.core: downloading /opt/memgraph/memgraph.log to memgraph.log
2024-09-12 02:20:58,188{GMT}	INFO	[jepsen node n2] jepsen.core: downloading /opt/memgraph/memgraph.log to memgraph.log
2024-09-12 02:31:08,308{GMT}	INFO	[jepsen test runner] jepsen.core: Snarfing log files
2024-09-12 02:31:08,317{GMT}	INFO	[jepsen node n1] jepsen.core: downloading /opt/memgraph/memgraph.log to memgraph.log
2024-09-12 02:31:08,317{GMT}	INFO	[jepsen node n5] jepsen.core: downloading /opt/memgraph/memgraph.log to memgraph.log
2024-09-12 02:31:08,318{GMT}	INFO	[jepsen node n4] jepsen.core: downloading /opt/memgraph/memgraph.log to memgraph.log
2024-09-12 02:31:08,318{GMT}	INFO	[jepsen node n2] jepsen.core: downloading /opt/memgraph/memgraph.log to memgraph.log
2024-09-12 02:31:08,318{GMT}	INFO	[jepsen node n3] jepsen.core: downloading /opt/memgraph/memgraph.log to memgraph.log
2024-09-12 02:32:20,072{GMT}	INFO	[jepsen node n1] jepsen.memgraph.support: n1 Tearing down Memgraph
2024-09-12 02:32:20,072{GMT}	INFO	[jepsen node n2] jepsen.memgraph.support: n2 Tearing down Memgraph

[aphyr]

There's a variant of the SSH remote which shells out to SCP--that might be faster! Generally the default remote is OK for files up to a GB or so, but if you find you're burning a ton of time in SSHJ, try replacing the remote in your test map:

(let [test {...
            :remote (jepsen.control.scp/remote jepsen.control/ssh)}]
  ...)