Skip to content

Failed to run lein test in tutorial due to ssh pam_systemd failures #612

New issue
New issue
Open
Open
Failed to run lein test in tutorial due to ssh pam_systemd failures#612
@Wyb1406272043

Description

@Wyb1406272043
Wyb1406272043
opened on Mar 11, 2025

I had an ssh connection error while running the tutorial:

ERROR [2025-03-11 14:36:25,635] main - jepsen.cli Oh jeez, I'm sorry, Jepsen broke. Here's why:
clojure.lang.ExceptionInfo: throw+: {:dir "/", :private-key-path nil, :password "root", :username "root", :type :jepsen.control/ssh-failed, :port 22, :strict-host-key-checking false, :host "n1", :sudo nil, :dummy false, :session {:open #object[jepsen.control$session$fn__3147 0x1e74d5d8 "jepsen.control$session$fn__3147@1e74d5d8"], :close #object[jepsen.control$fn__3018$G__2947__3023 0x79f1a2e3 "jepsen.control$fn__3018$G__2947__3023@79f1a2e3"], :log? true, :name [:control "n1"], :lock #object[java.util.concurrent.locks.ReentrantReadWriteLock 0x37b94e5 "java.util.concurrent.locks.ReentrantReadWriteLock@37b94e5[Write locks = 0, Read locks = 0]"], :conn #object[clojure.lang.Atom 0x7e79d881 {:status :ready, :val #jepsen.control.SSHRemote{:session #object[com.jcraft.jsch.Session 0x428cce61 "com.jcraft.jsch.Session@428cce61"]}}]}}
Detailed information 
root@control:/jepsen/jepsen.etcdemo# lein run test
14:36:03.643 [main] INFO jepsen.cli - Test options:
 {:concurrency 5,
 :leave-db-running? false,
 :logging-json? false,
 :ssh
 {:dummy? false,
  :username "root",
  :password "root",
  :strict-host-key-checking false,
  :private-key-path nil},
 :argv ("test"),
 :nodes ["n1" "n2" "n3" "n4" "n5"],
 :test-count 1,
 :time-limit 60}

INFO [2025-03-11 14:36:03,683] jepsen test runner - jepsen.core Command line:
lein run test
INFO [2025-03-11 14:36:03,711] jepsen test runner - jepsen.core Running test:
{:remote #jepsen.control.SSHRemote{:session nil}
 :concurrency 5
 :db
 #object[jepsen.etcdemo$db$reify__331
         "0x4331be30"
         "jepsen.etcdemo$db$reify__331@4331be30"]
 :leave-db-running? false
 :name "etcd"
 :logging-json? false
 :start-time
 #object[org.joda.time.DateTime "0x23e9dd78" "2025-03-11T14:36:03.000Z"]
 :net
 #object[jepsen.net$reify__9085 "0x470c4229" "jepsen.net$reify__9085@470c4229"]
 :client
 #object[jepsen.client$reify__8924
         "0x6408be5"
         "jepsen.client$reify__8924@6408be5"]
 :barrier
 #object[java.util.concurrent.CyclicBarrier
         "0x1e57b783"
         "java.util.concurrent.CyclicBarrier@1e57b783"]
 :pure-generators true
 :ssh
 {:dummy? false,
  :username "root",
  :password "root",
  :strict-host-key-checking false,
  :private-key-path nil}
 :checker
 #object[jepsen.checker$unbridled_optimism$reify__8609
         "0x73b8ab2c"
         "jepsen.checker$unbridled_optimism$reify__8609@73b8ab2c"]
 :argv ("test")
 :nemesis
 #object[jepsen.nemesis$reify__9186
         "0x5600a278"
         "jepsen.nemesis$reify__9186@5600a278"]
 :active-histories
 #object[clojure.lang.Atom "0x12421766" {:status :ready, :val #{}}]
 :nodes ["n1" "n2" "n3" "n4" "n5"]
 :test-count 1
 :generator nil
 :os
 #object[jepsen.os.debian.Debian
         "0x23c72128"
         "jepsen.os.debian.Debian@23c72128"]
 :time-limit 60}

INFO [2025-03-11 14:36:04,794] jepsen node n2 - jepsen.os.debian n2 setting up debian
INFO [2025-03-11 14:36:04,794] jepsen node n4 - jepsen.os.debian n4 setting up debian
INFO [2025-03-11 14:36:04,794] jepsen node n3 - jepsen.os.debian n3 setting up debian
INFO [2025-03-11 14:36:04,794] jepsen node n5 - jepsen.os.debian n5 setting up debian
INFO [2025-03-11 14:36:04,794] jepsen node n1 - jepsen.os.debian n1 setting up debian
WARN [2025-03-11 14:36:24,988] jepsen node n3 - jepsen.control Encountered error with conn [:control "n3"]; reopening
com.jcraft.jsch.JSchException: channel is not opened.
        at com.jcraft.jsch.Channel.sendChannelOpen(Channel.java:765)
        at com.jcraft.jsch.Channel.connect(Channel.java:151)
        at com.jcraft.jsch.Channel.connect(Channel.java:145)
        at clj_ssh.ssh$connect_channel.invokeStatic(ssh.clj:512)
        at clj_ssh.ssh$connect_channel.invoke(ssh.clj:509)
        at clj_ssh.ssh$ssh_exec_proc.invokeStatic(ssh.clj:667)
        at clj_ssh.ssh$ssh_exec_proc.invoke(ssh.clj:646)
        at clj_ssh.ssh$ssh_exec.invokeStatic(ssh.clj:677)
        at clj_ssh.ssh$ssh_exec.invoke(ssh.clj:670)
        at clj_ssh.ssh$ssh.invokeStatic(ssh.clj:723)
        at clj_ssh.ssh$ssh.invoke(ssh.clj:699)
        at jepsen.control.SSHRemote.execute_BANG_(control.clj:331)
        at jepsen.control$ssh_STAR_$fn__3063.invoke(control.clj:172)
        at jepsen.control$ssh_STAR_.invokeStatic(control.clj:172)
        at jepsen.control$ssh_STAR_.invoke(control.clj:168)
        at jepsen.control$exec_STAR_.invokeStatic(control.clj:194)
        at jepsen.control$exec_STAR_.doInvoke(control.clj:191)
        at clojure.lang.RestFn.applyTo(RestFn.java:137)
        at clojure.core$apply.invokeStatic(core.clj:665)
        at clojure.core$apply.invoke(core.clj:660)
        at jepsen.control$exec.invokeStatic(control.clj:210)
        at jepsen.control$exec.doInvoke(control.clj:204)
        at clojure.lang.RestFn.invoke(RestFn.java:408)
        at jepsen.os.debian$setup_hostfile_BANG_.invokeStatic(debian.clj:16)
        at jepsen.os.debian$setup_hostfile_BANG_.invoke(debian.clj:13)
        at jepsen.os.debian.Debian.setup_BANG_(debian.clj:162)
        at jepsen.os$fn__2459$G__2453__2463.invoke(os.clj:4)
        at jepsen.os$fn__2459$G__2452__2468.invoke(os.clj:4)
        at clojure.core$partial$fn__5824.invoke(core.clj:2625)
        at jepsen.control$on_nodes$fn__3161.invoke(control.clj:430)
        at clojure.lang.AFn.applyToHelper(AFn.java:154)
        at clojure.lang.AFn.applyTo(AFn.java:144)
        at clojure.core$apply.invokeStatic(core.clj:665)
        at clojure.core$with_bindings_STAR_.invokeStatic(core.clj:1973)
        at clojure.core$with_bindings_STAR_.doInvoke(core.clj:1973)
        at clojure.lang.RestFn.applyTo(RestFn.java:142)
        at clojure.core$apply.invokeStatic(core.clj:669)
        at clojure.core$bound_fn_STAR_$fn__5734.doInvoke(core.clj:2003)
        at clojure.lang.RestFn.invoke(RestFn.java:408)
        at dom_top.core$real_pmap_helper$build_thread__213$fn__214.invoke(core.clj:146)
        at clojure.lang.AFn.applyToHelper(AFn.java:152)
        at clojure.lang.AFn.applyTo(AFn.java:144)
        at clojure.core$apply.invokeStatic(core.clj:665)
        at clojure.core$with_bindings_STAR_.invokeStatic(core.clj:1973)
        at clojure.core$with_bindings_STAR_.doInvoke(core.clj:1973)
        at clojure.lang.RestFn.invoke(RestFn.java:425)
        at clojure.lang.AFn.applyToHelper(AFn.java:156)
        at clojure.lang.RestFn.applyTo(RestFn.java:132)
        at clojure.core$apply.invokeStatic(core.clj:669)
        at clojure.core$bound_fn_STAR_$fn__5734.doInvoke(core.clj:2003)
        at clojure.lang.RestFn.invoke(RestFn.java:397)
        at clojure.lang.AFn.run(AFn.java:22)
        at java.base/java.lang.Thread.run(Thread.java:1583)
WARN [2025-03-11 14:36:24,988] jepsen node n1 - jepsen.control Encountered error with conn [:control "n1"]; reopening
com.jcraft.jsch.JSchException: channel is not opened.
        at com.jcraft.jsch.Channel.sendChannelOpen(Channel.java:765)
        at com.jcraft.jsch.Channel.connect(Channel.java:151)
        at com.jcraft.jsch.Channel.connect(Channel.java:145)
        at clj_ssh.ssh$connect_channel.invokeStatic(ssh.clj:512)
        at clj_ssh.ssh$connect_channel.invoke(ssh.clj:509)
        at clj_ssh.ssh$ssh_exec_proc.invokeStatic(ssh.clj:667)
        at clj_ssh.ssh$ssh_exec_proc.invoke(ssh.clj:646)
        at clj_ssh.ssh$ssh_exec.invokeStatic(ssh.clj:677)
        at clj_ssh.ssh$ssh_exec.invoke(ssh.clj:670)
        at clj_ssh.ssh$ssh.invokeStatic(ssh.clj:723)
        at clj_ssh.ssh$ssh.invoke(ssh.clj:699)
        at jepsen.control.SSHRemote.execute_BANG_(control.clj:331)
        at jepsen.control$ssh_STAR_$fn__3063.invoke(control.clj:172)
        at jepsen.control$ssh_STAR_.invokeStatic(control.clj:172)
        at jepsen.control$ssh_STAR_.invoke(control.clj:168)
        at jepsen.control$exec_STAR_.invokeStatic(control.clj:194)
        at jepsen.control$exec_STAR_.doInvoke(control.clj:191)
        at clojure.lang.RestFn.applyTo(RestFn.java:137)
        at clojure.core$apply.invokeStatic(core.clj:665)
        at clojure.core$apply.invoke(core.clj:660)
        at jepsen.control$exec.invokeStatic(control.clj:210)
        at jepsen.control$exec.doInvoke(control.clj:204)
        at clojure.lang.RestFn.invoke(RestFn.java:408)
        at jepsen.os.debian$setup_hostfile_BANG_.invokeStatic(debian.clj:16)
        at jepsen.os.debian$setup_hostfile_BANG_.invoke(debian.clj:13)
        at jepsen.os.debian.Debian.setup_BANG_(debian.clj:162)
        at jepsen.os$fn__2459$G__2453__2463.invoke(os.clj:4)
        at jepsen.os$fn__2459$G__2452__2468.invoke(os.clj:4)
        at clojure.core$partial$fn__5824.invoke(core.clj:2625)
        at jepsen.control$on_nodes$fn__3161.invoke(control.clj:430)
        at clojure.lang.AFn.applyToHelper(AFn.java:154)
        at clojure.lang.AFn.applyTo(AFn.java:144)
        at clojure.core$apply.invokeStatic(core.clj:665)
        at clojure.core$with_bindings_STAR_.invokeStatic(core.clj:1973)
        at clojure.core$with_bindings_STAR_.doInvoke(core.clj:1973)
        at clojure.lang.RestFn.applyTo(RestFn.java:142)
        at clojure.core$apply.invokeStatic(core.clj:669)
        at clojure.core$bound_fn_STAR_$fn__5734.doInvoke(core.clj:2003)
        at clojure.lang.RestFn.invoke(RestFn.java:408)
        at dom_top.core$real_pmap_helper$build_thread__213$fn__214.invoke(core.clj:146)
        at clojure.lang.AFn.applyToHelper(AFn.java:152)
        at clojure.lang.AFn.applyTo(AFn.java:144)
        at clojure.core$apply.invokeStatic(core.clj:665)
        at clojure.core$with_bindings_STAR_.invokeStatic(core.clj:1973)
        at clojure.core$with_bindings_STAR_.doInvoke(core.clj:1973)
        at clojure.lang.RestFn.invoke(RestFn.java:425)
        at clojure.lang.AFn.applyToHelper(AFn.java:156)
        at clojure.lang.RestFn.applyTo(RestFn.java:132)
        at clojure.core$apply.invokeStatic(core.clj:669)
        at clojure.core$bound_fn_STAR_$fn__5734.doInvoke(core.clj:2003)
        at clojure.lang.RestFn.invoke(RestFn.java:397)
        at clojure.lang.AFn.run(AFn.java:22)
        at java.base/java.lang.Thread.run(Thread.java:1583)
WARN [2025-03-11 14:36:24,989] jepsen node n5 - jepsen.control Encountered error with conn [:control "n5"]; reopening
com.jcraft.jsch.JSchException: channel is not opened.
        at com.jcraft.jsch.Channel.sendChannelOpen(Channel.java:765)
        at com.jcraft.jsch.Channel.connect(Channel.java:151)
        at com.jcraft.jsch.Channel.connect(Channel.java:145)
        at clj_ssh.ssh$connect_channel.invokeStatic(ssh.clj:512)
        at clj_ssh.ssh$connect_channel.invoke(ssh.clj:509)
        at clj_ssh.ssh$ssh_exec_proc.invokeStatic(ssh.clj:667)
        at clj_ssh.ssh$ssh_exec_proc.invoke(ssh.clj:646)
        at clj_ssh.ssh$ssh_exec.invokeStatic(ssh.clj:677)
        at clj_ssh.ssh$ssh_exec.invoke(ssh.clj:670)
        at clj_ssh.ssh$ssh.invokeStatic(ssh.clj:723)
        at clj_ssh.ssh$ssh.invoke(ssh.clj:699)
        at jepsen.control.SSHRemote.execute_BANG_(control.clj:331)
        at jepsen.control$ssh_STAR_$fn__3063.invoke(control.clj:172)
        at jepsen.control$ssh_STAR_.invokeStatic(control.clj:172)
        at jepsen.control$ssh_STAR_.invoke(control.clj:168)
        at jepsen.control$exec_STAR_.invokeStatic(control.clj:194)
        at jepsen.control$exec_STAR_.doInvoke(control.clj:191)
        at clojure.lang.RestFn.applyTo(RestFn.java:137)
        at clojure.core$apply.invokeStatic(core.clj:665)
        at clojure.core$apply.invoke(core.clj:660)
        at jepsen.control$exec.invokeStatic(control.clj:210)
        at jepsen.control$exec.doInvoke(control.clj:204)
        at clojure.lang.RestFn.invoke(RestFn.java:408)
        at jepsen.os.debian$setup_hostfile_BANG_.invokeStatic(debian.clj:16)
        at jepsen.os.debian$setup_hostfile_BANG_.invoke(debian.clj:13)
        at jepsen.os.debian.Debian.setup_BANG_(debian.clj:162)
        at jepsen.os$fn__2459$G__2453__2463.invoke(os.clj:4)
        at jepsen.os$fn__2459$G__2452__2468.invoke(os.clj:4)
        at clojure.core$partial$fn__5824.invoke(core.clj:2625)
        at jepsen.control$on_nodes$fn__3161.invoke(control.clj:430)
        at clojure.lang.AFn.applyToHelper(AFn.java:154)
        at clojure.lang.AFn.applyTo(AFn.java:144)
        at clojure.core$apply.invokeStatic(core.clj:665)
        at clojure.core$with_bindings_STAR_.invokeStatic(core.clj:1973)
        at clojure.core$with_bindings_STAR_.doInvoke(core.clj:1973)
        at clojure.lang.RestFn.applyTo(RestFn.java:142)
        at clojure.core$apply.invokeStatic(core.clj:669)
        at clojure.core$bound_fn_STAR_$fn__5734.doInvoke(core.clj:2003)
        at clojure.lang.RestFn.invoke(RestFn.java:408)
        at dom_top.core$real_pmap_helper$build_thread__213$fn__214.invoke(core.clj:146)
        at clojure.lang.AFn.applyToHelper(AFn.java:152)
        at clojure.lang.AFn.applyTo(AFn.java:144)
        at clojure.core$apply.invokeStatic(core.clj:665)
        at clojure.core$with_bindings_STAR_.invokeStatic(core.clj:1973)
        at clojure.core$with_bindings_STAR_.doInvoke(core.clj:1973)
        at clojure.lang.RestFn.invoke(RestFn.java:425)
        at clojure.lang.AFn.applyToHelper(AFn.java:156)
        at clojure.lang.RestFn.applyTo(RestFn.java:132)
        at clojure.core$apply.invokeStatic(core.clj:669)
        at clojure.core$bound_fn_STAR_$fn__5734.doInvoke(core.clj:2003)
        at clojure.lang.RestFn.invoke(RestFn.java:397)
        at clojure.lang.AFn.run(AFn.java:22)
        at java.base/java.lang.Thread.run(Thread.java:1583)
WARN [2025-03-11 14:36:24,989] jepsen node n2 - jepsen.control Encountered error with conn [:control "n2"]; reopening
com.jcraft.jsch.JSchException: channel is not opened.
        at com.jcraft.jsch.Channel.sendChannelOpen(Channel.java:765)
        at com.jcraft.jsch.Channel.connect(Channel.java:151)
        at com.jcraft.jsch.Channel.connect(Channel.java:145)
        at clj_ssh.ssh$connect_channel.invokeStatic(ssh.clj:512)
        at clj_ssh.ssh$connect_channel.invoke(ssh.clj:509)
        at clj_ssh.ssh$ssh_exec_proc.invokeStatic(ssh.clj:667)
        at clj_ssh.ssh$ssh_exec_proc.invoke(ssh.clj:646)
        at clj_ssh.ssh$ssh_exec.invokeStatic(ssh.clj:677)
        at clj_ssh.ssh$ssh_exec.invoke(ssh.clj:670)
        at clj_ssh.ssh$ssh.invokeStatic(ssh.clj:723)
        at clj_ssh.ssh$ssh.invoke(ssh.clj:699)
        at jepsen.control.SSHRemote.execute_BANG_(control.clj:331)
        at jepsen.control$ssh_STAR_$fn__3063.invoke(control.clj:172)
        at jepsen.control$ssh_STAR_.invokeStatic(control.clj:172)
        at jepsen.control$ssh_STAR_.invoke(control.clj:168)
        at jepsen.control$exec_STAR_.invokeStatic(control.clj:194)
        at jepsen.control$exec_STAR_.doInvoke(control.clj:191)
        at clojure.lang.RestFn.applyTo(RestFn.java:137)
        at clojure.core$apply.invokeStatic(core.clj:665)
        at clojure.core$apply.invoke(core.clj:660)
        at jepsen.control$exec.invokeStatic(control.clj:210)
        at jepsen.control$exec.doInvoke(control.clj:204)
        at clojure.lang.RestFn.invoke(RestFn.java:408)
        at jepsen.os.debian$setup_hostfile_BANG_.invokeStatic(debian.clj:16)
        at jepsen.os.debian$setup_hostfile_BANG_.invoke(debian.clj:13)
        at jepsen.os.debian.Debian.setup_BANG_(debian.clj:162)
        at jepsen.os$fn__2459$G__2453__2463.invoke(os.clj:4)
        at jepsen.os$fn__2459$G__2452__2468.invoke(os.clj:4)
        at clojure.core$partial$fn__5824.invoke(core.clj:2625)
        at jepsen.control$on_nodes$fn__3161.invoke(control.clj:430)
        at clojure.lang.AFn.applyToHelper(AFn.java:154)
        at clojure.lang.AFn.applyTo(AFn.java:144)
        at clojure.core$apply.invokeStatic(core.clj:665)
        at clojure.core$with_bindings_STAR_.invokeStatic(core.clj:1973)
        at clojure.core$with_bindings_STAR_.doInvoke(core.clj:1973)
        at clojure.lang.RestFn.applyTo(RestFn.java:142)
        at clojure.core$apply.invokeStatic(core.clj:669)
        at clojure.core$bound_fn_STAR_$fn__5734.doInvoke(core.clj:2003)
        at clojure.lang.RestFn.invoke(RestFn.java:408)
        at dom_top.core$real_pmap_helper$build_thread__213$fn__214.invoke(core.clj:146)
        at clojure.lang.AFn.applyToHelper(AFn.java:152)
        at clojure.lang.AFn.applyTo(AFn.java:144)
        at clojure.core$apply.invokeStatic(core.clj:665)
        at clojure.core$with_bindings_STAR_.invokeStatic(core.clj:1973)
        at clojure.core$with_bindings_STAR_.doInvoke(core.clj:1973)
        at clojure.lang.RestFn.invoke(RestFn.java:425)
        at clojure.lang.AFn.applyToHelper(AFn.java:156)
        at clojure.lang.RestFn.applyTo(RestFn.java:132)
        at clojure.core$apply.invokeStatic(core.clj:669)
        at clojure.core$bound_fn_STAR_$fn__5734.doInvoke(core.clj:2003)
        at clojure.lang.RestFn.invoke(RestFn.java:397)
        at clojure.lang.AFn.run(AFn.java:22)
        at java.base/java.lang.Thread.run(Thread.java:1583)
WARN [2025-03-11 14:36:24,990] jepsen node n4 - jepsen.control Encountered error with conn [:control "n4"]; reopening
com.jcraft.jsch.JSchException: channel is not opened.
        at com.jcraft.jsch.Channel.sendChannelOpen(Channel.java:765)
        at com.jcraft.jsch.Channel.connect(Channel.java:151)
        at com.jcraft.jsch.Channel.connect(Channel.java:145)
        at clj_ssh.ssh$connect_channel.invokeStatic(ssh.clj:512)
        at clj_ssh.ssh$connect_channel.invoke(ssh.clj:509)
        at clj_ssh.ssh$ssh_exec_proc.invokeStatic(ssh.clj:667)
        at clj_ssh.ssh$ssh_exec_proc.invoke(ssh.clj:646)
        at clj_ssh.ssh$ssh_exec.invokeStatic(ssh.clj:677)
        at clj_ssh.ssh$ssh_exec.invoke(ssh.clj:670)
        at clj_ssh.ssh$ssh.invokeStatic(ssh.clj:723)
        at clj_ssh.ssh$ssh.invoke(ssh.clj:699)
        at jepsen.control.SSHRemote.execute_BANG_(control.clj:331)
        at jepsen.control$ssh_STAR_$fn__3063.invoke(control.clj:172)
        at jepsen.control$ssh_STAR_.invokeStatic(control.clj:172)
        at jepsen.control$ssh_STAR_.invoke(control.clj:168)
        at jepsen.control$exec_STAR_.invokeStatic(control.clj:194)
        at jepsen.control$exec_STAR_.doInvoke(control.clj:191)
        at clojure.lang.RestFn.applyTo(RestFn.java:137)
        at clojure.core$apply.invokeStatic(core.clj:665)
        at clojure.core$apply.invoke(core.clj:660)
        at jepsen.control$exec.invokeStatic(control.clj:210)
        at jepsen.control$exec.doInvoke(control.clj:204)
        at clojure.lang.RestFn.invoke(RestFn.java:408)
        at jepsen.os.debian$setup_hostfile_BANG_.invokeStatic(debian.clj:16)
        at jepsen.os.debian$setup_hostfile_BANG_.invoke(debian.clj:13)
        at jepsen.os.debian.Debian.setup_BANG_(debian.clj:162)
        at jepsen.os$fn__2459$G__2453__2463.invoke(os.clj:4)
        at jepsen.os$fn__2459$G__2452__2468.invoke(os.clj:4)
        at clojure.core$partial$fn__5824.invoke(core.clj:2625)
        at jepsen.control$on_nodes$fn__3161.invoke(control.clj:430)
        at clojure.lang.AFn.applyToHelper(AFn.java:154)
        at clojure.lang.AFn.applyTo(AFn.java:144)
        at clojure.core$apply.invokeStatic(core.clj:665)
        at clojure.core$with_bindings_STAR_.invokeStatic(core.clj:1973)
        at clojure.core$with_bindings_STAR_.doInvoke(core.clj:1973)
        at clojure.lang.RestFn.applyTo(RestFn.java:142)
        at clojure.core$apply.invokeStatic(core.clj:669)
        at clojure.core$bound_fn_STAR_$fn__5734.doInvoke(core.clj:2003)
        at clojure.lang.RestFn.invoke(RestFn.java:408)
        at dom_top.core$real_pmap_helper$build_thread__213$fn__214.invoke(core.clj:146)
        at clojure.lang.AFn.applyToHelper(AFn.java:152)
        at clojure.lang.AFn.applyTo(AFn.java:144)
        at clojure.core$apply.invokeStatic(core.clj:665)
        at clojure.core$with_bindings_STAR_.invokeStatic(core.clj:1973)
        at clojure.core$with_bindings_STAR_.doInvoke(core.clj:1973)
        at clojure.lang.RestFn.invoke(RestFn.java:425)
        at clojure.lang.AFn.applyToHelper(AFn.java:156)
        at clojure.lang.RestFn.applyTo(RestFn.java:132)
        at clojure.core$apply.invokeStatic(core.clj:669)
        at clojure.core$bound_fn_STAR_$fn__5734.doInvoke(core.clj:2003)
        at clojure.lang.RestFn.invoke(RestFn.java:397)
        at clojure.lang.AFn.run(AFn.java:22)
        at java.base/java.lang.Thread.run(Thread.java:1583)
WARN [2025-03-11 14:36:25,630] main - jepsen.core Test crashed!
clojure.lang.ExceptionInfo: throw+: {:dir "/", :private-key-path nil, :password "root", :username "root", :type :jepsen.control/ssh-failed, :port 22, :strict-host-key-checking false, :host "n1", :sudo nil, :dummy false, :session {:open #object[jepsen.control$session$fn__3147 0x1e74d5d8 "jepsen.control$session$fn__3147@1e74d5d8"], :close #object[jepsen.control$fn__3018$G__2947__3023 0x79f1a2e3 "jepsen.control$fn__3018$G__2947__3023@79f1a2e3"], :log? true, :name [:control "n1"], :lock #object[java.util.concurrent.locks.ReentrantReadWriteLock 0x37b94e5 "java.util.concurrent.locks.ReentrantReadWriteLock@37b94e5[Write locks = 0, Read locks = 0]"], :conn #object[clojure.lang.Atom 0x7e79d881 {:status :ready, :val #jepsen.control.SSHRemote{:session #object[com.jcraft.jsch.Session 0x428cce61 "com.jcraft.jsch.Session@428cce61"]}}]}}
        at slingshot.support$stack_trace.invoke(support.clj:201)
        at jepsen.control$ssh_STAR_$fn__3063.invoke(control.clj:172)
        at jepsen.control$ssh_STAR_.invokeStatic(control.clj:172)
        at jepsen.control$ssh_STAR_.invoke(control.clj:168)
        at jepsen.control$exec_STAR_.invokeStatic(control.clj:194)
        at jepsen.control$exec_STAR_.doInvoke(control.clj:191)
        at clojure.lang.RestFn.applyTo(RestFn.java:137)
        at clojure.core$apply.invokeStatic(core.clj:665)
        at clojure.core$apply.invoke(core.clj:660)
        at jepsen.control$exec.invokeStatic(control.clj:210)
        at jepsen.control$exec.doInvoke(control.clj:204)
        at clojure.lang.RestFn.invoke(RestFn.java:408)
        at jepsen.os.debian$setup_hostfile_BANG_.invokeStatic(debian.clj:16)
        at jepsen.os.debian$setup_hostfile_BANG_.invoke(debian.clj:13)
        at jepsen.os.debian.Debian.setup_BANG_(debian.clj:162)
        at jepsen.os$fn__2459$G__2453__2463.invoke(os.clj:4)
        at jepsen.os$fn__2459$G__2452__2468.invoke(os.clj:4)
        at clojure.core$partial$fn__5824.invoke(core.clj:2625)
        at jepsen.control$on_nodes$fn__3161.invoke(control.clj:430)
        at clojure.lang.AFn.applyToHelper(AFn.java:154)
        at clojure.lang.AFn.applyTo(AFn.java:144)
        at clojure.core$apply.invokeStatic(core.clj:665)
        at clojure.core$with_bindings_STAR_.invokeStatic(core.clj:1973)
        at clojure.core$with_bindings_STAR_.doInvoke(core.clj:1973)
        at clojure.lang.RestFn.applyTo(RestFn.java:142)
        at clojure.core$apply.invokeStatic(core.clj:669)
        at clojure.core$bound_fn_STAR_$fn__5734.doInvoke(core.clj:2003)
        at clojure.lang.RestFn.invoke(RestFn.java:408)
        at dom_top.core$real_pmap_helper$build_thread__213$fn__214.invoke(core.clj:146)
        at clojure.lang.AFn.applyToHelper(AFn.java:152)
        at clojure.lang.AFn.applyTo(AFn.java:144)
        at clojure.core$apply.invokeStatic(core.clj:665)
        at clojure.core$with_bindings_STAR_.invokeStatic(core.clj:1973)
        at clojure.core$with_bindings_STAR_.doInvoke(core.clj:1973)
        at clojure.lang.RestFn.invoke(RestFn.java:425)
        at clojure.lang.AFn.applyToHelper(AFn.java:156)
        at clojure.lang.RestFn.applyTo(RestFn.java:132)
        at clojure.core$apply.invokeStatic(core.clj:669)
        at clojure.core$bound_fn_STAR_$fn__5734.doInvoke(core.clj:2003)
        at clojure.lang.RestFn.invoke(RestFn.java:397)
        at clojure.lang.AFn.run(AFn.java:22)
        at java.base/java.lang.Thread.run(Thread.java:1583)
ERROR [2025-03-11 14:36:25,635] main - jepsen.cli Oh jeez, I'm sorry, Jepsen broke. Here's why:
clojure.lang.ExceptionInfo: throw+: {:dir "/", :private-key-path nil, :password "root", :username "root", :type :jepsen.control/ssh-failed, :port 22, :strict-host-key-checking false, :host "n1", :sudo nil, :dummy false, :session {:open #object[jepsen.control$session$fn__3147 0x1e74d5d8 "jepsen.control$session$fn__3147@1e74d5d8"], :close #object[jepsen.control$fn__3018$G__2947__3023 0x79f1a2e3 "jepsen.control$fn__3018$G__2947__3023@79f1a2e3"], :log? true, :name [:control "n1"], :lock #object[java.util.concurrent.locks.ReentrantReadWriteLock 0x37b94e5 "java.util.concurrent.locks.ReentrantReadWriteLock@37b94e5[Write locks = 0, Read locks = 0]"], :conn #object[clojure.lang.Atom 0x7e79d881 {:status :ready, :val #jepsen.control.SSHRemote{:session #object[com.jcraft.jsch.Session 0x428cce61 "com.jcraft.jsch.Session@428cce61"]}}]}}
        at slingshot.support$stack_trace.invoke(support.clj:201)
        at jepsen.control$ssh_STAR_$fn__3063.invoke(control.clj:172)
        at jepsen.control$ssh_STAR_.invokeStatic(control.clj:172)
        at jepsen.control$ssh_STAR_.invoke(control.clj:168)
        at jepsen.control$exec_STAR_.invokeStatic(control.clj:194)
        at jepsen.control$exec_STAR_.doInvoke(control.clj:191)
        at clojure.lang.RestFn.applyTo(RestFn.java:137)
        at clojure.core$apply.invokeStatic(core.clj:665)
        at clojure.core$apply.invoke(core.clj:660)
        at jepsen.control$exec.invokeStatic(control.clj:210)
        at jepsen.control$exec.doInvoke(control.clj:204)
        at clojure.lang.RestFn.invoke(RestFn.java:408)
        at jepsen.os.debian$setup_hostfile_BANG_.invokeStatic(debian.clj:16)
        at jepsen.os.debian$setup_hostfile_BANG_.invoke(debian.clj:13)
        at jepsen.os.debian.Debian.setup_BANG_(debian.clj:162)
        at jepsen.os$fn__2459$G__2453__2463.invoke(os.clj:4)
        at jepsen.os$fn__2459$G__2452__2468.invoke(os.clj:4)
        at clojure.core$partial$fn__5824.invoke(core.clj:2625)
        at jepsen.control$on_nodes$fn__3161.invoke(control.clj:430)
        at clojure.lang.AFn.applyToHelper(AFn.java:154)
        at clojure.lang.AFn.applyTo(AFn.java:144)
        at clojure.core$apply.invokeStatic(core.clj:665)
        at clojure.core$with_bindings_STAR_.invokeStatic(core.clj:1973)
        at clojure.core$with_bindings_STAR_.doInvoke(core.clj:1973)
        at clojure.lang.RestFn.applyTo(RestFn.java:142)
        at clojure.core$apply.invokeStatic(core.clj:669)
        at clojure.core$bound_fn_STAR_$fn__5734.doInvoke(core.clj:2003)
        at clojure.lang.RestFn.invoke(RestFn.java:408)
        at dom_top.core$real_pmap_helper$build_thread__213$fn__214.invoke(core.clj:146)
        at clojure.lang.AFn.applyToHelper(AFn.java:152)
        at clojure.lang.AFn.applyTo(AFn.java:144)
        at clojure.core$apply.invokeStatic(core.clj:665)
        at clojure.core$with_bindings_STAR_.invokeStatic(core.clj:1973)
        at clojure.core$with_bindings_STAR_.doInvoke(core.clj:1973)
        at clojure.lang.RestFn.invoke(RestFn.java:425)
        at clojure.lang.AFn.applyToHelper(AFn.java:156)
        at clojure.lang.RestFn.applyTo(RestFn.java:132)
        at clojure.core$apply.invokeStatic(core.clj:669)
        at clojure.core$bound_fn_STAR_$fn__5734.doInvoke(core.clj:2003)
        at clojure.lang.RestFn.invoke(RestFn.java:397)
        at clojure.lang.AFn.run(AFn.java:22)
        at java.base/java.lang.Thread.run(Thread.java:1583)

After debugging (sudo journalctl -u ssh --since "5 minutes ago") it was found that ssh timed out when executing pam_systemd:

pam_systemd(sshd:session): Failed to create session: Failed to activate service 'org.freedesktop.login1': timed out (service_start_timeout=25000ms)

I tried the ubuntu22.04 virtual machine and container running the official docker image provided by jepsen, with the same result, and ssh n1 took 25 seconds to connect.

When creating a node using docker, the systemd-logind service was not successfully started, but the node was still successfully established

Partial error information 
jepsen-n3       | run-parts: executing /usr/local/preinit/00-checkroot
jepsen-n3       | Cleaning up; ignore errors here:
jepsen-n3       | Cleaning up temporary files... /tmpfind: cannot delete './secrets/authorized_keys': Device or resource busy
jepsen-n3       | bootclean: Failure cleaning /run. ... failed!
jepsen-n3       |  /run/lock failed!
jepsen-n3       | run-parts: executing /usr/local/preinit/00-setinit
jepsen-n3       | run-parts: executing /usr/local/preinit/01-syslog
jepsen-n3       | Sending syslog to Docker (config file /etc/syslog.conf.stdout)
jepsen-n3       | run-parts: executing /usr/local/preinit/01-timezone
jepsen-n3       | run-parts: executing /usr/local/preinit/03-setup-jepsen
jepsen-n3       | mkdir: cannot create directory '/root/.ssh': File exists
jepsen-n3       | Starting systemd
jepsen-n3       | systemd 252.33-1~deb12u1 running in system mode (+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
jepsen-n3       | Detected virtualization docker.
jepsen-n3       | Detected architecture x86-64.
jepsen-n3       |
jepsen-n3       | Welcome to Debian GNU/Linux 12 (bookworm)!

jepsen-n4       | bootclean: Failure cleaning /run. ... failed!
jepsen-n3       |          Starting [email protected]…e - Load Kernel Module fuse...
jepsen-n2       |          Starting dbus.service - D-Bus System Message Bus...
jepsen-n3       |          Starting [email protected]…e - Load Kernel Module loop...
jepsen-n2       |          Starting e2scrub_reap.serv…e ext4 Metadata Check Snapshots...
jepsen-n3       |          Starting systemd-journald.service - Journal Service...
jepsen-n2       |          Starting rc-local.servicem - /etc/rc.local Compatibility...
jepsen-n3       |          Starting systemd-modules-l…rvice - Load Kernel Modules...
jepsen-n2       |          Starting rsyslog.service - System Logging Service...
jepsen-n2       |          Starting ssh.service - OpenBSD Secure Shell server...
jepsen-n4       |  /run/lock failed!
jepsen-n2       |          Starting systemd-logind.se…ice - User Login Management...
jepsen-n3       |          Starting systemd-remount-f…nt Root and Kernel File Systems...
jepsen-n2       |          Starting systemd-user-sess…vice - Permit User Sessions...
jepsen-n2       | [  OK  ] Started atd.service - Deferred execution scheduler.
jepsen-n3       | sys-kernel-debug.mount: Mount process exited, code=exited, status=32/n/a
jepsen-n2       | [FAILED] Failed to start systemd-lo…rvice - User Login Management.
jepsen-n2       | See 'systemctl status systemd-logind.service' for details.
jepsen-n2       | [  OK  ] Stopped systemd-logind.service - User Login Management.
jepsen-n3       | sys-kernel-debug.mount: Failed with result 'exit-code'.
jepsen-n2       |          Starting [email protected] - Load Kernel Module drm...
jepsen-n3       | [FAILED] Failed to mount sys-kernel…nt - Kernel Debug File System.
jepsen-n2       | [  OK  ] Finished [email protected] - Load Kernel Module drm.
jepsen-n3       | See 'systemctl status sys-kernel-debug.mount' for details.
jepsen-n2       |          Starting systemd-logind.se…ice - User Login Management...
jepsen-n3       | sys-kernel-tracing.mount: Mount process exited, code=exited, status=32/n/a
jepsen-n2       | [  OK  ] Started rsyslog.service - System Logging Service.
jepsen-n3       | sys-kernel-tracing.mount: Failed with result 'exit-code'.
jepsen-n2       | [  OK  ] Started rc-local.service - /etc/rc.local Compatibility.
jepsen-n3       | [FAILED] Failed to mount sys-kernel…nt - Kernel Trace File System.
jepsen-n2       | [  OK  ] Finished systemd-user-sess…ervice - Permit User Sessions.
jepsen-n4       | run-parts: executing /usr/local/preinit/00-setinit
jepsen-n3       | See 'systemctl status sys-kernel-tracing.mount' for details.
jepsen-n2       | [  OK  ] Started dbus.service - D-Bus System Message Bus.
jepsen-n4       | run-parts: executing /usr/local/preinit/01-syslog
jepsen-n3       | [email protected]: Deactivated successfully.
jepsen-n2       | [FAILED] Failed to start systemd-lo…rvice - User Login Management.
jepsen-n4       | Sending syslog to Docker (config file /etc/syslog.conf.stdout)
jepsen-n3       | [  OK  ] Finished modprobe@configfs…[0m - Load Kernel Module configfs.
jepsen-n2       | See 'systemctl status systemd-logind.service' for details.
jepsen-n3       | modprobe@dm_mod.service: Deactivated successfully.
jepsen-n2       | [FAILED] Failed to start e2scrub_re…ine ext4 Metadata Check Snapshots.
jepsen-n4       | run-parts: executing /usr/local/preinit/01-timezone
jepsen-n3       | [  OK  ] Finished modprobe@dm_mod.s…e - Load Kernel Module dm_mod.
jepsen-n2       | See 'systemctl status e2scrub_reap.service' for details.
jepsen-n4       | run-parts: executing /usr/local/preinit/03-setup-jepsen
jepsen-n3       | [email protected]: Deactivated successfully.
jepsen-n2       | [  OK  ] Started console-getty.service - Console Getty.
jepsen-n4       | mkdir: cannot create directory '/root/.ssh': File exists
jepsen-n3       | [  OK  ] Finished [email protected] - Load Kernel Module drm.
jepsen-n2       | [  OK  ] Reached target getty.target - Login Prompts.
jepsen-n4       | Starting systemd
jepsen-n3       | modprobe@efi_pstore.service: Deactivated successfully.
jepsen-n2       | [  OK  ] Stopped systemd-logind.service - User Login Management.
jepsen-n3       | [  OK  ] Finished modprobe@efi_psto…m - Load Kernel Module efi_pstore.
jepsen-n2       |          Starting [email protected] - Load Kernel Module drm...
jepsen-n2       | [  OK  ] Finished [email protected] - Load Kernel Module drm.
jepsen-n2       |          Starting systemd-logind.se…ice - User Login Management...
jepsen-n2       | [  OK  ] Started ssh.service - OpenBSD Secure Shell server.
jepsen-n2       | [FAILED] Failed to start systemd-lo…rvice - User Login Management.

One possible way to fix it

Changing UsePAM in /etc/ssh/sshd_config to no resolves this issue, but it seems that the root cause is that systemd-logind service is somehow not starting successfully,I don't know what caused it.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions