We have a background thread that's listening on a kernel's netlink socket for any changes to network interfaces. As these changes are happening, we're feeding info about them to sysrepo's operational DS. This is prone to races in several situations: - When the network is reconfigured in the running DS, we generate some config files on disk and then execute a `networkctl reload`. That happens form a module_change_cb. The other thread reacts to these events and attempts to push stuff to sysrepo -- all that while the first thread is still using that session for handling of the network config modification. - During the initial config sync, it's essentially the same situation, except now the race happens between the main thread and our background thread. - Also, the LLDP subscriptions can race with kernel's netlink updates. Here's (roughly) how scenario #1 looked like (note that line numbers are off because it's from a modified checkout): Thread 12 (Thread 0x7ffff1df56c0 (LWP 1238304) "test-sysrepo_in"): #0 0x00007ffff64e2250 in std::logic_error::logic_error(char const*) () from /nix/store/w47x7y9xjgq7n3171ckg722ddwf2pi3v-gcc-13.3.0-lib/lib/libstdc++.so.6 #1 0x00007ffff6d87114 in sysrepo::Session::mt_lock (mtx=...) at /home/jkt/work/cesnet/gerrit/CzechLight/sysrepo-cpp/include/sysrepo-cpp/Session.hpp:204 #2 0x00007ffff6d75054 in sysrepo::Session::activeDatastore (this=0x7ffff0ef5870) at /home/jkt/work/cesnet/gerrit/CzechLight/sysrepo-cpp/src/Session.cpp:118 #3 0x00005555558ef9be in velia::utils::ScopedDatastoreSwitch::ScopedDatastoreSwitch (this=0x7ffff0ef5870, session=..., ds=sysrepo::Datastore::Operational) at /home/jkt/work/cesnet/gerrit/CzechLight/velia/src/utils/sysrepo.cpp:163 #4 0x00005555558ee253 in velia::utils::valuesPush (session=..., values=std::vector of length 3, capacity 4 = {...}, foreignRemovals=..., ourRemovals=...) at /home/jkt/work/cesnet/gerrit/CzechLight/velia/src/utils/sysrepo.cpp:123 #5 0x00005555557e5bdb in velia::network::IETFInterfaces::onLinkUpdate (this=0x7ffff4bacaa0, link=<optimized out>, action=<optimized out>) at /home/jkt/work/cesnet/gerrit/CzechLight/velia/src/network/IETFInterfaces.cpp:262 #6 0x0000555555853167 in std::function<void(rtnl_link*, int)>::operator() (this=0x513000387ba8, __args=<optimized out>, __args=<optimized out>) at /nix/store/vl4zvn7ir6y4kzr99709xayck0mal8pn-gcc-13.3.0/include/c++/13.3.0/bits/std_function.h:591 #7 (anonymous namespace)::nlCacheMngrCallbackWrapper (obj=0x51a00058f880, action=<optimized out>, data=<optimized out>) at /home/jkt/work/cesnet/gerrit/CzechLight/velia/src/network/Rtnetlink.cpp:37 #8 0x00007ffff7e36a6c in cache_include () from /nix/store/aa90hcpy7dlm48ik7l7inrhnz14zv8m0-libnl-3.10.0/lib/libnl-3.so.200 #9 0x00007ffff6e70ea9 in link_msg_parser () from /nix/store/aa90hcpy7dlm48ik7l7inrhnz14zv8m0-libnl-3.10.0/lib/libnl-route-3.so.200 #10 0x00007ffff7e36337 in nl_cache_parse () from /nix/store/aa90hcpy7dlm48ik7l7inrhnz14zv8m0-libnl-3.10.0/lib/libnl-3.so.200 #11 0x00007ffff7e3763b in event_input () from /nix/store/aa90hcpy7dlm48ik7l7inrhnz14zv8m0-libnl-3.10.0/lib/libnl-3.so.200 #12 0x00007ffff7e3d1f1 in nl_recvmsgs_report () from /nix/store/aa90hcpy7dlm48ik7l7inrhnz14zv8m0-libnl-3.10.0/lib/libnl-3.so.200 #13 0x00007ffff7e37e8a in nl_cache_mngr_data_ready () from /nix/store/aa90hcpy7dlm48ik7l7inrhnz14zv8m0-libnl-3.10.0/lib/libnl-3.so.200 #14 0x00007ffff7e37f98 in nl_cache_mngr_poll () from /nix/store/aa90hcpy7dlm48ik7l7inrhnz14zv8m0-libnl-3.10.0/lib/libnl-3.so.200 #15 0x000055555585056b in velia::network::impl::nlCacheMngrWatcher::run (this=<optimized out>, manager=std::shared_ptr<nl_cache_mngr> (use count 2, weak count 0) = {...}) at /home/jkt/work/cesnet/gerrit/CzechLight/velia/src/network/Rtnetlink.cpp:90 #16 0x0000555555857e6f in std::__invoke_impl<void, void (velia::network::impl::nlCacheMngrWatcher::*)(std::shared_ptr<nl_cache_mngr>), velia::network::impl::nlCacheMngrWatcher*, std::shared_ptr<nl_cache_mngr> > (__f=<optimized out>, __t=<optimized out>, __args=...) at /nix/store/vl4zvn7ir6y4kzr99709xayck0mal8pn-gcc-13.3.0/include/c++/13.3.0/bits/invoke.h:74 #17 0x00007ffff64e86d3 in execute_native_thread_routine () from /nix/store/w47x7y9xjgq7n3171ckg722ddwf2pi3v-gcc-13.3.0-lib/lib/libstdc++.so.6 #18 0x0000555555639d24 in asan_thread_start(void*) () #19 0x00007ffff6298af3 in start_thread () from /nix/store/p9kdj55g5l39nbrxpjyz5wc1m0s7rzsx-glibc-2.40-66/lib/libc.so.6 #20 0x00007ffff6317f4c in __clone3 () from /nix/store/p9kdj55g5l39nbrxpjyz5wc1m0s7rzsx-glibc-2.40-66/lib/libc.so.6 Thread 1 (Thread 0x7ffff6ca2140 (LWP 1238177) "test-sysrepo_in"): #0 0x00005555556cc184 in ___interceptor_strcmp.part.0 () #1 0x00007ffff64bae14 in __dynamic_cast () from /nix/store/w47x7y9xjgq7n3171ckg722ddwf2pi3v-gcc-13.3.0-lib/lib/libstdc++.so.6 #2 0x0000555555750e07 in __ubsan::checkDynamicType(void*, void*, unsigned long) () #3 0x000055555574fc43 in HandleDynamicTypeCacheMiss(__ubsan::DynamicTypeCacheMissData*, unsigned long, unsigned long, __ubsan::ReportOptions) () #4 0x000055555575035b in __ubsan_handle_dynamic_type_cache_miss () #5 0x00005555557e1a3a in std::_Sp_counted_ptr_inplace<velia::network::LLDPDataProvider, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>::_Sp_counted_ptr_inplace<std::function<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >()>, velia::network::LLDPDataProvider::LocalData> (this=0x50c0006733c0, __a=..., __args=..., __args=...) at /nix/store/vl4zvn7ir6y4kzr99709xayck0mal8pn-gcc-13.3.0/include/c++/13.3.0/bits/shared_ptr_base.h:604 #6 0x00005555557e1431 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<velia::network::LLDPDataProvider, std::allocator<void>, std::function<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >()>, velia::network::LLDPDataProvider::LocalData> (this=<optimized out>, __p=<optimized out>, __a=..., __args=..., __args=...) at /nix/store/vl4zvn7ir6y4kzr99709xayck0mal8pn-gcc-13.3.0/include/c++/13.3.0/bits/shared_ptr_base.h:972 #7 0x0000555555751771 in std::__shared_ptr<velia::network::LLDPDataProvider, (__gnu_cxx::_Lock_policy)2>::__shared_ptr<std::allocator<void>, std::function<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >()>, velia::network::LLDPDataProvider::LocalData> (this=0x7ffff4826350, __tag=..., __args=..., __args=...) at /nix/store/vl4zvn7ir6y4kzr99709xayck0mal8pn-gcc-13.3.0/include/c++/13.3.0/bits/shared_ptr_base.h:1712 #8 std::shared_ptr<velia::network::LLDPDataProvider>::shared_ptr<std::allocator<void>, std::function<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >()>, velia::network::LLDPDataProvider::LocalData> (this=0x7ffff4826350, __tag=..., __args=..., __args=...) at /nix/store/vl4zvn7ir6y4kzr99709xayck0mal8pn-gcc-13.3.0/include/c++/13.3.0/bits/shared_ptr.h:464 #9 std::make_shared<velia::network::LLDPDataProvider, std::function<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >()>, velia::network::LLDPDataProvider::LocalData> (__args=..., __args=...) at /nix/store/vl4zvn7ir6y4kzr99709xayck0mal8pn-gcc-13.3.0/include/c++/13.3.0/bits/shared_ptr.h:1009 #10 velia::network::create (startupSess=..., persistentNetworkDirectory=Python Exception <class 'gdb.error'>: No type named std::filesystem::__cxx11::path::_Cmpt. filesystem::path "/home/jkt/work/prog/_build/czechlight-clang19-asan-ubsan/velia/tests/network-libnl-sd-networkd/startup", runningSess=..., runtimeNetworkDirectory=Python Exception <class 'gdb.error'>: No type named std::filesystem::__cxx11::path::_Cmpt. filesystem::path "/home/jkt/work/prog/_build/czechlight-clang19-asan-ubsan/velia/tests/network-libnl-sd-networkd/running", managedLinks=..., runningNetworkReloadCB=..., lldpCallback=..., lldpLocalData=...) at /home/jkt/work/cesnet/gerrit/CzechLight/velia/src/network/Factory.h:36 #11 0x00005555557684e8 in DOCTEST_ANON_FUNC_11 () at /home/jkt/work/cesnet/gerrit/CzechLight/velia/tests/sysrepo_interfaces-libnl.cpp:290 #12 0x000055555587c4b0 in doctest::Context::run (this=0x7ffff44006a0) at /home/jkt/work/prog/_build/czechlight-clang19-asan-ubsan/target/include/doctest/doctest.h:7007 #13 0x0000555555880d47 in main (argc=<optimized out>, argv=<optimized out>) at /home/jkt/work/prog/_build/czechlight-clang19-asan-ubsan/target/include/doctest/doctest.h:7085 The "good" thing is that this was quite deterministically happening on any HW box exactly *once* during the boot, *iff* the on-disk configuration and the configuration in sysrepo was different. After the first run, the configuration in /run/systemd/network was updated, so the velia-system (at that time) daemon had nothing to do, and stuff worked. As a result, this also worked on the majority of devices where the YANG config in sysrepo was "correct", but not on bidi amps where the default factory config has no data for ietf-network by mistake, while the persisted config files in /cfg/network (or /usr/lib/systemd/network) are OK. The fix is pretty obvious -- just let that dreaded netlink access a completely separate session. Oh well :]. Change-Id: I2bf25184003b7655f9e10e3e8b89a6f12824618f Bug: #12

SFP inventory & hwmon #11

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions