Description
Description
Version: v20.05-14~gd4b5fdde4
Reproduce: use multi threads vpp, set dhcp client intfc GigabitEthernet0/7/0
When vpp is used in multi threads/workers, the dhcp client causes the vpp crash.
the backtrace is
#0 0x00007ffff53c7387 in raise () from /lib64/libc.so.6
#1 0x00007ffff53c8a78 in abort () from /lib64/libc.so.6
#2 0x000000000040858e in os_panic () at /data/workspace/vpp/src/vpp/vnet/main.c:366
#3 0x00007ffff61688eb in debugger () at /data/workspace/vpp/src/vppinfra/error.c:84
#4 0x00007ffff6168cd6 in _clib_error (how_to_die=2, function_name=0x0, line_number=0, fmt=0x7fffb3d2b1d0 "%s:%d (%s) assertion `%s' fails") at /data/workspace/vpp/src/vppinfra/error.c:143 #5 0x00007fffb3cba1e5 in vlib_time_now (vm=0x7ffff7f81780 <vlib_global_main>) at /data/workspace/vpp/src/vlib/main.h:299
#6 0x00007fffb3cbd0db in dhcp_client_for_us (bi=636440, b=0x10026d8600, ip=0x10026d870e, udp=0x10026d8722, dhcp=0x10026d872a) at /data/workspace/vpp/src/plugins/dhcp/client.c:282
#7 0x00007fffb3ce0688 in dhcp_proxy_to_client_input (vm=0x7fffb474dcc0, node=0x7fffb4cbf900, from_frame=0x7fffb4c78000) at /data/workspace/vpp/src/plugins/dhcp/dhcp4_proxy_node.c:568
#8 0x00007ffff7ed7b44 in dispatch_node (vm=0x7fffb474dcc0, node=0x7fffb4cbf900, type=VLIB_NODE_TYPE_INTERNAL, dispatch_state=VLIB_NODE_STATE_POLLING, frame=0x7fffb4c78000, last_time_stamp=4188694816810460)
at /data/workspace/vpp/src/vlib/main.c:1235
#9 0x00007ffff7ed82ff in dispatch_pending_node (vm=0x7fffb474dcc0, pending_frame_index=4, last_time_stamp=4188694816810460) at /data/workspace/vpp/src/vlib/main.c:1403
#10 0x00007ffff7ed9fc2 in vlib_main_or_worker_loop (vm=0x7fffb474dcc0, is_main=0) at /data/workspace/vpp/src/vlib/main.c:1862
#11 0x00007ffff7edaa29 in vlib_worker_loop (vm=0x7fffb474dcc0) at /data/workspace/vpp/src/vlib/main.c:1996
#12 0x00007ffff7f1bda5 in vlib_worker_thread_fn (arg=0x7fffbc2bb180) at /data/workspace/vpp/src/vlib/threads.c:1842
#13 0x00007ffff61876b8 in clib_calljmp () at /data/workspace/vpp/src/vppinfra/longjmp.S:123
#14 0x00007fef4f5fec10 in ?? ()
#15 0x00007ffff7f15bc0 in vlib_worker_thread_bootstrap_fn (arg=0x7fffbc2bb180) at /data/workspace/vpp/src/vlib/threads.c:584
I checked the newest code, this commit 77d9838282 introduced this bug.
in vlib_time_now under this case,
vm->thread_index is main thread, __os_thread_index is the worker thread.
Assignee
Jan Cavojsky
Reporter
Faicker Mo
Comments
- jan_cavojsky (Mon, 21 Sep 2020 12:02:57 +0000): This bug is solved in master branch. Reason, why this patchset don't will be merged into branch 20.05 is in conversation on this link: https://gerrit.fd.io/r/c/vpp/+/28707
- jan_cavojsky (Tue, 8 Sep 2020 11:35:01 +0000):
Hi,
I know, why vpp on my PC worked correctly.
I tried reproduce this bug on branch, where was this bug fixed, and therefore didn't possible reproduced it.
I successful reproduce this bug on branch 2005, where was not fixed.
I tested behavior of vpp after applying of commit for solving this bug.
After applying this commit vpp works correct without crashing.
I made "cherry-pick" of commit, that solved this problem into branch 2005, that you was using: https://gerrit.fd.io/r/c/vpp/+/28707
- jan_cavojsky (Mon, 7 Sep 2020 07:44:46 +0000):
Hi,
I tried reproduced this bug, but not successfully.
My conditions was:
-
- simulating on virtualbox
- os version: Linux johny-VirtualBox 5.3.0-62-generic #56~18.04.1-Ubuntu SMP Wed Jun 24 16:17:03 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
- running of vpp on docker container
- vpp version: v20.09-rc0~370-g67fe77854
- dhcp server setting was directly in virtualbox by setting network card into NAT
- after binding of network interface was used vfio-pci driver, uio driver was loaded in linux, but not used, I don't understand why
- JIRAUSER13812 (Thu, 27 Aug 2020 01:46:14 +0000): /etc/vpp/startup.conf
- jan_cavojsky (Wed, 26 Aug 2020 12:06:08 +0000):
Hi
My config file is little different:
unix {
nodaemon
log /var/log/vpp.log
full-coredump
cli-listen /run/vpp/cli.sock
}
cpu {
main-core 1
corelist-workers 2
}
dpdk {
dev default {
num-rx-queues 1
num-tx-queues 1
num-rx-desc 256
num-tx-desc 256
}
dev 0000:00:08.0
dev 0000:00:09.0
uio-driver igb_uio
no-tx-checksum-offload
}
plugins {
path /opt/vpp/build-root/install-vpp_debug-native/vpp/lib/vpp_plugins
plugin default { disable }
plugin dpdk_plugin.so { enable }
plugin nat_plugin.so { enable }
plugin dhcp_plugin.so { enable }
plugin ping_plugin.so { enable }
}
used command for assign of ip address by command in vpp:
set interface state GigabitEthernet0/8/0 up
set dhcp client intfc GigabitEthernet0/8/0
Can you please give me more info about how reproduce this bug? How are your conditions?
Thanks.
unix {
nodaemon
log /var/log/vpp.log
full-coredump
cli-listen /run/vpp/cli.sock
gid vpp
}api-segment {
gid vpp
}socksvr {
default
}cpu {
main-core 1
corelist-workers 2
}dpdk {
dev default {
num-rx-queues 1
num-tx-queues 1num-rx-desc 256
num-tx-desc 256
}dev 0000:00:02.0
dev 0000:00:03.0uio-driver igb_uio
no-tx-checksum-offload
}plugins {
path /home/xxx/vpp/build-root/install-vpp_debug-native/vpp/lib/vpp_plugins
plugin default { disable }
plugin dpdk_plugin.so { enable }
plugin nat_plugin.so { enable }
plugin dhcp_plugin.so { enable }
plugin ping_plugin.so { enable }}
run vpp and vppctl to execute commands,
set interface state GigabitEthernet0/2/0 upset dhcp client intfc GigabitEthernet0/2/0
make sure GigabitEthernet0/2/0 dhcp client is OK(so you may need a dhcp server)
please how I can reproduce this bug? Can you give me config file for vpp or also commands for linux if are necessary?
Thanks.
Original issue: https://jira.fd.io/browse/VPP-1924