Low DPDK throughput in NFV benchmark

[eugeneia]

program/snabbnfv/packetblaster_bench.sh performs badly due to DPDK dropping packets. See #588.

[lukego]

@eugeneia Could you please update the issue text to say exactly what we know about the differences between the fast and slow test environments and how to run both tests? (In the discussion on the other issue I lose track a bit of exactly what code is being tested and what its results are.)

Standard question whenever a mysterious 30% performance difference appears: Can the NUMA affinity have changed? (Could check e.g. in htop which node's cores are being used in each run of the benchmark.)

[eugeneia]

@lukego I started testing from a clean slate, to be honest I didn't know anything and that's why I did not want to pollute the issue with previous speculations.

I am quite sure that its unrelated to NUMA affinity, as nothing changed there and the other benchmarks are not affected.

So here we go, fresh start. There are two different versions of DPDK being tested:

• “legacy” - this is the version we used previously in the opaque image, and we are using now for the CI
• v2.1.0-snabb - the current version of DPDK with our patches applied.

I also tested on two different machines:

• grindelwald - Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz
• davos - Intel(R) Xeon(R) CPU E5-2603 v2 @ 1.80GHz

Finally I tested with two different versions of QEMU (2.4.0 and 2.1.0). Now to the results:

1. On grindelwald, it performs worse (~25%) when using v2.1.0-snabb instead of legacy DPDK (4 Mpps vs 3 Mpps), but QEMU versions do not make a difference
2. On davos using legacy DPDK, it performs worse (~30%) when using QEMU 2.4 instead of 2.1 (2.4 Mpps vs 3.5 Mpps)

So while there is a performance hit when upgrading to DPDK 2.1, the reason for the decrease in performance on davos seems to be indeed the new QEMU (2.4.0) version.

@lukego On a side node, legacy DPDK does indeed seem to work with 1000 byte packets.

[lukego]

Can you tell me how to reproduce please?

We should not apply any patches to DPDK.

9000 byte is what I believe won't work in the older DPDK.

[eugeneia]

Good news, I have found the culprit: QEMU is the bottleneck, not DPDK as I thought.I have built a docker image identical to what the CI uses except that it contains QEMU https://github.com/SnabbCo/qemu/tree/v2.1.0-vhostuser plus the patch which increases hard-coded Virtio vring size (snabbco/qemu@7a94322).

(I actually reproduced the wrong QEMU version before, that's why I couldn't reproduce on grindelwald.)

This image should be an almost exact replica of the opaque blobs and QEMU we used in bench_env times (except a slight difference in the QEMU source for which I couldn't find the patch / context):

https://hub.docker.com/r/eugeneia/snabb-nfv-test-legacy/

You can reproduce like so:

$ docker pull eugeneia/snabb-nfv-test-legacy
$ SNABB_TEST_IMAGE=eugeneia/snabb-nfv-test-legacy \
  SNABB_PCI_INTEL0=0000:01:00.0 \
  SNABB_PCI_INTEL1=0000:01:00.1 \
  scripts/dock.sh program/snabbnfv/packetblaster_bench.sh

This yielded expected performance (5.3Mpps on grindelwald) and no DPDK packet loss when I ran it.

[kbara]

Nice work. :-)

[lukego]

Great detective work, Max!

The Docker workflow really does make it easy to reproduce tests! I ran this on chur and the results seem consistent when taking into account the slower CPU:

[luke@chur:~/git/snabbswitch/src]$ scripts/dock.sh "(cd ..; make -j)"
...
[luke@chur:~/git/snabbswitch/src]$ SNABB_TEST_IMAGE=eugeneia/snabb-nfv-test-legacy   SNABB_PCI_INTEL0=0000:01:00.0   SNABB_PCI_INTEL1=0000:01:00.1   scripts/dock.sh program/snabbnfv/packetblaster_bench.sh
...
Processed 100.0 million packets in 23.90 seconds (6400003840 bytes; 2.14 Gbps)
Made 891,403 breaths: 112.18 packets per breath; 26.81us per breath
Rate(Mpps): 4.184

Next I would really like to get in control of the patches. Specifically I would like to migrate over to testing with the latest releases of QEMU and DPDK without any patches applied. That is the intended target software environment. If the performance does not match expectations then we would dig in to look for the root cause and try to address this in the snabb code rather than with adding/reviving patches. Is this easy to setup?

Patches can really take on a life of their own :-). I have a little bit of Nix envy here: Nix makes the exact versions/patches being tested completely transparent, even down to kernels and libc both on the host and inside the VMs, whereas Docker seems to make this very opaque. Dockerhub makes it extremely easy to reproduce the test environment but the summary page doesn't provide any insight into what code is actually in the container.

(There is actually one QEMU patch that I do still recommend applying, "f393aea Add G_IO_HUP handler for socket chardev", but that should not be needed for the DPDK benchmark. It's only to allow the snabb process to restart without restarting QEMU.)

[eugeneia]

Next I would really like to get in control of the patches. Specifically I would like to migrate over to testing with the latest releases of QEMU and DPDK without any patches applied.

I am in the process of building a patch-less “vanilla” image.

[eugeneia]

Here is the “vanilla” image containing latest stable QEMU and DPDK (and kernel 3.19 instead of 3.13 because DPDK required 3.14+): https://hub.docker.com/r/eugeneia/snabb-nfv-test-vanilla/

[lukego]

Awesome, thanks! I am really impressed with the Docker workflow you have cooked up, I think that the effort has already paid off in terms of time saved on manually reproducing test environments.

Here is a braindump on the general theme of compatibility and performance with different software versions from our upstream Snabb Switch perspective:

The main thing is to make the latest upstream versions work and prevent them from breaking/regressing in the future. Then over time we build up a long trail of compatibility with consecutive versions. Supporting older and/or patched versions is less important unless specially required for some reason.

We want to take responsibility for making all the components work well together: Snabb, QEMU, DPDK/Linux guests, etc. If there is a problem then we need to find a solution even if that involves creating workarounds in Snabb Switch, making temporary patches to QEMU/DPDK, and working with upstream communities to merge fixes so that we can drop patches. In this sense it is better when a problem is caused by Snabb Switch, where it should be easy to fix, rather than a change in another project that we need to somehow deal with.

On performance issues like packet drops it can be that we need to take a "holistic" view of the relationships between all of the components rather than looking at one in isolation. For example if a new QEMU version leads to packet loss then this doesn't necessarily mean that there is a bug in QEMU but rather that the interactions between the components has changed. Particularly, QEMU is not involved in packet processing at all (this is done directly in shared memory between Snabb & DPDK) so it cannot be a direct processing bottleneck, but it is involved in negotiating the sizes and features of the shared memory rings and this can indirectly affect performance.

Likewise even between the processing components it is hard to point a finger at one and say that it is the problem. If the DPDK guest is dropping packets then all we know is that it is receiving faster than it is transmitting (and the difference is the dropped packets). This could be due to local DPDK behavior, or subtle Virtio-net behavior that gives the receive ring larger capacity than the transmit ring, or subtle Snabb Switch behavior that bursts packets onto the receive ring faster than it takes them off the transmit ring, and so on.

The most practical method I know of for diagnosing holistic performance problems is to bisect i.e. to identify two software versions that are as close as possible but with one being "good" and one being "bad". In this example it seems like we have "good" behavior from the fully patched QEMU 2.1 and "bad" behavior from the unpatched QEMU 2.4.1 and the next problem is to isolate this more narrowly e.g. does the problem appear when we drop one of our QEMU patches (like the vring size increase) or did it appear in QEMU 2.2 or 2.3 or 2.4 and so on.

I have a fantasty that our build infrastructure could make this easy to generate a test matrix e.g. by running commands like:

for snabb in v2015.08 v2015.09 v2015.10 v2015.11; do
  for qemu in 2.1-snabb 2.1 2.2 2.3 2.4; do
    for dpdk in 1.7-snabb 1.8 1.9 2.0 2.1; do
      run-test > result-$snabb-$qemu-$dpdk.txt
    done
  done
done

I am not sure if this is practical with the Docker-based test bootstrapping? If so that would be interesting!

I believe that this is a built-in feature of Hydra where build/test parameters can be specified as e.g. enums and all combinations can be tested (seen in a blog post on Hydra). I am keen to dig into this on the side in case it would be a nice solution for us in the future.

[lukego]

@eugeneia Running the "vanilla" test I don't see traffic passing so it seems like we have a compatibility issue between Snabb master + QEMU 2.4.1 + DPDK 2.1. Do you agree? If so I can create a separate ticket for that.

[eugeneia]

No I tested it and it works for me (albeit with decreased performance, since both the DPDK and QEMU “regressions” kick in.

chur$ SNABB_PCI_INTEL0=0000:03:00.0 \
      SNABB_PCI_INTEL1=0000:03:00.1 \
      SNABB_TEST_IMAGE=eugeneia/snabb-nfv-test-vanilla \
      scripts/dock.sh program/snabbnfv/packetblaster_bench.sh
[...]
Rate(Mpps): 2.076

Edit: I have only tested on chur.

[eugeneia]

My guess is that its these two patches that are impacting the throughput:

• virtualopensystems/dpdk@7807fbb ([l2fwd] let the driver select Rx/Tx desc number)
• snabbco/qemu@7a94322 (Increase hard-coded Virtio vring size from 256 to 8192)

And maybe this one (I am not sure what it does): virtualopensystems/dpdk@dae0a7f ( [virtio] Initialize the queues even if VIRTIO_NET_F_CTRL_VQ is not negotiated)

Will try to verify this guess.

[lukego]

Is there an easy way to confirm that based on the Docker workflow? This seems worth confirming before investing serious time in making a fix.