Cannot run mahimahi, seemingly because of permission issues.

[NotSpecial]

Hi everyone, and a happy new year!

I am trying to run mahimahi on Ubuntu 20.04LTS using binaries installed from the official repositories.

I'm trying to execute mm-delay 10, which instantly fails with the message:

Died on std::runtime_error: dnsmasq: did not start after 20 attempts
Died on std::runtime_error: `packetshell': process exited with failure status 

I initially thought the problem might be related to #143, but I do not experience any delay before the crash; it is instant.

Thus I did a bit of digging and compiled mahimahi myself from source to get additional debug output. The problem seems to come from lacking permissions:

• The crash happens unrelated to dnsmasq, but because of server.write("x") link to code.
• When attempting the write, a write: Connection refused is thrown.
• I checked which address the Socket connects to, and it's 0.0.0.0:53.

At this point, I am unsure what to do next to get mahimahi to work. I cannot run it as root (sudo mm-delay 10 throws mm-delay: please run as non-root). Is there any specific configuration I need to make mahimahi work?

Thanks and best,
Alex

[keithw]

Happy new year! I'm afraid I can't replicate this. I just installed a brand-new Ubuntu 20.04 VM on Amazon EC2, and then then logged in and ran:

$ sudo apt update
$ sudo apt install mahimahi
$ sudo sysctl -w net.ipv4.ip_forward=1
$ mm-delay 10
[delay 10 ms] ubuntu@ip-172-31-28-73:~$

and it started up fine.

In general, the server.write( "x" ) line is checking to see if anything is listening to UDP port 53 on localhost. If nothing is listening (meaning dnsmasq hasn't started yet), the kernel will send an ICMP port unreachable, which will trigger an error the next time the server.write("x") happens. So the code makes sure that it can send two UDP datagrams in a row (with a 10 ms delay between them) without getting an error, i.e. without an ICMP port unreachable. If it can't do that after 20 attempts (about 200 milliseconds) it concludes that dnsmasq didn't start successfully, or at least it's not listening on port 53.

Given that this works on a fresh Ubuntu install, it's possible the problem involves some sort of security hardening or firewall that has been configured on this computer, but it could also be something preventing dnsmasq from starting up properly.

[NotSpecial]

Thank you very much for checking and confirming that it works on your end.

Also thanks for the clarification on the two writes. Based on this, I believe the issue seems to be that dnsmasq does start properly, The first server.write("x") goes through without issue, and the second one raises the connection refused exception. I guess this means the ICMP unreachable happens in between, as you describe it.

Is there any particular configuration needed to make dnsmasq work for mahimahi?

In any case, I'll try to figure out what prevents it from starting properly.

[keithw]

mahimahi starts dnsmasq with a -C /dev/null option to make it ignore its default configuration file, so we don't intend for there to be any particular configuration needed. I mean it does work on a fresh install. I would wonder if dnsmasq is logging an error message if you start it manually with similar command-line options in a container that isn't already running a DNS server, e.g. if you run sudo dnsmasq --keep-in-foreground --no-resolv --no-hosts -C /dev/null in a fresh container [or, after temporarily killing anything bound to port 53 in your main namespace]. (You might also check the output in sudo journalctl -f to see if it's logging to syslog.)

[intothesea]

@keithw @NotSpecial
I solved this issue by changing ++attempts >= 20 to ++attempts >= 40. The reason is simple: the ‘dnsmasq’ process requires some time to start, which can sometimes exceed 200ms. This issue was particularly evident on a server with a Xeon(R) Gold 6138 CPU, but did not occur on a personal computer with an i9-14900 CPU. Therefore, modifying the condition to ++attempts >= 40 and rebuilding from source resolved the problem.

I hope the developers can come up with a more elegant fix for this issue, as it has troubled me for quite some time.