Docker DNS server returns NXDOMAIN for SRV records

[hawicz]

Docker's internal DNS server doesn't appear to function properly when attempting to resolve SRV records. A lookup like dig -t srv _imaps._tcp.gmail.com works fine on the host system, but returns NXDOMAIN inside a container.

Steps to reproduce:

# Regular rootless docker install:
sudo su -
apt-get install -y uidmap curl
# ... check /etc/subuid for proper entries ...
exit

curl -fsSL https://get.docker.com/rootless | sh

export PATH=${HOME}/bin:$PATH
export DOCKER_HOST=unix:///run/user/$(id -nu)/docker.sock

SRV lookup inside a container:

docker pull debian:buster   # or ubuntu:latest, or etc...
docker run --rm -it debian:buster bash -c 'apt-get update && apt-get install -y dnsutils && dig -t srv _imaps._tcp.gmail.com'

System info:

$  docker version
Client: Docker Engine - Community
 Version:           19.03.12
 API version:       1.40
 Go version:        go1.13.10
 Git commit:        48a66213fe
 Built:             Mon Jun 22 15:42:53 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.12
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.10
  Git commit:       48a66213fe
  Built:            Mon Jun 22 15:49:35 2020
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          v1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

$ docker info
Client:
 Debug Mode: false

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 3
 Server Version: 19.03.12
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: false
 Logging Driver: json-file
 Cgroup Driver: none
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
  rootless
 Kernel Version: 5.3.0-51-generic
 Operating System: Ubuntu 19.10
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 8.062GiB
 Name: testsystem
 ID: 7J5I:IG4C:QNFY:BOTC:E3QP:SH67:FTTP:EA4Z:UKJL:OK4N:XBU5:7PQT
 Docker Root Dir: /home/user/.local/share/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: true
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

WARNING: No swap limit support
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

[thaJeztah]

This should be supported by the embedded DNS server since

However, when running on the default ("bridge") network, Docker's embedded DNS server is not used, so not sure what would be causing this. Trying on a non-rootless install of Docker 19.03.12 on ubuntu 20.04 (note that Ubuntu 19.10 is a non-LTS release, and that is EOL), this works for me;

docker run --rm alpine sh -c 'apk add -q --no-cache bind-tools && dig -t srv _imaps._tcp.gmail.com'


; <<>> DiG 9.14.12 <<>> -t srv _imaps._tcp.gmail.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 56678
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;_imaps._tcp.gmail.com.		IN	SRV

;; ANSWER SECTION:
_imaps._tcp.gmail.com.	86062	IN	SRV	5 0 993 imap.gmail.com.

;; Query time: 0 msec
;; SERVER: 67.207.67.2#53(67.207.67.2)
;; WHEN: Thu Aug 27 12:30:48 UTC 2020
;; MSG SIZE  rcvd: 84

And with a custom network;

docker network create mynet
docker run --rm --network=mynet alpine sh -c 'apk add -q --no-cache bind-tools && dig -t srv _imaps._tcp.gmail.com'

; <<>> DiG 9.14.12 <<>> -t srv _imaps._tcp.gmail.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 34420
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;_imaps._tcp.gmail.com.		IN	SRV

;; ANSWER SECTION:
_imaps._tcp.gmail.com.	86025	IN	SRV	5 0 993 imap.gmail.com.

;; Query time: 1 msec
;; SERVER: 127.0.0.11#53(127.0.0.11)
;; WHEN: Thu Aug 27 12:31:25 UTC 2020
;; MSG SIZE  rcvd: 84

Are you seeing the same problem when running docker in non-rootless mode?

[hawicz]

Right, I do not see the problem in non-rootless mode, but I don't have root access where I need to be able to run my containers, so that's not going to help me much.

[thaJeztah]

@AkihiroSuda ptal

[AkihiroSuda]

This is a VPNKit issue.
Please install slirp4netns (v0.4.0 or later, v1.1.X is preferred).
When slirp4netns is installed, Docker automatically changes the network stack from VPNKit to slirp4netns.

[AkihiroSuda]

opened an issue in VPNKit repo: moby/vpnkit#509

[thaJeztah]

Thanks! Let me close this ticket in favour of moby/vpnkit#509

[hawicz]

I can confirm that using slirp4netns instead of vpnkit gets things working for me, thanks!
I'll follow that other issue for any vpnkit updates.