Ztunnel/DNS Capture: DNS Additional Section merged into DNS Answer

[zirkome]

Is this the right place to submit this?

• This is not a security vulnerability or a crashing bug
• This is not a question about how to use Istio

Bug Description

Hi!

We have a new Istio service which is now failing on SRV queries. After investigation we found out that it is due to a DNS additional section returned by Azure DNS that is merged in the Answer section by ztunnel DNS proxy.
I was able to reproduce the issue outside Azure (i.e. AWS) by mimicking the same DNS answer via CoreDNS override.

    template ANY ANY _tcp.db.srv.tld {
      answer "{{ .Name }} 8 IN SRV 0 0 2400 privatelink-db.srv.tld."
      answer "{{ .Name }} 8 IN SRV 0 0 2401 privatelink-db.srv.tld."
      answer "{{ .Name }} 8 IN SRV 0 0 2402 privatelink-db.srv.tld."
      additional "privatelink-db.srv.tld. 8 IN A 10.140.0.250"
      additional "privatelink-db.srv.tld. 8 IN A 10.140.0.250"
      additional "privatelink-db.srv.tld. 8 IN A 10.140.0.250"
    }

Without Istio (or w/ DNS Capture disabled) we get the following from dig:

;; ANSWER SECTION:
_tcp.db.srv.tld. 8	IN SRV 0 0 2400 privatelink-db.srv.tld.
_tcp.db.srv.tld. 8	IN SRV 0 0 2401 privatelink-db.srv.tld.
_tcp.db.srv.tld. 8	IN SRV 0 0 2402 privatelink-db.srv.tld.

;; ADDITIONAL SECTION:
privatelink-db.srv.tld. 8 IN A	10.140.0.250
privatelink-db.srv.tld. 8 IN A	10.140.0.250
privatelink-db.srv.tld. 8 IN A	10.140.0.250

However inside the pod, part of the Istio mesh, when we run dig we get the following:

;; ANSWER SECTION:
_tcp.db.srv.tld. 30	IN SRV 0 0 2400 privatelink-db.srv.tld.
_tcp.db.srv.tld. 30	IN SRV 0 0 2401 privatelink-db.srv.tld.
_tcp.db.srv.tld. 30	IN SRV 0 0 2402 privatelink-db.srv.tld.
privatelink-db.srv.tld. 30 IN A	10.140.0.250
privatelink-db.srv.tld. 30 IN A	10.140.0.250
privatelink-db.srv.tld. 30 IN A	10.140.0.250

Version

$ istioctl version
client version: 1.24.2
control plane version: 1.24.2
data plane version: 1.24.2 (5 proxies)
$ kubectl version
Client Version: v1.32.1
Kustomize Version: v5.5.0
Server Version: v1.30.9

Additional Information

No response

[bleggett]

Moving this to istio/ztunnel since it is likely a problem with the ztunnel DNS proxy.

[howardjohn]

Thanks for the report. I can reproduce and confirm its not impacting sidecars, only ztunnel

[howardjohn]

I think this is a bug in the DNS library we depend on. I opened an issue in hickory-dns/hickory-dns#2781.

[howardjohn]

I don't see many options here aside from contributing (or waiting) for a fix upstream

[zirkome]

A potential fix we found is using CoreDNS' minimal plugin which stripped the ADDITIONAL section