Spike of SERVFAIL responses after reboot since 14.0

[IngmarStein]

Whenever I restart the DNS server, I get a spike of SERVFAIL responses which lasts for several minutes before it drops to 0 again:

The log entries look like this:

TechnitiumLibrary.Net.Dns.DnsClientNoResponseException: DnsClient failed to recursively resolve the request 'technikblog.ch. AAAA IN': no response from name servers [ns1.cyon.ch ([2a01:ab20:0:185::2]), ns2.cyon.ch ([2001:67c:234::2]), ns1.cyon.ch (185.174.216.2), ns2.cyon.ch (91.206.24.2)] at delegation technikblog.ch.
 ---> TechnitiumLibrary.Net.Dns.DnsClientNoResponseException: DnsClient failed to resolve the request 'technikblog.ch. DNSKEY IN': request timed out for name server [ns1.cyon.ch (185.174.216.2)].
 ---> System.Net.Sockets.SocketException (60): Operation timed out
   at TechnitiumLibrary.Net.SocketExtensions.UdpQueryAsync(Socket socket, ArraySegment`1 request, ArraySegment`1 response, IPEndPoint remoteEP, Int32 timeout, Int32 retries, Boolean expBackoffTimeout, Func`2 isResponseValid, CancellationToken cancellationToken) in /private/tmp/technitium-library-20251119-6751-6f88eb/TechnitiumLibrary-dns-server-v14.1.0/TechnitiumLibrary.Net/SocketExtensions.cs:line 141
   at TechnitiumLibrary.Net.Dns.ClientConnection.UdpClientConnection.QueryAsync(DnsDatagram request, Int32 timeout, Int32 retries, CancellationToken cancellationToken) in /private/tmp/technitium-library-20251119-6751-6f88eb/TechnitiumLibrary-dns-server-v14.1.0/TechnitiumLibrary.Net/Dns/ClientConnection/UdpClientConnection.cs:line 330
   --- End of inner exception stack trace ---
   at TechnitiumLibrary.Net.Dns.ClientConnection.UdpClientConnection.QueryAsync(DnsDatagram request, Int32 timeout, Int32 retries, CancellationToken cancellationToken) in /private/tmp/technitium-library-20251119-6751-6f88eb/TechnitiumLibrary-dns-server-v14.1.0/TechnitiumLibrary.Net/Dns/ClientConnection/UdpClientConnection.cs:line 339
   at TechnitiumLibrary.Net.Dns.DnsClient.<>c__DisplayClass89_0.<<InternalResolveAsync>g__DoResolveAsync|1>d.MoveNext() in /private/tmp/technitium-library-20251119-6751-6f88eb/TechnitiumLibrary-dns-server-v14.1.0/TechnitiumLibrary.Net/Dns/DnsClient.cs:line 4512
--- End of stack trace from previous location ---
   at TechnitiumLibrary.Net.Dns.DnsClient.<>c__DisplayClass89_0.<<InternalResolveAsync>g__DoResolveAsync|1>d.MoveNext() in /private/tmp/technitium-library-20251119-6751-6f88eb/TechnitiumLibrary-dns-server-v14.1.0/TechnitiumLibrary.Net/Dns/DnsClient.cs:line 4502
--- End of stack trace from previous location ---
   at TechnitiumLibrary.Net.Dns.DnsClient.<>c__DisplayClass89_0.<<InternalResolveAsync>g__DoResolveAsync|1>d.MoveNext() in /private/tmp/technitium-library-20251119-6751-6f88eb/TechnitiumLibrary-dns-server-v14.1.0/TechnitiumLibrary.Net/Dns/DnsClient.cs:line 4428
--- End of stack trace from previous location ---
   at TechnitiumLibrary.Net.Dns.DnsClient.InternalResolveAsync(DnsDatagram request, Func`3 getValidatedResponseAsync, Boolean doNotReorderNameServers, CancellationToken cancellationToken) in /private/tmp/technitium-library-20251119-6751-6f88eb/TechnitiumLibrary-dns-server-v14.1.0/TechnitiumLibrary.Net/Dns/DnsClient.cs:line 4887
   at TechnitiumLibrary.Net.Dns.DnsClient.InternalResolveAsync(DnsDatagram request, Func`3 getValidatedResponseAsync, Boolean doNotReorderNameServers, CancellationToken cancellationToken) in /private/tmp/technitium-library-20251119-6751-6f88eb/TechnitiumLibrary-dns-server-v14.1.0/TechnitiumLibrary.Net/Dns/DnsClient.cs:line 4829
   at TechnitiumLibrary.Net.Dns.DnsClient.GetDnsKeyForAsync(IReadOnlyList`1 lastDSRecords, DnsClient dnsClient, IDnsCache cache, UInt16 udpPayloadSize, CancellationToken cancellationToken) in /private/tmp/technitium-library-20251119-6751-6f88eb/TechnitiumLibrary-dns-server-v14.1.0/TechnitiumLibrary.Net/Dns/DnsClient.cs:line 3218
   at TechnitiumLibrary.Net.Dns.DnsClient.DnssecValidateResponseAsync(DnsDatagram response, IReadOnlyList`1 lastDSRecords, DnsClient dnsClient, IDnsCache cache, UInt16 udpPayloadSize, CancellationToken cancellationToken) in /private/tmp/technitium-library-20251119-6751-6f88eb/TechnitiumLibrary-dns-server-v14.1.0/TechnitiumLibrary.Net/Dns/DnsClient.cs:line 2659
   at TechnitiumLibrary.Net.Dns.DnsClient.<>c__DisplayClass45_1.<<RecursiveResolveAsync>b__7>d.MoveNext() in /private/tmp/technitium-library-20251119-6751-6f88eb/TechnitiumLibrary-dns-server-v14.1.0/TechnitiumLibrary.Net/Dns/DnsClient.cs:line 1081
--- End of stack trace from previous location ---
   at TechnitiumLibrary.Net.Dns.DnsClient.<>c__DisplayClass89_0.<<InternalResolveAsync>g__DoResolveAsync|1>d.MoveNext() in /private/tmp/technitium-library-20251119-6751-6f88eb/TechnitiumLibrary-dns-server-v14.1.0/TechnitiumLibrary.Net/Dns/DnsClient.cs:line 4569
--- End of stack trace from previous location ---
   at TechnitiumLibrary.Net.Dns.DnsClient.<>c__DisplayClass89_0.<<InternalResolveAsync>g__DoResolveAsync|1>d.MoveNext() in /private/tmp/technitium-library-20251119-6751-6f88eb/TechnitiumLibrary-dns-server-v14.1.0/TechnitiumLibrary.Net/Dns/DnsClient.cs:line 4502
--- End of stack trace from previous location ---
   at TechnitiumLibrary.Net.Dns.DnsClient.<>c__DisplayClass89_0.<<InternalResolveAsync>g__DoResolveAsync|1>d.MoveNext() in /private/tmp/technitium-library-20251119-6751-6f88eb/TechnitiumLibrary-dns-server-v14.1.0/TechnitiumLibrary.Net/Dns/DnsClient.cs:line 4428
--- End of stack trace from previous location ---
   at TechnitiumLibrary.Net.Dns.DnsClient.InternalResolveAsync(DnsDatagram request, Func`3 getValidatedResponseAsync, Boolean doNotReorderNameServers, CancellationToken cancellationToken) in /private/tmp/technitium-library-20251119-6751-6f88eb/TechnitiumLibrary-dns-server-v14.1.0/TechnitiumLibrary.Net/Dns/DnsClient.cs:line 4887
   at TechnitiumLibrary.Net.Dns.DnsClient.InternalResolveAsync(DnsDatagram request, Func`3 getValidatedResponseAsync, Boolean doNotReorderNameServers, CancellationToken cancellationToken) in /private/tmp/technitium-library-20251119-6751-6f88eb/TechnitiumLibrary-dns-server-v14.1.0/TechnitiumLibrary.Net/Dns/DnsClient.cs:line 4829
   at TechnitiumLibrary.Net.Dns.DnsClient.RecursiveResolveAsync(DnsQuestionRecord question, IDnsCache cache, NetProxy proxy, Boolean preferIPv6, UInt16 udpPayloadSize, Boolean randomizeName, Boolean qnameMinimization, Boolean dnssecValidation, NetworkAddress eDnsClientSubnet, Int32 retries, Int32 timeout, Int32 concurrency, Int32 maxStackCount, Boolean minimalResponse, Boolean asyncNsResolution, List`1 rawResponses, CancellationToken cancellationToken) in /private/tmp/technitium-library-20251119-6751-6f88eb/TechnitiumLibrary-dns-server-v14.1.0/TechnitiumLibrary.Net/Dns/DnsClient.cs:line 1064
   --- End of inner exception stack trace ---
   at TechnitiumLibrary.Net.Dns.DnsClient.RecursiveResolveAsync(DnsQuestionRecord question, IDnsCache cache, NetProxy proxy, Boolean preferIPv6, UInt16 udpPayloadSize, Boolean randomizeName, Boolean qnameMinimization, Boolean dnssecValidation, NetworkAddress eDnsClientSubnet, Int32 retries, Int32 timeout, Int32 concurrency, Int32 maxStackCount, Boolean minimalResponse, Boolean asyncNsResolution, List`1 rawResponses, CancellationToken cancellationToken) in /private/tmp/technitium-library-20251119-6751-6f88eb/TechnitiumLibrary-dns-server-v14.1.0/TechnitiumLibrary.Net/Dns/DnsClient.cs:line 1807
   at TechnitiumLibrary.TaskExtensions.TimeoutAsync[T](Func`2 func, Int32 timeout, CancellationToken cancellationToken)
   at TechnitiumLibrary.TaskExtensions.TimeoutAsync[T](Func`2 func, Int32 timeout, CancellationToken cancellationToken) in /private/tmp/technitium-library-20251119-6751-6f88eb/TechnitiumLibrary-dns-server-v14.1.0/TechnitiumLibrary/TaskExtensions.cs:line 65
   at DnsServerCore.Dns.DnsServer.RecursiveResolverBackgroundTaskAsync(DnsQuestionRecord question, NetworkAddress eDnsClientSubnet, Boolean advancedForwardingClientSubnet, IReadOnlyList`1 conditionalForwarders, Boolean dnssecValidation, Boolean cachePrefetchOperation, Boolean cacheRefreshOperation, Boolean skipDnsAppAuthoritativeRequestHandlers, TaskCompletionSource`1 taskCompletionSource) in /private/tmp/technitium-dns-20251119-14310-akd84f/DnsServer-14.1.0/DnsServerCore/Dns/DnsServer.cs:line 4482

The "Operation timed out" error is curious because the WAN connection is stable all the time and I can reliably reproduce the issue just by restarting Technitium DNS.

Any idea how I can avoid this temporary degradation?
I can't remember seeing this prior to 14.0.

[ShreyasZare]

Thanks for the post. Please share a complete screenshot of the Dashboard stats so that the magnitude of this issue is clear and that it will help to understand the issue better.

If there are too many requests incoming when the DNS server starts, it may have a few resolutions fail to complete in time since the background recursive resolve may be too busy to complete all pending tasks in its internal queue. The Server Failure responses are also sent when there is no answer available within Client Timeout of default 2 sec while the resolution may take more time to complete.

The SocketException (60): Operation timed out error just means that the outbound query to the listed name servers did not yield in a response within the timeout period even after retrying. So, there is really nothing that can be done about it and its a common operational issue that comes with running recursive resolver. There will be times when some names servers do not answer quick enough and it can be due to intermittent network/routing issues at ISP level too.

Also, make sure that all the Cache settings are set to the recommended values and that Serve Stale and Save Cache To Disk are enabled. These options will help with mitigating these operational issues.

[IngmarStein]

Here's the complete screenshot with the DNS server restart visible at the 16:00 mark (please ignore the other spike at 19:00 which is when my gateway was rebooted to install a new firmware version):

I understand that "Operation timed out" can happen as part of normal operations, but the spike at restart seems odd because the errors also include entries within my localdomain zone which is just a Forwarder zone to the gateway which is certainly reachable and operational.

The cache settings are using the defaults and Serve Stale and Save Cache to Disk are enabled:

[IngmarStein]

Another benefit of using the more standard SIGTERM is that it would allow us to remoe the STOPSIGNAL and KillSignal customizations for Docker and systemd, respectively.

[ShreyasZare]

The issue is that .NET does not raise app close events for the code for SIGTERM but works for SIGINT. I am not sure if they have made any changes in .NET 9 but that was the issue in the previous runtime versions. Will need to revisit this once.

[IngmarStein]

@ShreyasZare wdyt about this diff?

diff --git a/DnsServerApp/Program.cs b/DnsServerApp/Program.cs
index 9c312e79..59b81e95 100644
--- a/DnsServerApp/Program.cs
+++ b/DnsServerApp/Program.cs
@@ -21,6 +21,7 @@ using DnsServerCore;
 using System;
 using System.Threading;
 using System.Threading.Tasks;
+using System.Runtime.InteropServices;
 
 namespace DnsServerApp
 {
@@ -83,6 +84,16 @@ namespace DnsServerApp
                     exitHandle.WaitOne();
                 };
 
+                if (RuntimeInformation.IsOSPlatform(OSPlatform.OSX) || RuntimeInformation.IsOSPlatform(OSPlatform.Linux))
+                {
+                    PosixSignalRegistration.Create(PosixSignal.SIGTERM, (context) =>
+                    {
+                        context.Cancel = true; // Prevent immediate hard termination
+                        waitHandle.Set();      // Signal main thread to shut down
+                        exitHandle.WaitOne();  // Wait for cleanup
+                    });
+                }
+
                 Console.WriteLine("Technitium DNS Server was started successfully.\r\nUsing config folder: " + service.ConfigFolder + "\r\n\r\nNote: Open http://" + Environment.MachineName.ToLowerInvariant() + ":" + service.WebServiceHttpPort + "/ in web browser to access web console.\r\n\r\nPress [CTRL + C] to stop...");
 
                 waitHandle.WaitOne();

or a bit shorter:

diff --git a/DnsServerApp/Program.cs b/DnsServerApp/Program.cs
index 9c312e79..0331c3cf 100644
--- a/DnsServerApp/Program.cs
+++ b/DnsServerApp/Program.cs
@@ -83,6 +83,12 @@ namespace DnsServerApp
                     exitHandle.WaitOne();
                 };
 
+                System.Runtime.Loader.AssemblyLoadContext.Default.Unloading += delegate (System.Runtime.Loader.AssemblyLoadContext obj)
+                {
+                    waitHandle.Set();
+                    exitHandle.WaitOne();
+                };
+
                 Console.WriteLine("Technitium DNS Server was started successfully.\r\nUsing config folder: " + service.ConfigFolder + "\r\n\r\nNote: Open http://" + Environment.MachineName.ToLowerInvariant() + ":" + service.WebServiceHttpPort + "/ in web browser to access web console.\r\n\r\nPress [CTRL + C] to stop...");
 
                 waitHandle.WaitOne();

[IngmarStein]

@ShreyasZare I sent you #1567 after testing locally on macOS.

When running brew services start technitium-dns; brew services stop technitium-dns, I can now see the graceful shutdown messages:

Technitium DNS Server was started successfully.
Using config folder: /opt/homebrew/etc/technitium-dns

Note: Open http://mac:5380/ in web browser to access web console.

Press [CTRL + C] to stop...

Technitium DNS Server is stopping...
Technitium DNS Server was stopped successfully.

[ShreyasZare]

Thanks for the details and PR. I think this issue was in .NET5 and there was no option to fix it back then. The PosixSignalRegistration came later in .NET 6 but this issue was never revisited till now. So its great to know that there is a proper solution available to this issue. Thanks for finding a working solution.

Some of the users that are using community scripts to install on LXC containers on proxmox too have this issue and this PR should fix it for them too.

[IngmarStein]

Closing this as v14.3 has resolved the issue. Thanks!

[ShreyasZare]

Thanks for confirming and your PR that fixed it!