At my client, they have an Active Directory domain with a few domain controllers which are also DNS servers. They all run Windows Server 2008 R2. All of these DNS servers/domain controllers all have forwarders enabled on them using the typical 126.96.36.199, 188.8.131.52 and 184.108.40.206.
A problem cropped up that unknown DNS traffic was being initiated from a DNS server out to the Internet. The network administrator had locked down outgoing DNS traffic only to the forwarders and was seeing a lot of hits on an ACL that was denying other DNS traffic to other public IPs. He needed me to figure out if this traffic was necessary to further open up DNS. I didn’t think so but I thought I’d investigate.
Step #1 – Fire up Wireshark.
With Wireshark now installed on this DNS server I opened it up and soon created a capture filter to narrow down interesting DNS activity as much as possible with this capture filter:
udp port 53 and not host 220.127.116.11 and not host 18.104.22.168 and not host 22.214.171.124. This capture filter narrows down the capture on UDP/53. I then exclude my forwarders because I know DNS traffic will be going to those.
I start the capture and then created a display filter. This display filter removes out all of the internal IPs I was seeing. I don’t care about any internal DNS activity; just to external DNS servers.
(!ip.dst==192.168.0.0/16) and (!ip.dst==126.96.36.199/8) and (!ip.dst==10.0.0.0/8) It’s always a good idea to create capture filters instead of display filters with Wireshark and, in hindsight, I probably should have added some additional subnet rules to the capture filter.
I was noticing TONS of DNS traffic going out to external DNS servers. I knew based on the article Recursive and Iterative Queries that when forwarders are used the queries are always recursive. This meant that no queries should be sent from this server to any others besides the forwarder IPs. However, I was seeing tons of iterative queries to other servers from the Wireshark packet capture.
After looking around a little bit, I noticed the option Use root hints if no forwarders are available. I thought well maybe the forwarders weren’t working for some reason and decided to compare the forwarders IP list with the list of servers that were showing up in the packet capture. Using PowerShell, I was able to easily get a list of the root hints.
(Get-DnsServerRootHint -ComputerName dc).IPAddress.RecordData.IPV4Address.IPAddressToString
I then took this list of IPs and compared it to the IPs that I copied out of the Endpoints summary field in Wireshark. I found none of the DNS servers that were being queried were any of the root hints IPs. Damn!
Since my consulting firm is a Microsoft partner, I was able to start a thread on the partner forums and the first thing the guys mentioned was to uncheck that check box. Being skeptical it was going to work I did it anyway and no and behold the queries stopped! I still don’t understand why none of the traffic was going to the root hint IPs directly but as soon as I unchecked the “use root hint” box the queries were gone.
The network administrator is happy and so am I finally figured this out. In hindsight, just open up a Microsoft thread first instead of going to the trouble of getting down to the packet capture level.