[squid-users] Bizarrely slow, timing out DNS only via Squid 😖

[Dan Charlesworth]

Right now we have 1 squid box (out of a lot), running 3.5.13, which does something like this for every request, taking about 10 seconds:

2016/03/03 16:30:48.883 kid1| 78,3| dns_internal.cc(1794) idnsPTRLookup: idnsPTRLookup: buf is 43 bytes for 10.100.128.1, id = 0x733a
2016/03/03 16:30:48.883 kid1| 78,3| dns_internal.cc(1745) idnsALookup: idnsALookup: buf is 29 bytes for httpbin.org, id = 0x8528
2016/03/03 16:30:48.883 kid1| 78,3| dns_internal.cc(1683) idnsSendSlaveAAAAQuery: buf is 29 bytes for httpbin.org, id = 0x69c2
2016/03/03 16:30:48.884 kid1| 78,3| dns_internal.cc(1277) idnsRead: idnsRead: starting with FD 7
2016/03/03 16:30:48.884 kid1| 78,3| dns_internal.cc(1323) idnsRead: idnsRead: FD 7: received 93 bytes from 192.231.203.132:53
2016/03/03 16:30:48.884 kid1| 78,3| dns_internal.cc(1130) idnsGrokReply: idnsGrokReply: QID 0x733a, -3 answers
2016/03/03 16:30:48.884 kid1| 78,3| dns_internal.cc(1195) idnsGrokReply: idnsGrokReply: error Name Error: The domain name does not exist. (3)
2016/03/03 16:30:53.884 kid1| 78,3| dns_internal.cc(1384) idnsCheckQueue: idnsCheckQueue: ID dns8 QID 0x8528: timeout
2016/03/03 16:30:53.884 kid1| 78,3| dns_internal.cc(1384) idnsCheckQueue: idnsCheckQueue: ID dns0 QID 0x69c2: timeout
2016/03/03 16:30:53.885 kid1| 78,3| dns_internal.cc(1277) idnsRead: idnsRead: starting with FD 7
2016/03/03 16:30:53.885 kid1| 78,3| dns_internal.cc(1323) idnsRead: idnsRead: FD 7: received 110 bytes from 172.16.100.4:53
2016/03/03 16:30:53.885 kid1| 78,3| dns_internal.cc(1130) idnsGrokReply: idnsGrokReply: QID 0x69c2, 0 answers
2016/03/03 16:30:58.885 kid1| 78,3| dns_internal.cc(1384) idnsCheckQueue: idnsCheckQueue: ID dns8 QID 0x8528: timeout
2016/03/03 16:30:58.886 kid1| 78,3| dns_internal.cc(1277) idnsRead: idnsRead: starting with FD 7
2016/03/03 16:30:58.886 kid1| 78,3| dns_internal.cc(1323) idnsRead: idnsRead: FD 7: received 246 bytes from 172.16.100.5:53
2016/03/03 16:30:58.886 kid1| 78,3| dns_internal.cc(1130) idnsGrokReply: idnsGrokReply: QID 0x8528, 1 answers

AND YET, every nslookup or dig done at the command line on the same server is lightning fast. I’ve tried local and ISP-level DNS servers and get the same result.

What could be going on here?