<div dir="auto"><div>What distro are you using?<br><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">בתאריך יום ב׳, 12 בפבר׳ 2024, 13:47, מאת Stephen Borrill <<a href="mailto:squid@borrill.org.uk">squid@borrill.org.uk</a>>:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On 16/01/2024 14:37, Alex Rousskov wrote:<br>
> On 2024-01-16 06:01, Stephen Borrill wrote:<br>
>> The problem is no different with 6.6. Is there any more debugging I <br>
>> can provide, Alex?<br>
> <br>
> Yes, but I need to give you a patch that adds that (temporary) debugging <br>
> first (assuming I fail to reproduce the problem in the lab). The ball is <br>
> on my side (unless somebody else steps in). Unfortunately, I do not have <br>
> any free time for any of that right now. If you do not hear from me <br>
> sooner, please ping me again on or after February 8, 2024.<br>
<br>
PING!<br>
<br>
I will get 6.7 compiled up so we can add debugging to it quickly. It <br>
would be good if we could get something in place this week as it is <br>
school holidays next week in the UK and so there will be little <br>
opportunity to test until afterwards.<br>
<br>
>> On 10/01/2024 12:40, Stephen Borrill wrote:<br>
>>> On 09/01/2024 15:42, Alex Rousskov wrote:<br>
>>>> On 2024-01-09 05:56, Stephen Borrill wrote:<br>
>>>>> On 09/01/2024 09:51, Stephen Borrill wrote:<br>
>>>>>> On 09/01/2024 03:41, Alex Rousskov wrote:<br>
>>>>>>> On 2024-01-08 08:31, Stephen Borrill wrote:<br>
>>>>>>>> I'm trying to determine why squid 6.x (seen with 6.5) connected <br>
>>>>>>>> via IPv4-only periodically fails to connect to the destination <br>
>>>>>>>> and then requires a restart to fix it (reload is not sufficient).<br>
>>>>>>>><br>
>>>>>>>> The problem appears to be that a host that has one address each <br>
>>>>>>>> of IPv4 and IPv6 occasionally has its IPv4 address go missing as <br>
>>>>>>>> a destination. On closer inspection, this appears to happen when <br>
>>>>>>>> the IPv6 address (not the IPv4) address is marked as bad.<br>
>>>><br>
>>>>> ipcache.cc(990) have: [2001:4860:4802:32::78]:443 at 0 in <br>
>>>>> 216.239.38.120 #1/2-0<br>
>>>><br>
>>>><br>
>>>> Thank you for sharing more debugging info!<br>
>>><br>
>>> The following seemed odd to. It finds an IPv4 address (this host does <br>
>>> not have IPv6), puts it in the cache and then says "No DNS records":<br>
>>><br>
>>> 2024/01/09 12:31:24.020 kid1| 14,4| ipcache.cc(617) nbgethostbyname: <br>
>>> schoolbase.online<br>
>>> 2024/01/09 12:31:24.020 kid1| 14,3| ipcache.cc(313) ipcacheRelease: <br>
>>> ipcacheRelease: Releasing entry for 'schoolbase.online'<br>
>>> 2024/01/09 12:31:24.020 kid1| 14,5| ipcache.cc(670) <br>
>>> ipcache_nbgethostbyname_: ipcache_nbgethostbyname: MISS for <br>
>>> 'schoolbase.online'<br>
>>> 2024/01/09 12:31:24.020 kid1| 14,3| ipcache.cc(480) ipcacheParse: 1 <br>
>>> answers for schoolbase.online<br>
>>> 2024/01/09 12:31:24.020 kid1| 14,7| ipcache.cc(995) have: no <br>
>>> 20.54.32.34 in [no cached IPs]<br>
>>> 2024/01/09 12:31:24.020 kid1| 14,7| ipcache.cc(995) have: no <br>
>>> 20.54.32.34 in [no cached IPs]<br>
>>> 2024/01/09 12:31:24.020 kid1| 14,5| ipcache.cc(549) updateTtl: use <br>
>>> first 69 from RR TTL 69<br>
>>> 2024/01/09 12:31:24.020 kid1| 14,3| ipcache.cc(535) addGood: <br>
>>> schoolbase.online #1 20.54.32.34<br>
>>> 2024/01/09 12:31:24.020 kid1| 14,7| ipcache.cc(253) forwardIp: <br>
>>> 20.54.32.34<br>
>>> 2024/01/09 12:31:24.020 kid1| 44,2| peer_select.cc(1174) handlePath: <br>
>>> PeerSelector72389 found conn564274 local=0.0.0.0 <br>
>>> remote=<a href="http://20.54.32.34:443" rel="noreferrer noreferrer" target="_blank">20.54.32.34:443</a> HIER_DIRECT flags=1, destination #1 for <br>
>>> schoolbase.online:443<br>
>>> 2024/01/09 12:31:24.020 kid1| 14,3| ipcache.cc(459) latestError: <br>
>>> ERROR: DNS failure while resolving schoolbase.online: No DNS records<br>
>>> 2024/01/09 12:31:24.020 kid1| 14,3| ipcache.cc(586) <br>
>>> ipcacheHandleReply: done with schoolbase.online: 20.54.32.34 #1/1-0<br>
>>> 2024/01/09 12:31:24.020 kid1| 14,7| ipcache.cc(236) finalCallback: <br>
>>> 0x1b7381f38 lookup_err=No DNS records<br>
>>><br>
>>> It seemed to happen about the same time as the other failure, so <br>
>>> perhaps another symptom of the same.<br>
>>><br>
>>>> The above log line is self-contradictory AFAICT: It says that the <br>
>>>> cache has both IPv6-looking and IPv4-looking address at the same <br>
>>>> cache position (0) and, judging by the corresponding code, those two <br>
>>>> IP addresses are equal. This is not possible (for those specific IP <br>
>>>> address values). The subsequent Squid behavior can be explained by <br>
>>>> this (unexplained) conflict.<br>
>>>><br>
>>>> I assume you are running official Squid v6.5 code.<br>
>>><br>
>>> Yes, compiled from source on NetBSD. I have the patch I refer to here <br>
>>> applied too:<br>
>>> <a href="https://lists.squid-cache.org/pipermail/squid-users/2023-November/026279.html" rel="noreferrer noreferrer" target="_blank">https://lists.squid-cache.org/pipermail/squid-users/2023-November/026279.html</a><br>
>>><br>
>>>> I can suggest the following two steps for going forward:<br>
>>>><br>
>>>> 1. Upgrade to the latest Squid v6 in hope that the problem goes away.<br>
>>><br>
>>> I have just upgraded to 6.6.<br>
>>><br>
>>>> 2. If the problem is still there, patch the latest Squid v6 to add <br>
>>>> more debugging in hope to explain what is going on. This may take a <br>
>>>> few iterations, and it will take me some time to produce the <br>
>>>> necessary debugging patch.<br>
>>><br>
>>> Unfortunately, I don't have a test case that will cause the problem <br>
>>> so I need to run this at a customer's production site that is <br>
>>> particularly affected by it. Luckily, the problem recurs pretty quickly.<br>
>>><br>
>>> Here's a run with 6.6 where the number of destinations drops from 2 <br>
>>> to 1 before reverting. Not seen this before - usually once it has <br>
>>> dropped to 1 (the IPv6 address), it stays there until a restart (and <br>
>>> this did happen about a minute after this log fragment). Happy to <br>
>>> test out any debugging patch.<br>
>>><br>
>>> 2024/01/10 11:55:49.849 kid1| 14,4| ipcache.cc(617) nbgethostbyname: <br>
>>> <a href="http://forcesafesearch.google.com" rel="noreferrer noreferrer" target="_blank">forcesafesearch.google.com</a><br>
>>> 2024/01/10 11:55:49.849 kid1| 14,3| Address.cc(389) lookupHostIP: <br>
>>> Given Non-IP '<a href="http://forcesafesearch.google.com" rel="noreferrer noreferrer" target="_blank">forcesafesearch.google.com</a>': hostname or servname not <br>
>>> provided or not known<br>
>>> 2024/01/10 11:55:49.849 kid1| 14,4| ipcache.cc(657) <br>
>>> ipcache_nbgethostbyname_: ipcache_nbgethostbyname: HIT for <br>
>>> '<a href="http://forcesafesearch.google.com" rel="noreferrer noreferrer" target="_blank">forcesafesearch.google.com</a>'<br>
>>> 2024/01/10 11:55:49.849 kid1| 14,7| ipcache.cc(253) forwardIp: <br>
>>> [2001:4860:4802:32::78]<br>
>>> 2024/01/10 11:55:49.849 kid1| 44,2| peer_select.cc(1174) handlePath: <br>
>>> PeerSelector300176 found conn2388484 local=[::] <br>
>>> remote=[2001:4860:4802:32::78]:443 HIER_DIRECT flags=1, destination <br>
>>> #1 for <a href="http://forcesafesearch.google.com:443" rel="noreferrer noreferrer" target="_blank">forcesafesearch.google.com:443</a><br>
>>> 2024/01/10 11:55:49.849 kid1| 44,2| peer_select.cc(1180) handlePath: <br>
>>> always_direct = ALLOWED<br>
>>> 2024/01/10 11:55:49.849 kid1| 44,2| peer_select.cc(1181) handlePath: <br>
>>> never_direct = DENIED<br>
>>> 2024/01/10 11:55:49.849 kid1| 44,2| peer_select.cc(1182) handlePath: <br>
>>> timedout = 0<br>
>>> 2024/01/10 11:55:49.849 kid1| 14,7| ipcache.cc(253) forwardIp: <br>
>>> 216.239.38.120<br>
>>> 2024/01/10 11:55:49.849 kid1| 44,2| peer_select.cc(1174) handlePath: <br>
>>> PeerSelector300176 found conn2388485 local=0.0.0.0 <br>
>>> remote=<a href="http://216.239.38.120:443" rel="noreferrer noreferrer" target="_blank">216.239.38.120:443</a> HIER_DIRECT flags=1, destination #2 for <br>
>>> <a href="http://forcesafesearch.google.com:443" rel="noreferrer noreferrer" target="_blank">forcesafesearch.google.com:443</a><br>
>>> 2024/01/10 11:55:49.849 kid1| 44,2| peer_select.cc(1180) handlePath: <br>
>>> always_direct = ALLOWED<br>
>>> 2024/01/10 11:55:49.849 kid1| 44,2| peer_select.cc(1181) handlePath: <br>
>>> never_direct = DENIED<br>
>>> 2024/01/10 11:55:49.849 kid1| 44,2| peer_select.cc(1182) handlePath: <br>
>>> timedout = 0<br>
>>> 2024/01/10 11:55:49.849 kid1| 14,7| ipcache.cc(236) finalCallback: <br>
>>> 0x12208e038<br>
>>> 2024/01/10 11:55:49.849 kid1| 44,2| peer_select.cc(479) <br>
>>> resolveSelected: PeerSelector300176 found all 2 destinations for <br>
>>> <a href="http://forcesafesearch.google.com:443" rel="noreferrer noreferrer" target="_blank">forcesafesearch.google.com:443</a><br>
>>> 2024/01/10 11:55:49.849 kid1| 44,2| peer_select.cc(480) <br>
>>> resolveSelected: always_direct = ALLOWED<br>
>>> 2024/01/10 11:55:49.849 kid1| 44,2| peer_select.cc(481) <br>
>>> resolveSelected: never_direct = DENIED<br>
>>> 2024/01/10 11:55:49.849 kid1| 44,2| peer_select.cc(482) <br>
>>> resolveSelected: timedout = 0<br>
>>> 2024/01/10 11:55:49.849 kid1| 14,7| ipcache.cc(990) have: <br>
>>> [2001:4860:4802:32::78]:443 at 0 in [2001:4860:4802:32::78] #2/2-0<br>
>>> 2024/01/10 11:55:49.849 kid1| 14,2| ipcache.cc(1031) markAsBad: <br>
>>> [2001:4860:4802:32::78]:443 of <a href="http://forcesafesearch.google.com" rel="noreferrer noreferrer" target="_blank">forcesafesearch.google.com</a><br>
>>> 2024/01/10 11:55:49.855 kid1| 14,7| ipcache.cc(990) have: <br>
>>> <a href="http://216.239.38.120:443" rel="noreferrer noreferrer" target="_blank">216.239.38.120:443</a> at 0 in [2001:4860:4802:32::78] #2/2-1<br>
>>> 2024/01/10 11:55:49.855 kid1| 14,2| ipcache.cc(1055) forgetMarking: <br>
>>> <a href="http://216.239.38.120:443" rel="noreferrer noreferrer" target="_blank">216.239.38.120:443</a> of <a href="http://forcesafesearch.google.com" rel="noreferrer noreferrer" target="_blank">forcesafesearch.google.com</a><br>
>>> 2024/01/10 11:55:49.877 kid1| 14,3| Address.cc(389) lookupHostIP: <br>
>>> Given Non-IP '<a href="http://forcesafesearch.google.com" rel="noreferrer noreferrer" target="_blank">forcesafesearch.google.com</a>': hostname or servname not <br>
>>> provided or not known<br>
>>> 2024/01/10 11:55:49.877 kid1| 44,2| peer_select.cc(460) <br>
>>> resolveSelected: Find IP destination for: <br>
>>> <a href="http://forcesafesearch.google.com:443" rel="noreferrer noreferrer" target="_blank">forcesafesearch.google.com:443</a>' via <a href="http://forcesafesearch.google.com" rel="noreferrer noreferrer" target="_blank">forcesafesearch.google.com</a><br>
>>> 2024/01/10 11:55:49.877 kid1| 14,4| ipcache.cc(617) nbgethostbyname: <br>
>>> <a href="http://forcesafesearch.google.com" rel="noreferrer noreferrer" target="_blank">forcesafesearch.google.com</a><br>
>>> 2024/01/10 11:55:49.877 kid1| 14,3| Address.cc(389) lookupHostIP: <br>
>>> Given Non-IP '<a href="http://forcesafesearch.google.com" rel="noreferrer noreferrer" target="_blank">forcesafesearch.google.com</a>': hostname or servname not <br>
>>> provided or not known<br>
>>> 2024/01/10 11:55:49.877 kid1| 14,4| ipcache.cc(657) <br>
>>> ipcache_nbgethostbyname_: ipcache_nbgethostbyname: HIT for <br>
>>> '<a href="http://forcesafesearch.google.com" rel="noreferrer noreferrer" target="_blank">forcesafesearch.google.com</a>'<br>
>>> 2024/01/10 11:55:49.877 kid1| 14,7| ipcache.cc(253) forwardIp: <br>
>>> [2001:4860:4802:32::78]<br>
>>> 2024/01/10 11:55:49.877 kid1| 44,2| peer_select.cc(1174) handlePath: <br>
>>> PeerSelector300177 found conn2388493 local=[::] <br>
>>> remote=[2001:4860:4802:32::78]:443 HIER_DIRECT flags=1, destination <br>
>>> #1 for <a href="http://forcesafesearch.google.com:443" rel="noreferrer noreferrer" target="_blank">forcesafesearch.google.com:443</a><br>
>>> 2024/01/10 11:55:49.877 kid1| 44,2| peer_select.cc(1180) handlePath: <br>
>>> always_direct = ALLOWED<br>
>>> 2024/01/10 11:55:49.877 kid1| 44,2| peer_select.cc(1181) handlePath: <br>
>>> never_direct = DENIED<br>
>>> 2024/01/10 11:55:49.877 kid1| 44,2| peer_select.cc(1182) handlePath: <br>
>>> timedout = 0<br>
>>> 2024/01/10 11:55:49.877 kid1| 14,7| ipcache.cc(253) forwardIp: <br>
>>> 216.239.38.120<br>
>>> 2024/01/10 11:55:49.877 kid1| 44,2| peer_select.cc(1174) handlePath: <br>
>>> PeerSelector300177 found conn2388494 local=0.0.0.0 <br>
>>> remote=<a href="http://216.239.38.120:443" rel="noreferrer noreferrer" target="_blank">216.239.38.120:443</a> HIER_DIRECT flags=1, destination #2 for <br>
>>> <a href="http://forcesafesearch.google.com:443" rel="noreferrer noreferrer" target="_blank">forcesafesearch.google.com:443</a><br>
>>> 2024/01/10 11:55:49.877 kid1| 44,2| peer_select.cc(1180) handlePath: <br>
>>> always_direct = ALLOWED<br>
>>> 2024/01/10 11:55:49.877 kid1| 44,2| peer_select.cc(1181) handlePath: <br>
>>> never_direct = DENIED<br>
>>> 2024/01/10 11:55:49.877 kid1| 44,2| peer_select.cc(1182) handlePath: <br>
>>> timedout = 0<br>
>>> 2024/01/10 11:55:49.877 kid1| 14,7| ipcache.cc(236) finalCallback: <br>
>>> 0x12208e038<br>
>>> 2024/01/10 11:55:49.877 kid1| 44,2| peer_select.cc(479) <br>
>>> resolveSelected: PeerSelector300177 found all 2 destinations for <br>
>>> <a href="http://forcesafesearch.google.com:443" rel="noreferrer noreferrer" target="_blank">forcesafesearch.google.com:443</a><br>
>>> 2024/01/10 11:55:49.877 kid1| 44,2| peer_select.cc(480) <br>
>>> resolveSelected: always_direct = ALLOWED<br>
>>> 2024/01/10 11:55:49.877 kid1| 44,2| peer_select.cc(481) <br>
>>> resolveSelected: never_direct = DENIED<br>
>>> 2024/01/10 11:55:49.877 kid1| 44,2| peer_select.cc(482) <br>
>>> resolveSelected: timedout = 0<br>
>>> 2024/01/10 11:55:49.877 kid1| 14,7| ipcache.cc(990) have: <br>
>>> [2001:4860:4802:32::78]:443 at 0 in [2001:4860:4802:32::78] #2/2-0<br>
>>> 2024/01/10 11:55:49.877 kid1| 14,2| ipcache.cc(1031) markAsBad: <br>
>>> [2001:4860:4802:32::78]:443 of <a href="http://forcesafesearch.google.com" rel="noreferrer noreferrer" target="_blank">forcesafesearch.google.com</a><br>
>>> 2024/01/10 11:55:49.882 kid1| 14,7| ipcache.cc(990) have: <br>
>>> <a href="http://216.239.38.120:443" rel="noreferrer noreferrer" target="_blank">216.239.38.120:443</a> at 0 in [2001:4860:4802:32::78] #2/2-1<br>
>>> 2024/01/10 11:55:49.882 kid1| 14,2| ipcache.cc(1055) forgetMarking: <br>
>>> <a href="http://216.239.38.120:443" rel="noreferrer noreferrer" target="_blank">216.239.38.120:443</a> of <a href="http://forcesafesearch.google.com" rel="noreferrer noreferrer" target="_blank">forcesafesearch.google.com</a><br>
>>><br>
>><br>
>> _______________________________________________<br>
>> squid-users mailing list<br>
>> <a href="mailto:squid-users@lists.squid-cache.org" target="_blank" rel="noreferrer">squid-users@lists.squid-cache.org</a><br>
>> <a href="https://lists.squid-cache.org/listinfo/squid-users" rel="noreferrer noreferrer" target="_blank">https://lists.squid-cache.org/listinfo/squid-users</a><br>
> <br>
> _______________________________________________<br>
> squid-users mailing list<br>
> <a href="mailto:squid-users@lists.squid-cache.org" target="_blank" rel="noreferrer">squid-users@lists.squid-cache.org</a><br>
> <a href="https://lists.squid-cache.org/listinfo/squid-users" rel="noreferrer noreferrer" target="_blank">https://lists.squid-cache.org/listinfo/squid-users</a><br>
> <br>
<br>
-- <br>
Dr. Stephen Borrill, Director and Solutions Architect<br>
Precedence Technologies Ltd T: +44 (0) 1223 359 900<br>
Technology House, 36a Union Lane E: <a href="mailto:sborrill@precedence.co.uk" target="_blank" rel="noreferrer">sborrill@precedence.co.uk</a><br>
Cambridge, CB4 1QB, United Kingdom W: <a href="http://www.precedence.co.uk/" rel="noreferrer noreferrer" target="_blank">http://www.precedence.co.uk/</a><br>
Limited company registered in England and Wales. Company number 3725626<br>
<br>
_______________________________________________<br>
squid-users mailing list<br>
<a href="mailto:squid-users@lists.squid-cache.org" target="_blank" rel="noreferrer">squid-users@lists.squid-cache.org</a><br>
<a href="https://lists.squid-cache.org/listinfo/squid-users" rel="noreferrer noreferrer" target="_blank">https://lists.squid-cache.org/listinfo/squid-users</a><br>
</blockquote></div></div></div>