[squid-users] IPv4 addresses go missing - markAsBad wrong?

[Stephen Borrill]

On 09/01/2024 03:41, Alex Rousskov wrote:
> On 2024-01-08 08:31, Stephen Borrill wrote:
>> I'm trying to determine why squid 6.x (seen with 6.5) connected via 
>> IPv4-only periodically fails to connect to the destination and then 
>> requires a restart to fix it (reload is not sufficient).
>>
>> The problem appears to be that a host that has one address each of 
>> IPv4 and IPv6 occasionally has its IPv4 address go missing as a 
>> destination. On closer inspection, this appears to happen when the 
>> IPv6 address (not the IPv4) address is marked as bad. A log fragment 
>> is as follows:
>>
>> 2024/01/08 13:18:39.974 kid1| 44,2| peer_select.cc(460) 
>> resolveSelected: Find IP destination for: 
>> clientservices.googleapis.com:443' via clientservices.googleapis.com
>> 2024/01/08 13:18:39.974 kid1| 44,2| peer_select.cc(1174) handlePath: 
>> PeerSelector82284 found conn696198 local=0.0.0.0 
>> remote=142.250.187.227:443 HIER_DIRECT flags=1, destination #1 for 
>> clientservices.googleapis.com:443
>> 2024/01/08 13:18:39.974 kid1| 44,2| peer_select.cc(1174) handlePath: 
>> PeerSelector82284 found conn696199 local=[::] 
>> remote=[2a00:1450:4009:820::2003]:443 HIER_DIRECT flags=1, destination 
>> #2 for clientservices.googleapis.com:443
>> 2024/01/08 13:18:39.974 kid1| 44,2| peer_select.cc(479) 
>> resolveSelected: PeerSelector82284 found all 2 destinations for 
>> clientservices.googleapis.com:443
>> 2024/01/08 13:18:40.245 kid1| 14,2| ipcache.cc(1031) markAsBad: 
>> [2a00:1450:4009:820::2003]:443 of clientservices.googleapis.com
>> 2024/01/08 13:18:40.245 kid1| 14,3| ipcache.cc(946) seekNewGood: 
>> succeeded for clientservices.googleapis.com: 
>> [2a00:1450:4009:820::2003] #2/2-1
>> 2024/01/08 13:18:40.245 kid1| 14,3| ipcache.cc(978) restoreGoodness: 
>> cleared all IPs for clientservices.googleapis.com; now back to 
>> [2a00:1450:4009:820::2003] #2/2-1
>> 2024/01/08 13:18:42.065 kid1| 14,3| Address.cc(389) lookupHostIP: 
>> Given Non-IP 'clientservices.googleapis.com': hostname or servname not 
>> provided or not known
>> 2024/01/08 13:18:42.065 kid1| 44,2| peer_select.cc(460) 
>> resolveSelected: Find IP destination for: 
>> clientservices.googleapis.com:443' via clientservices.googleapis.com
>> 2024/01/08 13:18:42.065 kid1| 14,3| Address.cc(389) lookupHostIP: 
>> Given Non-IP 'clientservices.googleapis.com': hostname or servname not 
>> provided or not known
>> 2024/01/08 13:18:42.065 kid1| 44,2| peer_select.cc(1174) handlePath: 
>> PeerSelector82372 found conn697148 local=[::] 
>> remote=[2a00:1450:4009:820::2003]:443 HIER_DIRECT flags=1, destination 
>> #1 for clientservices.googleapis.com:443
>> 2024/01/08 13:18:42.065 kid1| 44,2| peer_select.cc(479) 
>> resolveSelected: PeerSelector82372 found all 1 destinations for 
>> clientservices.googleapis.com:443
>>
>>
>> This shows two subsequent connection attempts to 
>> clientservices.googleapis.com. The first one has both IPv4 and IPv6 
>> destinations. The IPv6 address is passed to markAsBad. 
> 
> Yes.
> 
> 
>> After that the IPv4 address is not listed as a destination.
> 
> I do not see that. I see IPv6 address being selected as the first 
> destination (instead of the IPv4 address).

My reading of the the log files is that all possible destinations are 
listed, e.g. "found all 18 destinations for www.googleapis.com:443". 
Prior to markAsBad being called both IP addresses for 
clientservices.googleapis.com are shown, afterwards the IPv4 is not 
listed. It isn't just that IPv6 is being selected as the first 
destination, it is the ONLY destination as the IPv4 address is no longer 
a candidate.


> I cannot explain why that happens though. Moreover, a combination of 
> certain lines in your debug output near "seekNewGood" do not make sense 
> to me -- I do not see how it is possible for Squid to display those 
> exact debugging details, but I am probably missing something. Can you 
> retest and repost similar lines with 14,9 (or at least 14,7) added to 
> your debug_options (or share those lines privately; the more lines you 
> can share, the better)?

Thanks, I have started a new run with 14,7 and will pass it on privately 
unless I can distil out the relevant lines to be small enough for the list.

>> Note that there have been many connections to 
>> clientservices.googleapis.com prior to this where markAsBad was not 
>> called, even though IPv6 connectivity was never available.
> 
> No markAsBad() is probably normal if Squid did not try to establish an 
> IPv6 connection or did not wait long enough to know the result of that 
> attempt. However, that does not explain why Squid selected an IPv6 
> address as the next "good" address right after marking that IPv6 address 
> as bad (at "restoreGoodness" line) when there was another good IP 
> address available. It is as if Squid stored two identical IPv6 addresses 
> (and not IPv4 ones), but that should not happen either.

This is tangentially related to this thread too:
https://lists.squid-cache.org/pipermail/squid-users/2023-November/026266.html

Once only the IPv6 address is being used, then it returns 503 for that 
host and thus can quickly get marked as dead by a downstream squid 
meaning it does not get used at all (and if it's the only peer all 
access stops).

-- 
Stephen