[squid-users] Too many ERROR: Collapsed forwarding queue overflow for kid2 at 1024 items

[Alex Rousskov]

On 11/16/21 4:38 AM, Loučanský Lukáš wrote:
> is it going to be patched only in the v6 version? 

I hope the existing fix applies to v5 cleanly, and I am ready to help
with backporting if it does not. Beyond that, it is in the maintainer
hands. I cannot predict whether or when the fix will be officially
merged into v5 because I do not understand how those decisions are made.


> Anyway - in the morning I run debug with 20,9 to see:
> ...
> 2021/11/16 09:02:06.496 kid2| assertion failed: Transients.cc:221: "old == e"

Unfortunately, I cannot see the cause of the assertion in this
short/partial trace -- the problematic actions happened before the trace
or were not logged during the trace.

Patching your Squid with commit 5210df4 is the best next step IMO. If
that patch does not help, then there are probably other bugs that we
need to fix in v5 (at least).


HTH,

Alex.


> -----Original Message-----
> From: Alex Rousskov [mailto:rousskov at measurement-factory.com] 
> Sent: Monday, November 15, 2021 5:17 PM
> To: Squid Users
> Cc: Loučanský Lukáš
> Subject: Re: [squid-users] Too many ERROR: Collapsed forwarding queue overflow for kid2 at 1024 items
> 
> On 11/15/21 7:43 AM, Loučanský Lukáš wrote:
> 
>> 2021/11/14 10:13:30 kid2| assertion failed: Transients.cc:221: "old == e"
>> 2021/11/15 08:37:36 kid2| assertion failed: Transients.cc:221: "old == e"
>> 2021/11/15 11:54:14 kid1| assertion failed: Transients.cc:221: "old == e"
>> 2021/11/15 12:16:27 kid1| assertion failed: Transients.cc:221: "old == e"
> 
> I recommend ignoring queue overflows until the above assertions are fixed because worker deaths cause queue overflows. Your Squid is buggy, and those bugs essentially cause queue overflows.
> 
> The assertion itself is known as Bug 5134:
> https://bugs.squid-cache.org/show_bug.cgi?id=5134
> 
> That bug has a speculative fix (master/v6 commit 5210df4). Please try it if you can.
> 
> 
> HTH,
> 
> Alex.
> 
> 
>> -----Original Message-----
>> From: Alex Rousskov [mailto:rousskov at measurement-factory.com]
>> Sent: Friday, November 12, 2021 5:24 PM
>> To: Loučanský Lukáš; squid-users at lists.squid-cache.org
>> Subject: Re: [squid-users] Too many ERROR: Collapsed forwarding queue 
>> overflow for kid2 at 1024 items
>>
>> On 11/11/21 10:19 AM, Loučanský Lukáš wrote:
>>
>>> recently I'm facing too many ERROR: Collapsed forwarding queue 
>>> overflow for kid2 at 1024 items lines in my Squid 5.2 log files.
>>
>> We see those overflows when kids die. Do you see any FATAL messages, assertions, or similar deadly errors in cache.log?
>>
>>
>>> Could someone elaborate how the queue is filled - what is clogging it?
>>
>> The sender/writer sends messages faster than the recipient/reader is 
>> reading them, eventually exceeding the queue capacity (i.e. 1024 
>> messages). These messages are about Store entries that may need 
>> synchronization across workers. Each message is very sm
> all.
>>
>>
>>> I don't mind too much if I have to turn collapsed forwarding off
>>
>> Most likely, the problem is not tied to collapsed forwarding. These 
>> queues were used for collapsed forwarding when they were added, but 
>> they are used for regular traffic as well in modern SMP Squids. We 
>> need to change the queue names (and related code/m
> essage text) to reflect the expanded nature of these queues.
>>
>>
>> HTH,
>>
>> Alex.
>>