[squid-users] Too many ERROR: Collapsed forwarding queue overflow for kid2 at 1024 items

[Loučanský Lukáš]

I agree - it seems I'm catching red herrings here. Restarted squid yesterday at about 11:30. Everything seemed ok until 8am today - when this assert was again raised: (note - it's another from these in version 5)

2021/11/23 08:00:15 kid2| assertion failed: store.cc:1094: "store_status == STORE_PENDING"

then kid2 restarted itself and after another 9mins the "collapsed forwarding queue overflow" message began filling my log file.

Besides this there was one "unable to allocate 64kB mem" error

Nov 23 06:27:01 proxy-gw (squid-1): FATAL: xmalloc: Unable to allocate 65535 bytes!#012#012    current master transaction: master54
Nov 23 06:27:01 proxy-gw squid[20213]: Squid Parent: squid-1 process 20221 exited due to signal 6 with status 0
Nov 23 06:27:01 proxy-gw squid[20213]: Squid Parent: (squid-1) process 24763 started

But it does not trigger the 1024 items overflow error.

Nov 23 08:00:16 proxy-gw squid[20213]: Squid Parent: squid-2 process 20220 exited due to signal 6 with status 0
Nov 23 08:00:16 proxy-gw squid[20213]: Squid Parent: (squid-2) process 25372 started

Is it possible that after kid worker restarts it stops synchronizing or sending messages via this queues? ie. like the new one is unable to process or read the messages intended for the old one? In on of the old mails I've posted  list from the cachemgr showing transient queues being filled - which I don't see if the queue overflow messageis not logged. But it showed more items even for kid1 receiving from kid1 and kid1 sending to kid1 - so why would it stop sending it to itself? As I'm a laymen in this field (can not see the whole picture or how it works) I'm not currently able to tell how to induce such behaviour or speculate why it's filling. So far I've recompiled squid with disabled optimizations and enabled backtraces - let's see what it does while it crashes again....

LL 

-----Original Message-----
From: Alex Rousskov [mailto:rousskov at measurement-factory.com] 
Sent: Monday, November 22, 2021 5:35 PM
To: Loučanský Lukáš; squid-users at lists.squid-cache.org
Subject: Re: [squid-users] Too many ERROR: Collapsed forwarding queue overflow for kid2 at 1024 items

On 11/22/21 3:59 AM, Loučanský Lukáš wrote:

> I'm running Squid Object Cache: Version 6.0.0-20211116-r90086c5a8 for 
> about 14hrs now. I've noticed many new  "ERROR: Collapsed forwarding 
> queue overflow for kid2 at 1024 items" after about 11hrs runtime.

Ignore queue overflows (for now) because they are a known side effect of assertions.


> 2021/11/22 07:54:24 kid2| assertion failed: store.cc:1094: "store_status == STORE_PENDING"
> 2021/11/22 08:51:48 kid1| assertion failed: store.cc:1094: "store_status == STORE_PENDING"

I recommend filing a bug report with Squid Bugzilla and providing a stack trace ("bt full") from this assertion. If possible, please keep the core dump for further analysis.


Thank you,

Alex.



> -----Original Message-----
> From: Alex Rousskov [mailto:rousskov at measurement-factory.com]
> Sent: Sunday, November 21, 2021 8:28 PM
> To: Loučanský Lukáš; squid-users at lists.squid-cache.org
> Subject: Re: [squid-users] Too many ERROR: Collapsed forwarding queue 
> overflow for kid2 at 1024 items
> 
> On 11/21/21 2:08 PM, Loučanský Lukáš wrote:
>> Hello, I've replaced src/ipc/ReadWritelock.cc and ReadWriteLock.h 
>> with modified versions (with finalizeExclusive calls), but afeter 
>> some time I have got queue overflows agains.
> 
> After each bug fix, we have to restart the triage sequence from scratch:
> Any assertions, crashes, FATAL messages or similar things leading to kid deaths? If they are present, then they explain queue overflows.
> 
> 
>> So I've downloaded squid-6.0.0-20211116-r90086c5a8 - run it with my 
>> v5 configure and now I'm testing it. So far I get these:
>> 2021/11/21 19:05:06 kid1| BUG: missing ENTRY_REQUIRES_COLLAPSING for
> 
> Noted. If you cannot reproduce this bug at will, then I recommend focusing on other, easier-to-address problems first.
> 
> Alex.
>