<div dir="ltr"><div dir="ltr">Hello Alex,<div><br></div><div>Thanks for your reply!</div><div><br></div></div><div class="gmail_quote gmail_quote_container"><div dir="ltr" class="gmail_attr">On Mon, Mar 3, 2025 at 3:44 PM Alex Rousskov <<a href="mailto:rousskov@measurement-factory.com">rousskov@measurement-factory.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On 2025-03-02 15:47, Lubos Uhliarik wrote:<br>
<br>
<br>
> 2024/10/16 17:52:44 kid10| Adaptation support is off.<br>
> 2024/10/16 17:52:44 kid10| assertion failed: Queue.cc:388: "EX"<br>
<br>
Squid v6 has a few changes that affect SMP startup and shutdown <br>
sequences. Since you have ruled out upgrading to a supported version, I <br>
would start by turning on shared_memory_locking in hope that the problem <br>
lies in a problematic OS configuration.<br>
<br>
BTW, that bogus exception text "EX" is a bug. It was fixed in Squid v6. <br>
Squid v5.10 has a backport of that fix (commit 31f20fda).<br>
<br></blockquote><div><br></div><div><p>I was able to resolve the issue, so in case someone encounters the same backtrace, <br>here’s what happened: <br><br>In the end, we discovered that a customer was running two Squid instances on the same<br>machine—one SMP instance with 16 workers and one single-process instance. For a long <br>time, the single-process Squid instance would start first by chance, followed by <br>the SMP instance, which overwrote the shared memory. However, at some point, <br>the SMP instance started first, and when the single-process Squid instance started <br>second, it overwrote the SHM memory (resetting theProcessCount to 1) of the <br>SMP instance, causing it to crash with the mentioned backtrace.<br><br>The fix is simple—just specify a service name using the -n Squid parameter for <br>one of the Squid instances.<br></p></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Alex.<br>
<br>
<br>
> We have recently experienced Squid crashes when running Squid in SMP<br>
> mode with 16 workers (workers 16). While reviewing the Squid cache<br>
> logs, I noticed that assertion failures sometimes occurred (usually<br>
> during the shutdown process):<br>
> <br>
> 2024/10/16 17:52:44 kid10| Adaptation support is off.<br>
> 2024/10/16 17:52:44 kid10| assertion failed: Queue.cc:388: "EX"<br>
> 2024/10/16 17:52:44 kid7| Squid plugin modules loaded: 0<br>
> 2024/10/16 17:52:44 kid7| Adaptation support is off.<br>
> 2024/10/16 17:52:44 kid7| assertion failed: Queue.cc:388: "EX"<br>
> 2024/10/16 17:52:44 kid4| Finished loading MIME types and icons.<br>
> 2024/10/16 17:52:44 kid4| HTCP Disabled.<br>
> 2024/10/16 17:52:44 kid4| Squid plugin modules loaded: 0<br>
> 2024/10/16 17:52:44 kid4| Adaptation support is off.<br>
> 2024/10/16 17:52:44 kid4| assertion failed: Queue.cc:388: "EX"<br>
> 2024/10/16 17:52:44 kid3| Finished loading MIME types and icons.<br>
> 2024/10/16 17:52:44 kid2| Finished loading MIME types and icons.<br>
> 2024/10/16 17:52:44 kid5| Finished loading MIME types and icons.<br>
> 2024/10/16 17:52:45 kid3| HTCP Disabled.<br>
> 2024/10/16 17:52:45 kid2| HTCP Disabled.<br>
> 2024/10/16 17:52:45 kid5| HTCP Disabled.<br>
> 2024/10/16 17:52:45 kid5| Squid plugin modules loaded: 0<br>
> 2024/10/16 17:52:45 kid5| Adaptation support is off.<br>
> 2024/10/16 17:52:45 kid2| Squid plugin modules loaded: 0<br>
> 2024/10/16 17:52:45 kid2| Adaptation support is off.<br>
> 2024/10/16 17:52:45 kid3| Squid plugin modules loaded: 0<br>
> 2024/10/16 17:52:45 kid3| Adaptation support is off.<br>
> 2024/10/16 17:52:45 kid5| assertion failed: Queue.cc:388: "EX"<br>
> 2024/10/16 17:52:45 kid2| assertion failed: Queue.cc:388: "EX"<br>
> 2024/10/16 17:52:45 kid3| assertion failed: Queue.cc:388: "EX"<br>
> <br>
> I observed that each "kid" (worker) crashed once, and then Squid<br>
> successfully stopped and restarted.<br>
> <br>
> Recently, however, the Squid crashes started occurring during the<br>
> start process. The crashes now happen multiple times per worker,<br>
> causing Squid to enter a crash loop. This continues until the hard<br>
> drive fills up with core dumps:<br>
> <br>
> 2025/02/18 10:17:17 kid4| assertion failed: Queue.cc:388: "EX"<br>
> 2025/02/18 10:17:17 kid10| Current Directory is /<br>
> 2025/02/18 10:17:17 kid10| Starting Squid Cache version 5.5 for<br>
> x86_64-redhat-linux-gnu...<br>
> 2025/02/18 10:17:17 kid10| Service Name: squid<br>
> 2025/02/18 10:17:17 kid10| Process ID 18779<br>
> 2025/02/18 10:17:17 kid10| Process Roles: worker<br>
> 2025/02/18 10:17:17 kid10| With 65536 file descriptors available<br>
> 2025/02/18 10:17:17 kid10| Initializing IP Cache...<br>
> 2025/02/18 10:17:17 kid10| DNS Socket created at [::], FD 13<br>
> 2025/02/18 10:17:17 kid10| DNS Socket created at 0.0.0.0, FD 14<br>
> 2025/02/18 10:17:17 kid10| Adding nameserver 127.0.0.1 from squid.conf<br>
> 2025/02/18 10:17:17 kid10| Logfile: opening log /var/log/squid/access.log<br>
> 2025/02/18 10:17:17 kid10| WARNING: log name now starts with a module<br>
> name. Use 'stdio:/var/log/squid/access.log'<br>
> 2025/02/18 10:17:17 kid10| Local cache digest enabled; rebuild/rewrite<br>
> every 3600/3600 sec<br>
> 2025/02/18 10:17:17 kid10| Store logging disabled<br>
> 2025/02/18 10:17:17 kid10| Swap maxSize 0 + 262144 KB, estimated 20164 objects<br>
> 2025/02/18 10:17:17 kid10| Target number of buckets: 1008<br>
> 2025/02/18 10:17:17 kid10| Using 8192 Store buckets<br>
> 2025/02/18 10:17:17 kid10| Max Mem size: 262144 KB [shared]<br>
> 2025/02/18 10:17:17 kid10| Max Swap size: 0 KB<br>
> 2025/02/18 10:17:17 kid10| Using Least Load store dir selection<br>
> 2025/02/18 10:17:17 kid10| Current Directory is /<br>
> 2025/02/18 10:17:17 kid10| Finished loading MIME types and icons.<br>
> 2025/02/18 10:17:17 kid10| HTCP Disabled.<br>
> 2025/02/18 10:17:17 kid10| Squid plugin modules loaded: 0<br>
> 2025/02/18 10:17:17 kid10| Adaptation support is off.<br>
> 2025/02/18 10:17:17 kid10| assertion failed: Queue.cc:388: "EX"<br>
> 2025/02/18 10:17:17 kid7| Current Directory is /<br>
> <br>
> I investigated the core dump, and the code where Squid is crashing is<br>
> as follows:<br>
> <br>
> #0 0x00007f77b288ba6c in __pthread_kill_implementation () from /lib64/libc.so.6<br>
> Missing separate debuginfos, use: dnf debuginfo-install<br>
> glibc-2.34-125.el9_5.1.x86_64 keyutils-libs-1.6.3-1.el9.x86_64<br>
> krb5-libs-1.21.1-4.el9_5.x86_64 libcap-2.48-9.el9_2.x86_64<br>
> libcom_err-1.46.5-5.el9.x86_64 libecap-1.0.1-10.el9.x86_64<br>
> libgcc-11.5.0-2.el9.x86_64 libgcrypt-1.10.0-11.el9.x86_64<br>
> libgpg-error-1.42-5.el9.x86_64 libselinux-3.6-1.el9.x86_64<br>
> libstdc++-11.5.0-2.el9.x86_64 libtool-ltdl-2.4.6-46.el9.x86_64<br>
> libzstd-1.5.1-2.el9.x86_64 lz4-libs-1.9.3-5.el9.x86_64<br>
> openssl-libs-3.2.2-6.el9_5.x86_64 pcre2-10.40-6.el9.x86_64<br>
> sssd-client-2.9.5-4.el9_5.4.x86_64 systemd-libs-252-46.el9_5.2.x86_64<br>
> xz-libs-5.2.5-8.el9_0.x86_64 zlib-1.2.11-40.el9.x86_64<br>
> (gdb) bt<br>
> #0 0x00007f77b288ba6c in __pthread_kill_implementation () from /lib64/libc.so.6<br>
> #1 0x00007f77b283e686 in raise () from /lib64/libc.so.6<br>
> #2 0x00007f77b2828833 in abort () from /lib64/libc.so.6<br>
> #3 0x0000559bbb412a9d in xassert (msg=<optimized out>,<br>
> file=<optimized out>, line=<optimized out>) at<br>
> /usr/src/debug/squid-5.5-14.el9_5.3.x86_64/src/debug.cc:624<br>
> #4 0x0000559bbb7549f7 in Ipc::MultiQueue::reader (processId=3,<br>
> this=0x559bbccbf6e0) at ipc/Queue.cc:388<br>
> #5 Ipc::MultiQueue::localReader (this=0x559bbccbf6e0) at ipc/Queue.cc:408<br>
> #6 0x0000559bbb74c192 in Ipc::BaseMultiQueue::localReader<br>
> (this=<optimized out>) at ipc/Queue.cc:195<br>
> #7 Ipc::BaseMultiQueue::clearAllReaderSignals (this=<optimized out>)<br>
> at ipc/Queue.cc:155<br>
> #8 0x0000559bbb486724 in CollapsedForwarding::HandleNewDataAtStart ()<br>
> at /usr/src/debug/squid-5.5-14.el9_5.3.x86_64/src/CollapsedForwarding.cc:151<br>
> #9 0x0000559bbb6aa50b in AsyncCallQueue::fireNext<br>
> (this=this@entry=0x559bbccd18a0) at base/../../src/base/RefCount.h:73<br>
> #10 0x0000559bbb6aa89a in AsyncCallQueue::fire (this=0x559bbccd18a0)<br>
> at base/AsyncCallQueue.cc:43<br>
> #11 0x0000559bbb490da1 in EventLoop::dispatchCalls<br>
> (this=0x7fff2d084b90) at<br>
> /usr/src/debug/squid-5.5-14.el9_5.3.x86_64/src/EventLoop.cc:144<br>
> #12 EventLoop::runOnce (this=0x7fff2d084b90) at<br>
> /usr/src/debug/squid-5.5-14.el9_5.3.x86_64/src/EventLoop.cc:109<br>
> #13 0x0000559bbb5929a8 in EventLoop::run (this=0x7fff2d084b90) at<br>
> /usr/src/debug/squid-5.5-14.el9_5.3.x86_64/src/EventLoop.cc:83<br>
> #14 SquidMain (argc=argc@entry=5, argv=argv@entry=0x7fff2d084ea8) at<br>
> /usr/src/debug/squid-5.5-14.el9_5.3.x86_64/src/main.cc:1716<br>
> #15 0x0000559bbb436086 in SquidMainSafe (argv=0x7fff2d084ea8, argc=5)<br>
> at /usr/src/debug/squid-5.5-14.el9_5.3.x86_64/src/main.cc:1403<br>
> #16 main (argc=5, argv=0x7fff2d084ea8) at<br>
> /usr/src/debug/squid-5.5-14.el9_5.3.x86_64/src/main.cc:1391<br>
> (gdb) f 4<br>
> #4 0x0000559bbb7549f7 in Ipc::MultiQueue::reader (processId=3,<br>
> this=0x559bbccbf6e0) at ipc/Queue.cc:388<br>
> 388 assert(validProcessId(processId));<br>
> (gdb) list<br>
> 383 }<br>
> 384<br>
> 385 const Ipc::QueueReader &<br>
> 386 Ipc::MultiQueue::reader(const int processId) const<br>
> 387 {<br>
> 388 assert(validProcessId(processId));<br>
> 389 const int index = processId - metadata->theProcessIdOffset;<br>
> 390 return readers->theReaders[index];<br>
> 391 }<br>
> 392<br>
> (gdb) p processId<br>
> $1 = 3<br>
> (gdb) p metadata<br>
> $15 = {<RefCount<Ipc::Mem::Object<Ipc::MultiQueue::Metadata> >> = {p_<br>
> = 0x559bbccd9f30}, <No data fields>} (gdb) p *metadata.p_<br>
> $16 = {<Lock> = {_vptr.Lock = 0x559bbb96ea30 <vtable for<br>
> Ipc::Mem::Object<Ipc::MultiQueue::Metadata>+64>, count_ = 1},<br>
> _vptr.Object = 0x559bbb96ea08 <vtable for<br>
> Ipc::Mem::Object<Ipc::MultiQueue::Metadata>+24>, theSegment = {theFD =<br>
> 9, theName = {static npos = 18446744073709551615, size_ = 40, len_ =<br>
> 23,<br>
> static SizeMax_ = 196607, buf_ = 0x559bbccbc460<br>
> "/squid-cf__metadata.shm"}, theMem = 0x7f77b3720000, theSize = 8,<br>
> theReserved = 0, doUnlink = false}, theObject = 0x7f77b3720000}<br>
> (gdb) p metadata.p_->theObject<br>
> $17 = (Ipc::MultiQueue::Metadata *) 0x7f77b3720000<br>
> (gdb) p *metadata.p_->theObject<br>
> $18 = {theProcessCount = 0, theProcessIdOffset = 0}<br>
> (gdb) p metadata.p_->theObject->theProcessIdOffset<br>
> $19 = 0<br>
> <br>
> From this investigation, it appears that the theProcessCount and<br>
> theProcessIdOffset variables are set to 0. However, I don't think this<br>
> is actually the case, as these variables should be shared in SHM, and<br>
> the content of those files was not included in the core dump.<br>
> <br>
> Have any of you experienced similar crashes? I was trying to search<br>
> for a similar issue but without success.<br>
> <br>
> The backtrace above is from Squid version 5.5. While I understand it<br>
> is an older version, I believe the IPC code hasn’t changed much since<br>
> then. Unfortunately, we don't have a reproducer or the possibility of<br>
> upgrading Squid to a newer version on the running system for testing.<br>
> <br>
> It's also worth noting that after restarting Squid, which was stuck in<br>
> the crash loop, Squid started normally. I suspect it could be caused<br>
> by uninitialized SHM or something similar. What do you think?<br>
> <br>
> Would it help if I set some debug options and share the output with<br>
> you? I was considering something like:<br>
> <br>
> debug_options ALL,1 1,9 54,9<br>
> <br>
> Thanks in advance for your response.<br>
> <br>
> Regards,<br>
> Lubos<br>
> <br>
> _______________________________________________<br>
> squid-users mailing list<br>
> <a href="mailto:squid-users@lists.squid-cache.org" target="_blank">squid-users@lists.squid-cache.org</a><br>
> <a href="https://lists.squid-cache.org/listinfo/squid-users" rel="noreferrer" target="_blank">https://lists.squid-cache.org/listinfo/squid-users</a><br>
<br>
_______________________________________________<br>
squid-users mailing list<br>
<a href="mailto:squid-users@lists.squid-cache.org" target="_blank">squid-users@lists.squid-cache.org</a><br>
<a href="https://lists.squid-cache.org/listinfo/squid-users" rel="noreferrer" target="_blank">https://lists.squid-cache.org/listinfo/squid-users</a></blockquote><div><br></div><div>Lubos </div></div></div>