<div dir="ltr"><div>Hello, </div><div><br></div>> FWIW, related future Squid improvements may include:<br><br>> * Detecting such shared memory segments clashes; refusing to start.<br>> * Disabling shared memory use when caching is completely disabled.<br><br>> Quality pull requests welcome.<br><div><br></div><div><span class="gmail-EzKURWReUAB5oZgtQNkl" style="white-space-collapse: preserve;">I</span><span style="white-space-collapse: preserve;"> </span><span class="gmail-EzKURWReUAB5oZgtQNkl" style="white-space-collapse: preserve;">looked</span><span style="white-space-collapse: preserve;"> at the </span><span class="gmail-EzKURWReUAB5oZgtQNkl" style="white-space-collapse: preserve;">source</span><span style="white-space-collapse: preserve;"> </span><span class="gmail-EzKURWReUAB5oZgtQNkl" style="white-space-collapse: preserve;">code</span><span style="white-space-collapse: preserve;"> </span><span class="gmail-EzKURWReUAB5oZgtQNkl" style="white-space-collapse: preserve;">and</span><span style="white-space-collapse: preserve;"> found the detection of </span><span style="white-space-collapse: preserve;">the presence </span><span style="white-space-collapse: preserve;">of old shared memory </span><span class="gmail-EzKURWReUAB5oZgtQNkl" style="white-space-collapse: preserve;">segments ("src/ipc/mem/Segment.cc"):</span></div><div><span class="gmail-EzKURWReUAB5oZgtQNkl" style="white-space-collapse: preserve;"><font face="monospace">void<br>Ipc::Mem::Segment::create(const off_t aSize)<br>{<br> assert(aSize > 0);<br> assert(theFD < 0);<br><br> int xerrno = 0;<br><br> // Why a brand new segment? A Squid crash may leave a reusable segment, but<br> // our placement-new code requires an all-0s segment. We could truncate and<br> // resize the old segment, but OS X does not allow using O_TRUNC with<br> // shm_open() and does not support ftruncate() for old segments.<br> if (!createFresh(xerrno) && xerrno == EEXIST) {<br> unlink();<br> createFresh(xerrno);<br> }</font><br></span></div><div><span class="gmail-EzKURWReUAB5oZgtQNkl" style="white-space-collapse: preserve;"><br></span></div><div><span class="gmail-EzKURWReUAB5oZgtQNkl" style="white-space-collapse: preserve;">
<span class="gmail-EzKURWReUAB5oZgtQNkl">But</span><span class="gmail-EzKURWReUAB5oZgtQNkl">,</span> <span class="gmail-EzKURWReUAB5oZgtQNkl">as</span> the <span class="gmail-EzKURWReUAB5oZgtQNkl">comment</span> <span class="gmail-EzKURWReUAB5oZgtQNkl">says</span><span class="gmail-EzKURWReUAB5oZgtQNkl">,</span> the <span class="gmail-EzKURWReUAB5oZgtQNkl">segments</span> <span class="gmail-EzKURWReUAB5oZgtQNkl">may</span> <span class="gmail-EzKURWReUAB5oZgtQNkl">remain</span> <span class="gmail-EzKURWReUAB5oZgtQNkl">from</span> the <span class="gmail-EzKURWReUAB5oZgtQNkl">previous</span> Squid <span class="gmail-EzKURWReUAB5oZgtQNkl">crash</span><span class="gmail-EzKURWReUAB5oZgtQNkl">.</span> <span class="gmail-EzKURWReUAB5oZgtQNkl">Thus</span><span class="gmail-EzKURWReUAB5oZgtQNkl">,</span> in my opinion, it is <span class="gmail-EzKURWReUAB5oZgtQNkl">impractical</span> to <span class="gmail-EzKURWReUAB5oZgtQNkl">refuse</span> to launch the <span class="gmail-EzKURWReUAB5oZgtQNkl">program</span><span class="gmail-EzKURWReUAB5oZgtQNkl">.</span> <span class="gmail-EzKURWReUAB5oZgtQNkl">I</span> <span class="gmail-EzKURWReUAB5oZgtQNkl">think</span> it<span class="gmail-EzKURWReUAB5oZgtQNkl">'s</span> <span class="gmail-EzKURWReUAB5oZgtQNkl">enough</span> to <span class="gmail-EzKURWReUAB5oZgtQNkl">put</span> a <span class="gmail-EzKURWReUAB5oZgtQNkl">warning</span> <span class="gmail-EzKURWReUAB5oZgtQNkl">in</span> the <span class="gmail-EzKURWReUAB5oZgtQNkl">code</span> <span class="gmail-EzKURWReUAB5oZgtQNkl">when</span> <span class="gmail-EzKURWReUAB5oZgtQNkl">detecting</span> old segments<span class="gmail-EzKURWReUAB5oZgtQNkl">.</span></span></div><div><span class="gmail-EzKURWReUAB5oZgtQNkl" style="white-space-collapse: preserve;">Something like that:</span></div><div><span class="gmail-EzKURWReUAB5oZgtQNkl" style="white-space-collapse: preserve;"><font face="monospace">void<br>Ipc::Mem::Segment::create(const off_t aSize)<br>{<br> assert(aSize > 0);<br> assert(theFD < 0);<br><br> int xerrno = 0;<br><br> // Why a brand new segment? A Squid crash may leave a reusable segment, but<br> // our placement-new code requires an all-0s segment. We could truncate and<br> // resize the old segment, but OS X does not allow using O_TRUNC with<br> // shm_open() and does not support ftruncate() for old segments.<br> if (!createFresh(xerrno) && xerrno == EEXIST) {<br> <b>debugs(54, DBG_CRITICAL, "WARNING: there is an old shared memory segment: '" << theName << "'. Perhaps it was left after the previous crash of the Squid. We will remove it. Or it may be a sign that another instance of the Squid is running. In this case, you must launch the program with the -n option and specify the unique name of the service.");</b><br> unlink();<br> createFresh(xerrno);<br> }<br></font>
Kind regards,</span></div><div><span class="gmail-EzKURWReUAB5oZgtQNkl" style="white-space-collapse: preserve;"> Ankor.</span></div><div><span class="gmail-EzKURWReUAB5oZgtQNkl" style="white-space-collapse: preserve;"><br></span></div></div><br><div class="gmail_quote gmail_quote_container"><div dir="ltr" class="gmail_attr">сб, 8 мар. 2025 г. в 08:14, Andrey K <<a href="mailto:ankor2023@gmail.com">ankor2023@gmail.com</a>>:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hello Alex,<div><br></div><div>Thanks for the analysis.</div><div><span>Squid</span><span> </span><span>only</span><span> </span><span>allows</span><span> </span><span>alphanumeric</span><span> </span><span>characters</span><span> </span><span>in</span><span> the </span><span>service</span><span> </span><span>name, </span><span>so</span><span> </span><span>it</span><span> </span><span>refuses</span><span> to </span><span>use</span><span> the </span><span>original</span><span> service </span><span>name</span><span> </span><span>in the <font face="monospace">-n</font> option (<font face="monospace">-n squid.user.service)</font></span><span><font face="arial, sans-serif">.</font></span></div><div>I added<font face="monospace"> -n squiduser</font> option to the ExecStart string of the second instance.</div><div>Now it looks good:</div><div><br></div><div><font face="monospace"># the first instance</font></div><div><font face="monospace">lsof -p 3746105 | grep shm<br>squid 3746105 root mem REG 0,23 525572 1213614834 /dev/shm/squid-cf__queues.shm<br>squid 3746105 root mem REG 0,23 136 1213614835 /dev/shm/squid-cf__readers.shm<br>squid 3746105 root mem REG 0,23 8 1213614833 /dev/shm/squid-cf__metadata.shm<br>squid 3746105 root 7u REG 0,23 8 1213614833 /dev/shm/squid-cf__metadata.shm<br>squid 3746105 root 8u REG 0,23 525572 1213614834 /dev/shm/squid-cf__queues.shm<br>squid 3746105 root 9u REG 0,23 136 1213614835 /dev/shm/squid-cf__readers.shm</font></div><div><font face="monospace"><br></font></div><div><font face="monospace"># the second instance<br>lsof -p 3685356 | grep shm<br>squid.use 3685356 root mem REG 0,23 2093368 1212704041 /dev/shm/squiduser-tls_session_cache.shm<br>squid.use 3685356 root mem REG 0,23 525572 1212704039 /dev/shm/squiduser-cf__queues.shm<br>squid.use 3685356 root mem REG 0,23 136 1212704040 /dev/shm/squiduser-cf__readers.shm<br>squid.use 3685356 root mem REG 0,23 8 1212704038 /dev/shm/squiduser-cf__metadata.shm<br>squid.use 3685356 root 6u REG 0,23 8 1212704038 /dev/shm/squiduser-cf__metadata.shm<br>squid.use 3685356 root 7u REG 0,23 525572 1212704039 /dev/shm/squiduser-cf__queues.shm<br>squid.use 3685356 root 8u REG 0,23 136 1212704040 /dev/shm/squiduser-cf__readers.shm<br>squid.use 3685356 root 9u REG 0,23 2093368 1212704041 /dev/shm/squiduser-tls_session_cache.shm</font><br></div><div><br></div><div>Kind regards,</div><div> Ankor.</div><div><span><br></span></div><div><br></div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">пт, 7 мар. 2025 г. в 17:48, Alex Rousskov <<a href="mailto:rousskov@measurement-factory.com" target="_blank">rousskov@measurement-factory.com</a>>:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On 2025-03-07 06:50, Andrey K wrote:<br>
<br>
> Squid Cache: Version 6.13<br>
> Service Name: squid<br>
<br>
> Squid Cache: Version 6.10<br>
> Service Name: squid<br>
<br>
> # the first instance<br>
> 1318 DEL ... 30205 /dev/shm/squid-cf__queues.shm<br>
> 1318 DEL ... 30206 /dev/shm/squid-cf__readers.shm<br>
> 1318 DEL ... 30204 /dev/shm/squid-cf__metadata.shm<br>
> 1318 8u ... 8 30204 /dev/shm/squid-cf__metadata.shm (deleted)<br>
> 1318 9u ... 525572 30205 /dev/shm/squid-cf__queues.shm (deleted)<br>
> 1318 10u ... 136 30206 /dev/shm/squid-cf__readers.shm (deleted)<br>
> <br>
> # the second instance<br>
> 1514 mem ... 2093368 31497 /dev/shm/squid-tls_session_cache.shm<br>
> 1514 mem ... 525572 31495 /dev/shm/squid-cf__queues.shm<br>
> 1514 mem ... 136 31496 /dev/shm/squid-cf__readers.shm<br>
> 1514 mem ... 8 31494 /dev/shm/squid-cf__metadata.shm<br>
> 1514 6u ... 8 31494 /dev/shm/squid-cf__metadata.shm<br>
> 1514 7u ... 525572 31495 /dev/shm/squid-cf__queues.shm<br>
> 1514 8u ... 136 31496 /dev/shm/squid-cf__readers.shm<br>
> 1514 9u ... 2093368 31497 /dev/shm/squid-tls_session_cache.shm<br>
<br>
As suspected, these two Squid instances use the same shared memory <br>
segments (e.g., /dev/shm/squid-cf*). Such shared use violates critical <br>
code assumptions and results in undefined behavior.<br>
<br>
<br>
> Maybe I'm not experiencing any difficulties because I have caching turned off on <br>
> both instances?<br>
<br>
Well, you _are_ experiencing at least one difficulty -- the assertion <br>
that started this email thread. If you have fully disabled caching, the <br>
difficulties you experience should not include cache corruption. <br>
However, it looks like at least one of the two instances does cache TLS <br>
sessions (in /dev/shm/squid-tls_session_cache.shm). If both instances do <br>
that, then all bets are off!<br>
<br>
<br>
FWIW, related future Squid improvements may include:<br>
<br>
* Detecting such shared memory segments clashes; refusing to start.<br>
* Disabling shared memory use when caching is completely disabled.<br>
<br>
Quality pull requests welcome.<br>
<br>
<br>
Cheers,<br>
<br>
Alex.<br>
<br>
<br>
> чт, 6 мар. 2025 г. в 17:11, Alex Rousskov:<br>
> <br>
> On 2025-03-06 08:59, Amos Jeffries wrote:<br>
> > On 6/03/25 19:17, Andrey K wrote:<br>
> >> Hello,<br>
> >><br>
> >> I have a similar configuration: two SMP squids running on the<br>
> same OEL<br>
> >> host.<br>
> >><br>
> >> They were built with different configurations: with different<br>
> >> installation path prefixes and different names of binary files:<br>
> squid<br>
> >> and squid.user and they listen to different ports.<br>
> >> They are launched from two different services:squid.service and<br>
> >> squid.user.service with the service Type=forking:<br>
> >><br>
> >> ExecStart=/usr/sbin/squid -sYC<br>
> >> ExecStart=/sbin/squid.user -f /etc/squid.user/squid.conf<br>
> >><br>
> >> I have not experienced any troubles with this configuration yet.<br>
> >><br>
> >> /> Please be aware that "squid -n ..." is a REQUIREMENT for running/<br>
> >> /multiple Squid instances on the same machine regardless of what<br>
> features<br>
> >> are used./<br>
> >><br>
> >> Could you please tell me if I should use the -n option in the<br>
> >> ExecStart strings?<br>
> >> The arguments of the options should be the service names?<br>
> >><br>
> >> ExecStart=/usr/sbin/squid -sYC -n squid.service<br>
> >> ExecStart=/sbin/squid.user -f /etc/squid.user/squid.conf -n<br>
> >> squid.user.service<br>
> >><br>
> > Yes you should. The different ./configure options has helped you<br>
> avoid<br>
> > major issues, but some may still appear.<br>
> <br>
> I agree. Moreover, I do not understand how your two SMP Squids could<br>
> work correctly without distinct service names because (on OEL) I would<br>
> expect them to share the same shared memory segments (which they must<br>
> not do to remain distinct instances).<br>
> <br>
> What is your Squid version? Can you tell how your Squids name their<br>
> shared memory segment "files"? For example, on some Linux OSes, those<br>
> segments could be in /var/run/shm/ with names like<br>
> squid-tr_map_anchors.shm and squid-tr_spaces.shm.<br>
> <br>
> <br>
> Thank you,<br>
> <br>
> Alex.<br>
> <br>
> _______________________________________________<br>
> squid-users mailing list<br>
> <a href="mailto:squid-users@lists.squid-cache.org" target="_blank">squid-users@lists.squid-cache.org</a><br>
> <mailto:<a href="mailto:squid-users@lists.squid-cache.org" target="_blank">squid-users@lists.squid-cache.org</a>><br>
> <a href="https://lists.squid-cache.org/listinfo/squid-users" rel="noreferrer" target="_blank">https://lists.squid-cache.org/listinfo/squid-users</a><br>
> <<a href="https://lists.squid-cache.org/listinfo/squid-users" rel="noreferrer" target="_blank">https://lists.squid-cache.org/listinfo/squid-users</a>><br>
> <br>
<br>
</blockquote></div>
</blockquote></div>