[squid-users] assertion failed: Queue.cc:388: "EX"
Alex Rousskov
rousskov at measurement-factory.com
Thu Mar 13 17:55:32 UTC 2025
On 2025-03-13 07:49, Andrey K wrote:
> I implemented the A-example functionality and made PR:
> https://github.com/squid-cache/squid/pull/2023
Thank you.
> ButthemodifiedSquid projectis notbeingbuilt ongithub,althoughthere are
> noproblemsduringthe localbuilds.
> Alex, Amos and Francesco, could you see what the problem is?
Just to close this mailing list thread: PR #2023 problems are being
handled on GitHub.
Alex.
> вт, 11 мар. 2025 г. в 17:40, Andrey K:
>
> Hello Alex,
>
> Thank you for sharing your thoughts.
> The A-example looks easier to implement and more reliable.
>
> Kind regards,
> Ankor.
>
> вт, 11 мар. 2025 г. в 17:12, Alex Rousskov
> <rousskov at measurement-factory.com
> <mailto:rousskov at measurement-factory.com>>:
>
> On 2025-03-10 23:56, Andrey K wrote:
>
> > > Alex: FWIW, related future Squid improvements may include:
> > > * Detecting such shared memory segments clashes; refusing
> to start.
> > > * Disabling shared memory use when caching is completely
> disabled.
>
> > But ... segments may remain from the previous
> > Squid crash. Thus, in my opinion, it is impractical to refuse
> to launch
> > the program.
>
> "Squid should overwrite segments left by a crashed Squid" does
> not imply
> that "detecting segment clashes among independent Squid
> instances is
> impractical".
>
> The following rough examples are _not_ implementation
> blueprints, but
> they illustrate the lack of the above implication:
>
> A. If Squid were to incorporate PID filename[^1] into segment
> names,
> then segments from different Squid instances will not clash.
>
> B. If Squid were to write its PID filename[^1] into a shared memory
> segment, then Squid would be able to detect that another running
> instance is using the same shared memory segment (while still
> overwriting segments left by an earlier crash).
>
>
> [^1]: Full Squid PID filename are already guaranteed to be
> unique across
> Squid instances that may overwrite each other segments because
> Squid
> refuses to start if another Squid instance is using its PID
> file. The
> only red flag I can think of here is esoteric chrooted
> instances, but I
> would not be surprised if those instances cannot share segments.
>
>
> > I think it's enough to put a warning in the code when
> detecting old
> > segments.
>
> I disagree that a level-0 warning during regular/uneventful
> startups is
> a good idea, especially if there will be one warning for every
> segment.
> We should try to address the actual problem (i.e. refuse to start a
> segment-clashing concurrent instance) rather than allowing
> clashes but
> adding a yet another warning about a usually benign event.
>
> Alex.
>
>
> > Something like that:
> > void
> > Ipc::Mem::Segment::create(const off_t aSize)
> > {
> > assert(aSize > 0);
> > assert(theFD < 0);
> >
> > int xerrno = 0;
> >
> > // Why a brand new segment? A Squid crash may leave a
> reusable
> > segment, but
> > // our placement-new code requires an all-0s segment. We
> could
> > truncate and
> > // resize the old segment, but OS X does not allow using
> O_TRUNC with
> > // shm_open() and does not support ftruncate() for old
> segments.
> > if (!createFresh(xerrno) && xerrno == EEXIST) {
> > *debugs(54, DBG_CRITICAL, "WARNING: there is an old shared
> memory
> > segment: '" << theName << "'. Perhaps it was left after the
> previous
> > crash of the Squid. We will remove it. Or it may be a sign
> that another
> > instance of the Squid is running. In this case, you must
> launch the
> > program with the -n option and specify the unique name of the
> service.");*
> > unlink();
> > createFresh(xerrno);
> > }
> > Kind regards,
> > Ankor.
> >
> >
> > сб, 8 мар. 2025 г. в 08:14, Andrey K <ankor2023 at gmail.com
> <mailto:ankor2023 at gmail.com>
> > <mailto:ankor2023 at gmail.com <mailto:ankor2023 at gmail.com>>>:
> >
> > Hello Alex,
> >
> > Thanks for the analysis.
> > Squidonlyallowsalphanumericcharactersinthe servicename,
> > soitrefusesto usethe originalservice namein the -n option (-n
> > squid.user.service).
> > I added-n squiduser option to the ExecStart string of the
> second
> > instance.
> > Now it looks good:
> >
> > # the first instance
> > lsof -p 3746105 | grep shm
> > squid 3746105 root mem REG 0,23 525572
> > 1213614834 /dev/shm/squid-cf__queues.shm
> > squid 3746105 root mem REG 0,23 136
> > 1213614835 /dev/shm/squid-cf__readers.shm
> > squid 3746105 root mem REG 0,23 8
> > 1213614833 /dev/shm/squid-cf__metadata.shm
> > squid 3746105 root 7u REG 0,23 8
> > 1213614833 /dev/shm/squid-cf__metadata.shm
> > squid 3746105 root 8u REG 0,23 525572
> > 1213614834 /dev/shm/squid-cf__queues.shm
> > squid 3746105 root 9u REG 0,23 136
> > 1213614835 /dev/shm/squid-cf__readers.shm
> >
> > # the second instance
> > lsof -p 3685356 | grep shm
> > squid.use 3685356 root mem REG 0,23
> 2093368
> > 1212704041 /dev/shm/squiduser-tls_session_cache.shm
> > squid.use 3685356 root mem REG 0,23
> 525572
> > 1212704039 /dev/shm/squiduser-cf__queues.shm
> > squid.use 3685356 root mem REG 0,23
> 136
> > 1212704040 /dev/shm/squiduser-cf__readers.shm
> > squid.use 3685356 root mem REG 0,23
> 8
> > 1212704038 /dev/shm/squiduser-cf__metadata.shm
> > squid.use 3685356 root 6u REG 0,23
> 8
> > 1212704038 /dev/shm/squiduser-cf__metadata.shm
> > squid.use 3685356 root 7u REG 0,23
> 525572
> > 1212704039 /dev/shm/squiduser-cf__queues.shm
> > squid.use 3685356 root 8u REG 0,23
> 136
> > 1212704040 /dev/shm/squiduser-cf__readers.shm
> > squid.use 3685356 root 9u REG 0,23
> 2093368
> > 1212704041 /dev/shm/squiduser-tls_session_cache.shm
> >
> > Kind regards,
> > Ankor.
> >
> >
> >
> >
> >
> > пт, 7 мар. 2025 г. в 17:48, Alex Rousskov
> > <rousskov at measurement-factory.com
> <mailto:rousskov at measurement-factory.com>
> > <mailto:rousskov at measurement-factory.com
> <mailto:rousskov at measurement-factory.com>>>:
> >
> > On 2025-03-07 06:50, Andrey K wrote:
> >
> > > Squid Cache: Version 6.13
> > > Service Name: squid
> >
> > > Squid Cache: Version 6.10
> > > Service Name: squid
> >
> > > # the first instance
> > > 1318 DEL ... 30205
> /dev/shm/squid-cf__queues.shm
> > > 1318 DEL ... 30206
> /dev/shm/squid-cf__readers.shm
> > > 1318 DEL ... 30204
> /dev/shm/squid-cf__metadata.shm
> > > 1318 8u ... 8 30204
> /dev/shm/squid-cf__metadata.shm
> > (deleted)
> > > 1318 9u ... 525572 30205
> /dev/shm/squid-cf__queues.shm
> > (deleted)
> > > 1318 10u ... 136 30206
> /dev/shm/squid-cf__readers.shm
> > (deleted)
> > >
> > > # the second instance
> > > 1514 mem ... 2093368 31497
> > /dev/shm/squid-tls_session_cache.shm
> > > 1514 mem ... 525572 31495
> /dev/shm/squid-cf__queues.shm
> > > 1514 mem ... 136 31496
> /dev/shm/squid-cf__readers.shm
> > > 1514 mem ... 8 31494
> /dev/shm/squid-cf__metadata.shm
> > > 1514 6u ... 8 31494
> /dev/shm/squid-cf__metadata.shm
> > > 1514 7u ... 525572 31495
> /dev/shm/squid-cf__queues.shm
> > > 1514 8u ... 136 31496
> /dev/shm/squid-cf__readers.shm
> > > 1514 9u ... 2093368 31497
> > /dev/shm/squid-tls_session_cache.shm
> >
> > As suspected, these two Squid instances use the same
> shared memory
> > segments (e.g., /dev/shm/squid-cf*). Such shared use
> violates
> > critical
> > code assumptions and results in undefined behavior.
> >
> >
> > > Maybe I'm not experiencing any difficulties
> because I have
> > caching turned off on
> > > both instances?
> >
> > Well, you _are_ experiencing at least one difficulty
> -- the
> > assertion
> > that started this email thread. If you have fully
> disabled
> > caching, the
> > difficulties you experience should not include cache
> corruption.
> > However, it looks like at least one of the two
> instances does
> > cache TLS
> > sessions (in /dev/shm/squid-tls_session_cache.shm).
> If both
> > instances do
> > that, then all bets are off!
> >
> >
> > FWIW, related future Squid improvements may include:
> >
> > * Detecting such shared memory segments clashes;
> refusing to start.
> > * Disabling shared memory use when caching is
> completely disabled.
> >
> > Quality pull requests welcome.
> >
> >
> > Cheers,
> >
> > Alex.
> >
> >
> > > чт, 6 мар. 2025 г. в 17:11, Alex Rousskov:
> > >
> > > On 2025-03-06 08:59, Amos Jeffries wrote:
> > > > On 6/03/25 19:17, Andrey K wrote:
> > > >> Hello,
> > > >>
> > > >> I have a similar configuration: two SMP squids
> > running on the
> > > same OEL
> > > >> host.
> > > >>
> > > >> They were built with different
> configurations: with
> > different
> > > >> installation path prefixes and
> different names of
> > binary files:
> > > squid
> > > >> and squid.user and they listen to
> different ports.
> > > >> They are launched from two different
> > services:squid.service and
> > > >> squid.user.service with the service
> Type=forking:
> > > >>
> > > >> ExecStart=/usr/sbin/squid -sYC
> > > >> ExecStart=/sbin/squid.user -f
> > /etc/squid.user/squid.conf
> > > >>
> > > >> I have not experienced any troubles with this
> > configuration yet.
> > > >>
> > > >> /> Please be aware that "squid -n ..." is a
> > REQUIREMENT for running/
> > > >> /multiple Squid instances on the same machine
> > regardless of what
> > > features
> > > >> are used./
> > > >>
> > > >> Could you please tell me if I should use
> the -n
> > option in the
> > > >> ExecStart strings?
> > > >> The arguments of the options should be the
> service names?
> > > >>
> > > >> ExecStart=/usr/sbin/squid -sYC -n
> squid.service
> > > >> ExecStart=/sbin/squid.user -f
> > /etc/squid.user/squid.conf -n
> > > >> squid.user.service
> > > >>
> > > > Yes you should. The different ./configure
> options has
> > helped you
> > > avoid
> > > > major issues, but some may still appear.
> > >
> > > I agree. Moreover, I do not understand how
> your two SMP
> > Squids could
> > > work correctly without distinct service names
> because (on
> > OEL) I would
> > > expect them to share the same shared memory
> segments
> > (which they must
> > > not do to remain distinct instances).
> > >
> > > What is your Squid version? Can you tell how
> your Squids
> > name their
> > > shared memory segment "files"? For example, on
> some Linux
> > OSes, those
> > > segments could be in /var/run/shm/ with names like
> > > squid-tr_map_anchors.shm and squid-tr_spaces.shm.
> > >
> > >
> > > Thank you,
> > >
> > > Alex.
> > >
> > > _______________________________________________
> > > squid-users mailing list
> > > squid-users at lists.squid-cache.org
> <mailto:squid-users at lists.squid-cache.org>
> > <mailto:squid-users at lists.squid-cache.org
> <mailto:squid-users at lists.squid-cache.org>>
> > > <mailto:squid-users at lists.squid-cache.org
> <mailto:squid-users at lists.squid-cache.org>
> > <mailto:squid-users at lists.squid-cache.org
> <mailto:squid-users at lists.squid-cache.org>>>
> > > https://lists.squid-cache.org/listinfo/squid-users
> <https://lists.squid-cache.org/listinfo/squid-users>
> > <https://lists.squid-cache.org/listinfo/squid-users
> <https://lists.squid-cache.org/listinfo/squid-users>>
> > >
> <https://lists.squid-cache.org/listinfo/squid-users
> <https://lists.squid-cache.org/listinfo/squid-users>
> > <https://lists.squid-cache.org/listinfo/squid-users
> <https://lists.squid-cache.org/listinfo/squid-users>>>
> > >
> >
>
> _______________________________________________
> squid-users mailing list
> squid-users at lists.squid-cache.org
> <mailto:squid-users at lists.squid-cache.org>
> https://lists.squid-cache.org/listinfo/squid-users
> <https://lists.squid-cache.org/listinfo/squid-users>
>
More information about the squid-users
mailing list