[squid-dev] [RFC] Fix shared memory initialization, cleanup. Ensure its usability.

[Alex Rousskov]

Hello,

    The attached patch does four things:

1. Fixes shared memory initialization. The OS X portability fix (Bug
3805) is a gift that keeps on giving: After we fixed OS X compilation,
we stopped initializing previously used shared memory after crashes!
Search the patch for "truncate".

2. Fixes shared memory cleanup. One segment was not deleted when running
Squid in non-daemon mode (-N). Search the patch for "UsingSmp".

3. Makes sure using shared memory does not lead to SIGBUS crashes.
Search the patch for "mlock".

4. Adds tests to check whether shared memory is usable. Search the patch
for "memory checks" and "shouldTest".


My plan is:

* Commit #1 and #2 fixes to trunk without review unless somebody
requests one now: IMO, they are simple and non-controversial.

* Remove/forget #4. These paranoid checks should not be needed after #3,
which they have helped to test. We can always add them later if I am wrong.

* Discuss mlock() changes in #3 (below). Post the corresponding change
for the proper review. The code adding the mlock(2) call itself is
simple, but the side effects of this change are serious enough to
warrant proper review IMO.


Questions: Should we add a configuration directive to control whether
mlock(2) is called? If yes, should mlock(2) be called by default?
Should mlock(2) failures be fatal?


My answers are three YESes: I propose to call mlock(2) by default (if
available) to guarantee that mmapped memory is usable. I also propose to
make mlock(2) failures fatal. I hesitate offering a configuration option
to control this behavior, but I think we should offer it because mlock()
causes startup delays. I will discuss the problem and explain my
rationale below.

On Linux (at least), mmap(2) often does not allocate much memory.
Instead, the kernel tries to allocate memory when it is actually
accessed for the first time by the program. What happens when the shared
memory is not available at that time? Kernel kills Squid with SIGBUS,
and developers spend many days trying to find a Squid bug.

Technically, it is the responsibility of the admin to make sure the
Squid box has enough shared memory for the configured caches.  However,
it is difficult to figure out how much is "enough" _and_ correctly
configure the OS to have that much shared memory. Mistakes are very
common, especially for larger memory caches. Such mistakes go completely
unnoticed for many hours or days (as the memory cache gets filled) so
they often slip through pre-deployment tests, and the resulting SIGBUS
crashes are often too obscure to point to OS misconfiguration as the
true cause. Sometimes they are even disguised as segmentation faults.

The problem is so bad that we must call mlock() by default and make
mlock() failures fatal, even if this will break some "working" setups
that just did not happen to hit their shared memory limits yet (e.g.,
because they configured Squid to have a memory cache size larger than
Squid can ever fill in their environment).

mlock(2) is also a performance optimization as it prevents future paging
I/O. The memory is allocated and paged in at startup.

Unfortunately, calling mlock(2) introduces a startup delay because the
kernel has to prepare RAM for immediate use. A 10GB shared memory cache
incurs a ~6 second delay. A 100GB shared memory cache delays listening
by ~60 seconds.

AFAICT, this large delay is the only valid argument for making mlock()
calls optional -- if I am sure that Squid has enough RAM, then I might
want to trade one large startup delay for numerous tiny delays as the
mmapped memory gets allocated and used.

Needless to say, if we make mlock() calls optional, then some admins
will disable them without giving Squid enough RAM and will then complain
about mysterious crashes. On the other hand, all other factors being
equal, we should provide tools for knowledgeable admins even if those
tools may be misused by others.

How would you answer the three questions above?


Thank you,

Alex.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: SQUID-129-validate-shared-memory-bag14-t4.patch
Type: text/x-diff
Size: 19989 bytes
Desc: not available
URL: <http://lists.squid-cache.org/pipermail/squid-dev/attachments/20151209/f0b80f63/attachment-0001.patch>