[squid-users] shared_memory_locking failed to mlock

[Gordon Hsiao]

Turns out it is a ulimit-related issue, I bumped default mlock limit to a
large value and now I can start squid with memory-locked.

yes strace is only for syscalls, while ltrace shows all library calls.

Thanks for the help!

Gordon

On Mon, Jul 16, 2018 at 6:38 PM Alex Rousskov <
rousskov at measurement-factory.com> wrote:

> On 07/16/2018 05:08 PM, Gordon Hsiao wrote:
> > On a x86/64bit ubuntu machine if I set 'workers 4' and run:
>
> > squid --foreground -f /etc/squid.conf 2>&1 |grep mlock
> >   mlock(0x7f2e5bfb2000, 8)                = 0
> >   mlock(0x7f2e5bf9f000, 73912)            = -1 ENOMEM
>
> > squid -N -f /etc/squid.conf 2>& |grep mlock
> >   mlock(0x7f8e4b7c0000, 8)                = 0
> >   mlock(0x7f8e4b7ad000, 73912)            = -1 ENOMEM
>
> > Note 1; -N and --foreground made no difference as long as 'workers 4' is
> > set, I was expecting -N will ignore "worker 4", does it?
>
> IIRC, -N does not start workers. However, some (memory allocation) code
> may not honor -N and still allocate memory necessary for those (disabled
> by -N) workers. That would be a bug AFAICT.
>
>
> > Now I set 'workers 2' and run the same two commands above and I got the
> > output(both are the same), which means squid started successfully:
> >   mlock(0x7f0c441cc000, 8)                = 0
> >   mlock(0x7f0c441c3000, 32852)            = 0
> >   mlock(0x7f0c441c2000, 52)               = 0
>
> The second allocation is probably smaller because two workers need fewer
> SMP queues (or similar shared memory resources) than four workers.
>
>
> > I have more than 4GB RAM free(this is a 8GB RAM laptop) and this
> > is a Intel i7, the mlock failure is strange.
>
> The default amount of shared memory available to a program is often much
> smaller than the total amount of RAM. I do not recall which Ubuntu
> commands or sysctl settings control the former, but Squid wiki or other
> web resources should have that info. The question you should ask
> yourself is "How much shared memory is available for the Squid process"?
>
>
> > On my target system which has 512MB RAM, even 'workers 0' won't help, I
> > still get :
> >
> >   mlock(0x778de000, 2101212)              = -1 ENOMEM (Out of memory)
>
> For "workers 0" concerns, please see the -N discussion above. The two
> should be equivalent.
>
>
> > I have to disable lock-memory for now and it puzzles me why the very
> > first 2MB mlock can fail.
>
> Most likely, your OS is configured (or defaults) to provide very little
> shared memory to a process when the total RAM is only 512MB.
>
>
> > I strace|grep shm_get and shmat and found nothing,
>
> mlock() is a system call so strace should see it, but it may be called
> something else.
>
>
> > instead there are lots of mmap calls, so Squid is using mmap
> > for its shared memory mapping,
>
> Squid creates segments using shm_open() and attaches to them using mmap().
>
>
> > the only question is that, is this mlock
> > file-backed-up or is it anonymous mmaped(in this case on Linux it will
> > use /dev/shm by default)?
>
> On Ubuntu, Squid shared memory segments should all be in /dev/shm by
> default. Squid does not want them to be backed by real files. See
> shm_open(3).
>
> Please note that some libc calls manipulating regular files are
> translated into mmap() calls by the standard library (or some such). Not
> all mmap() calls you see in strace are Squid mmap() calls.
>
>
> HTH,
>
> Alex.
>
>
> > On Mon, Jul 16, 2018 at 11:58 AM Alex Rousskov wrote:
> >
> >     On 07/15/2018 08:47 PM, Gordon Hsiao wrote:
> >     > Just upgraded squid to 4.1, however if I enabled
> >     shared_memory_locking I
> >     > failed to start squid:
> >     >
> >     > "FATAL: shared_memory_locking on but failed to
> >     > mlock(/squid-tls_session_cache.shm, 2101212): (12) Out of memory"
> >
> >     > How do I know how much memory it is trying to mlock? is
> 2101212(~2MB)
> >     > the shm size of not,
> >
> >     Yes, Squid tried to lock a 2101212-byte segment and failed.
> >
> >
> >     > any way to debug/looking-into/config this size?
> >
> >     I am not sure what you mean, but please keep in mind that the failed
> >     segment could be the last straw -- most of the shared memory could be
> >     allocated earlier. You can observe all allocations/locks with 54,7
> >     debugging. Look for "mlock(".
> >
> >     You can also run "strace" or a similar command line tool to track
> >     allocations, but analyzing strace output may be more difficult than
> >     looking through Squid logs.
> >
> >
> >     > Again I disabled cache etc for a memory restricted environment,
> also
> >     > used the minimal configuration with a few enable-flags, in the
> >     meantime
> >     > I want to avoid memory overcommit from squid(thus mlock)
> >
> >     I am glad the new code is working to prevent runtime crashes in your
> >     memory-restricted environment. If studying previous mlock() calls
> does
> >     not help, please suggest what else Squid could do not help you.
> >
> >
> >     Thank you,
> >
> >     Alex.
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20180717/b24e62f2/attachment-0001.html>