[squid-users] tuning squid memory (aka avoiding the reaper)

[Alex Rousskov]

On 09/25/2017 05:23 PM, Aaron Turner wrote:
> So I'm testing squid 3.5.26 on an m3.xlarge w/ 14GB of RAM.  Squid is
> the only "real" service running (sshd and the like).  I'm running 4
> workers, and 2 rock cache.  The workers seem to be growing unbounded
> and given ~30min or so will cause the kernel to start killing off
> processes until memory is freed.  Yes, my clients (32 of them) are
> hitting this at about 250 URL's/min which doesn't seem that crazy, but
> ¯\_(ツ)_/¯
> 
> cache_mem 1 GB resulted in workers exceeding 4GB resident.  So I tried
> 500 MB, same problem.  Now I'm down to 250 MB and I'm still seeing
> workers using 3-4GB of RAM after a few minutes and still growing 

It is not the Squid memory cache that consumes your RAM, apparently.


> the docs indicate I should expect total memory to be roughly 3x cache_mem.

... which is an absurd formula for those using disk caches: Roughly
speaking, most large busy Squids spend most of their RAM on

* memory cache,
* disk cache indexes,
* SSL-related caches, and
* in-flight transactions.

Only one of those 4 components is proportional to cache_mem, with a
coefficient closer to 1 than to 3.


> mgr:info reports:

Thank you for posting this useful info. When you are using disk caching,
please also include the mgr:storedir report.


> I'm trying to figure out why and how to fix.

I recommend disabling all caching (memory and disk) and SslBump (if any)
to establish a baseline first. If everything looks stable and peachy for
a few hours, record/store the baseline measurements, and add one new
memory consumer (e.g., the memory cache). Ideally, this testing should
be done in a lab rather than on real users, but YMMV.


> One thing I've read about the cache_mem knob is:
> 
> "If circumstances require, this limit will be exceeded.
> 
> Specifically, if your incoming request rate requires more than
> 'cache_mem' of memory to hold in-transit objects, Squid will
> exceed this limit to satisfy the new requests.  When the load
> decreases, blocks will be freed until the high-water mark is
> reached.  Thereafter, blocks will be used to store hot
> objects."

The above is more-or-less accurate, but please note that in-transit
objects do not usually eat memory cache RAM in SMP mode. It is usually
best to think of in-flight transactions as a distinct SMP memory
consumer IMO.


> Not sure if this is the cause of my problem?

It could be -- it is difficult for me to say by looking at one random
mgr:info snapshot. If I have to guess based on that snapshot alone, then
my answer would be "no" because you have less than 4K concurrent
transactions and transaction response times are low. Hopefully somebody
else on the list can tell you more.



> The FAQ says try a different malloc, so tried recompiling with
> --enable-dlmalloc, but that had no impact.

Do not bother unless your deployment environment is very unusual. This
hint was helpful 20 years ago, but is rarely relevant these days AFAIK.
See above for a different attack plan.


HTH,

Alex.