[squid-users] High memory usage under load with caching disabled, memory is not being freed even with no load

[Amos Jeffries]

On 6/08/20 11:06 am, NgTech LTD wrote:
> Hey Ivan,
> 
> From what i remember there is a calculation for how much k per conn
> should squid use.

Aye;
 256KB * number of currently open FD
 + read_ahead_gap
 + received size of current in-transit response (if cacheable MISS)


> another thing is that squid is not returning memory once ot took it.

The calculation for this is _minimum_ 5MB per type of memory allocating
object is retained by Squid for quick re-use. The mgr:mem report lists
details of those allocations.



Alex didn't mention this earlier but what I am seeing in your "top" tool
output is that there are 5x 'squid' processes running. It looks like 4
of them are SMP worker or disker processes each using 2.5GB of RAM.

The "free" tool is confirming this with its report of "used: 10G" (4x
2.5GB) of memory actually being used on the machine.

Most kernels fork() implementation is terrible with virtual memory
calculations. Most of that number will never actually be used. So they
can be ignored so long as the per-process number does not exceed the
actual physical RAM installed (beyond that kernel refuses to spawn with
fork()).
 The numbers your tools are reporting are kind of reasonable - maximum
about 7GB *per process* allocated.


The 41GB "resident size" is from old memory allocation APIs in the
kernel which suffer from 32-bit issues. When this value has odd numbers
and/or conflicts with the system tools - believe the tools instead.



So to summarize; what I am seeing there is that during *Peak* load times
your proxy workers (combined) are *maybe* using up to 41GB of memory. At
the off-peak time you are doing your analysis reports they have dropped
down to 10GB.
 With one data point there is no sign of a memory leak happening. Just a
normal machine handling far more peak traffic than its available amount
of memory can cope with.

 That is not to rule out a leak entirely. More measurements over time
may show a pattern of increasing off-peak memory allocated. But just one
comparison of peak vs off-peak is not going to reveal that type of pattern.


Amos