[squid-users] High memory usage under load with caching disabled, memory is not being freed even with no load

Mon Aug 10 05:49:55 UTC 2020

It's 700+ connections per client... assuming it's only half ie 400+-
It's a lot.. for a ssl-bump proxy.
A simple tcp proxy can take it without stressing too much but TLS bump is another story.
Can you verify if there are sessions which are open more then 1 hour?

The basic suggestion for many proxies is to limit the time which a connection can stay open.
conntrack tools can help to identify idle connections on Ubuntu 18.04.

Eliezer

----
Eliezer Croitoru
Tech Support
Mobile: +972-5-28704261
Email: ngtech1ltd at gmail.com

-----Original Message-----
From: squid-users <squid-users-bounces at lists.squid-cache.org> On Behalf Of Ivan Bulatovic
Sent: Friday, August 7, 2020 8:30 PM
To: Amos Jeffries <squid3 at treenet.co.nz>
Cc: Squid Users <squid-users at lists.squid-cache.org>
Subject: Re: [squid-users] High memory usage under load with caching disabled, memory is not being freed even with no load

Hi Amos,

> > From what i remember there is a calculation for how much k per conn 
> > should squid use.
>
> Aye;
>  256KB * number of currently open FD
>  + read_ahead_gap
>  + received size of current in-transit response (if cacheable MISS)

I tried to reduce the number of in-memory objects using following (I read somewhere that 4KB is the minimum block of memory that squid can
allocate):
  cache_mem 8 MB
  maximum_object_size 4 KB
  maximum_object_size_in_memory 4 KB
  read_ahead_gap 4 KB

But the above settings did not help much.

At the moment I am running a much lighter load on the squid VM to see how it behaves.

So, right now, the machine has about 110.000 open TCP connections (I guess half are from clients and the other half is to the Internet, which my firewall also confirms). It has been running like this for the last 4 hours or so.

Here is the situation (in the attachment you will find full command print-outs and config file):

- Running squid 4.12 from diladele repository on Ubuntu 18.04 LTS

- RAM used: around 9 GB out of 16 GB (no of swap is used)

- I am running 2 squid workers at the moment (see attached squid.conf)

- Top reports this (removed other processes, they practically have no impact on the memory listing):

top - 10:22:03 up 18:00,  1 user,  load average: 1.88, 1.58, 1.43
Tasks: 169 total,   1 running,  94 sleeping,   0 stopped,   0 zombie
%Cpu(s): 13.6 us,  9.6 sy,  0.0 ni, 72.7 id,  0.1 wa,  0.0 hi,  3.9 si,  0.0 st KiB Mem : 16380456 total,  5011560 free,  9205660 used,  2163236 buff/cache
KiB Swap: 12582904 total, 12582904 free,        0 used.  7121124 avail Mem

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
  1515 proxy     20   0 5713808 4.396g  14188 S  36.5 28.1 130:03.34 squid
  1514 proxy     20   0 4360348 3.329g  14380 S  28.9 21.3 104:15.21 squid

 - mgr:info show some weird stats:
    Number of clients accessing cache:      156   (which is exactly
twice the number of actual clients, but this is probably due to the number of workers)
    Maximum Resident Size: 32467680 KB   (which is 32GB. At no time
during these 4 hours has this value of RAM consumption ever been reached. The memory is steadily, but slowly increasing to where it is now - at 9 GB. I have no idea what this value is.)

I have no idea if the rest of the stats from mgr:info are OK or not, I really have no way of checking that.

I added to the configuration memory pools option, we will see if it helps (I think I already tried this, but I can not be sure, I ran a lot of tests trying to fix this myself before I reached out to you):
memory_pools off

If there is anything else I can do to help with debugging this, please let me know.

Thank you for your time and help,
Ivan

On Fri, Aug 7, 2020 at 2:23 AM Amos Jeffries <squid3 at treenet.co.nz> wrote:
>
> On 6/08/20 11:06 am, NgTech LTD wrote:
> > Hey Ivan,
> >
> > From what i remember there is a calculation for how much k per conn 
> > should squid use.
>
> Aye;
>  256KB * number of currently open FD
>  + read_ahead_gap
>  + received size of current in-transit response (if cacheable MISS)
>
>
> > another thing is that squid is not returning memory once ot took it.
>
> The calculation for this is _minimum_ 5MB per type of memory 
> allocating object is retained by Squid for quick re-use. The mgr:mem 
> report lists details of those allocations.
>
>
>
> Alex didn't mention this earlier but what I am seeing in your "top" 
> tool output is that there are 5x 'squid' processes running. It looks 
> like 4 of them are SMP worker or disker processes each using 2.5GB of RAM.
>
> The "free" tool is confirming this with its report of "used: 10G" (4x
> 2.5GB) of memory actually being used on the machine.
>
> Most kernels fork() implementation is terrible with virtual memory 
> calculations. Most of that number will never actually be used. So they 
> can be ignored so long as the per-process number does not exceed the 
> actual physical RAM installed (beyond that kernel refuses to spawn 
> with fork()).
>  The numbers your tools are reporting are kind of reasonable - maximum 
> about 7GB *per process* allocated.
>
>
> The 41GB "resident size" is from old memory allocation APIs in the 
> kernel which suffer from 32-bit issues. When this value has odd 
> numbers and/or conflicts with the system tools - believe the tools instead.
>
>
>
> So to summarize; what I am seeing there is that during *Peak* load 
> times your proxy workers (combined) are *maybe* using up to 41GB of 
> memory. At the off-peak time you are doing your analysis reports they 
> have dropped down to 10GB.
>  With one data point there is no sign of a memory leak happening. Just 
> a normal machine handling far more peak traffic than its available 
> amount of memory can cope with.
>
>  That is not to rule out a leak entirely. More measurements over time 
> may show a pattern of increasing off-peak memory allocated. But just 
> one comparison of peak vs off-peak is not going to reveal that type of pattern.
>
>
> Amos
> _______________________________________________
> squid-users mailing list
> squid-users at lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users