[squid-users] High memory usage under load with caching disabled, memory is not being freed even with no load

NgTech LTD ngtech1ltd at gmail.com
Wed Aug 5 23:06:32 UTC 2020 

Hey Ivan,

>From what i remember there is a calculation for how much k per conn should
squid use.
another thing is that squid is not returning memory once ot took it.
Amos knows about this and might be able to respond.

Eliezer

On Wed, Aug 5, 2020, 20:34 Ivan Bulatovic <ivan.bulatovic at gmail.com> wrote:

> Hi Eliezer,
>
> In the original message I sent to the squid-users mail list, I
> attached listings from mgr:info and mgr:mem. The server is definitely
> using a lot of connections (close to 200K connections), which is why I
> increased the open files limits in linux as well as in squid.conf. And
> there are a lot of requests per second, when the server is running. I
> could understand that it needs a lot of memory for in-transit cache,
> but that memory should be later released  back to OS, once the
> requests load goes down. However, that is not the case, even days
> after there is no load on the server, it stays at 11GB of RAM and 5.5
> GB of swap used. The second I restart the squid process, everything
> goes to normal, memory is released. That is why I suspect there is
> some memory leak somewhere.
>
> The server is a Ubuntu 18.04 LTS VM (running on Hyper-V 2019 server),
> with 8 virtual processors and 12GB of RAM (although I can increase
> that if that is the problem, but I thought that without caching this
> would be more than enough).
>
> I am not using dynamic memory on Hyper-V (it is turned off for this VM).
>
> Best regards,
> Ivan
>
> On Wed, Aug 5, 2020 at 7:14 PM NgTech LTD <ngtech1ltd at gmail.com> wrote:
> >
> > I think that the mgr:info or another page there contains the amount of
> requests per second etc.
> > also netstat or ss -ntp might give some basic understanding about this
> server size.
> >
> > are you using dynamic memory on the hyper-v hypervisor?
> >
> > Eliezer
> >
> > On Wed, Aug 5, 2020, 19:59 Ivan Bulatovic <ivan.bulatovic at gmail.com>
> wrote:
> >>
> >> Hi Alex,
> >>
> >> Thank you very much for your help.
> >>
> >> I opened a bug on bugs.squid-cache.org
> >> (https://bugs.squid-cache.org/show_bug.cgi?id=5071).
> >>
> >> Best regards,
> >> Ivan
> >>
> >> On Mon, Aug 3, 2020 at 10:02 PM Alex Rousskov
> >> <rousskov at measurement-factory.com> wrote:
> >> >
> >> > On 8/3/20 9:11 AM, Ivan Bulatovic wrote:
> >> >
> >> > > Looks like squid has some serious memory issues when under heavy
> load
> >> > > (90 servers that crawl Internet sites).
> >> >
> >> > >         Maximum Resident Size: 41500720 KB
> >> >
> >> > If the above (unreliable) report matches your observations using
> system
> >> > tools like "top", then it is indeed likely that your Squid is
> suffering
> >> > from a memory leak -- 41GB is usually too much for most non-caching
> >> > Squid instances.
> >> >
> >> > Identifying the leak may take some time, and I am not volunteering to
> do
> >> > the necessary legwork personally, but the Squid Project does fix
> >> > virtually all runtime leaks that we know about. If you want to speed
> up
> >> > the process, one of the best things you can do is to run Squid under
> >> > valgrind with a good suppression file. This requires building Squid
> with
> >> > a special ./configure option. Several testing iterations may be
> >> > necessary. If you are willing to do this, please file a bug report and
> >> > somebody will guide you through the steps.
> >> >
> >> >
> >> > > It just eats up memory, and
> >> > > does not free it up even days after it is being used (with no load
> on
> >> > > the proxy for days).
> >> >
> >> > Some memory retention is expected by default. See
> >> > http://www.squid-cache.org/Doc/config/memory_pools/
> >> >
> >> > Unfortunately, AFAICT, your mgr:mem output does not show any obvious
> >> > leaks -- all numbers are very small. If something is leaking a lot,
> then
> >> > it is probably not pooled by Squid.
> >> >
> >> >
> >> > HTH,
> >> >
> >> > Alex.
> >> >
> >> >
> >> > > On Mon, Jul 20, 2020 at 10:46 PM Ivan Bulatovic wrote:
> >> > >>
> >> > >> Hi all,
> >> > >>
> >> > >> I am trying to configure squid to run as a forward proxy with no
> >> > >> caching (cache deny all) with an option to choose the outgoing IP
> >> > >> address based on the username. So all squid has to do is to use a
> >> > >> certain outgoing IP address for a certain user, return the data
> from
> >> > >> the server to that user and cache nothing.
> >> > >>
> >> > >> For that I created a special authentication helper and used the
> ACLs
> >> > >> and tcp_outgoing_address to create a lot of users and outgoing IP
> >> > >> addresses (about 260 at the moment). Example (not the real IP I
> use,
> >> > >> of course):
> >> > >>
> >> > >> acl use_IP1 proxy_auth user1
> >> > >> tcp_outgoing_address 1.2.3.4   use_IP1
> >> > >>
> >> > >> I also configured the squid to use 4 workers, but this happens even
> >> > >> when I use only one worker (default)
> >> > >>
> >> > >> And this works. However, under heavy load, Squid eats all of the
> RAM
> >> > >> and then starts going to swap. And the memory usage does not drop
> when
> >> > >> I remove all the load from squid (I shut down all clients).
> >> > >>
> >> > >> I left it to see if the memory will be freed but even after
> leaving it
> >> > >> for an hour the info page reports this:
> >> > >> Cache information for squid:
> >> > >>         Hits as % of all requests:      5min: 0.0%, 60min: 0.0%
> >> > >>         Hits as % of bytes sent:        5min: 0.0%, 60min: 1.1%
> >> > >>         Memory hits as % of hit requests:       5min: 0.0%, 60min:
> 0.0%
> >> > >>         Disk hits as % of hit requests: 5min: 0.0%, 60min: 100.0%
> >> > >>         Storage Swap size:      0 KB
> >> > >>         Storage Swap capacity:   0.0% used, 100.0% free
> >> > >>         Storage Mem size:       0 KB
> >> > >>         Storage Mem capacity:    0.0% used, 100.0% free
> >> > >>         Mean Object Size:       0.00 KB
> >> > >>         Requests given to unlinkd:      0
> >> > >>
> >> > >> Resource usage for squid:
> >> > >>         UP Time:        255334.875 seconds
> >> > >>         CPU Time:       7122.436 seconds
> >> > >>         CPU Usage:      2.79%
> >> > >>         CPU Usage, 5 minute avg:        0.05%
> >> > >>         CPU Usage, 60 minute avg:       37.66%
> >> > >>         Maximum Resident Size: 41500720 KB
> >> > >>         Page faults with physical i/o: 1003410
> >> > >>
> >> > >> And here is the listing of free and top commands (with no load on
> the server):
> >> > >>
> >> > >> # free -h
> >> > >>               total        used        free      shared
> buff/cache   available
> >> > >> Mem:            11G         10G        791M        676K
> 491M        1.0G
> >> > >> Swap:           11G        5.5G        6.5G
> >> > >>
> >> > >> # top
> >> > >> top - 14:12:32 up 3 days,  1:30,  1 user,  load average: 0.00,
> 0.00, 0.00
> >> > >> Tasks: 177 total,   1 running, 102 sleeping,   0 stopped,   0
> zombie
> >> > >> %Cpu0  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,
> 0.0 si,  0.0 st
> >> > >> %Cpu1  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,
> 0.0 si,  0.0 st
> >> > >> %Cpu2  :  0.0 us,  0.3 sy,  0.0 ni, 99.7 id,  0.0 wa,  0.0 hi,
> 0.0 si,  0.0 st
> >> > >> %Cpu3  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,
> 0.0 si,  0.0 st
> >> > >> %Cpu4  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,
> 0.0 si,  0.0 st
> >> > >> %Cpu5  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,
> 0.0 si,  0.0 st
> >> > >> %Cpu6  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,
> 0.0 si,  0.0 st
> >> > >> %Cpu7  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,
> 0.0 si,  0.0 st
> >> > >> KiB Mem : 91.2/12251688
> >> > >>
> [|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
> >> > >>         ]
> >> > >> KiB Swap: 45.8/12582904
> >> > >> [||||||||||||||||||||||||||||||||||||||||||||||
> >> > >>                               ]
> >> > >>
> >> > >>    PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM
>  TIME+ COMMAND
> >> > >>   7851 proxy     20   0 6946872 2.514g   8084 S   0.0 21.5
> 29:43.74 squid
> >> > >>   7832 proxy     20   0 6711480 2.464g   8040 S   0.0 21.1
> 29:58.17 squid
> >> > >>   7814 proxy     20   0 6834928 2.454g  10024 S   0.0 21.0
> 29:47.56 squid
> >> > >>   7843 proxy     20   0 6906252 2.436g   8208 S   0.0 20.8
> 29:15.60 squid
> >> > >>   1329 root      20   0 2416672 167272  12680 S   0.0  1.4
> 136:18.57 metricbeat
> >> > >>   1321 root      20   0 1831804  48364  11648 S   0.0  0.4
> 14:32.10 filebeat
> >> > >>    474 root      19  -1  127796  17576  17144 S   0.0  0.1
>  0:27.01
> >> > >> systemd-journal
> >> > >>   7811 proxy     20   0  549384  14168   8372 S   0.0  0.1
>  0:20.87 squid
> >> > >>   1166 root      20   0 1749724  10596   4468 S   0.0  0.1
>  0:31.83 snapd
> >> > >>  43940 proxy     20   0   28884   9608   5384 S   0.0  0.1
>  0:00.14 python3
> >> > >>  43941 proxy     20   0   28884   9552   5328 S   0.0  0.1
>  0:00.10 python3
> >> > >>  43939 proxy     20   0   28884   9524   5308 S   0.0  0.1
>  0:00.12 python3
> >> > >>  43938 proxy     20   0   28884   9452   5232 S   0.0  0.1
>  0:00.16 python3
> >> > >>  48848 root      20   0  105688   6960   5968 S   0.0  0.1
>  0:00.02 sshd
> >> > >>  48974 janitor   20   0  108120   5380   4372 S   0.0  0.0
>  0:00.00 sshd
> >> > >>      1 root      20   0   86360   4364   2488 S   0.0  0.0
> 32:46.22 systemd
> >> > >> ...
> >> > >> ... lines ommited
> >> > >> ...
> >> > >>
> >> > >> In the attachment you can find the printout from squidclient
> mgr:info
> >> > >> and squidclient mgr:mem. These are both taken at the moment when
> there
> >> > >> is no more load on the proxy. I also included my squid.conf file
> >> > >> (minus the two files where acls are defined and outgoing IP
> addresses,
> >> > >> these two contain only acl and tcp_outgoing_address lines as in the
> >> > >> example above).
> >> > >>
> >> > >> Machine info:
> >> > >> OS: Ubuntu 18.04 with latest updates
> >> > >> Squid version 4.12 (from diladele repository)
> >> > >> Hardware: Hyper-V virtual machine with 8 vCPU, 12GB of RAM
> >> > >>
> >> > >> I can not understand what is eating all of the memory, if I
> disabled the cache.
> >> > >>
> >> > >> Maybe I configured something wrong but I can not find what.
> >> > >>
> >> > >> Thank you for any help you can provide.
> >> > >>
> >> > >> Best regards,
> >> > >> Ivan
> >> > > _______________________________________________
> >> > > squid-users mailing list
> >> > > squid-users at lists.squid-cache.org
> >> > > http://lists.squid-cache.org/listinfo/squid-users
> >> > >
> >> >
> >> _______________________________________________
> >> squid-users mailing list
> >> squid-users at lists.squid-cache.org
> >> http://lists.squid-cache.org/listinfo/squid-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20200806/0515ad3c/attachment-0001.htm>

More information about the squid-users mailing list