[squid-users] High memory usage under load with caching disabled, memory is not being freed even with no load
NgTech LTD
ngtech1ltd at gmail.com
Wed Aug 5 17:14:23 UTC 2020
I think that the mgr:info or another page there contains the amount of
requests per second etc.
also netstat or ss -ntp might give some basic understanding about this
server size.
are you using dynamic memory on the hyper-v hypervisor?
Eliezer
On Wed, Aug 5, 2020, 19:59 Ivan Bulatovic <ivan.bulatovic at gmail.com> wrote:
> Hi Alex,
>
> Thank you very much for your help.
>
> I opened a bug on bugs.squid-cache.org
> (https://bugs.squid-cache.org/show_bug.cgi?id=5071).
>
> Best regards,
> Ivan
>
> On Mon, Aug 3, 2020 at 10:02 PM Alex Rousskov
> <rousskov at measurement-factory.com> wrote:
> >
> > On 8/3/20 9:11 AM, Ivan Bulatovic wrote:
> >
> > > Looks like squid has some serious memory issues when under heavy load
> > > (90 servers that crawl Internet sites).
> >
> > > Maximum Resident Size: 41500720 KB
> >
> > If the above (unreliable) report matches your observations using system
> > tools like "top", then it is indeed likely that your Squid is suffering
> > from a memory leak -- 41GB is usually too much for most non-caching
> > Squid instances.
> >
> > Identifying the leak may take some time, and I am not volunteering to do
> > the necessary legwork personally, but the Squid Project does fix
> > virtually all runtime leaks that we know about. If you want to speed up
> > the process, one of the best things you can do is to run Squid under
> > valgrind with a good suppression file. This requires building Squid with
> > a special ./configure option. Several testing iterations may be
> > necessary. If you are willing to do this, please file a bug report and
> > somebody will guide you through the steps.
> >
> >
> > > It just eats up memory, and
> > > does not free it up even days after it is being used (with no load on
> > > the proxy for days).
> >
> > Some memory retention is expected by default. See
> > http://www.squid-cache.org/Doc/config/memory_pools/
> >
> > Unfortunately, AFAICT, your mgr:mem output does not show any obvious
> > leaks -- all numbers are very small. If something is leaking a lot, then
> > it is probably not pooled by Squid.
> >
> >
> > HTH,
> >
> > Alex.
> >
> >
> > > On Mon, Jul 20, 2020 at 10:46 PM Ivan Bulatovic wrote:
> > >>
> > >> Hi all,
> > >>
> > >> I am trying to configure squid to run as a forward proxy with no
> > >> caching (cache deny all) with an option to choose the outgoing IP
> > >> address based on the username. So all squid has to do is to use a
> > >> certain outgoing IP address for a certain user, return the data from
> > >> the server to that user and cache nothing.
> > >>
> > >> For that I created a special authentication helper and used the ACLs
> > >> and tcp_outgoing_address to create a lot of users and outgoing IP
> > >> addresses (about 260 at the moment). Example (not the real IP I use,
> > >> of course):
> > >>
> > >> acl use_IP1 proxy_auth user1
> > >> tcp_outgoing_address 1.2.3.4 use_IP1
> > >>
> > >> I also configured the squid to use 4 workers, but this happens even
> > >> when I use only one worker (default)
> > >>
> > >> And this works. However, under heavy load, Squid eats all of the RAM
> > >> and then starts going to swap. And the memory usage does not drop when
> > >> I remove all the load from squid (I shut down all clients).
> > >>
> > >> I left it to see if the memory will be freed but even after leaving it
> > >> for an hour the info page reports this:
> > >> Cache information for squid:
> > >> Hits as % of all requests: 5min: 0.0%, 60min: 0.0%
> > >> Hits as % of bytes sent: 5min: 0.0%, 60min: 1.1%
> > >> Memory hits as % of hit requests: 5min: 0.0%, 60min:
> 0.0%
> > >> Disk hits as % of hit requests: 5min: 0.0%, 60min: 100.0%
> > >> Storage Swap size: 0 KB
> > >> Storage Swap capacity: 0.0% used, 100.0% free
> > >> Storage Mem size: 0 KB
> > >> Storage Mem capacity: 0.0% used, 100.0% free
> > >> Mean Object Size: 0.00 KB
> > >> Requests given to unlinkd: 0
> > >>
> > >> Resource usage for squid:
> > >> UP Time: 255334.875 seconds
> > >> CPU Time: 7122.436 seconds
> > >> CPU Usage: 2.79%
> > >> CPU Usage, 5 minute avg: 0.05%
> > >> CPU Usage, 60 minute avg: 37.66%
> > >> Maximum Resident Size: 41500720 KB
> > >> Page faults with physical i/o: 1003410
> > >>
> > >> And here is the listing of free and top commands (with no load on the
> server):
> > >>
> > >> # free -h
> > >> total used free shared buff/cache
> available
> > >> Mem: 11G 10G 791M 676K 491M
> 1.0G
> > >> Swap: 11G 5.5G 6.5G
> > >>
> > >> # top
> > >> top - 14:12:32 up 3 days, 1:30, 1 user, load average: 0.00, 0.00,
> 0.00
> > >> Tasks: 177 total, 1 running, 102 sleeping, 0 stopped, 0 zombie
> > >> %Cpu0 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0
> si, 0.0 st
> > >> %Cpu1 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0
> si, 0.0 st
> > >> %Cpu2 : 0.0 us, 0.3 sy, 0.0 ni, 99.7 id, 0.0 wa, 0.0 hi, 0.0
> si, 0.0 st
> > >> %Cpu3 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0
> si, 0.0 st
> > >> %Cpu4 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0
> si, 0.0 st
> > >> %Cpu5 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0
> si, 0.0 st
> > >> %Cpu6 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0
> si, 0.0 st
> > >> %Cpu7 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0
> si, 0.0 st
> > >> KiB Mem : 91.2/12251688
> > >>
> [|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
> > >> ]
> > >> KiB Swap: 45.8/12582904
> > >> [||||||||||||||||||||||||||||||||||||||||||||||
> > >> ]
> > >>
> > >> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
> COMMAND
> > >> 7851 proxy 20 0 6946872 2.514g 8084 S 0.0 21.5 29:43.74
> squid
> > >> 7832 proxy 20 0 6711480 2.464g 8040 S 0.0 21.1 29:58.17
> squid
> > >> 7814 proxy 20 0 6834928 2.454g 10024 S 0.0 21.0 29:47.56
> squid
> > >> 7843 proxy 20 0 6906252 2.436g 8208 S 0.0 20.8 29:15.60
> squid
> > >> 1329 root 20 0 2416672 167272 12680 S 0.0 1.4 136:18.57
> metricbeat
> > >> 1321 root 20 0 1831804 48364 11648 S 0.0 0.4 14:32.10
> filebeat
> > >> 474 root 19 -1 127796 17576 17144 S 0.0 0.1 0:27.01
> > >> systemd-journal
> > >> 7811 proxy 20 0 549384 14168 8372 S 0.0 0.1 0:20.87
> squid
> > >> 1166 root 20 0 1749724 10596 4468 S 0.0 0.1 0:31.83
> snapd
> > >> 43940 proxy 20 0 28884 9608 5384 S 0.0 0.1 0:00.14
> python3
> > >> 43941 proxy 20 0 28884 9552 5328 S 0.0 0.1 0:00.10
> python3
> > >> 43939 proxy 20 0 28884 9524 5308 S 0.0 0.1 0:00.12
> python3
> > >> 43938 proxy 20 0 28884 9452 5232 S 0.0 0.1 0:00.16
> python3
> > >> 48848 root 20 0 105688 6960 5968 S 0.0 0.1 0:00.02
> sshd
> > >> 48974 janitor 20 0 108120 5380 4372 S 0.0 0.0 0:00.00
> sshd
> > >> 1 root 20 0 86360 4364 2488 S 0.0 0.0 32:46.22
> systemd
> > >> ...
> > >> ... lines ommited
> > >> ...
> > >>
> > >> In the attachment you can find the printout from squidclient mgr:info
> > >> and squidclient mgr:mem. These are both taken at the moment when there
> > >> is no more load on the proxy. I also included my squid.conf file
> > >> (minus the two files where acls are defined and outgoing IP addresses,
> > >> these two contain only acl and tcp_outgoing_address lines as in the
> > >> example above).
> > >>
> > >> Machine info:
> > >> OS: Ubuntu 18.04 with latest updates
> > >> Squid version 4.12 (from diladele repository)
> > >> Hardware: Hyper-V virtual machine with 8 vCPU, 12GB of RAM
> > >>
> > >> I can not understand what is eating all of the memory, if I disabled
> the cache.
> > >>
> > >> Maybe I configured something wrong but I can not find what.
> > >>
> > >> Thank you for any help you can provide.
> > >>
> > >> Best regards,
> > >> Ivan
> > > _______________________________________________
> > > squid-users mailing list
> > > squid-users at lists.squid-cache.org
> > > http://lists.squid-cache.org/listinfo/squid-users
> > >
> >
> _______________________________________________
> squid-users mailing list
> squid-users at lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20200805/32c1e247/attachment-0001.htm>
More information about the squid-users
mailing list