[squid-users] High memory usage under load with caching disabled, memory is not being freed even with no load
Alex Rousskov
rousskov at measurement-factory.com
Mon Aug 3 20:01:47 UTC 2020
On 8/3/20 9:11 AM, Ivan Bulatovic wrote:
> Looks like squid has some serious memory issues when under heavy load
> (90 servers that crawl Internet sites).
> Maximum Resident Size: 41500720 KB
If the above (unreliable) report matches your observations using system
tools like "top", then it is indeed likely that your Squid is suffering
from a memory leak -- 41GB is usually too much for most non-caching
Squid instances.
Identifying the leak may take some time, and I am not volunteering to do
the necessary legwork personally, but the Squid Project does fix
virtually all runtime leaks that we know about. If you want to speed up
the process, one of the best things you can do is to run Squid under
valgrind with a good suppression file. This requires building Squid with
a special ./configure option. Several testing iterations may be
necessary. If you are willing to do this, please file a bug report and
somebody will guide you through the steps.
> It just eats up memory, and
> does not free it up even days after it is being used (with no load on
> the proxy for days).
Some memory retention is expected by default. See
http://www.squid-cache.org/Doc/config/memory_pools/
Unfortunately, AFAICT, your mgr:mem output does not show any obvious
leaks -- all numbers are very small. If something is leaking a lot, then
it is probably not pooled by Squid.
HTH,
Alex.
> On Mon, Jul 20, 2020 at 10:46 PM Ivan Bulatovic wrote:
>>
>> Hi all,
>>
>> I am trying to configure squid to run as a forward proxy with no
>> caching (cache deny all) with an option to choose the outgoing IP
>> address based on the username. So all squid has to do is to use a
>> certain outgoing IP address for a certain user, return the data from
>> the server to that user and cache nothing.
>>
>> For that I created a special authentication helper and used the ACLs
>> and tcp_outgoing_address to create a lot of users and outgoing IP
>> addresses (about 260 at the moment). Example (not the real IP I use,
>> of course):
>>
>> acl use_IP1 proxy_auth user1
>> tcp_outgoing_address 1.2.3.4 use_IP1
>>
>> I also configured the squid to use 4 workers, but this happens even
>> when I use only one worker (default)
>>
>> And this works. However, under heavy load, Squid eats all of the RAM
>> and then starts going to swap. And the memory usage does not drop when
>> I remove all the load from squid (I shut down all clients).
>>
>> I left it to see if the memory will be freed but even after leaving it
>> for an hour the info page reports this:
>> Cache information for squid:
>> Hits as % of all requests: 5min: 0.0%, 60min: 0.0%
>> Hits as % of bytes sent: 5min: 0.0%, 60min: 1.1%
>> Memory hits as % of hit requests: 5min: 0.0%, 60min: 0.0%
>> Disk hits as % of hit requests: 5min: 0.0%, 60min: 100.0%
>> Storage Swap size: 0 KB
>> Storage Swap capacity: 0.0% used, 100.0% free
>> Storage Mem size: 0 KB
>> Storage Mem capacity: 0.0% used, 100.0% free
>> Mean Object Size: 0.00 KB
>> Requests given to unlinkd: 0
>>
>> Resource usage for squid:
>> UP Time: 255334.875 seconds
>> CPU Time: 7122.436 seconds
>> CPU Usage: 2.79%
>> CPU Usage, 5 minute avg: 0.05%
>> CPU Usage, 60 minute avg: 37.66%
>> Maximum Resident Size: 41500720 KB
>> Page faults with physical i/o: 1003410
>>
>> And here is the listing of free and top commands (with no load on the server):
>>
>> # free -h
>> total used free shared buff/cache available
>> Mem: 11G 10G 791M 676K 491M 1.0G
>> Swap: 11G 5.5G 6.5G
>>
>> # top
>> top - 14:12:32 up 3 days, 1:30, 1 user, load average: 0.00, 0.00, 0.00
>> Tasks: 177 total, 1 running, 102 sleeping, 0 stopped, 0 zombie
>> %Cpu0 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
>> %Cpu1 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
>> %Cpu2 : 0.0 us, 0.3 sy, 0.0 ni, 99.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
>> %Cpu3 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
>> %Cpu4 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
>> %Cpu5 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
>> %Cpu6 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
>> %Cpu7 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
>> KiB Mem : 91.2/12251688
>> [|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
>> ]
>> KiB Swap: 45.8/12582904
>> [||||||||||||||||||||||||||||||||||||||||||||||
>> ]
>>
>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>> 7851 proxy 20 0 6946872 2.514g 8084 S 0.0 21.5 29:43.74 squid
>> 7832 proxy 20 0 6711480 2.464g 8040 S 0.0 21.1 29:58.17 squid
>> 7814 proxy 20 0 6834928 2.454g 10024 S 0.0 21.0 29:47.56 squid
>> 7843 proxy 20 0 6906252 2.436g 8208 S 0.0 20.8 29:15.60 squid
>> 1329 root 20 0 2416672 167272 12680 S 0.0 1.4 136:18.57 metricbeat
>> 1321 root 20 0 1831804 48364 11648 S 0.0 0.4 14:32.10 filebeat
>> 474 root 19 -1 127796 17576 17144 S 0.0 0.1 0:27.01
>> systemd-journal
>> 7811 proxy 20 0 549384 14168 8372 S 0.0 0.1 0:20.87 squid
>> 1166 root 20 0 1749724 10596 4468 S 0.0 0.1 0:31.83 snapd
>> 43940 proxy 20 0 28884 9608 5384 S 0.0 0.1 0:00.14 python3
>> 43941 proxy 20 0 28884 9552 5328 S 0.0 0.1 0:00.10 python3
>> 43939 proxy 20 0 28884 9524 5308 S 0.0 0.1 0:00.12 python3
>> 43938 proxy 20 0 28884 9452 5232 S 0.0 0.1 0:00.16 python3
>> 48848 root 20 0 105688 6960 5968 S 0.0 0.1 0:00.02 sshd
>> 48974 janitor 20 0 108120 5380 4372 S 0.0 0.0 0:00.00 sshd
>> 1 root 20 0 86360 4364 2488 S 0.0 0.0 32:46.22 systemd
>> ...
>> ... lines ommited
>> ...
>>
>> In the attachment you can find the printout from squidclient mgr:info
>> and squidclient mgr:mem. These are both taken at the moment when there
>> is no more load on the proxy. I also included my squid.conf file
>> (minus the two files where acls are defined and outgoing IP addresses,
>> these two contain only acl and tcp_outgoing_address lines as in the
>> example above).
>>
>> Machine info:
>> OS: Ubuntu 18.04 with latest updates
>> Squid version 4.12 (from diladele repository)
>> Hardware: Hyper-V virtual machine with 8 vCPU, 12GB of RAM
>>
>> I can not understand what is eating all of the memory, if I disabled the cache.
>>
>> Maybe I configured something wrong but I can not find what.
>>
>> Thank you for any help you can provide.
>>
>> Best regards,
>> Ivan
> _______________________________________________
> squid-users mailing list
> squid-users at lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users
>
More information about the squid-users
mailing list