[squid-users] High memory usage under load with caching disabled, memory is not being freed even with no load
Ivan Bulatovic
ivan.bulatovic at gmail.com
Mon Aug 3 13:11:31 UTC 2020
Hi all,
I tried with stock squid from Ubuntu 18.04 (version 4.10) and basic
config, but still no luck. Here is the config I tried:
-----------------------------------------
acl localnet src 10.20.0.0/16 # My network
acl localnet src fc00::/7 # RFC 4193 local private network range
acl localnet src fe80::/10 # RFC 4291 link-local
(directly plugged) machines
acl SSL_ports port 443
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 # https
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl CONNECT method CONNECT
cache deny all
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost manager
http_access allow localnet manager
http_access deny manager
http_access deny to_localhost
http_access allow localnet
http_access allow localhost
http_access deny all
http_port localhost:3128
http_port 10.20.6.8:50505
coredump_dir /var/spool/squid
shutdown_lifetime 2 seconds
max_filedescriptors 262143
server_persistent_connections off
client_persistent_connections off
request_header_access Via deny all
request_header_access X-Forwarded-For deny all
dns_v4_first on
------------------------------------
Looks like squid has some serious memory issues when under heavy load
(90 servers that crawl Internet sites). It just eats up memory, and
does not free it up even days after it is being used (with no load on
the proxy for days). So I guess I have to look for another solution.
Best regards,
Ivan Bulatovic
On Mon, Jul 20, 2020 at 10:46 PM Ivan Bulatovic
<ivan.bulatovic at gmail.com> wrote:
>
> Hi all,
>
> I am trying to configure squid to run as a forward proxy with no
> caching (cache deny all) with an option to choose the outgoing IP
> address based on the username. So all squid has to do is to use a
> certain outgoing IP address for a certain user, return the data from
> the server to that user and cache nothing.
>
> For that I created a special authentication helper and used the ACLs
> and tcp_outgoing_address to create a lot of users and outgoing IP
> addresses (about 260 at the moment). Example (not the real IP I use,
> of course):
>
> acl use_IP1 proxy_auth user1
> tcp_outgoing_address 1.2.3.4 use_IP1
>
> I also configured the squid to use 4 workers, but this happens even
> when I use only one worker (default)
>
> And this works. However, under heavy load, Squid eats all of the RAM
> and then starts going to swap. And the memory usage does not drop when
> I remove all the load from squid (I shut down all clients).
>
> I left it to see if the memory will be freed but even after leaving it
> for an hour the info page reports this:
> Cache information for squid:
> Hits as % of all requests: 5min: 0.0%, 60min: 0.0%
> Hits as % of bytes sent: 5min: 0.0%, 60min: 1.1%
> Memory hits as % of hit requests: 5min: 0.0%, 60min: 0.0%
> Disk hits as % of hit requests: 5min: 0.0%, 60min: 100.0%
> Storage Swap size: 0 KB
> Storage Swap capacity: 0.0% used, 100.0% free
> Storage Mem size: 0 KB
> Storage Mem capacity: 0.0% used, 100.0% free
> Mean Object Size: 0.00 KB
> Requests given to unlinkd: 0
>
> Resource usage for squid:
> UP Time: 255334.875 seconds
> CPU Time: 7122.436 seconds
> CPU Usage: 2.79%
> CPU Usage, 5 minute avg: 0.05%
> CPU Usage, 60 minute avg: 37.66%
> Maximum Resident Size: 41500720 KB
> Page faults with physical i/o: 1003410
>
> And here is the listing of free and top commands (with no load on the server):
>
> # free -h
> total used free shared buff/cache available
> Mem: 11G 10G 791M 676K 491M 1.0G
> Swap: 11G 5.5G 6.5G
>
> # top
> top - 14:12:32 up 3 days, 1:30, 1 user, load average: 0.00, 0.00, 0.00
> Tasks: 177 total, 1 running, 102 sleeping, 0 stopped, 0 zombie
> %Cpu0 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
> %Cpu1 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
> %Cpu2 : 0.0 us, 0.3 sy, 0.0 ni, 99.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
> %Cpu3 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
> %Cpu4 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
> %Cpu5 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
> %Cpu6 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
> %Cpu7 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
> KiB Mem : 91.2/12251688
> [|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
> ]
> KiB Swap: 45.8/12582904
> [||||||||||||||||||||||||||||||||||||||||||||||
> ]
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 7851 proxy 20 0 6946872 2.514g 8084 S 0.0 21.5 29:43.74 squid
> 7832 proxy 20 0 6711480 2.464g 8040 S 0.0 21.1 29:58.17 squid
> 7814 proxy 20 0 6834928 2.454g 10024 S 0.0 21.0 29:47.56 squid
> 7843 proxy 20 0 6906252 2.436g 8208 S 0.0 20.8 29:15.60 squid
> 1329 root 20 0 2416672 167272 12680 S 0.0 1.4 136:18.57 metricbeat
> 1321 root 20 0 1831804 48364 11648 S 0.0 0.4 14:32.10 filebeat
> 474 root 19 -1 127796 17576 17144 S 0.0 0.1 0:27.01
> systemd-journal
> 7811 proxy 20 0 549384 14168 8372 S 0.0 0.1 0:20.87 squid
> 1166 root 20 0 1749724 10596 4468 S 0.0 0.1 0:31.83 snapd
> 43940 proxy 20 0 28884 9608 5384 S 0.0 0.1 0:00.14 python3
> 43941 proxy 20 0 28884 9552 5328 S 0.0 0.1 0:00.10 python3
> 43939 proxy 20 0 28884 9524 5308 S 0.0 0.1 0:00.12 python3
> 43938 proxy 20 0 28884 9452 5232 S 0.0 0.1 0:00.16 python3
> 48848 root 20 0 105688 6960 5968 S 0.0 0.1 0:00.02 sshd
> 48974 janitor 20 0 108120 5380 4372 S 0.0 0.0 0:00.00 sshd
> 1 root 20 0 86360 4364 2488 S 0.0 0.0 32:46.22 systemd
> ...
> ... lines ommited
> ...
>
> In the attachment you can find the printout from squidclient mgr:info
> and squidclient mgr:mem. These are both taken at the moment when there
> is no more load on the proxy. I also included my squid.conf file
> (minus the two files where acls are defined and outgoing IP addresses,
> these two contain only acl and tcp_outgoing_address lines as in the
> example above).
>
> Machine info:
> OS: Ubuntu 18.04 with latest updates
> Squid version 4.12 (from diladele repository)
> Hardware: Hyper-V virtual machine with 8 vCPU, 12GB of RAM
>
> I can not understand what is eating all of the memory, if I disabled the cache.
>
> Maybe I configured something wrong but I can not find what.
>
> Thank you for any help you can provide.
>
> Best regards,
> Ivan
More information about the squid-users
mailing list