[squid-users] tuning squid memory (aka avoiding the reaper)

[Aaron Turner]

So I'm testing squid 3.5.26 on an m3.xlarge w/ 14GB of RAM.  Squid is
the only "real" service running (sshd and the like).  I'm running 4
workers, and 2 rock cache.  The workers seem to be growing unbounded
and given ~30min or so will cause the kernel to start killing off
processes until memory is freed.  Yes, my clients (32 of them) are
hitting this at about 250 URL's/min which doesn't seem that crazy, but
¯\_(ツ)_/¯

cache_mem 1 GB resulted in workers exceeding 4GB resident.  So I tried
500 MB, same problem.  Now I'm down to 250 MB and I'm still seeing
workers using 3-4GB of RAM after a few minutes and still growing which
is surprising since the docs indicate I should expect total memory to
be roughly 3x cache_mem.

mgr:info reports:

Squid Object Cache: Version 3.5.26
Build Info:
Service Name: squid
Start Time: Mon, 25 Sep 2017 22:53:22 GMT
Current Time: Mon, 25 Sep 2017 23:15:21 GMT
Connection information for squid:
Number of clients accessing cache: 12
Number of HTTP requests received: 568290
Number of ICP messages received: 0
Number of ICP messages sent: 0
Number of queued ICP replies: 0
Number of HTCP messages received: 0
Number of HTCP messages sent: 0
Request failure ratio: 0.00
Average HTTP requests per minute since start: 25851.6
Average ICP messages per minute since start: 0.0
Select loop called: 10686802 times, 2.009 ms avg
Cache information for squid:
Hits as % of all requests: 5min: 7.5%, 60min: 9.8%
Hits as % of bytes sent: 5min: 12.3%, 60min: 17.3%
Memory hits as % of hit requests: 5min: 36.2%, 60min: 40.0%
Disk hits as % of hit requests: 5min: 27.7%, 60min: 25.5%
Storage Swap size: 4481632 KB
Storage Swap capacity: 4.3% used, 95.7% free
Storage Mem size: 254656 KB
Storage Mem capacity: 99.5% used,  0.5% free
Mean Object Size: 26.30 KB
Requests given to unlinkd: 0
Median Service Times (seconds)  5 min    60 min:
HTTP Requests (All):   0.00800  0.00569
Cache Misses:          0.03766  0.04489
Cache Hits:            0.00030  0.00000
Near Hits:             0.05364  0.07135
Not-Modified Replies:  0.00236  0.00168
DNS Lookups:           0.04438  0.04540
ICP Queries:           0.00000  0.00000
Resource usage for squid:
UP Time: 1319.019 seconds
CPU Time: 1617.476 seconds
CPU Usage: 122.63%
CPU Usage, 5 minute avg: 107.50%
CPU Usage, 60 minute avg: 124.63%
Maximum Resident Size: 60715904 KB
Page faults with physical i/o: 8
Memory accounted for:
Total accounted:        44968 KB
memPoolAlloc calls: 158302074
memPoolFree calls:  159053228
File descriptor usage for squid:
Maximum number of file descriptors:   98304
Largest file desc currently in use:   1891
Number of file desc currently in use: 3423
Files queued for open:                   0
Available number of file descriptors: 94881
Reserved number of file descriptors:   600
Store Disk files open:                   8
Internal Data Structures:
 1702 StoreEntries
  454 StoreEntries with MemObjects
 4475 Hot Object Cache Items
170394 on-disk objects

top when things go bad:
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
21450 squid     20   0 5379004 3.984g 619260 D  28.5 27.9   8:54.37
(squid-4) -f /etc/squid/squid.conf
21451 squid     20   0 5253164 3.850g 601248 D  26.6 27.0   7:46.71
(squid-3) -f /etc/squid/squid.conf
21453 squid     20   0 4750376 3.299g 504192 D  17.3 23.1   5:36.19
(squid-1) -f /etc/squid/squid.conf
21452 squid     20   0 4448280 2.994g 482136 D  16.6 21.0   5:43.58
(squid-2) -f /etc/squid/squid.conf
21449 squid     20   0 1292376 358544 346416 D   1.4  2.4   0:45.83
(squid-disk-5) -f /etc/squid/squid.conf
21448 squid     20   0 1292376 356924 344788 D   1.2  2.4   0:46.51
(squid-disk-6) -f /etc/squid/squid.conf
21447 squid     20   0  943584  11624     20 S   0.0  0.1   0:00.19
(squid-coord-7) -f /etc/squid/squid.conf


I'm trying to figure out why and how to fix.  One thing I've read
about the cache_mem knob is:

"If circumstances require, this limit will be exceeded.

Specifically, if your incoming request rate requires more than
'cache_mem' of memory to hold in-transit objects, Squid will
exceed this limit to satisfy the new requests.  When the load
decreases, blocks will be freed until the high-water mark is
reached.  Thereafter, blocks will be used to store hot
objects."

Not sure if this is the cause of my problem?  Maybe something else?
The FAQ says try a different malloc, so tried recompiling with
--enable-dlmalloc, but that had no impact.


--
Aaron Turner
https://synfin.net/         Twitter: @synfinatic
My father once told me that respect for the truth comes close to being
the basis for all morality.  "Something cannot emerge from nothing,"
he said.  This is profound thinking if you understand how unstable
"the truth" can be.  -- Frank Herbert, Dune