[squid-users] squid 3.5.23 memory usage
Ivan Larionov
xeron.oskom at gmail.com
Fri Jan 20 00:23:54 UTC 2017
Hello.
I'm pretty sure this question has been asked multiple times already, but
after reading everything I found I still can't figure out squid memory
usage patterns.
We're currently trying to upgrade from squid 2.7 to squid 3.5 and memory
usage on squid 3 is much much higher compared to squid 2 with the same
configuration.
What do I see:
squid running for several days with low traffic:
# top
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
7367 squid 20 0 4780m 4.4g 5224 S 6.0 60.6 105:01.76 squid -N
So it uses 4.4GB resident memory. Ok, let's see important config options:
cache_mem 2298756 KB
maximum_object_size_in_memory 8 KB
memory_replacement_policy lru
cache_replacement_policy lru
cache_dir aufs /mnt/services/squid/cache 445644 16 256
minimum_object_size 64 bytes # none-zero so we dont cache mistakes
maximum_object_size 102400 KB
So we configured 2.2GB memory cache and 500GB disk cache. Disk cache is
quite big but current usage is only 3GB:
# du -sh /mnt/services/squid/cache # cache_dir
3.0G /mnt/services/squid/cache
Now I'm looking into this page
http://wiki.squid-cache.org/SquidFaq/SquidMemory and see:
14 MB of memory per 1 GB on disk for 64-bit Squid
Which means disk cache should use ~50MB of RAM.
All these means we have ~2.2GB ram used for everything else except
cache_mem and disk cache index.
Let's see top pools from mgr:mem:
Pool (KB) %Tot
mem_node 2298833 55.082
Short Strings 622365 14.913
HttpHeaderEntry 404531 9.693
Long Strings 284520 6.817
MemObject 182288 4.368
HttpReply 155612 3.729
StoreEntry 73965 1.772
Medium Strings 71152 1.705
cbdata MemBuf (12) 35573 0.852
LRU policy node 30403 0.728
MD5 digest 11380 0.273
16K Buffer 1056 0.025
These pools consume ~35% of total squid memory usage: Short Strings,
HttpHeaderEntry, Long Strings, HttpReply. Looks suspicious. On squid 2 same
pools use 10 times less memory.
I found a bug which looks similar to our experience:
http://bugs.squid-cache.org/show_bug.cgi?id=4084.
I'm attaching our config, mgr:info, mgr:mem and some system info I
collected.
Could someone say if this is normal and why it's so much different from
squid 2?
--
With best regards, Ivan Larionov.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20170119/6acfe4e2/attachment-0001.html>
-------------- next part --------------
HTTP/1.1 200 OK
Server: squid/3.5.23
Mime-Version: 1.0
Date: Thu, 19 Jan 2017 23:39:50 GMT
Content-Type: text/plain;charset=utf-8
Expires: Thu, 19 Jan 2017 23:39:50 GMT
Last-Modified: Thu, 19 Jan 2017 23:39:50 GMT
X-Cache: MISS from ip-172-22-10-120
X-Cache-Lookup: MISS from ip-172-22-10-120:3128
Connection: close
Squid Object Cache: Version 3.5.23
Build Info:
Service Name: squid
Start Time: Fri, 13 Jan 2017 23:35:32 GMT
Current Time: Thu, 19 Jan 2017 23:39:50 GMT
Connection information for squid:
Number of clients accessing cache: (client_db off)
Number of HTTP requests received: 8195690
Number of ICP messages received: 0
Number of ICP messages sent: 0
Number of queued ICP replies: 0
Number of HTCP messages received: 0
Number of HTCP messages sent: 0
Request failure ratio: 0.00
Average HTTP requests per minute since start: 948.1
Average ICP messages per minute since start: 0.0
Select loop called: 73529108 times, 7.054 ms avg
Cache information for squid:
Hits as % of all requests: 5min: 29.2%, 60min: 28.9%
Hits as % of bytes sent: 5min: 89.0%, 60min: 89.1%
Memory hits as % of hit requests: 5min: 0.0%, 60min: 0.0%
Disk hits as % of hit requests: 5min: 100.0%, 60min: 100.0%
Storage Swap size: 2915344 KB
Storage Swap capacity: 0.6% used, 99.4% free
Storage Mem size: 2276524 KB
Storage Mem capacity: 99.0% used, 1.0% free
Mean Object Size: 4.00 KB
Requests given to unlinkd: 0
Median Service Times (seconds) 5 min 60 min:
HTTP Requests (All): 0.01745 0.01745
Cache Misses: 0.02899 0.02451
Cache Hits: 0.00091 0.00091
Near Hits: 0.00000 0.00000
Not-Modified Replies: 0.00000 0.00000
DNS Lookups: 0.00000 0.00094
ICP Queries: 0.00000 0.00000
Resource usage for squid:
UP Time: 518657.265 seconds
CPU Time: 6265.444 seconds
CPU Usage: 1.21%
CPU Usage, 5 minute avg: 6.43%
CPU Usage, 60 minute avg: 5.11%
Maximum Resident Size: 18579360 KB
Page faults with physical i/o: 0
Memory accounted for:
Total accounted: -20826 KB
memPoolAlloc calls: 2192400061
memPoolFree calls: 2194290230
File descriptor usage for squid:
Maximum number of file descriptors: 524288
Largest file desc currently in use: 70
Number of file desc currently in use: 44
Files queued for open: 0
Available number of file descriptors: 524244
Reserved number of file descriptors: 100
Store Disk files open: 0
Internal Data Structures:
728426 StoreEntries
569027 StoreEntries with MemObjects
569025 Hot Object Cache Items
728372 on-disk objects
-------------- next part --------------
A non-text attachment was scrubbed...
Name: squid_memory_3.tsv
Type: text/tab-separated-values
Size: 14636 bytes
Desc: not available
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20170119/6acfe4e2/attachment-0001.tsv>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: squid.conf
Type: application/octet-stream
Size: 3298 bytes
Desc: not available
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20170119/6acfe4e2/attachment-0001.obj>
-------------- next part --------------
# free -m
total used free shared buffers cached
Mem: 7482 7349 133 0 189 1354
-/+ buffers/cache: 5805 1677
Swap: 0 0 0
# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 9.8G 3.3G 6.4G 34% /
devtmpfs 3.7G 68K 3.7G 1% /dev
tmpfs 3.7G 44K 3.7G 1% /dev/shm
/dev/mapper/vg-data--master 29G 1.1G 28G 4% /mnt
/dev/xvdh 504G 3.0G 476G 1% /mnt/services/squid/cache
# du -sh /mnt/services/squid/cache
3.0G /mnt/services/squid/cache
# top
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
7367 squid 20 0 4780m 4.4g 5224 S 6.0 60.6 105:01.76 squid -N
# squid -v
Squid Cache: Version 3.5.23
Service Name: squid
configure options: '--program-prefix=' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/sbin' '--sbindir=/usr/sbin' '--sysconfdir=/etc/squid' '--libdir=/usr/lib' '--libexecdir=/usr/lib/squid' '--includedir=/usr/include' '--datadir=/usr/share' '--sharedstatedir=/usr/com' '--localstatedir=/var' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--enable-epoll' '--enable-removal-policies=heap,lru' '--enable-storeio=aufs' '--enable-delay-pools' '--with-pthreads' '--enable-cache-digests' '--enable-useragent-log' '--enable-referer-log' '--with-large-files' '--with-maxfd=16384' '--enable-err-languages=English' '--enable-htcp'
More information about the squid-users
mailing list