[squid-users] Memory Leak Squid 3.4.9 on FreeBSD 10.0 x64
Amos Jeffries
squid3 at treenet.co.nz
Tue Nov 25 01:53:05 UTC 2014
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 25/11/2014 9:06 a.m., Doug Sampson wrote:
> Recently due to squid 2.7 being EOL'ed, we migrated our squid
> server to version 3.4.9 on a FreeBSD 10.0-RELEASE running on 64-bit
> hardware. We started seeing paging file being swapped out
> eventually running out of available memory. From the time squid
> gets started it usually takes about two days before we see these
> entries in /var/log/messages as follows:
>
> +swap_pager_getswapspace(16): failed +swap_pager_getswapspace(16):
> failed +swap_pager_getswapspace(16): failed
> +swap_pager_getswapspace(12): failed +swap_pager_getswapspace(16):
> failed +swap_pager_getswapspace(12): failed
> +swap_pager_getswapspace(6): failed +swap_pager_getswapspace(16):
> failed
>
> Looking at the 'top' results, I see that the swap file has been
> totally exhausted. Memory used by squid hovers around 2.3GB out of
> the total 3GB of system memory.
>
> I am not sure what is causing these memory leaks. After rebooting,
> squid-internal-mgr/info shows the following statistics:
>
> Squid Object Cache: Version 3.4.9 Build Info: Start Time: Mon, 24
> Nov 2014 18:39:08 GMT Current Time: Mon, 24 Nov 2014 19:39:13 GMT
> Connection information for squid: Number of clients accessing
> cache: 18 Number of HTTP requests received: 10589 Number of ICP
> messages received: 0 Number of ICP messages sent: 0 Number of
> queued ICP replies: 0 Number of HTCP messages received: 0 Number of
> HTCP messages sent: 0 Request failure ratio: 0.00 Average HTTP
> requests per minute since start: 176.2 Average ICP messages per
> minute since start: 0.0 Select loop called: 763993 times, 4.719 ms
> avg Cache information for squid: Hits as % of all requests: 5min:
> 3.2%, 60min: 17.0% Hits as % of bytes sent: 5min: 2.0%, 60min:
> 6.7% Memory hits as % of hit requests: 5min: 0.0%, 60min: 37.2%
> Disk hits as % of hit requests: 5min: 22.2%, 60min: 33.2% Storage
> Swap size: 7361088 KB Storage Swap capacity: 58.5% used, 41.5%
> free Storage Mem size: 54348 KB Storage Mem capacity: 3.9% used,
> 96.1% free Mean Object Size: 23.63 KB Requests given to unlinkd: 1
> Median Service Times (seconds) 5 min 60 min: HTTP Requests
> (All): 0.10857 0.19742 Cache Misses: 0.10857 0.32154
> Cache Hits: 0.08265 0.01387 Near Hits:
> 0.15048 0.12106 Not-Modified Replies: 0.00091 0.00091 DNS
> Lookups: 0.05078 0.05078 ICP Queries: 0.00000
> 0.00000 Resource usage for squid: UP Time: 3605.384 seconds CPU
> Time: 42.671 seconds CPU Usage: 1.18% CPU Usage, 5 minute avg:
> 0.72% CPU Usage, 60 minute avg: 1.17% Maximum Resident Size: 845040
> KB Page faults with physical i/o: 20 Memory accounted for: Total
> accounted: 105900 KB memPoolAlloc calls: 2673353
> memPoolFree calls: 2676487 File descriptor usage for squid:
> Maximum number of file descriptors: 87516 Largest file desc
> currently in use: 310 Number of file desc currently in use:
> 198 Files queued for open: 0 Available number of
> file descriptors: 87318 Reserved number of file descriptors: 100
> Store Disk files open: 0 Internal Data
> Structures: 311543 StoreEntries 4421 StoreEntries with MemObjects
> 4416 Hot Object Cache Items 311453 on-disk objects
>
> I will post another one tomorrow that will indicate growing
> memory/swapfile consumption.
>
> Here is my squid.conf:
>
> # OPTIONS FOR AUTHENTICATION #
> -----------------------------------------------------------------------------
>
>
# 1st four lines for
> auth_param basic children 5 auth_param basic realm Squid
> proxy-caching web server auth_param basic credentialsttl 2 hours
> auth_param basic casesensitive off # next three lines for kerberos
> authentication (needed to use usernames) # used in conjunction
> with "acl auth proxy_auth" line below #auth_param negotiate program
> /usr/local/libexec/squid/negotiate_kerberos_auth -i #auth_param
> negotiate children 50 startup=10 idle=5 #auth_param negotiate
> keep_alive on
>
>
> # ACCESS CONTROLS #
> -----------------------------------------------------------------------------
>
>
# Example rule allowing access from your local networks.
> # Adapt to list your (internal) IP networks from where browsing #
> should be allowed #acl manager proto cache_object acl manager
> url_regex -i ^cache_object:// /squid-internal-mgr/ acl adminhost
> src 192.168.1.149 acl localnet src 192.168.1.0/24 # RFC1918
> possible internal network acl localnet src fc00::/7 # RFC
> 4193 local private network range acl localnet src fe80::/10
> # RFC 4291 link-local (directly plugged) machines acl webserver src
> 198.168.1.35 acl some_big_clients src 192.168.1.149/32 #CI53
>
> # We want to limit downloads of these type of files # Put this all
> in one line acl magic_words url_regex -i ftp .exe .mp3 .vqf .tar.gz
> .gz .rpm .zip .rar .avi .mpeg .mpe .mpg .qt .ram .rm .iso .raw .wav
> .dmg .mp4 .img # We don't block .html, .gif, .jpg and similar
> files, because they # generally don't consume much bandwidth
But you do. Whenever the domain name or path contains any of the byte
sequences in that regex above. The entire websites
http://www.divx.com/ and http://isohunt.com/ for example.
And whats wrong with adding more HITs ? even if they are small enough
not to use much cache space.
<snip>
>
> # OPTIONS WHICH AFFECT THE NEIGHBOR SELECTION ALGORITHM #
> -----------------------------------------------------------------------------
>
>
hierarchy_stoplist cgi-bin ?
... but you dont have neighbours. This is also deprecated anyway.
>
> # MEMORY CACHE OPTIONS #
> -----------------------------------------------------------------------------
>
>
cache_mem 1366 MB
> #cache_mem 2134 MB #maximum_object_size_in_memory 64 KB
> maximum_object_size_in_memory 128 KB
>
> # DISK CACHE OPTIONS #
> -----------------------------------------------------------------------------
>
>
cache_replacement_policy heap LFUDA
> cache_dir aufs /data/squid/aufs_cache 4096 16 256 min-size=131073
> cache_dir diskd /data/squid/diskd_cache 8192 16 256 Q1=64 Q2=72
> max-size=131072
Why the segregation between diskd and aufs?
The only difference between these cache types is the method if I/O
performed accessing the disk. AUFS is threaded SMP, diskd is
multi-process SMP.
NP: FreeBSD 10 seem to have resolved the issues Squid AUFS has with
older BSD and people are now noticing the speed issues with diskd.
The official recommendation is currently to use AUFS with FreeBSD 10+
and diskd with older FreeBSD.
> #maximum_object_size 122880 KB maximum_object_size 153600 KB
> cache_swap_low 90 cache_swap_high 95
>
> # LOGFILE OPTIONS #
> -----------------------------------------------------------------------------
>
>
access_log daemon:/data/squid/logs/access.log
> cache_store_log daemon:/data/squid/logs/store.log cache_swap_log
> /var/spool/squid/%s
What is this %s ??
> logfile_rotate 28
>
> # OPTIONS FOR TROUBLESHOOTING #
> -----------------------------------------------------------------------------
>
>
cache_log /data/squid/logs/cache.log
> # Leave coredumps in the first cache dir coredump_dir /data/squid
>
> # OPTIONS FOR EXTERNAL SUPPORT PROGRAMS #
> -----------------------------------------------------------------------------
>
>
diskd_program /usr/local/libexec/squid/diskd
>
Unless you are replacing this helper with a custom-built one with
strange name this should not be configured explicitly in Squid-3.
> # OPTIONS FOR TUNING THE CACHE #
> -----------------------------------------------------------------------------
>
>
refresh_pattern http://.*\.windowsupdate\.microsoft\.com/ 0 80% 20160
> refresh_pattern http://office\.microsoft\.com/ 0 80% 20160
> refresh_pattern http://windowsupdate\.microsoft\.com/ 0 80% 20160
> refresh_pattern http://w?xpsp[0-9]\.microsoft\.com/ 0 80% 20160
> refresh_pattern http://w2ksp[0-9]\.microsoft\.com/ 0 80% 20160
> refresh_pattern http://download\.microsoft\.com/ 0 80% 20160
> refresh_pattern http://download\.macromedia\.com/ 0 80% 20160
> refresh_pattern http://ftp\.software\.ibm\.com/ 0 80% 20160
> refresh_pattern cgi-bin 1 20% 2 refresh_pattern
> \.asp$ 1 20% 2 refresh_pattern \.acgi$ 1
> 20% 2 refresh_pattern \.cgi$ 1 20% 2
> refresh_pattern \.pl$ 1 20% 2 refresh_pattern
> \.shtml$ 1 20% 2 refresh_pattern \.php3$ 1
> 20% 2 refresh_pattern \? 1 20% 2
> refresh_pattern \.gif$ 10080 90% 43200
> refresh_pattern \.png$ 10080 90% 43200
> refresh_pattern \.jpg$ 10080 90% 43200
> refresh_pattern \.ico$ 10080 90% 43200
> refresh_pattern \.bom\.gov\.au 30 20% 120
> refresh_pattern \.html$ 480 50% 22160
> refresh_pattern \.htm$ 480 50% 22160
> refresh_pattern \.css$ 480 50% 22160
> refresh_pattern \.js$ 480 50% 22160
> refresh_pattern \.class$ 10080 90% 43200
> refresh_pattern \.zip$ 10080 90% 43200
> refresh_pattern \.jpeg$ 10080 90% 43200
> refresh_pattern \.mid$ 10080 90% 43200
> refresh_pattern \.shtml$ 480 50% 22160
> refresh_pattern \.exe$ 10080 90% 43200
> refresh_pattern \.thm$ 10080 90% 43200
> refresh_pattern \.wav$ 10080 90% 43200
> refresh_pattern \.mp4$ 10080 90% 43200
> refresh_pattern \.txt$ 10080 90% 43200
> refresh_pattern \.cab$ 10080 90% 43200
> refresh_pattern \.au$ 10080 90% 43200
> refresh_pattern \.mov$ 10080 90% 43200
> refresh_pattern \.xbm$ 10080 90% 43200
> refresh_pattern \.ram$ 10080 90% 43200
> refresh_pattern \.iso$ 10080 90% 43200
> refresh_pattern \.avi$ 10080 90% 43200
> refresh_pattern \.chtml$ 480 50% 22160
> refresh_pattern \.thb$ 10080 90% 43200
> refresh_pattern \.dcr$ 10080 90% 43200
> refresh_pattern \.bmp$ 10080 90% 43200
> refresh_pattern \.phtml$ 480 50% 22160
> refresh_pattern \.mpg$ 10080 90% 43200
> refresh_pattern \.pdf$ 10080 90% 43200
> refresh_pattern \.art$ 10080 90% 43200
> refresh_pattern \.swf$ 10080 90% 43200
> refresh_pattern \.flv$ 10080 90% 43200
> refresh_pattern \.x-flv$ 10080 90% 43200
> refresh_pattern \.mp3$ 10080 90% 43200
> refresh_pattern \.ra$ 10080 90% 43200
> refresh_pattern \.spl$ 10080 90% 43200
> refresh_pattern \.viv$ 10080 90% 43200
> refresh_pattern \.doc$ 10080 90% 43200
> refresh_pattern \.gz$ 10080 90% 43200
> refresh_pattern \.Z$ 10080 90% 43200
> refresh_pattern \.tgz$ 10080 90% 43200
> refresh_pattern \.tar$ 10080 90% 43200
> refresh_pattern \.vrm$ 10080 90% 43200
> refresh_pattern \.vrml$ 10080 90% 43200
> refresh_pattern \.aif$ 10080 90% 43200
> refresh_pattern \.aifc$ 10080 90% 43200
> refresh_pattern \.aiff$ 10080 90% 43200
> refresh_pattern \.arj$ 10080 90% 43200
> refresh_pattern \.c$ 10080 90% 43200
> refresh_pattern \.cpt$ 10080 90% 43200
> refresh_pattern \.dir$ 10080 90% 43200
> refresh_pattern \.dxr$ 10080 90% 43200
> refresh_pattern \.hqx$ 10080 90% 43200
> refresh_pattern \.jpe$ 10080 90% 43200
> refresh_pattern \.lha$ 10080 90% 43200
> refresh_pattern \.lzh$ 10080 90% 43200
> refresh_pattern \.midi$ 10080 90% 43200
> refresh_pattern \.movie$ 10080 90% 43200
> refresh_pattern \.mp2$ 10080 90% 43200
> refresh_pattern \.mpe$ 10080 90% 43200
> refresh_pattern \.mpeg$ 10080 90% 43200
> refresh_pattern \.mpga$ 10080 90% 43200
> refresh_pattern \.pl$ 10080 90% 43200
> refresh_pattern \.ppt$ 10080 90% 43200
> refresh_pattern \.ps$ 10080 90% 43200
> refresh_pattern \.qt$ 10080 90% 43200
> refresh_pattern \.qtm$ 10080 90% 43200
> refresh_pattern \.rar$ 10080 90% 43200
> refresh_pattern \.ras$ 10080 90% 43200
> refresh_pattern \.sea$ 10080 90% 43200
> refresh_pattern \.sit$ 10080 90% 43200
> refresh_pattern \.tif$ 10080 90% 43200
> refresh_pattern \.tiff$ 10080 90% 43200
> refresh_pattern \.snd$ 10080 90% 43200
> refresh_pattern \.wrl$ 10080 90% 43200
> refresh_pattern ^ftp: 1440 60% 22160
> refresh_pattern ^gopher: 1440 20% 1440
> refresh_pattern -i (cgi-bin|\?) 0 0% 0
> refresh_pattern . 480 50% 22160
>
That is a LOT of regex comparisions the proxy is having to do at least
once per-request.
The special rules you have up the top for "cgi-bin" and "\?" are also
violating HTTP safe behaviour. The default rule we provide is highly
tuned to handle caching of those responses safely without breaking old
legacy scripts.
At least most of them end with $ anchor point to prevent random URLs
matching.
> # ADMINISTRATIVE PARAMETERS #
> -----------------------------------------------------------------------------
>
>
cache_mgr admin at example.com
> mail_from squid at example.com cache_effective_user squid
> cache_effective_group squid
>
> # DELAY POOL PARAMETERS #
> -----------------------------------------------------------------------------
>
>
delay_pools 2
> delay_class 1 2 # When big_files are being downloaded, the first
> 5MB (625000 * 8 bits) are # downloaded at max network speed. Once
> the file size limit of 5MB is reached, # download speed drops to
> 438,000 bits or 3,504,000 MB per sec. Current # contracted Internet
> connection speed w/ TP is at 7MB per sec. delay_parameters 1
> 750000/750000 438000/625000
> acl big_files url_regex -i ftp .exe .mp3 .vqf .tar.gz .gz .rpm .zip
> .rar .avi .mpeg .mpe .mpg .qt .ram .rm .iso .raw .wav .dmg .mp4
> .img .flv .wmv .divx .mov .bz2 .deb
Another long list of regex patterns. Notice how these are permitted to
match anywhere in the entie URL. Including domain names.
FTP traffic in particular is not guaranteed to be "big files".
<snip>
> Intially, I set mem_cache=2134MB and after noticing these memory
> leaks, I dropped it down to 1344MB. Memory leaks are still
> occurring.
>
> Am I using anything that is known to cause memory leaks?
>
> If there is additional information that you need, please do not
> hesitate to ask! Thanks.
A copy of the manager "mem" report would be very useful to see whats
using the memory.
Note that it is a TSV format, so please save as .tsv file and attach.
rather than cut-n-pasting inline.
Amos
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (MingW32)
iQEcBAEBAgAGBQJUc+EAAAoJELJo5wb/XPRj7TMH/jko8cuy/iURLknvFadt6Yn7
QKh8LSV63yLClh6dWxVGvHLbFcF+GQDEn2SGaZ9ZGxt5yEjAJjwoSZDTpFcFNZPf
ocROu8/R1Ys69PCGHMLJh8DzRaXrLW1/OPrt1hcSuogWKUnNCEbgg+g3MpQO4AmM
4AgwIQyx8O3kE39CHTSKy5luCFzj8pMB/qr20AwjWiM4eG+MV81OWQpJL+AnhH1s
5LYZdLXtQ16BR0TT8uokYmnzS2B+B51VC9HYKEJdWz9BALgjMHQhcDtYtRZh7pV/
ppdARfj984xYk9l4wsmkFMIlBp4aDNMhJiIZcNp8t96MKwLeDgoCaaGh6dDBa/Q=
=wKvq
-----END PGP SIGNATURE-----
More information about the squid-users
mailing list