[squid-users] Memory Leak Squid 3.4.9 on FreeBSD 10.0 x64
Doug Sampson
dougs at dawnsign.com
Tue Nov 25 19:59:27 UTC 2014
> On 25/11/2014 9:06 a.m., Doug Sampson wrote:
> > Recently due to squid 2.7 being EOL'ed, we migrated our squid
> > server to version 3.4.9 on a FreeBSD 10.0-RELEASE running on 64-bit
> > hardware. We started seeing paging file being swapped out
> > eventually running out of available memory. From the time squid
> > gets started it usually takes about two days before we see these
> > entries in /var/log/messages as follows:
> >
> > +swap_pager_getswapspace(16): failed +swap_pager_getswapspace(16):
> > failed +swap_pager_getswapspace(16): failed
> > +swap_pager_getswapspace(12): failed +swap_pager_getswapspace(16):
> > failed +swap_pager_getswapspace(12): failed
> > +swap_pager_getswapspace(6): failed +swap_pager_getswapspace(16):
> > failed
> >
> > Looking at the 'top' results, I see that the swap file has been
> > totally exhausted. Memory used by squid hovers around 2.3GB out of
> > the total 3GB of system memory.
> >
> > I am not sure what is causing these memory leaks. After rebooting,
> > squid-internal-mgr/info shows the following statistics:
> >
> > Squid Object Cache: Version 3.4.9 Build Info: Start Time: Mon, 24
> > Nov 2014 18:39:08 GMT Current Time: Mon, 24 Nov 2014 19:39:13 GMT
> > Connection information for squid: Number of clients accessing
> > cache: 18 Number of HTTP requests received: 10589 Number of ICP
> > messages received: 0 Number of ICP messages sent: 0 Number of
> > queued ICP replies: 0 Number of HTCP messages received: 0 Number of
> > HTCP messages sent: 0 Request failure ratio: 0.00 Average HTTP
> > requests per minute since start: 176.2 Average ICP messages per
> > minute since start: 0.0 Select loop called: 763993 times, 4.719 ms
> > avg Cache information for squid: Hits as % of all requests: 5min:
> > 3.2%, 60min: 17.0% Hits as % of bytes sent: 5min: 2.0%, 60min:
> > 6.7% Memory hits as % of hit requests: 5min: 0.0%, 60min: 37.2%
> > Disk hits as % of hit requests: 5min: 22.2%, 60min: 33.2% Storage
> > Swap size: 7361088 KB Storage Swap capacity: 58.5% used, 41.5%
> > free Storage Mem size: 54348 KB Storage Mem capacity: 3.9%
> used,
> > 96.1% free Mean Object Size: 23.63 KB Requests given to unlinkd: 1
> > Median Service Times (seconds) 5 min 60 min: HTTP Requests
> > (All): 0.10857 0.19742 Cache Misses: 0.10857 0.32154
> > Cache Hits: 0.08265 0.01387 Near Hits:
> > 0.15048 0.12106 Not-Modified Replies: 0.00091 0.00091 DNS
> > Lookups: 0.05078 0.05078 ICP Queries: 0.00000
> > 0.00000 Resource usage for squid: UP Time: 3605.384 seconds CPU
> > Time: 42.671 seconds CPU Usage: 1.18% CPU Usage, 5 minute avg:
> > 0.72% CPU Usage, 60 minute avg: 1.17% Maximum Resident Size: 845040
> > KB Page faults with physical i/o: 20 Memory accounted for: Total
> > accounted: 105900 KB memPoolAlloc calls: 2673353
> > memPoolFree calls: 2676487 File descriptor usage for squid:
> > Maximum number of file descriptors: 87516 Largest file desc
> > currently in use: 310 Number of file desc currently in use:
> > 198 Files queued for open: 0 Available number of
> > file descriptors: 87318 Reserved number of file descriptors: 100
> > Store Disk files open: 0 Internal Data
> > Structures: 311543 StoreEntries 4421 StoreEntries with MemObjects
> > 4416 Hot Object Cache Items 311453 on-disk objects
> >
> > I will post another one tomorrow that will indicate growing
> > memory/swapfile consumption.
> >
> > Here is my squid.conf:
> >
> > # OPTIONS FOR AUTHENTICATION #
> > ------------------------------------------------------------------------
> -----
> >
> >
> # 1st four lines for
> > auth_param basic children 5 auth_param basic realm Squid
> > proxy-caching web server auth_param basic credentialsttl 2 hours
> > auth_param basic casesensitive off # next three lines for kerberos
> > authentication (needed to use usernames) # used in conjunction
> > with "acl auth proxy_auth" line below #auth_param negotiate program
> > /usr/local/libexec/squid/negotiate_kerberos_auth -i #auth_param
> > negotiate children 50 startup=10 idle=5 #auth_param negotiate
> > keep_alive on
> >
> >
> > # ACCESS CONTROLS #
> > ------------------------------------------------------------------------
> -----
> >
> >
> # Example rule allowing access from your local networks.
> > # Adapt to list your (internal) IP networks from where browsing #
> > should be allowed #acl manager proto cache_object acl manager
> > url_regex -i ^cache_object:// /squid-internal-mgr/ acl adminhost
> > src 192.168.1.149 acl localnet src 192.168.1.0/24 # RFC1918
> > possible internal network acl localnet src fc00::/7 # RFC
> > 4193 local private network range acl localnet src fe80::/10
> > # RFC 4291 link-local (directly plugged) machines acl webserver src
> > 198.168.1.35 acl some_big_clients src 192.168.1.149/32 #CI53
> >
> > # We want to limit downloads of these type of files # Put this all
> > in one line acl magic_words url_regex -i ftp .exe .mp3 .vqf .tar.gz
> > .gz .rpm .zip .rar .avi .mpeg .mpe .mpg .qt .ram .rm .iso .raw .wav
> > .dmg .mp4 .img # We don't block .html, .gif, .jpg and similar
> > files, because they # generally don't consume much bandwidth
>
> But you do. Whenever the domain name or path contains any of the byte
> sequences in that regex above. The entire websites
> http://www.divx.com/ and http://isohunt.com/ for example.
>
> And whats wrong with adding more HITs ? even if they are small enough
> not to use much cache space.
>
> <snip>
> >
> > # OPTIONS WHICH AFFECT THE NEIGHBOR SELECTION ALGORITHM #
> > ------------------------------------------------------------------------
> -----
> >
> >
> hierarchy_stoplist cgi-bin ?
>
>
> ... but you dont have neighbours. This is also deprecated anyway.
>
> >
> > # MEMORY CACHE OPTIONS #
> > ------------------------------------------------------------------------
> -----
> >
> >
> cache_mem 1366 MB
> > #cache_mem 2134 MB #maximum_object_size_in_memory 64 KB
> > maximum_object_size_in_memory 128 KB
> >
> > # DISK CACHE OPTIONS #
> > ------------------------------------------------------------------------
> -----
> >
> >
> cache_replacement_policy heap LFUDA
> > cache_dir aufs /data/squid/aufs_cache 4096 16 256 min-size=131073
> > cache_dir diskd /data/squid/diskd_cache 8192 16 256 Q1=64 Q2=72
> > max-size=131072
>
> Why the segregation between diskd and aufs?
>
> The only difference between these cache types is the method if I/O
> performed accessing the disk. AUFS is threaded SMP, diskd is
> multi-process SMP.
>
> NP: FreeBSD 10 seem to have resolved the issues Squid AUFS has with
> older BSD and people are now noticing the speed issues with diskd.
>
> The official recommendation is currently to use AUFS with FreeBSD 10+
> and diskd with older FreeBSD.
>
>
> > #maximum_object_size 122880 KB maximum_object_size 153600 KB
> > cache_swap_low 90 cache_swap_high 95
> >
> > # LOGFILE OPTIONS #
> > ------------------------------------------------------------------------
> -----
> >
> >
> access_log daemon:/data/squid/logs/access.log
> > cache_store_log daemon:/data/squid/logs/store.log cache_swap_log
> > /var/spool/squid/%s
>
> What is this %s ??
>
> > logfile_rotate 28
> >
> > # OPTIONS FOR TROUBLESHOOTING #
> > ------------------------------------------------------------------------
> -----
> >
> >
> cache_log /data/squid/logs/cache.log
> > # Leave coredumps in the first cache dir coredump_dir /data/squid
> >
> > # OPTIONS FOR EXTERNAL SUPPORT PROGRAMS #
> > ------------------------------------------------------------------------
> -----
> >
> >
> diskd_program /usr/local/libexec/squid/diskd
> >
>
> Unless you are replacing this helper with a custom-built one with
> strange name this should not be configured explicitly in Squid-3.
>
>
> > # OPTIONS FOR TUNING THE CACHE #
> > ------------------------------------------------------------------------
> -----
> >
> >
> refresh_pattern http://.*\.windowsupdate\.microsoft\.com/ 0 80% 20160
> > refresh_pattern http://office\.microsoft\.com/ 0 80% 20160
> > refresh_pattern http://windowsupdate\.microsoft\.com/ 0 80% 20160
> > refresh_pattern http://w?xpsp[0-9]\.microsoft\.com/ 0 80% 20160
> > refresh_pattern http://w2ksp[0-9]\.microsoft\.com/ 0 80% 20160
> > refresh_pattern http://download\.microsoft\.com/ 0 80% 20160
> > refresh_pattern http://download\.macromedia\.com/ 0 80% 20160
> > refresh_pattern http://ftp\.software\.ibm\.com/ 0 80% 20160
> > refresh_pattern cgi-bin 1 20% 2 refresh_pattern
> > \.asp$ 1 20% 2 refresh_pattern \.acgi$ 1
> > 20% 2 refresh_pattern \.cgi$ 1 20% 2
> > refresh_pattern \.pl$ 1 20% 2 refresh_pattern
> > \.shtml$ 1 20% 2 refresh_pattern \.php3$ 1
> > 20% 2 refresh_pattern \? 1 20% 2
> > refresh_pattern \.gif$ 10080 90% 43200
> > refresh_pattern \.png$ 10080 90% 43200
> > refresh_pattern \.jpg$ 10080 90% 43200
> > refresh_pattern \.ico$ 10080 90% 43200
> > refresh_pattern \.bom\.gov\.au 30 20% 120
> > refresh_pattern \.html$ 480 50% 22160
> > refresh_pattern \.htm$ 480 50% 22160
> > refresh_pattern \.css$ 480 50% 22160
> > refresh_pattern \.js$ 480 50% 22160
> > refresh_pattern \.class$ 10080 90% 43200
> > refresh_pattern \.zip$ 10080 90% 43200
> > refresh_pattern \.jpeg$ 10080 90% 43200
> > refresh_pattern \.mid$ 10080 90% 43200
> > refresh_pattern \.shtml$ 480 50% 22160
> > refresh_pattern \.exe$ 10080 90% 43200
> > refresh_pattern \.thm$ 10080 90% 43200
> > refresh_pattern \.wav$ 10080 90% 43200
> > refresh_pattern \.mp4$ 10080 90% 43200
> > refresh_pattern \.txt$ 10080 90% 43200
> > refresh_pattern \.cab$ 10080 90% 43200
> > refresh_pattern \.au$ 10080 90% 43200
> > refresh_pattern \.mov$ 10080 90% 43200
> > refresh_pattern \.xbm$ 10080 90% 43200
> > refresh_pattern \.ram$ 10080 90% 43200
> > refresh_pattern \.iso$ 10080 90% 43200
> > refresh_pattern \.avi$ 10080 90% 43200
> > refresh_pattern \.chtml$ 480 50% 22160
> > refresh_pattern \.thb$ 10080 90% 43200
> > refresh_pattern \.dcr$ 10080 90% 43200
> > refresh_pattern \.bmp$ 10080 90% 43200
> > refresh_pattern \.phtml$ 480 50% 22160
> > refresh_pattern \.mpg$ 10080 90% 43200
> > refresh_pattern \.pdf$ 10080 90% 43200
> > refresh_pattern \.art$ 10080 90% 43200
> > refresh_pattern \.swf$ 10080 90% 43200
> > refresh_pattern \.flv$ 10080 90% 43200
> > refresh_pattern \.x-flv$ 10080 90% 43200
> > refresh_pattern \.mp3$ 10080 90% 43200
> > refresh_pattern \.ra$ 10080 90% 43200
> > refresh_pattern \.spl$ 10080 90% 43200
> > refresh_pattern \.viv$ 10080 90% 43200
> > refresh_pattern \.doc$ 10080 90% 43200
> > refresh_pattern \.gz$ 10080 90% 43200
> > refresh_pattern \.Z$ 10080 90% 43200
> > refresh_pattern \.tgz$ 10080 90% 43200
> > refresh_pattern \.tar$ 10080 90% 43200
> > refresh_pattern \.vrm$ 10080 90% 43200
> > refresh_pattern \.vrml$ 10080 90% 43200
> > refresh_pattern \.aif$ 10080 90% 43200
> > refresh_pattern \.aifc$ 10080 90% 43200
> > refresh_pattern \.aiff$ 10080 90% 43200
> > refresh_pattern \.arj$ 10080 90% 43200
> > refresh_pattern \.c$ 10080 90% 43200
> > refresh_pattern \.cpt$ 10080 90% 43200
> > refresh_pattern \.dir$ 10080 90% 43200
> > refresh_pattern \.dxr$ 10080 90% 43200
> > refresh_pattern \.hqx$ 10080 90% 43200
> > refresh_pattern \.jpe$ 10080 90% 43200
> > refresh_pattern \.lha$ 10080 90% 43200
> > refresh_pattern \.lzh$ 10080 90% 43200
> > refresh_pattern \.midi$ 10080 90% 43200
> > refresh_pattern \.movie$ 10080 90% 43200
> > refresh_pattern \.mp2$ 10080 90% 43200
> > refresh_pattern \.mpe$ 10080 90% 43200
> > refresh_pattern \.mpeg$ 10080 90% 43200
> > refresh_pattern \.mpga$ 10080 90% 43200
> > refresh_pattern \.pl$ 10080 90% 43200
> > refresh_pattern \.ppt$ 10080 90% 43200
> > refresh_pattern \.ps$ 10080 90% 43200
> > refresh_pattern \.qt$ 10080 90% 43200
> > refresh_pattern \.qtm$ 10080 90% 43200
> > refresh_pattern \.rar$ 10080 90% 43200
> > refresh_pattern \.ras$ 10080 90% 43200
> > refresh_pattern \.sea$ 10080 90% 43200
> > refresh_pattern \.sit$ 10080 90% 43200
> > refresh_pattern \.tif$ 10080 90% 43200
> > refresh_pattern \.tiff$ 10080 90% 43200
> > refresh_pattern \.snd$ 10080 90% 43200
> > refresh_pattern \.wrl$ 10080 90% 43200
> > refresh_pattern ^ftp: 1440 60% 22160
> > refresh_pattern ^gopher: 1440 20% 1440
> > refresh_pattern -i (cgi-bin|\?) 0 0% 0
> > refresh_pattern . 480 50% 22160
> >
>
> That is a LOT of regex comparisions the proxy is having to do at least
> once per-request.
>
> The special rules you have up the top for "cgi-bin" and "\?" are also
> violating HTTP safe behaviour. The default rule we provide is highly
> tuned to handle caching of those responses safely without breaking old
> legacy scripts.
>
>
> At least most of them end with $ anchor point to prevent random URLs
> matching.
>
>
> > # ADMINISTRATIVE PARAMETERS #
> > ------------------------------------------------------------------------
> -----
> >
> >
> cache_mgr admin at example.com
> > mail_from squid at example.com cache_effective_user squid
> > cache_effective_group squid
> >
> > # DELAY POOL PARAMETERS #
> > ------------------------------------------------------------------------
> -----
> >
> >
> delay_pools 2
> > delay_class 1 2 # When big_files are being downloaded, the first
> > 5MB (625000 * 8 bits) are # downloaded at max network speed. Once
> > the file size limit of 5MB is reached, # download speed drops to
> > 438,000 bits or 3,504,000 MB per sec. Current # contracted Internet
> > connection speed w/ TP is at 7MB per sec. delay_parameters 1
> > 750000/750000 438000/625000
>
> > acl big_files url_regex -i ftp .exe .mp3 .vqf .tar.gz .gz .rpm .zip
> > .rar .avi .mpeg .mpe .mpg .qt .ram .rm .iso .raw .wav .dmg .mp4
> > .img .flv .wmv .divx .mov .bz2 .deb
>
> Another long list of regex patterns. Notice how these are permitted to
> match anywhere in the entie URL. Including domain names.
>
> FTP traffic in particular is not guaranteed to be "big files".
>
> <snip>
> > Intially, I set mem_cache=2134MB and after noticing these memory
> > leaks, I dropped it down to 1344MB. Memory leaks are still
> > occurring.
> >
> > Am I using anything that is known to cause memory leaks?
> >
> > If there is additional information that you need, please do not
> > hesitate to ask! Thanks.
>
> A copy of the manager "mem" report would be very useful to see whats
> using the memory.
> Note that it is a TSV format, so please save as .tsv file and attach.
> rather than cut-n-pasting inline.
>
Thanks, Amos, for your pointers.
I've commented out all the fresh_patterns lines appearing above the last two lines.
I also have dropped diskd in favor of using aufs exclusively, taking out the min-size parameter. I've commented out the diskd_program support option. In the previous version of squid (2.7) I had split the cache_dir into two types with great success using coss and aufs. Previously I had only aufs and performance wasn't where I wanted it. Apparently coss is no longer supported in the 3.x version of squid atop FreeBSD.
The pathname for the cache swap logs have been fixed. Apparently this came from a squid.conf example that I copied in parts. Would this be the reason why we are seeing the error messages in /var/log/messages regarding swapping mentioned in my original post?
The hierarchy_stoplist line has been stripped out as you say it is deprecated.
The mem .TSV file is attached herewith.
Currently I have the cache_dir located on the OS disk and all of the cache logging files on a second drive. Is this the optimal setup of cache-dir and logs?
Your comments are much appreciated!
~Doug
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: squid-internal-mgr_MEM.txt
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20141125/29500fc5/attachment-0001.txt>
More information about the squid-users
mailing list