[squid-users] Memory Leak Squid 3.4.9 on FreeBSD 10.0 x64
Doug Sampson
dougs at dawnsign.com
Mon Nov 24 20:06:52 UTC 2014
Recently due to squid 2.7 being EOL'ed, we migrated our squid server to version 3.4.9 on a FreeBSD 10.0-RELEASE running on 64-bit hardware. We started seeing paging file being swapped out eventually running out of available memory. From the time squid gets started it usually takes about two days before we see these entries in /var/log/messages as follows:
+swap_pager_getswapspace(16): failed
+swap_pager_getswapspace(16): failed
+swap_pager_getswapspace(16): failed
+swap_pager_getswapspace(12): failed
+swap_pager_getswapspace(16): failed
+swap_pager_getswapspace(12): failed
+swap_pager_getswapspace(6): failed
+swap_pager_getswapspace(16): failed
Looking at the 'top' results, I see that the swap file has been totally exhausted. Memory used by squid hovers around 2.3GB out of the total 3GB of system memory.
I am not sure what is causing these memory leaks. After rebooting, squid-internal-mgr/info shows the following statistics:
Squid Object Cache: Version 3.4.9
Build Info:
Start Time: Mon, 24 Nov 2014 18:39:08 GMT
Current Time: Mon, 24 Nov 2014 19:39:13 GMT
Connection information for squid:
Number of clients accessing cache: 18
Number of HTTP requests received: 10589
Number of ICP messages received: 0
Number of ICP messages sent: 0
Number of queued ICP replies: 0
Number of HTCP messages received: 0
Number of HTCP messages sent: 0
Request failure ratio: 0.00
Average HTTP requests per minute since start: 176.2
Average ICP messages per minute since start: 0.0
Select loop called: 763993 times, 4.719 ms avg
Cache information for squid:
Hits as % of all requests: 5min: 3.2%, 60min: 17.0%
Hits as % of bytes sent: 5min: 2.0%, 60min: 6.7%
Memory hits as % of hit requests: 5min: 0.0%, 60min: 37.2%
Disk hits as % of hit requests: 5min: 22.2%, 60min: 33.2%
Storage Swap size: 7361088 KB
Storage Swap capacity: 58.5% used, 41.5% free
Storage Mem size: 54348 KB
Storage Mem capacity: 3.9% used, 96.1% free
Mean Object Size: 23.63 KB
Requests given to unlinkd: 1
Median Service Times (seconds) 5 min 60 min:
HTTP Requests (All): 0.10857 0.19742
Cache Misses: 0.10857 0.32154
Cache Hits: 0.08265 0.01387
Near Hits: 0.15048 0.12106
Not-Modified Replies: 0.00091 0.00091
DNS Lookups: 0.05078 0.05078
ICP Queries: 0.00000 0.00000
Resource usage for squid:
UP Time: 3605.384 seconds
CPU Time: 42.671 seconds
CPU Usage: 1.18%
CPU Usage, 5 minute avg: 0.72%
CPU Usage, 60 minute avg: 1.17%
Maximum Resident Size: 845040 KB
Page faults with physical i/o: 20
Memory accounted for:
Total accounted: 105900 KB
memPoolAlloc calls: 2673353
memPoolFree calls: 2676487
File descriptor usage for squid:
Maximum number of file descriptors: 87516
Largest file desc currently in use: 310
Number of file desc currently in use: 198
Files queued for open: 0
Available number of file descriptors: 87318
Reserved number of file descriptors: 100
Store Disk files open: 0
Internal Data Structures:
311543 StoreEntries
4421 StoreEntries with MemObjects
4416 Hot Object Cache Items
311453 on-disk objects
I will post another one tomorrow that will indicate growing memory/swapfile consumption.
Here is my squid.conf:
# OPTIONS FOR AUTHENTICATION
# -----------------------------------------------------------------------------
# 1st four lines for
auth_param basic children 5
auth_param basic realm Squid proxy-caching web server
auth_param basic credentialsttl 2 hours
auth_param basic casesensitive off
# next three lines for kerberos authentication (needed to use usernames)
# used in conjunction with "acl auth proxy_auth" line below
#auth_param negotiate program /usr/local/libexec/squid/negotiate_kerberos_auth -i
#auth_param negotiate children 50 startup=10 idle=5
#auth_param negotiate keep_alive on
# ACCESS CONTROLS
# -----------------------------------------------------------------------------
# Example rule allowing access from your local networks.
# Adapt to list your (internal) IP networks from where browsing
# should be allowed
#acl manager proto cache_object
acl manager url_regex -i ^cache_object:// /squid-internal-mgr/
acl adminhost src 192.168.1.149
acl localnet src 192.168.1.0/24 # RFC1918 possible internal network
acl localnet src fc00::/7 # RFC 4193 local private network range
acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged) machines
acl webserver src 198.168.1.35
acl some_big_clients src 192.168.1.149/32 #CI53
# We want to limit downloads of these type of files
# Put this all in one line
acl magic_words url_regex -i ftp .exe .mp3 .vqf .tar.gz .gz .rpm .zip .rar .avi .mpeg .mpe .mpg .qt .ram .rm .iso .raw .wav .dmg .mp4 .img
# We don't block .html, .gif, .jpg and similar files, because they
# generally don't consume much bandwidth
acl SSL_ports port 443
acl SSL_ports port 8443
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 # https
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl CONNECT method CONNECT
acl CONNECT method CONNECT
acl Winupdate dstdomain .microsoft.com # Dot is important
acl social-sites dstdomain .facebook.com .fbcdn.net
acl ZipInfo dstdomain .dial-a-zip.com
# in conjunction with negotiate_kerberos_auth line above
#acl auth proxy_auth REQUIRED
#
# Recommended minimum Access Permission configuration:
#
http_access allow manager localnet
http_access allow manager localhost
http_access allow manager webserver
http_access allow manager adminhost
http_access deny manager
acl PURGE method PURGE
http_access allow PURGE localhost
http_access deny PURGE
# Deny requests to certain unsafe ports
http_access deny !Safe_ports
# Deny CONNECT to other than secure SSL ports
http_access deny CONNECT !SSL_ports
# We strongly recommend the following be uncommented to protect innocent
# web applications running on the proxy server who think the only
# one who can access services on "localhost" is a local user
#http_access deny to_localhost
#
# INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS
#
redirector_access deny Winupdate
# in conjunction with negotiate_kerberos_auth line above
#http_access deny !auth
#http_access allow auth
# Example rule allowing access from your local networks.
# Adapt localnet in the ACL section to list your (internal) IP networks
# from where browsing should be allowed
http_access allow localnet
http_access allow localhost
http_access allow ZipInfo localnet
# And finally deny all other access to this proxy
http_access deny all
# NETWORK OPTIONS
# -----------------------------------------------------------------------------
# Squid normally listens to port 3128
http_port 3128
# OPTIONS WHICH AFFECT THE NEIGHBOR SELECTION ALGORITHM
# -----------------------------------------------------------------------------
hierarchy_stoplist cgi-bin ?
# MEMORY CACHE OPTIONS
# -----------------------------------------------------------------------------
cache_mem 1366 MB
#cache_mem 2134 MB
#maximum_object_size_in_memory 64 KB
maximum_object_size_in_memory 128 KB
# DISK CACHE OPTIONS
# -----------------------------------------------------------------------------
cache_replacement_policy heap LFUDA
cache_dir aufs /data/squid/aufs_cache 4096 16 256 min-size=131073
cache_dir diskd /data/squid/diskd_cache 8192 16 256 Q1=64 Q2=72 max-size=131072
#maximum_object_size 122880 KB
maximum_object_size 153600 KB
cache_swap_low 90
cache_swap_high 95
# LOGFILE OPTIONS
# -----------------------------------------------------------------------------
access_log daemon:/data/squid/logs/access.log
cache_store_log daemon:/data/squid/logs/store.log
cache_swap_log /var/spool/squid/%s
logfile_rotate 28
# OPTIONS FOR TROUBLESHOOTING
# -----------------------------------------------------------------------------
cache_log /data/squid/logs/cache.log
# Leave coredumps in the first cache dir
coredump_dir /data/squid
# OPTIONS FOR EXTERNAL SUPPORT PROGRAMS
# -----------------------------------------------------------------------------
diskd_program /usr/local/libexec/squid/diskd
# OPTIONS FOR TUNING THE CACHE
# -----------------------------------------------------------------------------
refresh_pattern http://.*\.windowsupdate\.microsoft\.com/ 0 80% 20160
refresh_pattern http://office\.microsoft\.com/ 0 80% 20160
refresh_pattern http://windowsupdate\.microsoft\.com/ 0 80% 20160
refresh_pattern http://w?xpsp[0-9]\.microsoft\.com/ 0 80% 20160
refresh_pattern http://w2ksp[0-9]\.microsoft\.com/ 0 80% 20160
refresh_pattern http://download\.microsoft\.com/ 0 80% 20160
refresh_pattern http://download\.macromedia\.com/ 0 80% 20160
refresh_pattern http://ftp\.software\.ibm\.com/ 0 80% 20160
refresh_pattern cgi-bin 1 20% 2
refresh_pattern \.asp$ 1 20% 2
refresh_pattern \.acgi$ 1 20% 2
refresh_pattern \.cgi$ 1 20% 2
refresh_pattern \.pl$ 1 20% 2
refresh_pattern \.shtml$ 1 20% 2
refresh_pattern \.php3$ 1 20% 2
refresh_pattern \? 1 20% 2
refresh_pattern \.gif$ 10080 90% 43200
refresh_pattern \.png$ 10080 90% 43200
refresh_pattern \.jpg$ 10080 90% 43200
refresh_pattern \.ico$ 10080 90% 43200
refresh_pattern \.bom\.gov\.au 30 20% 120
refresh_pattern \.html$ 480 50% 22160
refresh_pattern \.htm$ 480 50% 22160
refresh_pattern \.css$ 480 50% 22160
refresh_pattern \.js$ 480 50% 22160
refresh_pattern \.class$ 10080 90% 43200
refresh_pattern \.zip$ 10080 90% 43200
refresh_pattern \.jpeg$ 10080 90% 43200
refresh_pattern \.mid$ 10080 90% 43200
refresh_pattern \.shtml$ 480 50% 22160
refresh_pattern \.exe$ 10080 90% 43200
refresh_pattern \.thm$ 10080 90% 43200
refresh_pattern \.wav$ 10080 90% 43200
refresh_pattern \.mp4$ 10080 90% 43200
refresh_pattern \.txt$ 10080 90% 43200
refresh_pattern \.cab$ 10080 90% 43200
refresh_pattern \.au$ 10080 90% 43200
refresh_pattern \.mov$ 10080 90% 43200
refresh_pattern \.xbm$ 10080 90% 43200
refresh_pattern \.ram$ 10080 90% 43200
refresh_pattern \.iso$ 10080 90% 43200
refresh_pattern \.avi$ 10080 90% 43200
refresh_pattern \.chtml$ 480 50% 22160
refresh_pattern \.thb$ 10080 90% 43200
refresh_pattern \.dcr$ 10080 90% 43200
refresh_pattern \.bmp$ 10080 90% 43200
refresh_pattern \.phtml$ 480 50% 22160
refresh_pattern \.mpg$ 10080 90% 43200
refresh_pattern \.pdf$ 10080 90% 43200
refresh_pattern \.art$ 10080 90% 43200
refresh_pattern \.swf$ 10080 90% 43200
refresh_pattern \.flv$ 10080 90% 43200
refresh_pattern \.x-flv$ 10080 90% 43200
refresh_pattern \.mp3$ 10080 90% 43200
refresh_pattern \.ra$ 10080 90% 43200
refresh_pattern \.spl$ 10080 90% 43200
refresh_pattern \.viv$ 10080 90% 43200
refresh_pattern \.doc$ 10080 90% 43200
refresh_pattern \.gz$ 10080 90% 43200
refresh_pattern \.Z$ 10080 90% 43200
refresh_pattern \.tgz$ 10080 90% 43200
refresh_pattern \.tar$ 10080 90% 43200
refresh_pattern \.vrm$ 10080 90% 43200
refresh_pattern \.vrml$ 10080 90% 43200
refresh_pattern \.aif$ 10080 90% 43200
refresh_pattern \.aifc$ 10080 90% 43200
refresh_pattern \.aiff$ 10080 90% 43200
refresh_pattern \.arj$ 10080 90% 43200
refresh_pattern \.c$ 10080 90% 43200
refresh_pattern \.cpt$ 10080 90% 43200
refresh_pattern \.dir$ 10080 90% 43200
refresh_pattern \.dxr$ 10080 90% 43200
refresh_pattern \.hqx$ 10080 90% 43200
refresh_pattern \.jpe$ 10080 90% 43200
refresh_pattern \.lha$ 10080 90% 43200
refresh_pattern \.lzh$ 10080 90% 43200
refresh_pattern \.midi$ 10080 90% 43200
refresh_pattern \.movie$ 10080 90% 43200
refresh_pattern \.mp2$ 10080 90% 43200
refresh_pattern \.mpe$ 10080 90% 43200
refresh_pattern \.mpeg$ 10080 90% 43200
refresh_pattern \.mpga$ 10080 90% 43200
refresh_pattern \.pl$ 10080 90% 43200
refresh_pattern \.ppt$ 10080 90% 43200
refresh_pattern \.ps$ 10080 90% 43200
refresh_pattern \.qt$ 10080 90% 43200
refresh_pattern \.qtm$ 10080 90% 43200
refresh_pattern \.rar$ 10080 90% 43200
refresh_pattern \.ras$ 10080 90% 43200
refresh_pattern \.sea$ 10080 90% 43200
refresh_pattern \.sit$ 10080 90% 43200
refresh_pattern \.tif$ 10080 90% 43200
refresh_pattern \.tiff$ 10080 90% 43200
refresh_pattern \.snd$ 10080 90% 43200
refresh_pattern \.wrl$ 10080 90% 43200
refresh_pattern ^ftp: 1440 60% 22160
refresh_pattern ^gopher: 1440 20% 1440
refresh_pattern -i (cgi-bin|\?) 0 0% 0
refresh_pattern . 480 50% 22160
# ADMINISTRATIVE PARAMETERS
# -----------------------------------------------------------------------------
cache_mgr admin at example.com
mail_from squid at example.com
cache_effective_user squid
cache_effective_group squid
# DELAY POOL PARAMETERS
# -----------------------------------------------------------------------------
delay_pools 2
delay_class 1 2
# When big_files are being downloaded, the first 5MB (625000 * 8 bits) are
# downloaded at max network speed. Once the file size limit of 5MB is reached,
# download speed drops to 438,000 bits or 3,504,000 MB per sec. Current
# contracted Internet connection speed w/ TP is at 7MB per sec.
delay_parameters 1 750000/750000 438000/625000
acl big_files url_regex -i ftp .exe .mp3 .vqf .tar.gz .gz .rpm .zip .rar .avi .mpeg .mpe .mpg .qt .ram .rm .iso .raw .wav .dmg .mp4 .img .flv .wmv .divx .mov .bz2 .deb
delay_access 1 allow big_files
delay_access 1 deny all
delay_class 2 2
# Any files other than big_files are downloaded at wire speed (currently 7MB)
# until 6MB file size limit is reached and thereafter at 6MB per sec (750,000
# * 8 bits)
delay_parameters 2 750000/750000 750000/750000
delay_access 2 allow localnet
delay_access 2 deny big_files
delay_access 2 deny all
delay_initial_bucket_level 25
# DNS OPTIONS
# -----------------------------------------------------------------------------
dns_nameservers 192.168.1.1
append_domain .example.com
# MISCELLANEOUS
# -----------------------------------------------------------------------------
memory_pools on
memory_pools_limit none
cachemgr_passwd none all
Squid was compiled with the following options:
===> The following configuration options are available for squid-3.4.9:
ARP_ACL=on: ARP/MAC/EUI based authentification
AUTH_KERB=on: Install Kerberos authentication helpers
AUTH_LDAP=on: Install LDAP authentication helpers
AUTH_NIS=off: Install NIS/YP authentication helpers
AUTH_SASL=off: Install SASL authentication helpers
AUTH_SMB=on: Install SMB auth. helpers (req. Samba)
AUTH_SQL=off: Install SQL based auth (uses MySQL)
CACHE_DIGESTS=on: Use cache digests
DEBUG=off: Build with extended debugging support
DELAY_POOLS=on: Delay pools (bandwidth limiting)
DNS_HELPER=on: Use external dnsserver processes for DNS
DOCS=on: Build and/or install documentation
ECAP=off: Loadable content adaptation modules
ESI=off: ESI support
EXAMPLES=on: Build and/or install examples
FOLLOW_XFF=on: Support for the X-Following-For header
FS_AUFS=on: AUFS (threaded-io) support
FS_DISKD=on: DISKD storage engine controlled by separate service
FS_ROCK=off: ROCK (unstable)
HTCP=on: HTCP support
ICAP=off: the ICAP client
ICMP=off: ICMP pinging and network measurement
IDENT=on: Ident lookups (RFC 931)
IPV6=on: IPv6 protocol support
KQUEUE=on: Kqueue(2) support
LARGEFILE=on: Support large (>2GB) cache and log files
LAX_HTTP=off: Do not enforce strict HTTP compliance
SNMP=on: SNMP support
SSL=on: SSL gatewaying support
SSL_CRTD=off: Use ssl_crtd to handle SSL cert requests
STACKTRACES=off: Enable automatic backtraces on fatal errors
TP_IPF=off: Transparent proxying with IPFilter
TP_IPFW=off: Transparent proxying with IPFW
TP_PF=off: Transparent proxying with PF
VIA_DB=off: Forward/Via database
WCCP=on: Web Cache Coordination Protocol
WCCPV2=on: Web Cache Coordination Protocol v2
Intially, I set mem_cache=2134MB and after noticing these memory leaks, I dropped it down to 1344MB. Memory leaks are still occurring.
Am I using anything that is known to cause memory leaks?
If there is additional information that you need, please do not hesitate to ask! Thanks.
~Doug
More information about the squid-users
mailing list