[squid-users] Huge amount of time_wait connections after upgrade from v2 to v3
Amos Jeffries
squid3 at treenet.co.nz
Wed Jun 7 11:34:03 UTC 2017
On 07/06/17 12:13, Ivan Larionov wrote:
> Hi!
>
> We recently updated from squid v2 to v3 and now see huge increase in
> connections in TIME_WAIT state on our squid servers (verified that
> this is clients connections).
The biggest change between 2.7 and 3.5 in this area is that 2.7 was
HTTP/1.0 which closed TCP connections after each request by default, and
3.5 is HTTP/1.1 which does not. So connections are more likely to
persist until they hit some TCP timeout then enter the slow TIME_WAIT
process.
There were also some other bugs identified in older 3.5 releases which
increased the TIME_WAIT specifically. I thought those were almost all
fixed by now, but YMMV whether you hit the remaining issues.
A workaround it to set
<http://www.squid-cache.org/Doc/config/client_idle_pconn_timeout/> to a
shorter value than the default 2min. eg you might want it to be 30sec
or so.
>
> See versions and amount of such connections under the same load with
> the same configs (except some incompatible stuff):
>
> squid 2.7.STABLE9
>
> configure options: '--program-prefix=' '--prefix=/usr'
> '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin'
> '--sysconfdir=/etc' '--includedir=/usr/include' '--libdir=/usr/lib'
> '--libexecdir=/usr/libexec' '--sharedstatedir=/usr/com'
> '--mandir=/usr/share/man' '--infodir=/usr/share/info'
> '--exec_prefix=/usr' '--bindir=/usr/sbin'
> '--libexecdir=/usr/lib/squid' '--localstatedir=/var'
> '--datadir=/usr/share' '--sysconfdir=/etc/squid' '--enable-epoll'
> '--enable-removal-policies=heap,lru' '--enable-storeio=aufs'
> '--enable-delay-pools' '--with-pthreads' '--enable-cache-digests'
> '--enable-useragent-log' '--enable-referer-log' '--with-large-files'
> '--with-maxfd=16384' '--enable-err-languages=English'
>
> # netstat -tn | grep TIME_WAIT | grep 3128 | wc -l
> 95
>
> squid 3.5.25
>
> configure options: '--program-prefix=' '--prefix=/usr'
> '--exec-prefix=/usr' '--bindir=/usr/sbin' '--sbindir=/usr/sbin'
> '--sysconfdir=/etc/squid' '--libdir=/usr/lib'
> '--libexecdir=/usr/lib/squid' '--includedir=/usr/include'
> '--datadir=/usr/share' '--sharedstatedir=/usr/com'
> '--localstatedir=/var' '--mandir=/usr/share/man'
> '--infodir=/usr/share/info' '--enable-epoll'
> '--enable-removal-policies=heap,lru' '--enable-storeio=aufs'
> '--enable-delay-pools' '--with-pthreads' '--enable-cache-digests'
> '--enable-useragent-log' '--enable-referer-log' '--with-large-files'
> '--with-maxfd=16384' '--enable-err-languages=English' '--enable-htcp'
FYI, these options are not doing anything for Squid-3:
'--enable-useragent-log' '--enable-referer-log'
'--enable-err-languages=English'
>
> # netstat -tn | grep TIME_WAIT | grep 3128 | wc -l
> 11277
>
> Config:
>
> http_port 0.0.0.0:3128 <http://0.0.0.0:3128>
>
> acl localnet src 10.0.0.0/8 <http://10.0.0.0/8> # RFC1918 possible
> internal network
> acl localnet src 172.16.0.0/12 <http://172.16.0.0/12> # RFC1918
> possible internal network
> acl localnet src 192.168.0.0/16 <http://192.168.0.0/16> # RFC1918
> possible internal network
> acl localnet src fc00::/7 # RFC 4193 local private network range
> acl localnet src fe80::/10 # RFC 4291 link-local (directly
> plugged) machines
>
> acl SSL_ports port 443
>
> acl Safe_ports port 80 # http
> acl Safe_ports port 21 # ftp
> acl Safe_ports port 443 # https
> acl Safe_ports port 70 # gopher
> acl Safe_ports port 210 # wais
> acl Safe_ports port 280 # http-mgmt
> acl Safe_ports port 488 # gss-http
> acl Safe_ports port 591 # filemaker
> acl Safe_ports port 777 # multiling http
> acl Safe_ports port 1025-65535 # unregistered ports
>
> acl CONNECT method CONNECT
>
> ### START CUSTOM
> acl Purge_method method PURGE
>
> # Allow localhost to selectively flush the cache
> http_access allow localhost Purge_method
> http_access deny Purge_method
> ### END CUSTOM
>
> ### ALLOW ACCESS TO ALL PORTS
> # http_access deny !Safe_ports
> http_access deny CONNECT !SSL_ports
> http_access allow localhost manager
> http_access deny manager
>
> http_access allow localnet
> http_access allow localhost
> http_access deny all
>
> ### START CUSTOM
> # Disable icp
> icp_port 0
> # Allow ICP queries from local networks only
> icp_access allow localnet
> icp_access allow localhost
> icp_access deny all
>
> # Disable htcp
> htcp_port 0
> # Allow HTCP queries from local networks only
> htcp_access allow localnet
> htcp_access allow localhost
> htcp_access deny all
FYI: setting icp_access and htcp_access is pointless when the relevant
port is 0. That port 0 disables the entire component.
>
> # Check for custom request header
> acl custom_acl req_header x-use-custom-proxy -i true
> # Check for x-use-new-proxy request header
> acl custom_new_acl req_header x-use-new-proxy -i true
>
> # first_proxy
> cache_peer 127.0.0.1 parent 18070 0 no-query no-digest name=first_proxy
> cache_peer_access first_proxy deny custom_acl
> cache_peer_access first_proxy deny custom_new_acl
>
> # second_proxy
> cache_peer 127.0.0.1 parent 18079 0 no-query no-digest name=second_proxy
> cache_peer_access second_proxy allow custom_acl
> cache_peer_access second_proxy allow custom_new_acl
> cache_peer_access second_proxy deny all
>
> never_direct allow all
>
> cache_mem 4620591 KB
> maximum_object_size_in_memory 8 KB
> memory_replacement_policy heap LRU
> cache_replacement_policy heap LRU
>
> cache_dir aufs /mnt/services/squid/cache 891289 16 256
>
> minimum_object_size 64 bytes # none-zero so we dont cache mistakes
> maximum_object_size 102400 KB
>
> logformat combined %>a %ui %un [%tl] "%rm %ru HTTP/%rv" %>Hs %<st %tr
> "%{Referer}>h" "%{User-Agent}>h" %Ss:%Sh
> logformat squid %ts.%03tu %6tr %>a %Ss/%03>Hs %<st %rm %ru %un %Sh/%<A %mt
Please do not re-define these formats. If you want to use the default
format they are defined internally by Squid3, if you want any
customizations use a different format name.
>
> access_log stdio:/var/log/squid/access.log combined
> cache_log /var/log/squid/cache.log
> cache_store_log none
> logfile_rotate 0
>
> client_db off
>
> pid_filename /var/run/squid.pid
>
>
> coredump_dir /var/cache
> ### END CUSTOM
>
> refresh_pattern ^ftp: 1440 20% 10080
> refresh_pattern ^gopher: 1440 0% 1440
> # refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
Please do not remove that cgi-bin pattern. It is there to protect the
cache against servers with broken/ancient CGI engines. It is designed
explicitly so modern dynamic sites that provide proper cacheability
headers can still be stored. So no harm and only benefits from in
leaving it there.
Amos
More information about the squid-users
mailing list