[squid-users] Huge amount of time_wait connections after upgrade from v2 to v3
Ivan Larionov
xeron.oskom at gmail.com
Fri Jul 7 01:55:48 UTC 2017
Hi. Sorry that I'm answering to the old thread. I was on vacation and
didn't have a chance to test the proposed solution.
Dieter, yes, I'm on the old CentOS 6 based OS (Amazon Linux) but with a new
kernel 4.9.27.
Amos, thank you for the suggestions about configure flags and squid config
options, I fixed all issues you pointed to.
Unfortunately following workarounds didn't help:
* client_idle_pconn_timeout 30 seconds
* half_closed_clients on
* client_persistent_connections off
* server_persistent_connections off
However I assumed that this is a bug and that I can find older version
which worked fine. I started testing from 3.1.x all the way to 3.5.26 and
this is what I found:
* All versions until 3.5.21 work fine. There no issues with huge amount of
TIME_WAIT connections under load.
* 3.5.20 is the latest stable version.
* 3.5.21 is the first broken version.
* 3.5.23, 3.5.25, 3.5.26 are broken as well.
This effectively means that bug is somewhere in between 3.5.20 and 3.5.21.
I hope this helps and I hope you'll be able to find an issue. If you can
create a bug report based on this information and post it here it would be
awesome.
Thank you.
On Wed, Jun 7, 2017 at 4:34 AM, Amos Jeffries <squid3 at treenet.co.nz> wrote:
> On 07/06/17 12:13, Ivan Larionov wrote:
>
>> Hi!
>>
>> We recently updated from squid v2 to v3 and now see huge increase in
>> connections in TIME_WAIT state on our squid servers (verified that this is
>> clients connections).
>>
>
> The biggest change between 2.7 and 3.5 in this area is that 2.7 was
> HTTP/1.0 which closed TCP connections after each request by default, and
> 3.5 is HTTP/1.1 which does not. So connections are more likely to persist
> until they hit some TCP timeout then enter the slow TIME_WAIT process.
>
> There were also some other bugs identified in older 3.5 releases which
> increased the TIME_WAIT specifically. I thought those were almost all fixed
> by now, but YMMV whether you hit the remaining issues.
> A workaround it to set <http://www.squid-cache.org/Do
> c/config/client_idle_pconn_timeout/> to a shorter value than the default
> 2min. eg you might want it to be 30sec or so.
>
>
>
>
>> See versions and amount of such connections under the same load with the
>> same configs (except some incompatible stuff):
>>
>> squid 2.7.STABLE9
>>
>> configure options: '--program-prefix=' '--prefix=/usr'
>> '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin'
>> '--sysconfdir=/etc' '--includedir=/usr/include' '--libdir=/usr/lib'
>> '--libexecdir=/usr/libexec' '--sharedstatedir=/usr/com'
>> '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--exec_prefix=/usr'
>> '--bindir=/usr/sbin' '--libexecdir=/usr/lib/squid' '--localstatedir=/var'
>> '--datadir=/usr/share' '--sysconfdir=/etc/squid' '--enable-epoll'
>> '--enable-removal-policies=heap,lru' '--enable-storeio=aufs'
>> '--enable-delay-pools' '--with-pthreads' '--enable-cache-digests'
>> '--enable-useragent-log' '--enable-referer-log' '--with-large-files'
>> '--with-maxfd=16384' '--enable-err-languages=English'
>>
>> # netstat -tn | grep TIME_WAIT | grep 3128 | wc -l
>> 95
>>
>> squid 3.5.25
>>
>> configure options: '--program-prefix=' '--prefix=/usr'
>> '--exec-prefix=/usr' '--bindir=/usr/sbin' '--sbindir=/usr/sbin'
>> '--sysconfdir=/etc/squid' '--libdir=/usr/lib' '--libexecdir=/usr/lib/squid'
>> '--includedir=/usr/include' '--datadir=/usr/share'
>> '--sharedstatedir=/usr/com' '--localstatedir=/var'
>> '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--enable-epoll'
>> '--enable-removal-policies=heap,lru' '--enable-storeio=aufs'
>> '--enable-delay-pools' '--with-pthreads' '--enable-cache-digests'
>> '--enable-useragent-log' '--enable-referer-log' '--with-large-files'
>> '--with-maxfd=16384' '--enable-err-languages=English' '--enable-htcp'
>>
>
> FYI, these options are not doing anything for Squid-3:
> '--enable-useragent-log' '--enable-referer-log'
> '--enable-err-languages=English'
>
>
>
>> # netstat -tn | grep TIME_WAIT | grep 3128 | wc -l
>> 11277
>>
>> Config:
>>
>> http_port 0.0.0.0:3128 <http://0.0.0.0:3128>
>>
>> acl localnet src 10.0.0.0/8 <http://10.0.0.0/8> # RFC1918 possible
>> internal network
>> acl localnet src 172.16.0.0/12 <http://172.16.0.0/12> # RFC1918
>> possible internal network
>> acl localnet src 192.168.0.0/16 <http://192.168.0.0/16> # RFC1918
>> possible internal network
>>
>> acl localnet src fc00::/7 # RFC 4193 local private network range
>> acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged)
>> machines
>>
>> acl SSL_ports port 443
>>
>> acl Safe_ports port 80 # http
>> acl Safe_ports port 21 # ftp
>> acl Safe_ports port 443 # https
>> acl Safe_ports port 70 # gopher
>> acl Safe_ports port 210 # wais
>> acl Safe_ports port 280 # http-mgmt
>> acl Safe_ports port 488 # gss-http
>> acl Safe_ports port 591 # filemaker
>> acl Safe_ports port 777 # multiling http
>> acl Safe_ports port 1025-65535 # unregistered ports
>>
>> acl CONNECT method CONNECT
>>
>> ### START CUSTOM
>> acl Purge_method method PURGE
>>
>> # Allow localhost to selectively flush the cache
>> http_access allow localhost Purge_method
>> http_access deny Purge_method
>> ### END CUSTOM
>>
>> ### ALLOW ACCESS TO ALL PORTS
>> # http_access deny !Safe_ports
>> http_access deny CONNECT !SSL_ports
>> http_access allow localhost manager
>> http_access deny manager
>>
>> http_access allow localnet
>> http_access allow localhost
>> http_access deny all
>>
>> ### START CUSTOM
>> # Disable icp
>> icp_port 0
>> # Allow ICP queries from local networks only
>> icp_access allow localnet
>> icp_access allow localhost
>> icp_access deny all
>>
>> # Disable htcp
>> htcp_port 0
>> # Allow HTCP queries from local networks only
>> htcp_access allow localnet
>> htcp_access allow localhost
>> htcp_access deny all
>>
>
> FYI: setting icp_access and htcp_access is pointless when the relevant
> port is 0. That port 0 disables the entire component.
>
>
>> # Check for custom request header
>> acl custom_acl req_header x-use-custom-proxy -i true
>> # Check for x-use-new-proxy request header
>> acl custom_new_acl req_header x-use-new-proxy -i true
>>
>> # first_proxy
>> cache_peer 127.0.0.1 parent 18070 0 no-query no-digest name=first_proxy
>> cache_peer_access first_proxy deny custom_acl
>> cache_peer_access first_proxy deny custom_new_acl
>>
>> # second_proxy
>> cache_peer 127.0.0.1 parent 18079 0 no-query no-digest name=second_proxy
>> cache_peer_access second_proxy allow custom_acl
>> cache_peer_access second_proxy allow custom_new_acl
>> cache_peer_access second_proxy deny all
>>
>> never_direct allow all
>>
>> cache_mem 4620591 KB
>> maximum_object_size_in_memory 8 KB
>> memory_replacement_policy heap LRU
>> cache_replacement_policy heap LRU
>>
>> cache_dir aufs /mnt/services/squid/cache 891289 16 256
>>
>> minimum_object_size 64 bytes # none-zero so we dont cache mistakes
>> maximum_object_size 102400 KB
>>
>> logformat combined %>a %ui %un [%tl] "%rm %ru HTTP/%rv" %>Hs %<st %tr
>> "%{Referer}>h" "%{User-Agent}>h" %Ss:%Sh
>> logformat squid %ts.%03tu %6tr %>a %Ss/%03>Hs %<st %rm %ru %un %Sh/%<A %mt
>>
>
> Please do not re-define these formats. If you want to use the default
> format they are defined internally by Squid3, if you want any
> customizations use a different format name.
>
>
>> access_log stdio:/var/log/squid/access.log combined
>> cache_log /var/log/squid/cache.log
>> cache_store_log none
>> logfile_rotate 0
>>
>> client_db off
>>
>> pid_filename /var/run/squid.pid
>>
>>
>> coredump_dir /var/cache
>> ### END CUSTOM
>>
>> refresh_pattern ^ftp: 1440 20% 10080
>> refresh_pattern ^gopher: 1440 0% 1440
>> # refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
>>
>
> Please do not remove that cgi-bin pattern. It is there to protect the
> cache against servers with broken/ancient CGI engines. It is designed
> explicitly so modern dynamic sites that provide proper cacheability headers
> can still be stored. So no harm and only benefits from in leaving it there.
>
>
> Amos
>
>
> _______________________________________________
> squid-users mailing list
> squid-users at lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users
>
--
With best regards, Ivan Larionov.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20170706/1fd9e614/attachment-0001.html>
More information about the squid-users
mailing list