<div dir="ltr">Hi. Sorry that I'm answering to the old thread. I was on vacation and didn't have a chance to test the proposed solution.<div><br></div><div>Dieter, yes, I'm on the old CentOS 6 based OS (Amazon Linux) but with a new kernel 4.9.27.</div><div><br></div><div>Amos, thank you for the suggestions about configure flags and squid config options, I fixed all issues you pointed to.</div><div><br></div><div>Unfortunately following workarounds didn't help:</div><div><div><br></div><div>* client_idle_pconn_timeout 30 seconds<br></div><div>* half_closed_clients on</div><div>* client_persistent_connections off</div><div>* server_persistent_connections off</div></div><div><br></div><div>However I assumed that this is a bug and that I can find older version which worked fine. I started testing from 3.1.x all the way to 3.5.26 and this is what I found:</div><div><br></div><div>* All versions until 3.5.21 work fine. There no issues with huge amount of TIME_WAIT connections under load.</div><div>* 3.5.20 is the latest stable version.</div><div>* 3.5.21 is the first broken version.</div><div>* 3.5.23, 3.5.25, 3.5.26 are broken as well.</div><div><br></div><div>This effectively means that bug is somewhere in between 3.5.20 and 3.5.21.</div><div><br></div><div>I hope this helps and I hope you'll be able to find an issue. If you can create a bug report based on this information and post it here it would be awesome.</div><div><br></div><div>Thank you.</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jun 7, 2017 at 4:34 AM, Amos Jeffries <span dir="ltr"><<a href="mailto:squid3@treenet.co.nz" target="_blank">squid3@treenet.co.nz</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On 07/06/17 12:13, Ivan Larionov wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi!<br>
<br>
We recently updated from squid v2 to v3 and now see huge increase in connections in TIME_WAIT state on our squid servers (verified that this is clients connections).<br>
</blockquote>
<br></span>
The biggest change between 2.7 and 3.5 in this area is that 2.7 was HTTP/1.0 which closed TCP connections after each request by default, and 3.5 is HTTP/1.1 which does not. So connections are more likely to persist until they hit some TCP timeout then enter the slow TIME_WAIT process.<br>
<br>
There were also some other bugs identified in older 3.5 releases which increased the TIME_WAIT specifically. I thought those were almost all fixed by now, but YMMV whether you hit the remaining issues.<br>
A workaround it to set <<a href="http://www.squid-cache.org/Doc/config/client_idle_pconn_timeout/" rel="noreferrer" target="_blank">http://www.squid-cache.org/Do<wbr>c/config/client_idle_pconn_tim<wbr>eout/</a>> to a shorter value than the default 2min. eg you might want it to be 30sec or so.<span class=""><br>
<br>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
See versions and amount of such connections under the same load with the same configs (except some incompatible stuff):<br>
<br>
squid 2.7.STABLE9<br>
<br>
configure options: '--program-prefix=' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--includedir=/usr/include' '--libdir=/usr/lib' '--libexecdir=/usr/libexec' '--sharedstatedir=/usr/com' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--exec_prefix=/usr' '--bindir=/usr/sbin' '--libexecdir=/usr/lib/squid' '--localstatedir=/var' '--datadir=/usr/share' '--sysconfdir=/etc/squid' '--enable-epoll' '--enable-removal-policies=hea<wbr>p,lru' '--enable-storeio=aufs' '--enable-delay-pools' '--with-pthreads' '--enable-cache-digests' '--enable-useragent-log' '--enable-referer-log' '--with-large-files' '--with-maxfd=16384' '--enable-err-languages=Englis<wbr>h'<br>
<br>
# netstat -tn | grep TIME_WAIT | grep 3128 | wc -l<br>
95<br>
<br>
squid 3.5.25<br>
<br>
configure options: '--program-prefix=' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/sbin' '--sbindir=/usr/sbin' '--sysconfdir=/etc/squid' '--libdir=/usr/lib' '--libexecdir=/usr/lib/squid' '--includedir=/usr/include' '--datadir=/usr/share' '--sharedstatedir=/usr/com' '--localstatedir=/var' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--enable-epoll' '--enable-removal-policies=hea<wbr>p,lru' '--enable-storeio=aufs' '--enable-delay-pools' '--with-pthreads' '--enable-cache-digests' '--enable-useragent-log' '--enable-referer-log' '--with-large-files' '--with-maxfd=16384' '--enable-err-languages=Englis<wbr>h' '--enable-htcp'<br>
</blockquote>
<br></span>
FYI, these options are not doing anything for Squid-3:<br>
'--enable-useragent-log' '--enable-referer-log' '--enable-err-languages=Englis<wbr>h'<br>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">
<br>
# netstat -tn | grep TIME_WAIT | grep 3128 | wc -l<br></span>
11277<br>
<br>
Config:<br>
<br>
http_port <a href="http://0.0.0.0:3128" rel="noreferrer" target="_blank">0.0.0.0:3128</a> <<a href="http://0.0.0.0:3128" rel="noreferrer" target="_blank">http://0.0.0.0:3128</a>><br>
<br>
acl localnet src <a href="http://10.0.0.0/8" rel="noreferrer" target="_blank">10.0.0.0/8</a> <<a href="http://10.0.0.0/8" rel="noreferrer" target="_blank">http://10.0.0.0/8</a>> # RFC1918 possible internal network<br>
acl localnet src <a href="http://172.16.0.0/12" rel="noreferrer" target="_blank">172.16.0.0/12</a> <<a href="http://172.16.0.0/12" rel="noreferrer" target="_blank">http://172.16.0.0/12</a>> # RFC1918 possible internal network<br>
acl localnet src <a href="http://192.168.0.0/16" rel="noreferrer" target="_blank">192.168.0.0/16</a> <<a href="http://192.168.0.0/16" rel="noreferrer" target="_blank">http://192.168.0.0/16</a>> # RFC1918 possible internal network<div><div class="h5"><br>
acl localnet src fc00::/7 # RFC 4193 local private network range<br>
acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged) machines<br>
<br>
acl SSL_ports port 443<br>
<br>
acl Safe_ports port 80 # http<br>
acl Safe_ports port 21 # ftp<br>
acl Safe_ports port 443 # https<br>
acl Safe_ports port 70 # gopher<br>
acl Safe_ports port 210 # wais<br>
acl Safe_ports port 280 # http-mgmt<br>
acl Safe_ports port 488 # gss-http<br>
acl Safe_ports port 591 # filemaker<br>
acl Safe_ports port 777 # multiling http<br>
acl Safe_ports port 1025-65535 # unregistered ports<br>
<br>
acl CONNECT method CONNECT<br>
<br>
### START CUSTOM<br>
acl Purge_method method PURGE<br>
<br>
# Allow localhost to selectively flush the cache<br>
http_access allow localhost Purge_method<br>
http_access deny Purge_method<br>
### END CUSTOM<br>
<br>
### ALLOW ACCESS TO ALL PORTS<br>
# http_access deny !Safe_ports<br>
http_access deny CONNECT !SSL_ports<br>
http_access allow localhost manager<br>
http_access deny manager<br>
<br>
http_access allow localnet<br>
http_access allow localhost<br>
http_access deny all<br>
<br>
### START CUSTOM<br>
# Disable icp<br>
icp_port 0<br>
# Allow ICP queries from local networks only<br>
icp_access allow localnet<br>
icp_access allow localhost<br>
icp_access deny all<br>
<br>
# Disable htcp<br>
htcp_port 0<br>
# Allow HTCP queries from local networks only<br>
htcp_access allow localnet<br>
htcp_access allow localhost<br>
htcp_access deny all<br>
</div></div></blockquote>
<br>
FYI: setting icp_access and htcp_access is pointless when the relevant port is 0. That port 0 disables the entire component.<span class=""><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
# Check for custom request header<br>
acl custom_acl req_header x-use-custom-proxy -i true<br>
# Check for x-use-new-proxy request header<br>
acl custom_new_acl req_header x-use-new-proxy -i true<br>
<br>
# first_proxy<br>
cache_peer 127.0.0.1 parent 18070 0 no-query no-digest name=first_proxy<br>
cache_peer_access first_proxy deny custom_acl<br>
cache_peer_access first_proxy deny custom_new_acl<br>
<br>
# second_proxy<br>
cache_peer 127.0.0.1 parent 18079 0 no-query no-digest name=second_proxy<br>
cache_peer_access second_proxy allow custom_acl<br>
cache_peer_access second_proxy allow custom_new_acl<br>
cache_peer_access second_proxy deny all<br>
<br>
never_direct allow all<br>
<br>
cache_mem 4620591 KB<br>
maximum_object_size_in_memory 8 KB<br>
memory_replacement_policy heap LRU<br>
cache_replacement_policy heap LRU<br>
<br>
cache_dir aufs /mnt/services/squid/cache 891289 16 256<br>
<br>
minimum_object_size 64 bytes # none-zero so we dont cache mistakes<br>
maximum_object_size 102400 KB<br>
<br>
logformat combined %>a %ui %un [%tl] "%rm %ru HTTP/%rv" %>Hs %<st %tr "%{Referer}>h" "%{User-Agent}>h" %Ss:%Sh<br>
logformat squid %ts.%03tu %6tr %>a %Ss/%03>Hs %<st %rm %ru %un %Sh/%<A %mt<br>
</blockquote>
<br></span>
Please do not re-define these formats. If you want to use the default format they are defined internally by Squid3, if you want any customizations use a different format name.<span class=""><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
access_log stdio:/var/log/squid/access.lo<wbr>g combined<br>
cache_log /var/log/squid/cache.log<br>
cache_store_log none<br>
logfile_rotate 0<br>
<br>
client_db off<br>
<br>
pid_filename /var/run/squid.pid<br>
<br>
<br>
coredump_dir /var/cache<br>
### END CUSTOM<br>
<br>
refresh_pattern ^ftp: 1440 20% 10080<br>
refresh_pattern ^gopher: 1440 0% 1440<br>
# refresh_pattern -i (/cgi-bin/|\?) 0 0% 0<br>
</blockquote>
<br></span>
Please do not remove that cgi-bin pattern. It is there to protect the cache against servers with broken/ancient CGI engines. It is designed explicitly so modern dynamic sites that provide proper cacheability headers can still be stored. So no harm and only benefits from in leaving it there.<span class="HOEnZb"><font color="#888888"><br>
<br>
<br>
Amos</font></span><div class="HOEnZb"><div class="h5"><br>
<br>
______________________________<wbr>_________________<br>
squid-users mailing list<br>
<a href="mailto:squid-users@lists.squid-cache.org" target="_blank">squid-users@lists.squid-cache.<wbr>org</a><br>
<a href="http://lists.squid-cache.org/listinfo/squid-users" rel="noreferrer" target="_blank">http://lists.squid-cache.org/l<wbr>istinfo/squid-users</a><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature">With best regards, Ivan Larionov.</div>
</div>