<div dir="ltr"><div><span style="color:rgb(80,0,80);font-size:12.8px">I think I found part of the config which triggers the retry.</span></div><div><br></div><div>======<span style="color:rgb(80,0,80);font-size:12.8px"><br></span></div><div><span style="color:rgb(80,0,80);font-size:12.8px"><br></span></div><div><div># Check for newproxy request header</div><div>acl newproxy_acl req_header x-use-newproxy -i true</div><div><br></div><div># proxy</div><div>cache_peer 127.0.0.1 parent 18070 0 no-query no-digest no-netdb-exchange name=proxy</div><div>cache_peer_access proxy deny newproxy_acl</div><div><br></div><div># newproxy</div><div>cache_peer 127.0.0.1 parent 18079 0 no-query no-digest no-netdb-exchange name=newproxy</div><div>cache_peer_access newproxy allow newproxy_acl</div><div>cache_peer_access newproxy deny all</div><div><br></div><div>never_direct allow all</div></div><div><br></div><div>======</div><div><br></div><div>I see retries only when squid config has 2 parents. If I comment out everything related to "newproxy" I can't reproduce this behavior anymore.</div><div><br></div><div>BUT</div><div><br></div><div>My test request could only be forwarded to "proxy" since it doesn't have "x-use-newproxy" header.</div><div><br></div><div>Which results in the following:</div><div><br></div><div>squid has 2 parents</div><div>request could only be forwarded to the 1st parent due to ACL</div><div>parent doesn't respond and closes the TCP connection</div><div>squid retries with the same parent ignoring "forward_max_tries 1"</div><div><span style="color:rgb(80,0,80);font-size:12.8px"><br></span></div><div><span style="color:rgb(80,0,80);font-size:12.8px">I also want to clarify some facts which make me think that it could be a bug.</span></div><div><span style="color:rgb(80,0,80);font-size:12.8px"><br></span></div><div><span style="color:rgb(80,0,80);font-size:12.8px">1. There're no issues with TCP connection. Squid successfully connects to parent and sends an HTTP request.</span></div><div><span style="color:rgb(80,0,80);font-size:12.8px">2. Parent ACKs HTTP request and then correctly closes the connection with FIN,ACK after 40 seconds. There're no TCP timeouts/reconnects involved. </span><span style="color:rgb(80,0,80);font-size:12.8px">The only issue here is that parent doesn't send any HTTP response.</span></div><div><span style="color:rgb(80,0,80);font-size:12.8px">3. </span><font color="#500050"><span style="font-size:12.8px">forward_max_tries is set to 1 to make sure squid won't retry. Parent handles the retry so we don't want squid to make any additional retries.</span></font></div><div><span style="color:rgb(80,0,80);font-size:12.8px"><br></span></div><div><span style="color:rgb(80,0,80);font-size:12.8px">Also see </span><font color="#500050"><span style="font-size:12.8px"><a href="https://wiki.squid-cache.org/SquidFaq/InnerWorkings#When_does_Squid_re-forward_a_client_request.3F">https://wiki.squid-cache.org/SquidFaq/InnerWorkings#When_does_Squid_re-forward_a_client_request.3F</a></span></font></div><div><font color="#500050"><span style="font-size:12.8px"><br></span></font></div><div><font color="#500050"><span style="font-size:12.8px">> Squid does not try to re-forward a request if at least one of the following conditions is true:</span><br></font></div><div><font color="#500050"><span style="font-size:12.8px">> …</span></font></div><div><font color="#500050"><span style="font-size:12.8px">> The number of forwarding attempts exceeded forward_max_tries. For example, if you set forward_max_tries to 1 (one), then no requests will be re-forwarded.</span></font></div><div><span style="color:rgb(80,0,80);font-size:12.8px">> …</span><font color="#500050"><span style="font-size:12.8px"><br></span></font></div><div><font color="#500050"><span style="font-size:12.8px">> Squid has no alternative destinations to try. Please note that alternative destinations may include multiple next hop IP addresses and multiple peers.</span></font></div><div><span style="color:rgb(80,0,80);font-size:12.8px">> …</span><span style="color:rgb(80,0,80);font-size:12.8px"><br></span></div><div><br></div><div><span style="color:rgb(80,0,80);font-size:12.8px">Another part of config which may or may not be related (this is to increase the amount of local ports to use):</span></div><div><span style="color:rgb(80,0,80);font-size:12.8px"><br></span></div><div><div># 33% of traffic per local IP</div><div>acl third random 1/3</div><div>acl half random 1/2</div><div>tcp_outgoing_address 127.0.0.2 third</div><div>tcp_outgoing_address 127.0.0.3 half</div><div>tcp_outgoing_address 127.0.0.4</div></div><div><span style="color:rgb(80,0,80);font-size:12.8px"><br></span></div><div><span style="color:rgb(80,0,80);font-size:12.8px">Logs:</span></div><div><span style="color:rgb(80,0,80);font-size:12.8px"><br></span></div><div>ALL,2 (includes 44,2):<br></div><div><br></div><div><div>2017/11/27 15:53:40.542| 5,2| TcpAcceptor.cc(220) doAccept: New connection on FD 15</div><div>2017/11/27 15:53:40.542| 5,2| TcpAcceptor.cc(295) acceptNext: connection on local=<a href="http://0.0.0.0:3128">0.0.0.0:3128</a> remote=[::] FD 15 flags=9</div><div>2017/11/27 15:53:40.543| 11,2| client_side.cc(2372) parseHttpRequest: HTTP Client local=<a href="http://127.0.0.1:3128">127.0.0.1:3128</a> remote=<a href="http://127.0.0.1:53798">127.0.0.1:53798</a> FD 45 flags=1</div><div>2017/11/27 15:53:40.543| 11,2| client_side.cc(2373) parseHttpRequest: HTTP Client REQUEST:</div><div>---------</div><div>GET <a href="http://HOST:12345/">http://HOST:12345/</a> HTTP/1.1</div><div>Host: HOST:12345</div><div>User-Agent: curl/7.51.0</div><div>Accept: */*</div><div>Proxy-Connection: Keep-Alive</div><div><br></div><div><br></div><div>----------</div><div>2017/11/27 15:53:40.543| 85,2| client_side_request.cc(745) clientAccessCheckDone: The request GET <a href="http://HOST:12345/">http://HOST:12345/</a> is ALLOWED; last ACL checked: localhost</div><div>2017/11/27 15:53:40.543| 85,2| client_side_request.cc(721) clientAccessCheck2: No adapted_http_access configuration. default: ALLOW</div><div>2017/11/27 15:53:40.543| 85,2| client_side_request.cc(745) clientAccessCheckDone: The request GET <a href="http://HOST:12345/">http://HOST:12345/</a> is ALLOWED; last ACL checked: localhost</div><div>2017/11/27 15:53:40.543| 17,2| FwdState.cc(133) FwdState: Forwarding client request local=<a href="http://127.0.0.1:3128">127.0.0.1:3128</a> remote=<a href="http://127.0.0.1:53798">127.0.0.1:53798</a> FD 45 flags=1, url=<a href="http://HOST:12345/">http://HOST:12345/</a></div><div>2017/11/27 15:53:40.543| 44,2| peer_select.cc(258) peerSelectDnsPaths: Find IP destination for: <a href="http://HOST:12345/">http://HOST:12345/</a>' via 127.0.0.1</div><div>2017/11/27 15:53:40.543| 44,2| peer_select.cc(280) peerSelectDnsPaths: Found sources for '<a href="http://HOST:12345/">http://HOST:12345/</a>'</div><div>2017/11/27 15:53:40.543| 44,2| peer_select.cc(281) peerSelectDnsPaths: always_direct = DENIED</div><div>2017/11/27 15:53:40.543| 44,2| peer_select.cc(282) peerSelectDnsPaths: never_direct = ALLOWED</div><div>2017/11/27 15:53:40.543| 44,2| peer_select.cc(292) peerSelectDnsPaths: cache_peer = local=127.0.0.3 remote=<a href="http://127.0.0.1:18070">127.0.0.1:18070</a> flags=1</div><div>2017/11/27 15:53:40.543| 44,2| peer_select.cc(295) peerSelectDnsPaths: timedout = 0</div><div>2017/11/27 15:53:40.543| 11,2| http.cc(2229) sendRequest: HTTP Server local=<a href="http://127.0.0.3:57091">127.0.0.3:57091</a> remote=<a href="http://127.0.0.1:18070">127.0.0.1:18070</a> FD 40 flags=1</div><div>2017/11/27 15:53:40.543| 11,2| http.cc(2230) sendRequest: HTTP Server REQUEST:</div><div>---------</div><div>GET <a href="http://HOST:12345/">http://HOST:12345/</a> HTTP/1.1</div><div>User-Agent: curl/7.51.0</div><div>Accept: */*</div><div>Host: HOST:12345</div><div>Cache-Control: max-age=259200</div><div>Connection: keep-alive</div><div><br></div><div><br></div><div>----------</div><div><br></div><div>[SKIPPED 40 seconds until parent closes TCP connection with FIN,ACK]</div><div><br></div><div>2017/11/27 15:54:20.627| 11,2| http.cc(1299) continueAfterParsingHeader: WARNING: HTTP: Invalid Response: No object data received for <a href="http://HOST:12345/">http://HOST:12345/</a> AKA HOST/</div><div>2017/11/27 15:54:20.627| 17,2| FwdState.cc(655) handleUnregisteredServerEnd: self=0x3e31838*2 err=0x409b338 <a href="http://HOST:12345/">http://HOST:12345/</a></div><div>2017/11/27 15:54:20.627| 11,2| http.cc(2229) sendRequest: HTTP Server local=<a href="http://127.0.0.3:41355">127.0.0.3:41355</a> remote=<a href="http://127.0.0.1:18070">127.0.0.1:18070</a> FD 40 flags=1</div><div>2017/11/27 15:54:20.627| 11,2| http.cc(2230) sendRequest: HTTP Server REQUEST:</div><div>---------</div><div>GET <a href="http://HOST:12345/">http://HOST:12345/</a> HTTP/1.1</div><div>User-Agent: curl/7.51.0</div><div>Accept: */*</div><div>Host: HOST:12345</div><div>Cache-Control: max-age=259200</div><div>Connection: keep-alive</div><div><br></div><div><br></div><div>----------</div><div><br></div><div>[SKIPPED 40 seconds again until parent closes TCP connection with FIN,ACK]</div><div><br></div><div>2017/11/27 15:55:00.728| ctx: enter level 0: '<a href="http://HOST:12345/">http://HOST:12345/</a>'</div><div>2017/11/27 15:55:00.728| 11,2| http.cc(719) processReplyHeader: HTTP Server local=<a href="http://127.0.0.3:41355">127.0.0.3:41355</a> remote=<a href="http://127.0.0.1:18070">127.0.0.1:18070</a> FD 40 flags=1</div><div>2017/11/27 15:55:00.728| 11,2| http.cc(720) processReplyHeader: HTTP Server REPLY:</div><div>---------</div><div>HTTP/1.0 502 Bad Gateway</div><div>Cache-Control: no-cache</div><div>Connection: close</div><div>Content-Type: text/html</div><div><br></div><div><html><body><h1>502 Bad Gateway</h1></div><div>The server returned an invalid or incomplete response.</div><div></body></html></div><div><br></div><div>----------</div><div>2017/11/27 15:55:00.728| ctx: exit level 0</div><div>2017/11/27 15:55:00.728| 20,2| store.cc(996) checkCachable: StoreEntry::checkCachable: NO: not cachable</div><div>2017/11/27 15:55:00.728| 20,2| store.cc(996) checkCachable: StoreEntry::checkCachable: NO: not cachable</div><div>2017/11/27 15:55:00.728| 88,2| client_side_reply.cc(2073) processReplyAccessResult: The reply for GET <a href="http://HOST:12345/">http://HOST:12345/</a> is ALLOWED, because it matched (access_log stdio:/var/log/squid/access.log line)</div><div>2017/11/27 15:55:00.728| 11,2| client_side.cc(1409) sendStartOfMessage: HTTP Client local=<a href="http://127.0.0.1:3128">127.0.0.1:3128</a> remote=<a href="http://127.0.0.1:53798">127.0.0.1:53798</a> FD 45 flags=1</div><div>2017/11/27 15:55:00.728| 11,2| client_side.cc(1410) sendStartOfMessage: HTTP Client REPLY:</div><div>---------</div><div>HTTP/1.1 502 Bad Gateway</div><div>Date: Mon, 27 Nov 2017 23:54:20 GMT</div><div>Cache-Control: no-cache</div><div>Content-Type: text/html</div><div>X-Cache: MISS from ip-172-23-18-130</div><div>X-Cache-Lookup: MISS from ip-172-23-18-130:3128</div><div>Transfer-Encoding: chunked</div><div>Connection: keep-alive</div><div><br></div><div><br></div><div>----------</div></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Nov 23, 2017 at 11:43 PM, Amos Jeffries <span dir="ltr"><<a href="mailto:squid3@treenet.co.nz" target="_blank">squid3@treenet.co.nz</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class=""><br>
On 24/11/17 10:03, Ivan Larionov wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
On Nov 23, 2017, at 12:32 AM, Amos Jeffries <<a href="mailto:squid3@treenet.co.nz" target="_blank">squid3@treenet.co.nz</a>> wrote:<br>
<br>
On 23/11/17 14:20, Ivan Larionov wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hello.<br>
We have an issue with squid when it tries to re-forward / retry failed request even when forward_max_tries is set to 1. The situation when it happens is when there's no response, parent just closes the connection.<br>
</blockquote>
...<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
It doesn't happen 100% times. Sometimes squid returns 502 after the 1st try, sometimes it retries once. Also I haven't seen more than 1 retry.<br>
</blockquote>
<br>
Please enable debug_options 44,2 to see what destinations your Squid is actually finding.<br>
</blockquote>
<br>
I'll check this on Monday.<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
max_forward_tries is just a rough cap on the number of server names which can be found when generating that list. The actual destinations count can exceed it if one or more of the servers happens to have multiple IPs to try.<br>
<br>
The overall transaction can involve retries if one of the other layers (TCP or HTTP) contains retry semantics to a single server.<br>
<br>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Could it be a bug? We'd really like to disable these retries.<br>
</blockquote>
<br>
Why are trying to break HTTP?<br>
What is the actual problem you are trying to resolve here?<br>
<br>
</blockquote>
<br>
Why do you think I'm trying to break HTTP?<br>
<br>
squid forwards the request to parent but parent misbehaves and just closes the connection after 40 seconds. I'm trying to prevent retry of request in such situation. Why squid retries if I never asked him to do it and specifically said "forward_max_tries 1".<br>
<br>
And this is not a connection failure, squid successfully establishes the connection and sends the request, parent ACKs it, just never responses back and proactively closes the connection.<br>
<br>
</blockquote>
<br></span>
This is not misbehaviour on the part of either Squid nor the parent.<br>
<<a href="https://tools.ietf.org/html/rfc7230#section-6.3.1" rel="noreferrer" target="_blank">https://tools.ietf.org/html/r<wbr>fc7230#section-6.3.1</a>><br>
"Connections can be closed at any time, with or without intention."<br>
<br>
<br>
As has been discussed in other threads recently there are servers out there starting to greylist TCP connections, closing the first one some time *after* SYN+ACK regardless of what the proxy sends and accepting any followup connection attempts.<br>
<br>
NP: That can result in exactly the behaviour you describe from the peer as Squid does not wait for a FIN to arrive before sending its upstream HTTP request - Squid will "randomly" get a FIN or a RST depending on whether the FIN or the DATA packet wins the race into the Squid machines TCP stack. FIN and RST have different retry properties which might explain your "sometimes retries" behaviour.<br>
<br>
<br>
Also, TCP connections fail quite often for many other reasons anyway. Anything from power fluctuations at a router to BGP switching the packet route dropping packets. They are most often a short-term situation which is resolved by the time the repeat is attempted.<br>
<br>
What you are trying to do will result in Squid being unable to cope with any of these transitory restrictions from the TCP environment and force the client to receive a terminal error page.<br>
That will greatly slow down detection and recovery from the slightly longer-lived TCP issues in Squid itself and may result in N other clients also unnecessarily receiving the same error response as bad connection attempts gets spread between many clients (all getting errors) instead of isolated to the one/few who hit it when the issue initially occurs.<br>
<br>
Expanding the retries to large numbers (ie the recent default change to 25), or to low numbers (eg the old default of 5) are reasonable things to do depending on the network stability to your upstreams. But going all the way to 0 retries is guaranteed to lead to more client visible problems than necessary.<br>
<br>
<br>
All that asside I phrased it as a question because you might have had a good reason for increasing the visible failure rates.<span class=""><br>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
We're already fixing parent behavior, but still want to disable retries on squid side.<br>
<br>
</blockquote>
<br>
<br></span>
Since you describe this as peer misbehaviour, then treating it to Squids normal TCP failure recovery is the best behaviour. Retry is the intended correct behaviour for a proxy to perform on any non-idempotent requests. In your case up to a value of 1 retry before declaring non-temporary route failure.<br>
<br>
NP: idempotent vs non-idempotent may be another reason behind the observed behaviour of retry happening only sometimes.<br>
<br>
<br>
If you are doing this due to overall latency/delay on the affected client traffic you would be better off reducing the timeouts involved (cache_peer connect-timeout= parameter AFAICS) than aiming at a retry count of 0. Perhapse also requiring a few standby=N persistent connections to be maintained if the peer is HTTP/1.1 capable.<span class="HOEnZb"><font color="#888888"><br>
<br>
Amos<br>
</font></span></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature">With best regards, Ivan Larionov.</div>
</div>