<div dir="ltr">Hi,<div><br></div><div>Thank you all for such detailed responses.</div><div><br></div><div>I still don't understand whether it is possible to reuse ICAP connections for cases other than retries.</div><div><br></div><div>As far as I know, Squid is able to save connections in a pool called `<span style="font-size:12.8px">theIdleConns`. Can these connection be reused for other transactions in the future?</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">The flow I am trying to accomplish is the following:</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">1. Client performs a requests and a response arrives</span></div><div><span style="font-size:12.8px">2. Squid catches the response and invokes the ICAP adaptation routine</span></div><div><span style="font-size:12.8px">3. A new connection the the ICAP server is created</span></div><div><span style="font-size:12.8px">4. The RESPMOD transaction is issued successfully</span></div><div><span style="font-size:12.8px">5. The connection is returned to the pool</span></div><div><span style="font-size:12.8px">6. The response is then sent to the client</span></div><div><br></div><div>... Moments later:</div><div><br></div><div>7. Another client performs a request a a response arrives</div><div>8. <span style="font-size:12.8px">Squid catches the response and invokes the ICAP adaptation routine</span></div><div><span style="font-size:12.8px">9. Squid reuses the connection which was created previously</span></div><div><span style="font-size:12.8px">10. </span><span style="font-size:12.8px">The RESPMOD transaction is issued successfully</span></div><div><span style="font-size:12.8px">11. The connection is returned to the pool</span><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">12. The response is then sent to the client</span></div><div><br></div><div>That behavior does not happen, going back to this code:</div><div><br></div><div><div style="font-size:12.8px"><div> if (retriableXact)</div><div> connection = theIdleConns->pop();</div><div> else</div><div> theIdleConns->closeN(1);</div><div><br></div></div></div><div>The line `<span style="font-size:12.8px">theIdleConns->closeN(1);` is the one that gets executed all the time with no exception. Well, it is actually an exception to the case, and I'll explain it later. Have in mind that I modified the code in order to prove that:</span></div><div><span style="font-size:12.8px"><br></span></div><div><div><span style="font-size:12.8px"> if (retriableXact) {</span></div><div><span style="font-size:12.8px"> debugs(93, 1, HERE << "Calling theIdleConns->pop()");</span></div><div><span style="font-size:12.8px"> connection = theIdleConns->pop();</span></div><div><span style="font-size:12.8px"> } else {</span></div><div><span style="font-size:12.8px"> debugs(93, 1, HERE << "Calling theIdleConns->closeN(1)");</span></div><div><span style="font-size:12.8px"> theIdleConns->closeN(1);</span></div><div><span style="font-size:12.8px"> }</span></div></div><div><span style="font-size:12.8px"><br></span></div><div><br></div><div>Based on that log, I could confirm that connections are reused properly only for small response sizes. </div><div>Digging a little bit into the source code I finally understood what Alex mean with "<span style="font-size:12.8px">Squid can buffer the entire HTTP message body.": </span></div><div><span style="font-size:12.8px"> Retries are disabled if the message body is to big, this is a protective measure which prevents from using to much RAM, it is clearly documented and I understand it.</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">For those curious enough, the file in question is `ModXact.cc` (functions </span><span style="font-size:12.8px">`</span><span style="font-size:12.8px">decideOnRetries()</span><span style="font-size:12.8px">`</span><span style="font-size:12.8px"> and </span><span style="font-size:12.8px">`</span><span style="font-size:12.8px">canBackupEverything()</span><span style="font-size:12.8px">`</span><span style="font-size:12.8px">).</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">Wrapping up: </span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">* Why does disabling retries in a connection dooms the connection?</span></div><div><span style="font-size:12.8px">* Why does the size of the payload affect the way connections are reused</span></div><div><span style="font-size:12.8px">* Does Squid need a finer control on this matter in order to ensure connection reusability is not affected by the size of the payload?</span></div><div><br></div><div>In the first response, Amos wrote the following:</div><div><br></div><div><span style="font-size:12.8px">>> Requests which are not retriable are not able to be re-sent if it turns </span><span style="font-size:12.8px">out </span></div><div><span style="font-size:12.8px">>> they got even partially delivered on a pre-opened persistent connection </span></div><div><span style="font-size:12.8px">>> which happened to be in the process of closing by the other endpoint </span></div><div><span style="font-size:12.8px">>> unknown to Squid. For example if the timing of the write to deliver the </span></div><div><span style="font-size:12.8px">>> ICAP RESPMOD headers overlapped with the TCP FIN/RST packets </span></div><div><span style="font-size:12.8px">>> coming from a router along the network path, or the ICAP service itself.</span><br style="font-size:12.8px"><br style="font-size:12.8px"><span style="font-size:12.8px">>> As such those requests need a new TCP connection to be opened to </span></div><div><span style="font-size:12.8px">>> guarantee the absence of immediate closure. When they complete with </span></div><div><span style="font-size:12.8px">>> their transaction it gets added to the pool for other traffic to use if they can.</span><br></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px"><br></span></div><div>This allows me to possibly answer the questions above and create new ones (a clever reader may notice I understand much more now than when I started writing this message).</div><div><br></div><div>+ Squid uses a self-defense mechanisms which allows to do two things:</div><div> 1. Use a fresh connection for non-retriable transactions, minimizing the possibility of using a connection which may be in the process of being closed.</div><div> 2. Control the number of open connections so it doesn't go out of bounds. This is a little redundant because `ServiceRep::putConnection()` is already taken care of that.</div><div>+ I am working with payloads that have more than 64K in size, which makes me fall in the conditions stated above.</div><div>+ I am not sure if it needs finer control, but the fact that is closing connections in advance seems dangerous.</div><div>+ It also seems dangerous to me this protection mechanism given the fact that opening/closing TCP connections everytime can become a bottleneck.</div><div><br></div><div>So, and this is the final question, I promise:</div><div><br></div><div>I disabled the call to `<span style="font-size:12.8px">theIdleConns->closeN(1);` and it seems to work well. </span></div><div><span style="font-size:12.8px">The pool doesn't go out of bounds because the connection is closed inside </span>`ServiceRep::putConnection()` <span style="font-size:12.8px">whenever `ServiceRep::excessConnections()` returns true.</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">Do you think there would be any issues with this change?</span></div><div><br></div><div><br></div><div>Thanks in advance</div><div><br></div><div>Juan</div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, May 12, 2017 at 11:08 AM, Alex Rousskov <span dir="ltr"><<a href="mailto:rousskov@measurement-factory.com" target="_blank">rousskov@measurement-factory.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On 05/11/2017 07:30 PM, Amos Jeffries wrote:<br>
<br>
> Requests which are not retriable are not able to be re-sent [...]<br>
<span class="">> As such those requests need a new TCP connection to be opened to<br>
> guarantee the absence of immediate closure. When they complete with<br>
> their transaction it gets added to the pool for other traffic to use if<br>
> they can.<br>
<br>
</span>The above is accurate, including the snipped parts documenting the<br>
rationale. BTW, since HTTP and ICAP share the same basic connection<br>
persistency model, you may find more information reading about these<br>
general problems in HTTP-related documents.<br>
<span class=""><br>
<br>
> As far as I know RESPMOD transactions should all be retriable unless the<br>
> body payload length is unknown at the HTTP level (lack of Content-Length<br>
> header).<br>
<br>
</span>This is inaccurate. A REQMOD or RESPMOD transaction is deemed retriable<br>
at the time when a connection is chosen/open if:<br>
<br>
* Squid can send a preview or<br>
* Squid can buffer the entire HTTP message body.<br>
<br>
The "known length" aspect affects the second bullet, but that bullet<br>
covers more cases than just messages with Content-Length. And there may<br>
be more conditions in the code than the bullets cover -- I have not checked.<br>
<span class=""><br>
<br>
> They can also become non-retriable when the response from the ICAP<br>
> server has been received by Squid and already started delivery to the<br>
> HTTP client.<br>
<br>
</span>This is accurate (including the snipped part about the irrelevance to<br>
the connection choice).<br>
<br>
<br>
HTH,<br>
<br>
Alex.<br>
<br>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>Juan</div><div>:wq</div><div><br></div></div></div>
</div>