<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Dec 30, 2020 at 4:54 PM Alex Rousskov <<a href="mailto:rousskov@measurement-factory.com">rousskov@measurement-factory.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On 12/30/20 9:09 AM, Moti Berger wrote:<br>
> I have a setup with squid 5.0.4 with ICAP server handling responses. <br>
<br>
FTR: This part seems to be about an ICAP RESPMOD service generating a<br>
307 redirect (AFAICT).<br>
<br>
<br>
> When it hangs, I see this as the output of cURL <br>
<br>
> > GET / HTTP/1.1<br>
> > Host: <a href="http://www.one.co.il" rel="noreferrer" target="_blank">www.one.co.il</a> <<a href="http://www.one.co.il" rel="noreferrer" target="_blank">http://www.one.co.il</a>><br>
> > User-Agent: curl/7.58.0<br>
> > Accept: */*<br>
<br>
> < HTTP/1.1 307 Temporary Redirect<br>
> < Location: <REDIRECTION-URL><br>
> < Cache-Control: no-cache, no-store, must-revalidate<br>
> < Pragma: no-cache<br>
> < Date: Wed, 30 Dec 2020 13:58:51 GMT<br>
> < X-Cache: MISS from a0e59ea22cf8<br>
> < X-Cache-Lookup: MISS from a0e59ea22cf8:3128<br>
> < Transfer-Encoding: chunked<br>
> < Via: 1.1 a0e59ea22cf8 (squid/5.0.4)<br>
> < Connection: keep-alive<br>
<br>
Did curl receive the last-chunk from Squid? If not, did Squid received<br>
the last-chunk from the ICAP service? See below for more details.<br>
<br>
<br>
> For some of the URLs where it hangs, I saw that running the same cURL<br>
> command with the --compressed switch, makes cURL exit as expected:<br>
<br>
Squid ignores Accept-Encoding so the --compressed switch is not relevant<br>
to Squid. Perhaps the ICAP service or the origin server react to it, but<br>
the headers you have provided do not show any important differences:<br>
<br>
> > GET / HTTP/1.1<br>
> > Host: <a href="http://www.one.co.il" rel="noreferrer" target="_blank">www.one.co.il</a> <<a href="http://www.one.co.il" rel="noreferrer" target="_blank">http://www.one.co.il</a>><br>
> > User-Agent: curl/7.58.0<br>
> > Accept: */*<br>
> > Accept-Encoding: deflate, gzip<br>
<br>
> < HTTP/1.1 307 Temporary Redirect<br>
> < Location: <REDIRECTION URL><br>
> < Cache-Control: no-cache, no-store, must-revalidate<br>
> < Pragma: no-cache<br>
> < Date: Wed, 30 Dec 2020 14:00:31 GMT<br>
> < X-Cache: MISS from a0e59ea22cf8<br>
> < X-Cache-Lookup: MISS from a0e59ea22cf8:3128<br>
> < Transfer-Encoding: chunked<br>
> < Via: 1.1 a0e59ea22cf8 (squid/5.0.4)<br>
> < Connection: keep-alive<br>
<br>
> When I skip the adaptation in REQMOD, I get the page and the connection<br>
> is terminated.<br>
<br>
... but REQMOD does not redirect when you do _not_ skip it, right? That<br>
is, it is RESPMOD that always redirects (until a separate test below)?<br>
If that is what happens, then it is possible that something goes wrong<br>
inside the ICAP service, and turning off REQMOD that does not redirect<br>
fixes the redirect in RESPMOD. However, it is a bit difficult for me to<br>
follow the REQMOD/RESPMOD roles in your email so I may be<br>
misunderstanding something.</blockquote><div><br></div><div>>>> My ICAP server handles both REQMOD and RESPMOD. I can switch on and off</div><div>the adaptation in both modes. The adaptation, when take place, is to do redirects.</div><div>So, for example, when I turn on the adaptation in REQMOD, meaning, redirecting,</div><div>the connection terminates like expected.</div><div>When the adaptation in REQMOD is turned off (meaning, the ICAP server still receives</div><div>requests to modify, but it doesn't change them), and RESPMOD is turned on and</div><div>adaptation occurs (meaning, redirecting) for some URLs, it get stuck and for some it doesn't.</div><div>Regarding no adaptation in REQMOD that might affect the RESPMOD, I removed from</div><div>squid.conf the related configs and the issue persists.</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <br></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
> When the ICAP works in REQMOD and redirects on the same URLs, everything<br>
> seems to work properly.<br>
<br>
OK. Redirection in REQMOD is quite a different beast -- a lot of things<br>
change, on both Squid and ICAP sides.<br>
<br>
<br>
> What could make squid not to terminate the connection? Could it be that<br>
> it still holds connection with the HTTP server?<br>
<br>
The actual question here is not about the connections but about the<br>
messages inside those connections. It sounds like Squid does not send<br>
the entire redirect response body to curl in some cases. _That_ could be<br>
caused by the ICAP service not sending the entire redirect response body<br>
to Squid.<br>
<br>
You can probably confirm or deny the last bit by looking at ICAP traffic<br>
in wireshark. Does the ICAP response contain the last-chunk (which is<br>
usually the "0 CR LF CR LF" sequence in the chunked-encoded message body)?<br>
<br></blockquote><div>>>> I don't see the "0 CR LF CR LF" at all, not only for those URLs that cause the curl to hang.</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
* If the last-chunk is missing, then Squid is waiting for the ICAP<br>
service. I do not know what the ICAP RESPMOD service is waiting for, but<br>
it could be the last-chunk of the virgin HTTP response that Squid is<br>
supposed to send to the RESPMOD service. Wireshark should show that<br>
exchange as well.<br></blockquote><div><br></div><div>>>> I'm attaching a tcpdump file. </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
* Otherwise, it could be a Squid bug -- post a link to a compressed<br>
ALL,9 cache.log of the problematic transaction as discussed at<br>
<a href="https://wiki.squid-cache.org/SquidFaq/BugReporting#Debugging_a_single_transaction" rel="noreferrer" target="_blank">https://wiki.squid-cache.org/SquidFaq/BugReporting#Debugging_a_single_transaction</a><br>
<br>
<br>
HTH,<br>
<br>
Alex.<br>
</blockquote></div></div>