[squid-users] Connection occasionally not ending after adapting response with ICAP

[Moti Berger]

On Wed, Dec 30, 2020 at 4:54 PM Alex Rousskov <
rousskov at measurement-factory.com> wrote:

> On 12/30/20 9:09 AM, Moti Berger wrote:
> > I have a setup with squid 5.0.4 with ICAP server handling responses.
>
> FTR: This part seems to be about an ICAP RESPMOD service generating a
> 307 redirect (AFAICT).
>
>
> > When it hangs, I see this as the output of cURL
>
> >     > GET / HTTP/1.1
> >     > Host: www.one.co.il <http://www.one.co.il>
> >     > User-Agent: curl/7.58.0
> >     > Accept: */*
>
> >     < HTTP/1.1 307 Temporary Redirect
> >     < Location: <REDIRECTION-URL>
> >     < Cache-Control: no-cache, no-store, must-revalidate
> >     < Pragma: no-cache
> >     < Date: Wed, 30 Dec 2020 13:58:51 GMT
> >     < X-Cache: MISS from a0e59ea22cf8
> >     < X-Cache-Lookup: MISS from a0e59ea22cf8:3128
> >     < Transfer-Encoding: chunked
> >     < Via: 1.1 a0e59ea22cf8 (squid/5.0.4)
> >     < Connection: keep-alive
>
> Did curl receive the last-chunk from Squid? If not, did Squid received
> the last-chunk from the ICAP service? See below for more details.
>
>
> > For some of the URLs where it hangs, I saw that running the same cURL
> > command with the --compressed switch, makes cURL exit as expected:
>
> Squid ignores Accept-Encoding so the --compressed switch is not relevant
> to Squid. Perhaps the ICAP service or the origin server react to it, but
> the headers you have provided do not show any important differences:
>
> >     > GET / HTTP/1.1
> >     > Host: www.one.co.il <http://www.one.co.il>
> >     > User-Agent: curl/7.58.0
> >     > Accept: */*
> >     > Accept-Encoding: deflate, gzip
>
> >     < HTTP/1.1 307 Temporary Redirect
> >     < Location: <REDIRECTION URL>
> >     < Cache-Control: no-cache, no-store, must-revalidate
> >     < Pragma: no-cache
> >     < Date: Wed, 30 Dec 2020 14:00:31 GMT
> >     < X-Cache: MISS from a0e59ea22cf8
> >     < X-Cache-Lookup: MISS from a0e59ea22cf8:3128
> >     < Transfer-Encoding: chunked
> >     < Via: 1.1 a0e59ea22cf8 (squid/5.0.4)
> >     < Connection: keep-alive
>
> > When I skip the adaptation in REQMOD, I get the page and the connection
> > is terminated.
>
> ... but REQMOD does not redirect when you do _not_ skip it, right? That
> is, it is RESPMOD that always redirects (until a separate test below)?
> If that is what happens, then it is possible that something goes wrong
> inside the ICAP service, and turning off REQMOD that does not redirect
> fixes the redirect in RESPMOD. However, it is a bit difficult for me to
> follow the REQMOD/RESPMOD roles in your email so I may be
> misunderstanding something.


>>> My ICAP server handles both REQMOD and RESPMOD. I can switch on and off
the adaptation in both modes. The adaptation, when take place, is to do
redirects.
So, for example, when I turn on the adaptation in REQMOD, meaning,
redirecting,
the connection terminates like expected.
When the adaptation in REQMOD is turned off (meaning, the ICAP server still
receives
requests to modify, but it doesn't change them), and RESPMOD is turned on
and
adaptation occurs (meaning, redirecting) for some URLs, it get stuck and
for some it doesn't.
Regarding no adaptation in REQMOD that might affect the RESPMOD, I removed
from
squid.conf the related configs and the issue persists.

>
>

> > When the ICAP works in REQMOD and redirects on the same URLs, everything
> > seems to work properly.
>
> OK. Redirection in REQMOD is quite a different beast -- a lot of things
> change, on both Squid and ICAP sides.
>
>
> > What could make squid not to terminate the connection? Could it be that
> > it still holds connection with the HTTP server?
>
> The actual question here is not about the connections but about the
> messages inside those connections. It sounds like Squid does not send
> the entire redirect response body to curl in some cases. _That_ could be
> caused by the ICAP service not sending the entire redirect response body
> to Squid.
>
> You can probably confirm or deny the last bit by looking at ICAP traffic
> in wireshark. Does the ICAP response contain the last-chunk (which is
> usually the "0 CR LF CR LF" sequence in the chunked-encoded message body)?
>
> >>> I don't see the "0 CR LF CR LF" at all, not only for those URLs that
cause the curl to hang.

> * If the last-chunk is missing, then Squid is waiting for the ICAP
> service. I do not know what the ICAP RESPMOD service is waiting for, but
> it could be the last-chunk of the virgin HTTP response that Squid is
> supposed to send to the RESPMOD service. Wireshark should show that
> exchange as well.
>

>>> I'm attaching a tcpdump file.

>
> * Otherwise, it could be a Squid bug -- post a link to a compressed
> ALL,9 cache.log of the problematic transaction as discussed at
>
> https://wiki.squid-cache.org/SquidFaq/BugReporting#Debugging_a_single_transaction
>
>
> HTH,
>
> Alex.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20201230/2e754567/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: squid_icap_tcpdump
Type: application/octet-stream
Size: 3556 bytes
Desc: not available
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20201230/2e754567/attachment-0001.obj>