[squid-users] Connection occasionally not ending after adapting response with ICAP

[Alex Rousskov]

On 12/30/20 9:09 AM, Moti Berger wrote:
> I have a setup with squid 5.0.4 with ICAP server handling responses. 

FTR: This part seems to be about an ICAP RESPMOD service generating a
307 redirect (AFAICT).


> When it hangs, I see this as the output of cURL 

>     > GET / HTTP/1.1
>     > Host: www.one.co.il <http://www.one.co.il>
>     > User-Agent: curl/7.58.0
>     > Accept: */*

>     < HTTP/1.1 307 Temporary Redirect
>     < Location: <REDIRECTION-URL>
>     < Cache-Control: no-cache, no-store, must-revalidate
>     < Pragma: no-cache
>     < Date: Wed, 30 Dec 2020 13:58:51 GMT
>     < X-Cache: MISS from a0e59ea22cf8
>     < X-Cache-Lookup: MISS from a0e59ea22cf8:3128
>     < Transfer-Encoding: chunked
>     < Via: 1.1 a0e59ea22cf8 (squid/5.0.4)
>     < Connection: keep-alive

Did curl receive the last-chunk from Squid? If not, did Squid received
the last-chunk from the ICAP service? See below for more details.


> For some of the URLs where it hangs, I saw that running the same cURL
> command with the --compressed switch, makes cURL exit as expected:

Squid ignores Accept-Encoding so the --compressed switch is not relevant
to Squid. Perhaps the ICAP service or the origin server react to it, but
the headers you have provided do not show any important differences:

>     > GET / HTTP/1.1
>     > Host: www.one.co.il <http://www.one.co.il>
>     > User-Agent: curl/7.58.0
>     > Accept: */*
>     > Accept-Encoding: deflate, gzip

>     < HTTP/1.1 307 Temporary Redirect
>     < Location: <REDIRECTION URL>
>     < Cache-Control: no-cache, no-store, must-revalidate
>     < Pragma: no-cache
>     < Date: Wed, 30 Dec 2020 14:00:31 GMT
>     < X-Cache: MISS from a0e59ea22cf8
>     < X-Cache-Lookup: MISS from a0e59ea22cf8:3128
>     < Transfer-Encoding: chunked
>     < Via: 1.1 a0e59ea22cf8 (squid/5.0.4)
>     < Connection: keep-alive

> When I skip the adaptation in REQMOD, I get the page and the connection
> is terminated.

... but REQMOD does not redirect when you do _not_ skip it, right? That
is, it is RESPMOD that always redirects (until a separate test below)?
If that is what happens, then it is possible that something goes wrong
inside the ICAP service, and turning off REQMOD that does not redirect
fixes the redirect in RESPMOD. However, it is a bit difficult for me to
follow the REQMOD/RESPMOD roles in your email so I may be
misunderstanding something.


> When the ICAP works in REQMOD and redirects on the same URLs, everything
> seems to work properly.

OK. Redirection in REQMOD is quite a different beast -- a lot of things
change, on both Squid and ICAP sides.


> What could make squid not to terminate the connection? Could it be that
> it still holds connection with the HTTP server?

The actual question here is not about the connections but about the
messages inside those connections. It sounds like Squid does not send
the entire redirect response body to curl in some cases. _That_ could be
caused by the ICAP service not sending the entire redirect response body
to Squid.

You can probably confirm or deny the last bit by looking at ICAP traffic
in wireshark. Does the ICAP response contain the last-chunk (which is
usually the "0 CR LF CR LF" sequence in the chunked-encoded message body)?

* If the last-chunk is missing, then Squid is waiting for the ICAP
service. I do not know what the ICAP RESPMOD service is waiting for, but
it could be the last-chunk of the virgin HTTP response that Squid is
supposed to send to the RESPMOD service. Wireshark should show that
exchange as well.

* Otherwise, it could be a Squid bug -- post a link to a compressed
ALL,9 cache.log of the problematic transaction as discussed at
https://wiki.squid-cache.org/SquidFaq/BugReporting#Debugging_a_single_transaction


HTH,

Alex.