[squid-users] Squid 3.5.25 does not recognise ICAP 408 status code

[Alex Rousskov]

On 10/31/18 11:20 AM, Arunabha Saha wrote:
>>As with any timeout, it is impossible to say in general which side of
>>the connection is at fault. This case has at least three sides: It could
>>be the HTTP agent, Squid, and/or the ICAP service. Did one of them stall
>>the transaction? Or was the ICAP service just too impatient? See option
>>#4 below.

> I've tried to track this down.   There are some  persistent sockets used
> by SaaS apps for APIs (otservice api from google sites) and sometimes
> the HTTP response takes a long time to trickle in.  I have seen upto 25
> seconds for the response body to trickle in after the response header.

Glad you found the culprit!

 
> The timeout at 10 seconds is somewhat aggressive so moving
> that up should help but some code changes in either squid or icap as
> suggested look necessary.

If you do not want to modify Squid, then the ICAP service should tell
Squid what to do with the timed out HTTP transaction (e.g., respond with
ICAP 204 or ICAP 200) instead of telling Squid that the service does not
know what to do (i.e. respond with ICAP 408). When the service does not
know what to do, Squid does not know what to do either, resulting in
transaction errors (that Squid blames the ICAP service for).

Please note that teaching Squid about the special meaning of ICAP 408
(Timeout) responses is not enough to address the problem -- Squid would
still need to know what to do with the HTTP transaction (e.g. block it,
pass through as is, or abort) and with the service (count a service
failure or ignore the timeout). Those decisions are likely to be
different for different admins/deployments.

Alex.

> I was referring to the c-icap implementation.

>     On 10/30/18 6:45 PM, Arunabha Saha wrote:
> 
>     > Squid 3.5.25 does not seem to recognise the 408 request timeout error
>     > code from ICAP.
> 
>     Squid effectively recognizes ICAP 408 response as an ICAP transaction
>     error response and blames the ICAP service for that error. That
>     (minimal) support can be improved, of course. See options #1 and #3
>     below.
> 
> 
>  
> 
> 
>     Needless to say, treating all ICAP service timeouts as if nothing bad
>     happened would break some existing Squid deployments (while possibly
>     fixing yours). A proper general solution (option #3 below) would most
>     likely require making Squid behavior configurable.
> 
> 
>     > The more troublesome issue for me is the exception it generates
>     and then
>     > declares ICAP down after a certain number of such exceptions.    
>     >
>     > I don't want to disable the failure limit entirely given that we can
>     > often have genuine failures that squid needs to detect.   
>     >
>     > What i'd like to see is squid should not throw an exception in this
>     > case.
> 
>     The "exception" is a minor low-level/technical detail. What you really
>     want to see is Squid blaming itself (rather than the ICAP service) for
>     the problem. Squid indeed lacks that kind of functionality, but it can
>     be added if really needed. See options #1 and #3 below.
> 
> 
>     > The timeout is somewhat aggressive but works with an earlier
>     > version of ICAP (0.1.x).  The one i'm testing is 0.5.3.
> 
>     Please note that ICAP is a protocol, not a product/software name. It
>     probably does not matter what ICAP service you are using though.
> 
> 
>     > Any suggestions?
> 
>     I can suggest a few options, in no particular order:
> 
>     1. Modify your Squid to treat 408 differently.
>     2. Modify your ICAP service to stop sending ICAP 408 responses to Squid.
>     3. Add proper ICAP timeout support feature to Squid.
>     4. Investigate why your ICAP service times out. If you are lucky, you
>        may be able to fix or work around the problem by adjusting Squid
>        and/or your ICAP service configuration.
> 
>     For option #1, Adaptation::Icap::ModXact::parseIcapHead() may be a good
>     starting point.
> 
>     For options #1 and #3, see also
>     https://wiki.squid-cache.org/SquidFaq/AboutSquid#How_to_add_a_new_Squid_feature.2C_enhance.2C_of_fix_something.3F
> 
>     In most cases, option #4 is the best first step but YMMV.
> 
> 
>     HTH,
> 
>     Alex.
>