[squid-users] Squid "suspending ICAP service for too many failures"

[Andrea Venturoli]

On 1/29/21 8:38 PM, Alex Rousskov wrote:

> IIRC, you did not disclose timeout suspicions before. This explanation
> is news to me, and it eliminates several suspects.

Sorry, I didn't say much in fact.
I gave for granted that it was C-ICAP who stopped answering; I didn't 
suspect a Squid bug and had no other idea.



> If you are talking about Squid timing out when attempting to establish a
> TCP connection with the ICAP server, then this may by as much insight as
> you can get from the Squid side.

What I hoped to find in Squid logs was *what* was being passed to C-ICAP 
when it locked.
I'll try on the C-ICAP side then.



> I do not know much about c-icap, but I would check whether its
> configuration or something like crontab results in hourly restarts and
> associated loss of connectivity.

AFAIK no.



> The network interface or the routing tables might also be reset hourly

They live on the same host.



> The ICAP server/service might be running out of descriptors or memory.

I'd expect it to log that, but I'll investigate better.



> One potentially useful test is to try to connect to the ICAP server
> _while the problem is happening_ using telnet or netcat. When Squid
> cannot establish a connection, can you?

I'll try, but it's going to be hard, since this happens for a few 
minutes once a day at most.



> Packet captures can tell you whether other Squid-ICAP server connections
> were active at the time, whether from-Squid SYN packets were able to
> reach the ICAP server, etc.
> 
> In other words, basic network troubleshooting steps...

As I said, they live on the same host, so it can't be a network problem.



> Higher timeout will delay HTTP client transactions for longer periods of
> time, of course. If you want to go down the road of finding workarounds,
> then check whether raising that timeout actually helps. It is not yet
> clear (to me) whether the connections just need more time to be
> established or are simply doomed.

It's not clear to me either, but I suspect so, given the trouble only 
last a few minutes.




>> Same for disabling icap_service_failure_limit?
> 
> This is an essential ICAP service (icap_service bypass=off). I assume
> there is no backup service -- no adaptation_service_set in play here. If
> so, disabling the limit means that fewer HTTP transactions will be
> inconvenienced in the long run than if the service were to be suspended.
>   Hence, fewer ICAP errors will be delivered to Squid clients.

Agreed.



> You can also enable bypass.

I guess this would open a potential for an attack.
DoS the service (antivirus), then let something nasty pass...



> Fixing the problem would be a much better solution, of course.

Sure, I know these are workarounds and I'd rather avoid them, but I'll 
need to consider them as a last resort.



  bye & Thanks
	av.