[squid-users] Squid 3.5 ICAP Problems

[Flashdown]

Hey Guys,

I also have a long term issue with C-ICAP on many servers in different 
countries.

It looks like an ICAP Queue overload happening within squid. Since after 
taking a lot time in order to fine tune and debug the C-ICAP Service and 
the clamav-daemon in order to ensure that it can handle the high load 
and is configured properly, the ICAP suspended, down and UP messages 
dissappeared entirely in the cache.log, which I had after some days, 
then some weeks (after tuning the ICAP Server even more) occuring and 
after some messages the performance become steady slow until squid 
service is restarted. So now I have fixed the C-ICAP Server that is 
using squidclamav as C-ICAP Module for in-stream scanning. 
http://squidclamav.darold.net/, So I no longer can see any C-ICAP DOWN 
and UP Messages anymore. Which tells me that C-ICAP seems to always have 
enough workers to take the load.

So as I said now after finalizing my C-ICAP Server fine tuning, 
reconfiguration and debugging. I have no more ICAP Up and down messages 
and now after some weeks squid became slow again. When restarting the 
squid service all works fine again until some day in some weeks where it 
becomes slow again until service restart.

So I guess I need to set the right debug options for you guys, but that 
would create a very big log file since it takes days or weeks until this 
issue is happening. Any suggestions on what you think how I can do 
debugging in the best way?


BTW: I hope I do not mix up issues here, but it seems to be related to 
my issue that I am face since I was moving to a newer Version back in 
the old days then squid 3.5.12 <- somewhere there this story started for 
me. Unsure if it was exactliy starting with this version, some versions 
later or some earlier. But well it definitely was not happening on 
3.5.10 or earlier, so here I can cut it a  bit. So the issue was 
definitely introduced somewhere between 3.5.11 until 3.5.18. But as I 
said now I can at least fully outline the C-ICAP Server, which the cache 
log confirms by not showing any down, suspended and up messages any 
more.

I believe when I enforce a service restart once in the night, I will 
never see this issue of slow performance caused by the C-ICAP Module 
within squid anymore.

Best regards,
Enrico/Flashdown



Am 2017-11-03 12:54, schrieb Eliezer Croitoru:
> I'm not sure but after testing with telnet  or\and nc you can try to
> verify the open files limits on the system which might cause this
> issue.
> To identify how many connections are opened to the service you can use
> netstat or ss tools.
> netstat -ntp
> 
> and a similar on ss.
> 
> Also it's a good practice to put an ICAP periodic(2-20 seconds
> apparat) with an OPTIONS request to make sure that the service is
> alive.
> If you want my testing script let me know.
> 
> Eliezer
> 
> ----
> Eliezer Croitoru
> Linux System Administrator
> Mobile: +972-5-28704261
> Email: eliezer at ngtech.co.il
> 
> 
> 
> -----Original Message-----
> From: squid-users [mailto:squid-users-bounces at lists.squid-cache.org]
> On Behalf Of Alex Rousskov
> Sent: Thursday, November 2, 2017 19:22
> To: Stephen Stark <logic4life at gmail.com>; 
> squid-users at lists.squid-cache.org
> Subject: Re: [squid-users] Squid 3.5 ICAP Problems
> 
> On 11/02/2017 10:29 AM, Stephen Stark wrote:
>> Adaptation::Icap::Xaction::noteCommConnected(local=[::]
>> remote=127.0.0.1:1344 flags=1, errno=101, ...
> 
> The logs you have provided do not show where/why exactly the TCP
> connection to the ICAP service fails, but error number 101 is probably
> "network unreachable". This is unusual but not impossible for
> localhost traffic. The next step depends on the failure cause. There
> are at least two major cases to consider:
> 
> A) If Squid sends packets to 127.0.0.1:1344, then you can easily
> reproduce the problem using something like "telnet" or "nc" on the
> Squid box command line. Just make sure you use the right _source_ IP
> address for the connection! It has to be the same source IP address
> that Squid is using. It might not be 127.0.0.1. Running that command
> as Squid user might also be important if the Squid box have some fancy
> user-specific networking restrictions.
> 
> B) If Squid does not send packets to 127.0.0.1:1344, then one can
> figure out what goes wrong by studying relevant ALL,9 logs. You may
> also want to address other errors or warnings Squid logs to cache.log
> (if any).
> 
> You can determine whether Squid sends packets to 127.0.0.1:1344 by
> collecting a packet trace (for all Squid box interfaces!) and/or
> running strace (for the Squid worker process).
> 
> 
> HTH,
> 
> Alex.
> _______________________________________________
> squid-users mailing list
> squid-users at lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users
> 
> _______________________________________________
> squid-users mailing list
> squid-users at lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users