[squid-users] url_rewrite_program and ACLs

[Amos Jeffries]

On 20/11/17 22:15, Vieri wrote:
> ________________________________
> From: Amos Jeffries <squid3 at treenet.co.nz>
>>
>> I would compare your custom script to the ext_sql_session_acl.pl.in
>> script we bundle with current Squid.
> 
> 
> I've rewritten my perl script, and have been running it for a full week now without any issues. Free RAM drops down to alarming values, but then rises back up again. In any case, "used swap" is always the same. The only thing that keeps be edgy is the fact that the open FDs keep growing (slowly but steadily). After a few days the value is around 6000, but after a week (today) it's:
> 


> Squid Object Cache: Version 3.5.27-20171101-re69e56c
> Build Info:
> Service Name: squid
> Start Time:     Mon, 13 Nov 2017 11:06:36 GMT
> Current Time:   Mon, 20 Nov 2017 08:48:00 GMT

> Average HTTP requests per minute since start:   647.3
...
> File descriptor usage for squid:
> Maximum number of file descriptors:   65536
> Largest file desc currently in use:   12714
> Number of file desc currently in use: 11998

So ~100 RPS, I would expect the open FDs used by ICAP to be around 
299-300. Not the 11K+ you are seeing.

If we assume that each request opens a new connection and they are not 
closed until TCP times out on the socket we do get numbers much more 
like that 11K+ you are seeing.

That implies that ICAP transactions are probably not finishing 
completely. Is the ICAP service finishing the ICAP reply messages 
properly? (eg with a 0-size chunk)

Or maybe the error that led to you deciding to configure 
"icap_service_failure_limit -1" was actually a real problem.


> 
> mgr:filedescriptors shows a great deal of these:
> 
> Remote Address        Description
> --------------------- ------------------------------
> 127.0.0.1:1344        127.0.0.1
> 
> 
> # squidclient mgr:filedescriptors | grep -c "127.0.0.1:1344"
> 11578
> 
> 
> Port 1344 is where the c-icap daemon listens on.
> This is the relevant part in squid.conf:
> 
> icap_enable on
> icap_send_client_ip on
> icap_send_client_username on
> icap_client_username_encode off
> icap_client_username_header X-Authenticated-User
> icap_preview_enable on
> icap_preview_size 1024
> icap_service squidclamav respmod_precache bypass=0 icap://127.0.0.1:1344/clamav
> adaptation_access squidclamav allow all
> icap_service_failure_limit -1
> 
> 
> The number of connections to this port fluctuates over time (it also decreases), but overall it clearly increases day by day.
> I could have an issue with either c-icap itself or one of its modules.
> I'll keep an eye on it.

I think those 11K open FDs is sufficient evidence to say something is 
wrong with the ICAP transactions. If you can get a packet trace of some 
of some ICAP connections it might give better clues to what is happening 
there.

Amos