[squid-users] Cache ran out of descriptors due to ICAP service/TCP SYNs ?

[Ahmad, Sarfaraz]

Can somebody please explain what could have happened here?

First squid(4.0.25) encountered a URL > 8K bytes. I think this caused it to crash.

Jul 13 11:04:13 <hostname> squid[9102]: parse URL too large (9697 bytes)
Jul 13 11:04:13 <hostname> squid[29254]: Squid Parent: squid-1 process 9102 exited due to signal 11 with status 0

squid-1 was respawned by the parent squid process.

Then I see ,
WARNING: ICAP Max-Connections limit exceeded for service icap://127.0.0.1:1344/reqmod. Open connections now: 16, including 0 idle persistent connections.
The newly spawned squid-1  crashes yet again. As seen below,
Jul 13 11:16:14 <hostname> squid[29254]: Squid Parent: squid-1 process 10951 exited due to signal 11 with status 0
Logs don't explain why squid-1 crashed here. ICAP message above is just a warning.

squid-1 is respawned a second time and I see,

Jul 13 11:22:18 <hostname> squid[13123]: ERROR: negotiating TLS on FD 1722: error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed (1/-1/0)
Jul 13 11:22:18 <hostname> squid[13123]: Error negotiating SSL connection on FD 1400: (104) Connection reset by peer
Jul 13 11:23:14 <hostname> squid[13123]: Error negotiating SSL connection on FD 1046: (104) Connection reset by peer
Jul 13 11:23:14 <hostname> squid[13123]: Error negotiating SSL connection on FD 582: (104) Connection reset by peer
Jul 13 11:23:15 <hostname> squid[13123]: Error negotiating SSL connection on FD 61: (104) Connection reset by peer
Jul 13 11:23:16 <hostname> squid[13123]: Error negotiating SSL connection on FD 1150: (104) Connection reset by peer
Jul 13 11:23:18 <hostname> squid[13123]: Error negotiating SSL connection on FD 1674: (104) Connection reset by peer
Jul 13 11:23:18 <hostname> squid[13123]: Error negotiating SSL connection on FD 1519: (104) Connection reset by peer
Jul 13 11:23:18 <hostname> squid[13123]: Error negotiating SSL connection on FD 1292: (104) Connection reset by peer
Jul 13 11:23:18 <hostname> squid[13123]: Error negotiating SSL connection on FD 1631: (104) Connection reset by peer
Jul 13 11:35:17 <hostname> squid[13123]: Error negotiating SSL connection on FD 1331: (104) Connection reset by peer
Jul 13 11:35:24 <hostname> squid[13123]: WARNING! Your cache is running out of filedescriptors
Jul 13 11:35:56 <hostname> squid[13123]: Error negotiating SSL connection on FD 1867: (104) Connection reset by peer
Jul 13 11:35:58 <hostname> squid[13123]: Error negotiating SSL connection on FD 1715: (104) Connection reset by peer
Jul 13 11:35:59 <hostname> squid[13123]: suspending ICAP service for too many failures
Jul 13 11:35:59 <hostname> squid[13123]: optional ICAP service is suspended: icap://127.0.0.1:1344/reqmod [down,susp,fail11]
Jul 13 11:36:00 <hostname> squid[13123]: comm_openex socket failure: (24) Too many open files
Jul 13 11:36:00 <hostname> squid[13123]: comm_openex socket failure: (24) Too many open files
Jul 13 11:36:00 <hostname> squid[13123]: comm_openex socket failure: (24) Too many open files
Jul 13 11:36:00 <hostname> squid[13123]: comm_openex socket failure: (24) Too many open files
Jul 13 11:36:00 <hostname> squid[13123]: comm_openex socket failure: (24) Too many open files


There is only one icap service defined as below :

icap_enable on
icap_service test_icap reqmod_precache icap://127.0.0.1:1344/reqmod bypass=on routing=off on-overload=wait

The open file ulimit is set to 16k. How many TCP connections would Squid have opened up that it exhausted 16k file descriptors ?  Some sort of file descriptor leak ?
I am unable to connect the dots where an unresponsive ICAP service lead to the proxy running out of file descriptors ?  Too many TCP SYN attempts ?

When in working condition, this is what it looks like, from cachemgr,

File descriptor usage for squid:
        Maximum number of file descriptors:   16384
        Largest file desc currently in use:     58
        Number of file desc currently in use:   27
        Files queued for open:                   0
        Available number of file descriptors: 16357
        Reserved number of file descriptors:   100
        Store Disk files open:                   0

I will be installing Squid4.1 shortly but I need an explanation for what happened here. Please provide some pointers or let me know if any other information is needed to figure this out.

Regards,
Sarfaraz
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20180717/fc15c9ef/attachment.html>