<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:"Courier New";}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal">Can somebody please explain what could have happened here? <o:p>
</o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">First squid(4.0.25) encountered a URL > 8K bytes. I think this caused it to crash.
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Jul 13 11:04:13 <hostname> squid[9102]: parse URL too large (9697 bytes)<o:p></o:p></p>
<p class="MsoNormal">Jul 13 11:04:13 <hostname> squid[29254]: Squid Parent: squid-1 process 9102 exited due to signal 11 with status 0<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">squid-1 was respawned by the parent squid process.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Then I see ,<o:p></o:p></p>
<p class="MsoNormal">WARNING: ICAP Max-Connections limit exceeded for service icap://127.0.0.1:1344/reqmod. Open connections now: 16, including 0 idle persistent connections.<o:p></o:p></p>
<p class="MsoNormal">The newly spawned squid-1 crashes yet again. As seen below,
<o:p></o:p></p>
<p class="MsoNormal">Jul 13 11:16:14 <hostname> squid[29254]: Squid Parent: squid-1 process 10951 exited due to signal 11 with status 0<o:p></o:p></p>
<p class="MsoNormal">Logs don’t explain why squid-1 crashed here. ICAP message above is just a warning.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">squid-1 is respawned a second time and I see,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Jul 13 11:22:18 <hostname> squid[13123]: ERROR: negotiating TLS on FD 1722: error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed (1/-1/0)<o:p></o:p></p>
<p class="MsoNormal">Jul 13 11:22:18 <hostname> squid[13123]: Error negotiating SSL connection on FD 1400: (104) Connection reset by peer<o:p></o:p></p>
<p class="MsoNormal">Jul 13 11:23:14 <hostname> squid[13123]: Error negotiating SSL connection on FD 1046: (104) Connection reset by peer<o:p></o:p></p>
<p class="MsoNormal">Jul 13 11:23:14 <hostname> squid[13123]: Error negotiating SSL connection on FD 582: (104) Connection reset by peer<o:p></o:p></p>
<p class="MsoNormal">Jul 13 11:23:15 <hostname> squid[13123]: Error negotiating SSL connection on FD 61: (104) Connection reset by peer<o:p></o:p></p>
<p class="MsoNormal">Jul 13 11:23:16 <hostname> squid[13123]: Error negotiating SSL connection on FD 1150: (104) Connection reset by peer<o:p></o:p></p>
<p class="MsoNormal">Jul 13 11:23:18 <hostname> squid[13123]: Error negotiating SSL connection on FD 1674: (104) Connection reset by peer<o:p></o:p></p>
<p class="MsoNormal">Jul 13 11:23:18 <hostname> squid[13123]: Error negotiating SSL connection on FD 1519: (104) Connection reset by peer<o:p></o:p></p>
<p class="MsoNormal">Jul 13 11:23:18 <hostname> squid[13123]: Error negotiating SSL connection on FD 1292: (104) Connection reset by peer<o:p></o:p></p>
<p class="MsoNormal">Jul 13 11:23:18 <hostname> squid[13123]: Error negotiating SSL connection on FD 1631: (104) Connection reset by peer<o:p></o:p></p>
<p class="MsoNormal">Jul 13 11:35:17 <hostname> squid[13123]: Error negotiating SSL connection on FD 1331: (104) Connection reset by peer<o:p></o:p></p>
<p class="MsoNormal">Jul 13 11:35:24 <hostname> squid[13123]: WARNING! Your cache is running out of filedescriptors<o:p></o:p></p>
<p class="MsoNormal">Jul 13 11:35:56 <hostname> squid[13123]: Error negotiating SSL connection on FD 1867: (104) Connection reset by peer<o:p></o:p></p>
<p class="MsoNormal">Jul 13 11:35:58 <hostname> squid[13123]: Error negotiating SSL connection on FD 1715: (104) Connection reset by peer<o:p></o:p></p>
<p class="MsoNormal">Jul 13 11:35:59 <hostname> squid[13123]: suspending ICAP service for too many failures<o:p></o:p></p>
<p class="MsoNormal">Jul 13 11:35:59 <hostname> squid[13123]: optional ICAP service is suspended: icap://127.0.0.1:1344/reqmod [down,susp,fail11]<o:p></o:p></p>
<p class="MsoNormal">Jul 13 11:36:00 <hostname> squid[13123]: comm_openex socket failure: (24) Too many open files<o:p></o:p></p>
<p class="MsoNormal">Jul 13 11:36:00 <hostname> squid[13123]: comm_openex socket failure: (24) Too many open files<o:p></o:p></p>
<p class="MsoNormal">Jul 13 11:36:00 <hostname> squid[13123]: comm_openex socket failure: (24) Too many open files<o:p></o:p></p>
<p class="MsoNormal">Jul 13 11:36:00 <hostname> squid[13123]: comm_openex socket failure: (24) Too many open files<o:p></o:p></p>
<p class="MsoNormal">Jul 13 11:36:00 <hostname> squid[13123]: comm_openex socket failure: (24) Too many open files<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">There is only one icap service defined as below :<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">icap_enable on<o:p></o:p></p>
<p class="MsoNormal">icap_service test_icap reqmod_precache icap://127.0.0.1:1344/reqmod bypass=on routing=off on-overload=wait<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The open file ulimit is set to 16k. How many TCP connections would Squid have opened up that it exhausted 16k file descriptors ? Some sort of file descriptor leak ?<o:p></o:p></p>
<p class="MsoNormal">I am unable to connect the dots where an unresponsive ICAP service lead to the proxy running out of file descriptors ? Too many TCP SYN attempts ?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">When in working condition, this is what it looks like, from cachemgr,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">File descriptor usage for squid:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> Maximum number of file descriptors: 16384<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> Largest file desc currently in use: 58<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> Number of file desc currently in use: 27<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> Files queued for open: 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> Available number of file descriptors: 16357<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> Reserved number of file descriptors: 100<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New""> Store Disk files open: 0<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I will be installing Squid4.1 shortly but I need an explanation for what happened here. Please provide some pointers or let me know if any other information is needed to figure this out.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Regards,<o:p></o:p></p>
<p class="MsoNormal">Sarfaraz<o:p></o:p></p>
</div>
</body>
</html>