<html xmlns="http://www.w3.org/1999/xhtml" xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office"><head><!--[if gte mso 9]><xml><o:OfficeDocumentSettings><o:AllowPNG/><o:PixelsPerInch>96</o:PixelsPerInch></o:OfficeDocumentSettings></xml><![endif]--></head><body>
Thanks. We have limits set at 100K, squid can easily reach that. The problem is that the number of FD in use keeps increasing. A workaround is to restart squid every time it goes over a certain value, but it’s not really a solution. <div>In the same situation, with centos and squid 3.5, we seldom went over 20k FD in use. <br>Thanks for your reply. <br><br><div class="yahoo-signature">Panem et circenses</div><br><p class="yahoo-quoted-begin" style="font-size: 15px; color: rgb(132, 115, 255); padding-top: 15px; margin-top: 0px;">On Friday, July 12, 2024, 7:05 PM, Yvain PAYEN <yvain.payen@tessi.fr> wrote:</p><blockquote class="iosymail"><div id="yiv4238416288"><style>
#yiv4238416288 filtered {}
#yiv4238416288 filtered {}
#yiv4238416288 filtered {}
#yiv4238416288 filtered {}
#yiv4238416288 filtered {}
#yiv4238416288 filtered {}
#yiv4238416288 filtered {}
#yiv4238416288 p.yiv4238416288MsoNormal, #yiv4238416288 li.yiv4238416288MsoNormal, #yiv4238416288 div.yiv4238416288MsoNormal
{margin:0cm;font-size:11.0pt;font-family:sans-serif;}
#yiv4238416288 a:link, #yiv4238416288 span.yiv4238416288MsoHyperlink
{color:blue;text-decoration:underline;}
#yiv4238416288 code
{}
#yiv4238416288 span.yiv4238416288EmailStyle19
{font-family:sans-serif;color:windowtext;}
#yiv4238416288 .yiv4238416288MsoChpDefault
{font-size:10.0pt;}
#yiv4238416288 filtered {}
#yiv4238416288 div.yiv4238416288WordSection1
{}
#yiv4238416288 filtered {}
#yiv4238416288 filtered {}
#yiv4238416288 filtered {}
#yiv4238416288 filtered {}
#yiv4238416288 filtered {}
#yiv4238416288 filtered {}
#yiv4238416288 filtered {}
#yiv4238416288 filtered {}
#yiv4238416288 filtered {}
#yiv4238416288 filtered {}
#yiv4238416288 ol
{margin-bottom:0cm;}
#yiv4238416288 ul
{margin-bottom:0cm;}
</style><div>
<div class="yiv4238416288WordSection1">
<p class="yiv4238416288MsoNormal"><span style="">Hi,</span></p>
<p class="yiv4238416288MsoNormal"><span style=""> </span></p>
<p class="yiv4238416288MsoNormal"><span lang="EN-US" style="">I my setup (also ubuntu) I have made these changes :
</span></p>
<p class="yiv4238416288MsoNormal"><span lang="EN-US" style=""> </span></p>
<p class="yiv4238416288MsoNormal"><span lang="EN-US" style="">root@proxy: # cat /etc/security/limits.d/squid.conf</span></p>
<p class="yiv4238416288MsoNormal"><span lang="EN-US" style="">squid soft nofile 64000</span></p>
<p class="yiv4238416288MsoNormal"><span lang="EN-US" style="">squid hard nofile 65500</span></p>
<p class="yiv4238416288MsoNormal"><span lang="EN-US" style=""> </span></p>
<p class="yiv4238416288MsoNormal"><span style="">root@proxy: # cat /etc/squid/squid.conf | grep max_file</span></p>
<p class="yiv4238416288MsoNormal"><span lang="EN-US" style="">max_filedesc 64000</span></p>
<p class="yiv4238416288MsoNormal"><span lang="EN-US" style=""> </span></p>
<p class="yiv4238416288MsoNormal"><span lang="EN-US" style="">This force the system limits for squid process and tell squid how much FD it can consume.</span></p>
<p class="yiv4238416288MsoNormal"><span lang="EN-US" style=""> </span></p>
<p class="yiv4238416288MsoNormal"><span lang="EN-US" style="">Regards,</span></p>
<p class="yiv4238416288MsoNormal"><span lang="EN-US" style=""> </span></p>
<div>
<p style="line-height:105%;" class="yiv4238416288MsoNormal"><b><span style="font-size:10.0pt;line-height:105%;font-family:sans-serif;color:#1A214D;">Yvain PAYEN</span></b></p>
</div>
<p class="yiv4238416288MsoNormal"><span style=""> </span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm;">
<p class="yiv4238416288MsoNormal"><b>De :</b> squid-users <squid-users-bounces@lists.squid-cache.org>
<b>De la part de</b> paolo.prinx@gmail.com<br clear="none">
<b>Envoyé :</b> vendredi 12 juillet 2024 12:58<br clear="none">
<b>À :</b> squid-users@lists.squid-cache.org<br clear="none">
<b>Objet :</b> [squid-users] Socket handle leak?</p>
</div>
</div>
<p class="yiv4238416288MsoNormal"> </p>
<div style="border:dashed #9C6500 1.0pt;padding:1.0pt 1.0pt 1.0pt 1.0pt;">
<p style="line-height: 10pt;" class="yiv4238416288MsoNormal"><span style="font-size:10.0pt;font-family:UI sans-serif;color:black;">⚠</span><span style="font-size:10.0pt;color:black;"> FR : Ce message provient de l'extérieur de l'organisation.
N'ouvrez pas de liens ou de pièces jointes à moins que vous ne sachiez que le contenu est fiable. </span><span style="font-size:10.0pt;font-family:UI sans-serif;color:black;">⚠</span><span style="font-size:10.0pt;color:black;"></span></p><div id="yiv4238416288yqtfd81317" class="yiv4238416288yqt3065869294"></div><div id="yiv4238416288yqtfd78194" class="yiv4238416288yqt3065869294">
</div></div><div id="yiv4238416288yqtfd28061" class="yiv4238416288yqt3065869294">
<p style="margin-bottom:12.0pt;" class="yiv4238416288MsoNormal"> </p>
<div>
<div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;font-family:sans-serif;">Hello,</span></p>
</div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;font-family:sans-serif;"> apologies in advance for the silly question.</span></p>
</div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;font-family:sans-serif;"> </span></p>
</div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;font-family:sans-serif;">We are having some stability issues with our squid farms after a recent upgrade from Centos/Squid 3.5.x to Ubuntu/Squid 5.7/6.9. I wonder if anyone here has seen something
similar, and might have some suggestion about what we are obviously missing?</span></p>
</div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;font-family:sans-serif;"> </span></p>
</div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;font-family:sans-serif;"> </span></p>
</div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;font-family:sans-serif;">In short, after running for a certain period the servers run out of file descriptors. We see a slowly growing number of TCP or TCPv6 socket handles, that eventually hits
the configured maximum. The handles do not get released until after squid is restarted (-k restart)</span></p>
</div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;font-family:sans-serif;"> </span></p>
</div>
<div>
<div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;font-family:sans-serif;"> </span></p>
</div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;font-family:sans-serif;color:black;">It is somewhat similar to what reported under </span><span style="font-size:12.0pt;font-family:sans-serif;"><a rel="nofollow noopener noreferrer" shape="rect" target="_blank" href="https://access.redhat.com/solutions/3362211"><span style="color:#196AD4;">https://access.redhat.com/solutions/3362211</span></a><span style="color:black;"> .
They state that </span></span></p>
</div>
<div>
<ul type="disc"><li style="" class="yiv4238416288MsoNormal">
<i><span style="font-size:12.0pt;font-family:sans-serif;color:#252525;">If an application fails to </span></i><code><i><span style="font-family: Consolas; color: rgb(37, 37, 37);">close()</span></i></code><i><span style="font-size:12.0pt;font-family:sans-serif;color:#252525;"> it's
socket descriptors and continues to allocate new sockets then it can use up all the system memory on TCP(v6) slab objects.</span></i><span style="font-size:12.0pt;font-family:sans-serif;"></span></li><li style="" class="yiv4238416288MsoNormal">
<i><span style="font-size:12.0pt;font-family:sans-serif;color:#252525;">Note some of these sockets will not show up in </span></i><code><i><span style="font-family: Consolas; color: rgb(37, 37, 37);">/proc/net/sockstat(6)</span></i></code><i><span style="font-size:12.0pt;font-family:sans-serif;color:#252525;">.
Sockets that still have a file descriptor but are in the </span></i><code><i><span style="font-family: Consolas; color: rgb(37, 37, 37);">TCP_CLOSE</span></i></code><i><span style="font-size:12.0pt;font-family:sans-serif;color:#252525;"> state
will consume a slab object. But will not be accounted for in </span></i><code><i><span style="font-family: Consolas; color: rgb(37, 37, 37);">/proc/net/sockstat(6)</span></i></code><i><span style="font-size:12.0pt;font-family:sans-serif;color:#252525;"> or
"ss" or "netstat".</span></i><span style="font-size:12.0pt;font-family:sans-serif;"></span></li><li style="" class="yiv4238416288MsoNormal">
<i><span style="font-size:12.0pt;font-family:sans-serif;color:#252525;">It can be determined whether this is an application sockets leak, by stopping the application processes that are consuming sockets. If the slab objects in </span></i><code><i><span style="font-family: Consolas; color: rgb(37, 37, 37);">/proc/slabinfo</span></i></code><i><span style="font-size:12.0pt;font-family:sans-serif;color:#252525;"> are
freed then the application is responsible. As that means that destructor routines have found open file descriptors to sockets in the process.</span></i><span style="font-size:12.0pt;font-family:sans-serif;"></span></li></ul>
</div>
</div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;font-family:sans-serif;"> </span></p>
</div>
<div>
<p class="yiv4238416288MsoNormal"><i><span style="font-size:12.0pt;font-family:sans-serif;color:black;">"</span></i><i><span style="font-size:12.0pt;font-family:sans-serif;color:#252525;">This is most likely to be a case of the application not handling
error conditions correctly and not calling </span></i><code><i><span style="font-family: Consolas; color: rgb(37, 37, 37);">close()</span></i></code><i><span style="font-size:12.0pt;font-family:sans-serif;color:#252525;"> to free the FD and
socket."</span></i><span style="font-size:12.0pt;font-family:sans-serif;"></span></p>
</div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;font-family:sans-serif;"> </span></p>
</div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;font-family:sans-serif;"> </span></p>
</div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;font-family:sans-serif;">For example, on a server with squid 5.7, unmodified package:</span></p>
</div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;font-family:sans-serif;"> </span></p>
</div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;font-family:sans-serif;">list of open files;</span></p>
</div>
<blockquote style="margin-left:30.0pt;margin-right:0cm;">
<div>
<div>
<div>
<div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;">lsof |wc -l</span><span style="font-size:12.0pt;font-family:sans-serif;"></span></p>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;">56963</span><span style="font-size:12.0pt;font-family:sans-serif;"></span></p>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<div>
<div>
<div>
<div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;font-family:sans-serif;"> </span></p>
</div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;font-family:sans-serif;">of which 35K in TCPv6:</span></p>
</div>
</div>
</div>
</div>
</div>
<blockquote style="margin-left:30.0pt;margin-right:0cm;">
<div>
<div>
<div>
<div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;">lsof |grep proxy |grep TCPv6 |wc -l</span><span style="font-size:12.0pt;font-family:sans-serif;"></span></p>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<div>
<div>
<div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;"> 35301</span><span style="font-size:12.0pt;font-family:sans-serif;"></span></p>
</div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;font-family:sans-serif;"> </span></p>
</div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;font-family:sans-serif;">under /proc I see less objects<br clear="none">
</span><span style="font-size:12.0pt;"> cat /proc/net/tcp6 |wc -l</span><span style="font-size:12.0pt;font-family:sans-serif;"></span></p>
</div>
</div>
</div>
<div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;"> 3095</span><span style="font-size:12.0pt;font-family:sans-serif;"></span></p>
</div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;font-family:sans-serif;"> </span></p>
</div>
</div>
<div>
<div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;">b</span><span style="font-size:12.0pt;font-family:sans-serif;">ut the number of objects in the slabs is high</span></p>
</div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;"> cat /proc/slabinfo |grep TCPv6</span><span style="font-size:12.0pt;font-family:sans-serif;"></span></p>
</div>
<div>
<div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;"> MPTCPv6 0 0 2048 16 8 : tunables 0 0 0 : slabdata 0 0 0</span><span style="font-size:12.0pt;font-family:sans-serif;"></span></p>
</div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;"> tw_sock_TCPv6 1155 1155 248 33 2 : tunables 0 0 0 : slabdata 35 35 0</span><span style="font-size:12.0pt;font-family:sans-serif;"></span></p>
</div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;"> request_sock_TCPv6 0 0 304 26 2 : tunables 0 0 0 : slabdata 0 0 0</span><span style="font-size:12.0pt;font-family:sans-serif;"></span></p>
</div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;"> TCPv6
<b>38519 38519</b> 2432 13 8 : tunables 0 0 0 : slabdata 2963 2963 0</span><span style="font-size:12.0pt;font-family:sans-serif;"></span></p>
</div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;font-family:sans-serif;"> </span></p>
</div>
</div>
</div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;font-family:sans-serif;">I have 35K of lines like this</span></p>
</div>
</div>
</div>
<div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;"> lsof |grep proxy |grep TCPv6 |more</span><span style="font-size:12.0pt;font-family:sans-serif;"></span></p>
</div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;"> squid 1049 proxy 13u sock 0,8 0t0 5428173 protocol: TCPv6</span><span style="font-size:12.0pt;font-family:sans-serif;"></span></p>
</div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;"> squid 1049 proxy 14u sock 0,8 0t0 27941608 protocol: TCPv6</span><span style="font-size:12.0pt;font-family:sans-serif;"></span></p>
</div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;"> squid 1049 proxy 24u sock 0,8 0t0 45124047 protocol: TCPv6</span><span style="font-size:12.0pt;font-family:sans-serif;"></span></p>
</div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;"> squid 1049 proxy 25u sock 0,8 0t0 50689821 protocol: TCPv6</span><span style="font-size:12.0pt;font-family:sans-serif;"></span></p>
</div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;">...</span><span style="font-size:12.0pt;font-family:sans-serif;"></span></p>
</div>
</div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;font-family:sans-serif;"> </span></p>
</div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;font-family:sans-serif;"> </span></p>
</div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;font-family:sans-serif;">We thought maybe this is a weird IPv6 thing, as we only route IPv4, so we compiled a more recent version of squid with no v6 support. The thing just moved to TCP4..</span></p>
</div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;font-family:sans-serif;"> </span></p>
</div>
<div>
<div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;color:black;">lsof |wc -l</span><span style="font-size:12.0pt;font-family:sans-serif;color:black;"></span></p>
</div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;color:black;">120313</span><span style="font-size:12.0pt;font-family:sans-serif;color:black;"></span></p>
</div>
</div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;font-family:sans-serif;"> </span></p>
</div>
<div>
<div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;">cat /proc/slabinfo |grep TCP</span><span style="font-size:12.0pt;font-family:sans-serif;"></span></p>
</div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;">MPTCPv6 0 0 2048 16 8 : tunables 0 0 0 : slabdata 0 0 0</span><span style="font-size:12.0pt;font-family:sans-serif;"></span></p>
</div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;">tw_sock_TCPv6 0 0 248 33 2 : tunables 0 0 0 : slabdata 0 0 0</span><span style="font-size:12.0pt;font-family:sans-serif;"></span></p>
</div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;">request_sock_TCPv6 0 0 304 26 2 : tunables 0 0 0 : slabdata 0 0 0</span><span style="font-size:12.0pt;font-family:sans-serif;"></span></p>
</div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;">TCPv6 208 208 2432 13 8 : tunables 0 0 0 : slabdata 16 16 0</span><span style="font-size:12.0pt;font-family:sans-serif;"></span></p>
</div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;">MPTCP 0 0 1856 17 8 : tunables 0 0 0 : slabdata 0 0 0</span><span style="font-size:12.0pt;font-family:sans-serif;"></span></p>
</div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;">tw_sock_TCP 5577 5577 248 33 2 : tunables 0 0 0 : slabdata 169 169 0</span><span style="font-size:12.0pt;font-family:sans-serif;"></span></p>
</div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;">request_sock_TCP 1898 2002 304 26 2 : tunables 0 0 0 : slabdata 77 77 0</span><span style="font-size:12.0pt;font-family:sans-serif;"></span></p>
</div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;">TCP <b>102452 113274
</b> 2240 14 8 : tunables 0 0 0 : slabdata 8091 8091 0</span><span style="font-size:12.0pt;font-family:sans-serif;"></span></p>
</div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;font-family:sans-serif;"> </span></p>
</div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;font-family:sans-serif;"> </span></p>
</div>
</div>
<div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;">cat /proc/net/tcp |wc -l</span><span style="font-size:12.0pt;font-family:sans-serif;"></span></p>
</div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;">255</span><span style="font-size:12.0pt;font-family:sans-serif;"></span></p>
</div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;font-family:sans-serif;"> </span></p>
</div>
</div>
<p style="margin-bottom:12.0pt;" class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;font-family:sans-serif;">After restarting squid the slab objects are released and the open file descriptors drop to a reasonable value. This further suggests it is squid
hanging on to these FDs.</span></p>
</div>
<div>
<div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;">lsof |grep proxy |wc -l</span><span style="font-size:12.0pt;font-family:sans-serif;"></span></p>
</div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;">1221</span><span style="font-size:12.0pt;font-family:sans-serif;"></span></p>
</div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;font-family:sans-serif;"> </span></p>
</div>
</div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;font-family:sans-serif;"> </span></p>
</div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;font-family:sans-serif;">Any suggestion? I guess it's something blatantly obvious, but it's a couple of days we look at this and we're not going anywhere...</span></p>
</div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;font-family:sans-serif;"> </span></p>
</div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;font-family:sans-serif;">Thanks again</span></p>
</div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;font-family:sans-serif;"> </span></p>
</div>
<div>
<p class="yiv4238416288MsoNormal"><span style="font-size:12.0pt;font-family:sans-serif;"> </span></p>
</div>
</div>
</div>
</div></div><div id="yiv4238416288yqtfd92411" class="yiv4238416288yqt3065869294">
</div></div></div><blockquote></blockquote></blockquote></div>
</body></html>