<html><head></head><body><div class="yahoo-style-wrap" style="font-family:Helvetica Neue, Helvetica, Arial, sans-serif;font-size:16px;"><div dir="ltr" data-setdir="false">Hello,</div><div dir="ltr" data-setdir="false"> apologies in advance for the silly question.</div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false">We are having some stability issues with our squid farms after a recent upgrade from Centos/Squid 3.5.x to Ubuntu/Squid 5.7/6.9. I wonder if anyone here has seen something similar, and might have some suggestion about what we are obviously missing?</div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false">In short, after running for a certain period the servers run out of file descriptors. We see a slowly growing number of TCP or TCPv6 socket handles, that eventually hits the configured maximum. The handles do not get released until after squid is restarted (-k restart)</div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false"><div><div><br></div><div><span style="color: rgb(0, 0, 0); font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 16px;">It is somewhat similar to what reported under </span><a href="https://access.redhat.com/solutions/3362211" style="color: rgb(25, 106, 212); text-decoration-line: underline; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 16px;" rel="nofollow" target="_blank">https://access.redhat.com/solutions/3362211</a><span style="color: rgb(0, 0, 0); font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 16px;"> . They state that </span></div><div><ul><li><i style="color: rgb(37, 37, 37); font-family: "Red Hat Text", RedHatText, "Helvetica Neue", Arial, sans-serif;">If an application fails to <code style="font-family: Consolas, Monaco, "Andale Mono", monospace; font-size: 14.4px; background-color: rgb(245, 245, 245); border-radius: 0px;">close()</code> it's socket descriptors and continues to allocate new sockets then it can use up all the system memory on TCP(v6) slab objects.</i></li><li><i style="color: rgb(37, 37, 37); font-family: "Red Hat Text", RedHatText, "Helvetica Neue", Arial, sans-serif;">Note some of these sockets will not show up in <code style="font-family: Consolas, Monaco, "Andale Mono", monospace; font-size: 14.4px; background-color: rgb(245, 245, 245); border-radius: 0px;">/proc/net/sockstat(6)</code>. Sockets that still have a file descriptor but are in the <code style="font-family: Consolas, Monaco, "Andale Mono", monospace; font-size: 14.4px; background-color: rgb(245, 245, 245); border-radius: 0px;">TCP_CLOSE</code> state will consume a slab object. But will not be accounted for in <code style="font-family: Consolas, Monaco, "Andale Mono", monospace; font-size: 14.4px; background-color: rgb(245, 245, 245); border-radius: 0px;">/proc/net/sockstat(6)</code> or "ss" or "netstat".</i></li><li><span style="color: rgb(37, 37, 37); font-family: "Red Hat Text", RedHatText, "Helvetica Neue", Arial, sans-serif; font-style: italic;">It can be determined whether this is an application sockets leak, by stopping the application processes that are consuming sockets. If the slab objects in </span><code style="color: rgb(37, 37, 37); font-style: italic; font-variant-ligatures: ; font-variant-caps: ; font-weight: ; font-family: Consolas, Monaco, "Andale Mono", monospace; font-size: 14.4px; background-color: rgb(245, 245, 245); border-radius: 0px;">/proc/slabinfo</code><span style="color: rgb(37, 37, 37); font-family: "Red Hat Text", RedHatText, "Helvetica Neue", Arial, sans-serif; font-style: italic;"> are freed then the application is responsible. As that means that destructor routines have found open file descriptors to sockets in the process.</span></li></ul></div></div><div dir="ltr" data-setdir="false"><i style="color: rgb(37, 37, 37); font-family: "Red Hat Text", RedHatText, "Helvetica Neue", Arial, sans-serif;"><br></i></div><div dir="ltr" data-setdir="false"><i style="color: rgb(37, 37, 37); font-family: "Red Hat Text", RedHatText, "Helvetica Neue", Arial, sans-serif;"><span><span style="color: rgb(0, 0, 0); font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 16px;">"</span><span style="color: rgb(0, 0, 0); font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 16px;"><i><span style="color: rgb(37, 37, 37); font-family: "Red Hat Text", RedHatText, "Helvetica Neue", Arial, sans-serif;">This is most likely to be a case of the application not handling error conditions correctly and not calling </span><code style="font-family: Consolas, Monaco, "Andale Mono", monospace; font-size: 14.4px; color: rgb(37, 37, 37); background-color: rgb(245, 245, 245); border-radius: 0px;">close()</code><span style="color: rgb(37, 37, 37); font-family: "Red Hat Text", RedHatText, "Helvetica Neue", Arial, sans-serif;"> to free the FD and socket."</span></i></span></span><br></i></div><br></div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false">For example, on a server with squid 5.7, unmodified package:<br></div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false">list of open files;</div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;"><div dir="ltr" data-setdir="false"><div><div dir="ltr" data-setdir="false"><div><div><font face="lucida console, sans-serif">lsof |wc -l</font></div></div></div></div></div><div dir="ltr" data-setdir="false"><div><div dir="ltr" data-setdir="false"><div><div><font face="lucida console, sans-serif">56963</font></div></div></div></div></div></blockquote><div dir="ltr" data-setdir="false"><div><div dir="ltr" data-setdir="false"> <div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false">of which 35K in TCPv6:</div></div></div></div></div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;"><div dir="ltr" data-setdir="false"><div><div dir="ltr" data-setdir="false"><div><div><font face="lucida console, sans-serif">lsof |grep proxy |grep TCPv6 |wc -l</font></div></div></div></div></div></blockquote><div dir="ltr" data-setdir="false"><div><div dir="ltr" data-setdir="false"><div><font face="lucida console, sans-serif"> 35301</font></div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false">under /proc I see less objects<br><span style="font-family: "lucida console", sans-serif;"> cat /proc/net/tcp6 |wc -l</span><br></div></div></div><div><div><font face="lucida console, sans-serif"> 3095</font></div><div><font face="lucida console, sans-serif"><br></font></div></div><div dir="ltr" data-setdir="false"><div><div dir="ltr" data-setdir="false"><font face="lucida console, sans-serif">b</font>ut the number of objects in the slabs is high</div><div><font face="lucida console, sans-serif"> cat /proc/slabinfo |grep TCPv6</font></div><div dir="ltr" data-setdir="false"> <div><div><font face="lucida console, sans-serif"> MPTCPv6 0 0 2048 16 8 : tunables 0 0 0 : slabdata 0 0 0</font></div><div><font face="lucida console, sans-serif"> tw_sock_TCPv6 1155 1155 248 33 2 : tunables 0 0 0 : slabdata 35 35 0</font></div><div><font face="lucida console, sans-serif"> request_sock_TCPv6 0 0 304 26 2 : tunables 0 0 0 : slabdata 0 0 0</font></div><div><font face="lucida console, sans-serif"> TCPv6 <b>38519 38519</b> 2432 13 8 : tunables 0 0 0 : slabdata 2963 2963 0</font></div><div><br></div></div></div><div dir="ltr" data-setdir="false">I have 35K of lines like this</div></div></div><div><div><font face="lucida console, sans-serif"> lsof |grep proxy |grep TCPv6 |more</font></div><div><font face="lucida console, sans-serif"> squid 1049 proxy 13u sock 0,8 0t0 5428173 protocol: TCPv6</font></div><div><font face="lucida console, sans-serif"> squid 1049 proxy 14u sock 0,8 0t0 27941608 protocol: TCPv6</font></div><div><font face="lucida console, sans-serif"> squid 1049 proxy 24u sock 0,8 0t0 45124047 protocol: TCPv6</font></div><div><font face="lucida console, sans-serif"> squid 1049 proxy 25u sock 0,8 0t0 50689821 protocol: TCPv6</font></div><div><font face="lucida console, sans-serif">...</font></div></div><br></div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false">We thought maybe this is a weird IPv6 thing, as we only route IPv4, so we compiled a more recent version of squid with no v6 support. The thing just moved to TCP4..</div><div dir="ltr" data-setdir="false"><font face="lucida console, sans-serif"><br></font></div><div dir="ltr" data-setdir="false"><div><div style="color: rgb(0, 0, 0); font-size: 16px;"><font face="lucida console, sans-serif">lsof |wc -l</font></div><div style="color: rgb(0, 0, 0); font-size: 16px;"><font face="lucida console, sans-serif">120313</font></div></div><font face="lucida console, sans-serif"><br></font></div><div dir="ltr" data-setdir="false"><div><div dir="ltr" data-setdir="false"><font face="lucida console, sans-serif">cat /proc/slabinfo |grep TCP</font></div><div><font face="lucida console, sans-serif">MPTCPv6 0 0 2048 16 8 : tunables 0 0 0 : slabdata 0 0 0</font></div><div><font face="lucida console, sans-serif">tw_sock_TCPv6 0 0 248 33 2 : tunables 0 0 0 : slabdata 0 0 0</font></div><div><font face="lucida console, sans-serif">request_sock_TCPv6 0 0 304 26 2 : tunables 0 0 0 : slabdata 0 0 0</font></div><div><font face="lucida console, sans-serif">TCPv6 208 208 2432 13 8 : tunables 0 0 0 : slabdata 16 16 0</font></div><div><font face="lucida console, sans-serif">MPTCP 0 0 1856 17 8 : tunables 0 0 0 : slabdata 0 0 0</font></div><div><font face="lucida console, sans-serif">tw_sock_TCP 5577 5577 248 33 2 : tunables 0 0 0 : slabdata 169 169 0</font></div><div><font face="lucida console, sans-serif">request_sock_TCP 1898 2002 304 26 2 : tunables 0 0 0 : slabdata 77 77 0</font></div><div><font face="lucida console, sans-serif">TCP <b>102452 113274 </b> 2240 14 8 : tunables 0 0 0 : slabdata 8091 8091 0</font></div><div><br></div><div><br></div></div><div><div><font face="lucida console, sans-serif">cat /proc/net/tcp |wc -l</font></div><div><font face="lucida console, sans-serif">255</font></div><div><br></div></div>After restarting squid the slab objects are released and the open file descriptors drop to a reasonable value. This further suggests it is squid hanging on to these FDs.<br><br></div><div dir="ltr" data-setdir="false"><div><div><font face="lucida console, sans-serif">lsof |grep proxy |wc -l</font></div><div><font face="lucida console, sans-serif">1221</font></div><div><br></div></div><br></div><div dir="ltr" data-setdir="false">Any suggestion? I guess it's something blatantly obvious, but it's a couple of days we look at this and we're not going anywhere...</div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false">Thanks again</div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false"><br></div></div></body></html>