<div dir="ltr"><div><span style="font-family:"Helvetica Neue";font-size:13px">Hi,</span><br></div><div>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica Neue";min-height:15px"><br></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica Neue"">We are running the squid proxy for servicing outbound HTTP quests from our network and have observed a significant memory leak with 5.x versions. While there are several discussions about memory leaks with recent versions, just wanted to list out what we have observed in case this is an unknown leak.</p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica Neue";min-height:15px"><br></p>
<ul class="gmail-ul1">
<li class="gmail-li1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica Neue""><span class="gmail-s1" style="font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:9px;line-height:normal;font-family:Menlo"></span>The request-rate through our squid proxy currently ranges between a daily low of 10 rps to a daily high 125 rps.</li>
<li class="gmail-li1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica Neue""><span class="gmail-s1" style="font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:9px;line-height:normal;font-family:Menlo"></span>About mid-way during the daily ramp in request rates, the memory usage of the squid proxy starts to increase by about 1-2.5 G / day before leveling off till the next day's ramp in reqs. 4.17 does not exhibit this memory leak (or at least not at anything close to this rate).<span class="gmail-Apple-converted-space"> </span></li>
<li class="gmail-li1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica Neue""><span class="gmail-s1" style="font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:9px;line-height:normal;font-family:Menlo"></span>We are running 30 squid workers and the cache is set to deny all. Besides this, we have a single tcp_outgoing_address, some site specific ACL’s (both IP & domain acl's), and a custom access log format with a UDP target.</li>
<li class="gmail-li1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica Neue""><span class="gmail-s1" style="font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:9px;line-height:normal;font-family:Menlo"></span>Both versions of squid run on similar hosts (64 cores each) and receive identical traffic patterns throughout the day. Version 5.3 & 5.1 have similar rates of memory leak.<span class="gmail-Apple-converted-space"> </span></li>
</ul>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica Neue";min-height:15px"><br></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica Neue"">We are unable to use version 5.X in our production environment as we will have a much higher rate of requests through the proxy later on.</p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica Neue";min-height:15px"><br></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica Neue"">Due to the nature of the memory leak, it appears something with the memory pool management has been broken with version 5.x.</p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica Neue";min-height:15px"><br></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica Neue"">I have attempted to build squid with -with-valgrind-debug, and run it in a test env. However valgrind appears to collect some data from the parse config functions and then the squid proxy restarts. Valgrind no longer reports memory leak stats afterwards.</p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica Neue";min-height:15px"><br></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica Neue"">Snips from squid-internal-mgr/info. Note the data below does not appear to account for all the memory used by the squid process(es), which is also reported by the squid-exporter container. Our node level stats show that the 5.x squid is currently using up more than 28G, while the 4.17 version is under 7G. Both instances were setup to take traffic about 10 days ago.</p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica Neue";min-height:15px"><br></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica Neue"">Thanks</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica Neue"">Praveen</p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica Neue";min-height:15px"><br></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica Neue"">Version 5.3</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica Neue"">——————</p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica Neue";min-height:15px"><br></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica Neue"">Memory accounted for:</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica Neue""><span class="gmail-Apple-tab-span" style="white-space:pre">  </span>Total accounted: <span class="gmail-Apple-converted-space">      </span>1215892 KB</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica Neue""><span class="gmail-Apple-tab-span" style="white-space:pre">  </span>memPoolAlloc calls: 7529400915</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica Neue""><span class="gmail-Apple-tab-span" style="white-space:pre">  </span>memPoolFree calls:<span class="gmail-Apple-converted-space">  </span>7624989161</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica Neue"">File descriptor usage for squid:</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica Neue""><span class="gmail-Apple-tab-span" style="white-space:pre">  </span>Maximum number of file descriptors: <span class="gmail-Apple-converted-space">  </span>31457280</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica Neue""><span class="gmail-Apple-tab-span" style="white-space:pre">  </span>Largest file desc currently in use: <span class="gmail-Apple-converted-space">  </span>24962</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica Neue""><span class="gmail-Apple-tab-span" style="white-space:pre">  </span>Number of file desc currently in use: 4486</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica Neue""><span class="gmail-Apple-tab-span" style="white-space:pre">  </span>Files queued for open: <span class="gmail-Apple-converted-space">                  </span>0</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica Neue""><span class="gmail-Apple-tab-span" style="white-space:pre">  </span>Available number of file descriptors: 31452794</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica Neue""><span class="gmail-Apple-tab-span" style="white-space:pre">  </span>Reserved number of file descriptors:<span class="gmail-Apple-converted-space">  </span>3000</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica Neue""><span class="gmail-Apple-tab-span" style="white-space:pre">  </span>Store Disk files open: <span class="gmail-Apple-converted-space">                  </span>0</p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica Neue";min-height:15px"><br></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica Neue"">Version 4.17</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica Neue"">——————</p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica Neue";min-height:15px"><br></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica Neue"">Memory accounted for:</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica Neue""><span class="gmail-Apple-tab-span" style="white-space:pre">  </span>Total accounted:<span class="gmail-Apple-converted-space">        </span>76878 KB</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica Neue""><span class="gmail-Apple-tab-span" style="white-space:pre">  </span>memPoolAlloc calls: 6217787434</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica Neue""><span class="gmail-Apple-tab-span" style="white-space:pre">  </span>memPoolFree calls:<span class="gmail-Apple-converted-space">  </span>6301483686</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica Neue"">File descriptor usage for squid:</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica Neue""><span class="gmail-Apple-tab-span" style="white-space:pre">  </span>Maximum number of file descriptors: <span class="gmail-Apple-converted-space">  </span>31457280</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica Neue""><span class="gmail-Apple-tab-span" style="white-space:pre">  </span>Largest file desc currently in use: <span class="gmail-Apple-converted-space">  </span>20184</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica Neue""><span class="gmail-Apple-tab-span" style="white-space:pre">  </span>Number of file desc currently in use: 4419</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica Neue""><span class="gmail-Apple-tab-span" style="white-space:pre">  </span>Files queued for open: <span class="gmail-Apple-converted-space">                  </span>0</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica Neue""><span class="gmail-Apple-tab-span" style="white-space:pre">  </span>Available number of file descriptors: 31452861</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica Neue""><span class="gmail-Apple-tab-span" style="white-space:pre">  </span>Reserved number of file descriptors:<span class="gmail-Apple-converted-space">  </span>3000</p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica Neue""><span class="gmail-Apple-tab-span" style="white-space:pre">  </span>Store Disk files open: <span class="gmail-Apple-converted-space">                  </span>0</p></div></div>