<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>ok - I'dont want to tell stories how I went from rockd and
workers back to non-SMP and aufs. But I want to mention I face
these kind of issues too. I do not have too much evidence nor
expirience to call it memory leak - but - whatever I set to
cache_mem, all buffers, timeouts etc., reducing sysctl buffer
bloat for tcp/udp connections etc. my squid eats all my RAM and
starts to swap - and then crash by low memory (could not start
storeid helpers etc.)</p>
<p>After one day the resources taken look like this</p>
<p>15487 proxy 20 0 8478512 7.9g 17280 S 6.0 50.5
64:15.84 squid</p>
<p>and raising. I have about 370 clients, avg rps (now) 480 and
about 1000 peak. In this config 2GB (of 16GB) cache_mem (same
behaviour with 256MB) and about 80GB in two aufs cache dirs. I'm
doing an ssl "inspection" by looking at sni server names and cert
fingerprints during peek and splice steps (peek and terminate to
be precise), everything other is spliced. Would it be SSL context
memory bug like in the 3.5.x?</p>
<p>Resource usage for squid: <br>
</p>
<p>Maximum Resident Size: 33619392 KB <br>
</p>
<p>Memory accounted for: <br>
Total accounted: 1421104 KB <br>
memPoolAlloc calls: 226605414
<br>
memPoolFree calls: 229519316</p>
<p>Frankly - I don't have too much xmas power to go back to 4.x and
investigate. I went ahead to 6.x already... no change.</p>
<p>L<br>
</p>
<div class="moz-cite-prefix">Dne 22.12.2021 v 1:48 Praveen Ponakanti
napsal(a):<br>
</div>
<blockquote type="cite"
cite="mid:CACabJxOfSqVkN63DN=N=_J=B7CtH-ahRkFaAtM93H67BpQx9eA@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div><span style="font-family:"Helvetica
Neue";font-size:13px">Hi,</span><br>
</div>
<div>
<p class="gmail-p2"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue";min-height:15px"><br>
</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue"">We are running the squid proxy for servicing
outbound HTTP quests from our network and have observed a
significant memory leak with 5.x versions. While there are
several discussions about memory leaks with recent versions,
just wanted to list out what we have observed in case this
is an unknown leak.</p>
<p class="gmail-p2"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue";min-height:15px"><br>
</p>
<ul class="gmail-ul1">
<li class="gmail-li1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue""><span class="gmail-s1"
style="font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:9px;line-height:normal;font-family:Menlo"></span>The
request-rate through our squid proxy currently ranges
between a daily low of 10 rps to a daily high 125 rps.</li>
<li class="gmail-li1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue""><span class="gmail-s1"
style="font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:9px;line-height:normal;font-family:Menlo"></span>About
mid-way during the daily ramp in request rates, the memory
usage of the squid proxy starts to increase by about 1-2.5
G / day before leveling off till the next day's ramp in
reqs. 4.17 does not exhibit this memory leak (or at least
not at anything close to this rate).<span
class="gmail-Apple-converted-space"> </span></li>
<li class="gmail-li1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue""><span class="gmail-s1"
style="font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:9px;line-height:normal;font-family:Menlo"></span>We
are running 30 squid workers and the cache is set to deny
all. Besides this, we have a single tcp_outgoing_address,
some site specific ACL’s (both IP & domain acl's), and
a custom access log format with a UDP target.</li>
<li class="gmail-li1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue""><span class="gmail-s1"
style="font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:9px;line-height:normal;font-family:Menlo"></span>Both
versions of squid run on similar hosts (64 cores each) and
receive identical traffic patterns throughout the day.
Version 5.3 & 5.1 have similar rates of memory leak.<span
class="gmail-Apple-converted-space"> </span></li>
</ul>
<p class="gmail-p2"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue";min-height:15px"><br>
</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue"">We are unable to use version 5.X in our
production environment as we will have a much higher rate of
requests through the proxy later on.</p>
<p class="gmail-p2"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue";min-height:15px"><br>
</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue"">Due to the nature of the memory leak, it appears
something with the memory pool management has been broken
with version 5.x.</p>
<p class="gmail-p2"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue";min-height:15px"><br>
</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue"">I have attempted to build squid with
-with-valgrind-debug, and run it in a test env. However
valgrind appears to collect some data from the parse config
functions and then the squid proxy restarts. Valgrind no
longer reports memory leak stats afterwards.</p>
<p class="gmail-p2"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue";min-height:15px"><br>
</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue"">Snips from squid-internal-mgr/info. Note the
data below does not appear to account for all the memory
used by the squid process(es), which is also reported by the
squid-exporter container. Our node level stats show that the
5.x squid is currently using up more than 28G, while the
4.17 version is under 7G. Both instances were setup to take
traffic about 10 days ago.</p>
<p class="gmail-p2"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue";min-height:15px"><br>
</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue"">Thanks</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue"">Praveen</p>
<p class="gmail-p2"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue";min-height:15px"><br>
</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue"">Version 5.3</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue"">——————</p>
<p class="gmail-p2"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue";min-height:15px"><br>
</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue"">Memory accounted for:</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue""><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Total
accounted: <span class="gmail-Apple-converted-space">
</span>1215892 KB</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue""><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>memPoolAlloc
calls: 7529400915</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue""><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>memPoolFree
calls:<span class="gmail-Apple-converted-space"> </span>7624989161</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue"">File descriptor usage for squid:</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue""><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Maximum
number of file descriptors: <span
class="gmail-Apple-converted-space"> </span>31457280</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue""><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Largest
file desc currently in use: <span
class="gmail-Apple-converted-space"> </span>24962</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue""><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Number
of file desc currently in use: 4486</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue""><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Files
queued for open: <span class="gmail-Apple-converted-space">
</span>0</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue""><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Available
number of file descriptors: 31452794</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue""><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Reserved
number of file descriptors:<span
class="gmail-Apple-converted-space"> </span>3000</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue""><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Store
Disk files open: <span class="gmail-Apple-converted-space">
</span>0</p>
<p class="gmail-p2"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue";min-height:15px"><br>
</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue"">Version 4.17</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue"">——————</p>
<p class="gmail-p2"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue";min-height:15px"><br>
</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue"">Memory accounted for:</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue""><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Total
accounted:<span class="gmail-Apple-converted-space">
</span>76878 KB</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue""><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>memPoolAlloc
calls: 6217787434</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue""><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>memPoolFree
calls:<span class="gmail-Apple-converted-space"> </span>6301483686</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue"">File descriptor usage for squid:</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue""><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Maximum
number of file descriptors: <span
class="gmail-Apple-converted-space"> </span>31457280</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue""><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Largest
file desc currently in use: <span
class="gmail-Apple-converted-space"> </span>20184</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue""><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Number
of file desc currently in use: 4419</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue""><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Files
queued for open: <span class="gmail-Apple-converted-space">
</span>0</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue""><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Available
number of file descriptors: 31452861</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue""><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Reserved
number of file descriptors:<span
class="gmail-Apple-converted-space"> </span>3000</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue""><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Store
Disk files open: <span class="gmail-Apple-converted-space">
</span>0</p>
</div>
</div>
<br>
<fieldset class="moz-mime-attachment-header"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
squid-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:squid-users@lists.squid-cache.org">squid-users@lists.squid-cache.org</a>
<a class="moz-txt-link-freetext" href="http://lists.squid-cache.org/listinfo/squid-users">http://lists.squid-cache.org/listinfo/squid-users</a>
</pre>
</blockquote>
<div id="DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2"><br /> <table style="border-top: 1px solid #D3D4DE;">
<tr>
<td style="width: 55px; padding-top: 18px;"><a href="https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient" target="_blank"><img src="https://ipmcdn.avast.com/images/icons/icon-envelope-tick-round-orange-animated-no-repeat-v1.gif" alt="" width="46" height="29" style="width: 46px; height: 29px;" /></a></td>
<td style="width: 470px; padding-top: 17px; color: #41424e; font-size: 13px; font-family: Arial, Helvetica, sans-serif; line-height: 18px;">Bez virů. <a href="https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient" target="_blank" style="color: #4453ea;">www.avast.com</a> </td>
</tr>
</table>
<a href="#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2" width="1" height="1"> </a></div></body>
</html>