<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>ups - i mean reqs per minute :-], and I'm doing icap to clamav
daemon at localhost</p>
<p>L<br>
</p>
<div class="moz-cite-prefix">Dne 22.12.2021 v 10:36 Lukáš Loučanský
napsal(a):<br>
</div>
<blockquote type="cite"
cite="mid:b6a228c7-2b57-31e3-e2b3-1a386cfb63f1@kjj.cz">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<p>ok - I'dont want to tell stories how I went from rockd and
workers back to non-SMP and aufs. But I want to mention I face
these kind of issues too. I do not have too much evidence nor
expirience to call it memory leak - but - whatever I set to
cache_mem, all buffers, timeouts etc., reducing sysctl buffer
bloat for tcp/udp connections etc. my squid eats all my RAM and
starts to swap - and then crash by low memory (could not start
storeid helpers etc.)</p>
<p>After one day the resources taken look like this</p>
<p>15487 proxy 20 0 8478512 7.9g 17280 S 6.0 50.5
64:15.84 squid</p>
<p>and raising. I have about 370 clients, avg rps (now) 480 and
about 1000 peak. In this config 2GB (of 16GB) cache_mem (same
behaviour with 256MB) and about 80GB in two aufs cache dirs. I'm
doing an ssl "inspection" by looking at sni server names and
cert fingerprints during peek and splice steps (peek and
terminate to be precise), everything other is spliced. Would it
be SSL context memory bug like in the 3.5.x?</p>
<p>Resource usage for squid: <br>
</p>
<p>Maximum Resident Size: 33619392 KB <br>
</p>
<p>Memory accounted for: <br>
Total accounted: 1421104 KB <br>
memPoolAlloc calls: 226605414 <br>
memPoolFree calls: 229519316</p>
<p>Frankly - I don't have too much xmas power to go back to 4.x
and investigate. I went ahead to 6.x already... no change.</p>
<p>L<br>
</p>
<div class="moz-cite-prefix">Dne 22.12.2021 v 1:48 Praveen
Ponakanti napsal(a):<br>
</div>
<blockquote type="cite"
cite="mid:CACabJxOfSqVkN63DN=N=_J=B7CtH-ahRkFaAtM93H67BpQx9eA@mail.gmail.com">
<meta http-equiv="content-type" content="text/html;
charset=UTF-8">
<div dir="ltr">
<div><span style="font-family:"Helvetica
Neue";font-size:13px">Hi,</span><br>
</div>
<div>
<p class="gmail-p2"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue";min-height:15px"><br>
</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue"">We are running the squid proxy for servicing
outbound HTTP quests from our network and have observed a
significant memory leak with 5.x versions. While there are
several discussions about memory leaks with recent
versions, just wanted to list out what we have observed in
case this is an unknown leak.</p>
<p class="gmail-p2"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue";min-height:15px"><br>
</p>
<ul class="gmail-ul1">
<li class="gmail-li1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue""><span class="gmail-s1"
style="font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:9px;line-height:normal;font-family:Menlo"></span>The
request-rate through our squid proxy currently ranges
between a daily low of 10 rps to a daily high 125 rps.</li>
<li class="gmail-li1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue""><span class="gmail-s1"
style="font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:9px;line-height:normal;font-family:Menlo"></span>About
mid-way during the daily ramp in request rates, the
memory usage of the squid proxy starts to increase by
about 1-2.5 G / day before leveling off till the next
day's ramp in reqs. 4.17 does not exhibit this memory
leak (or at least not at anything close to this rate).<span
class="gmail-Apple-converted-space"> </span></li>
<li class="gmail-li1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue""><span class="gmail-s1"
style="font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:9px;line-height:normal;font-family:Menlo"></span>We
are running 30 squid workers and the cache is set to
deny all. Besides this, we have a single
tcp_outgoing_address, some site specific ACL’s (both IP
& domain acl's), and a custom access log format with
a UDP target.</li>
<li class="gmail-li1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue""><span class="gmail-s1"
style="font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:9px;line-height:normal;font-family:Menlo"></span>Both
versions of squid run on similar hosts (64 cores each)
and receive identical traffic patterns throughout the
day. Version 5.3 & 5.1 have similar rates of memory
leak.<span class="gmail-Apple-converted-space"> </span></li>
</ul>
<p class="gmail-p2"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue";min-height:15px"><br>
</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue"">We are unable to use version 5.X in our
production environment as we will have a much higher rate
of requests through the proxy later on.</p>
<p class="gmail-p2"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue";min-height:15px"><br>
</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue"">Due to the nature of the memory leak, it
appears something with the memory pool management has been
broken with version 5.x.</p>
<p class="gmail-p2"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue";min-height:15px"><br>
</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue"">I have attempted to build squid with
-with-valgrind-debug, and run it in a test env. However
valgrind appears to collect some data from the parse
config functions and then the squid proxy restarts.
Valgrind no longer reports memory leak stats afterwards.</p>
<p class="gmail-p2"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue";min-height:15px"><br>
</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue"">Snips from squid-internal-mgr/info. Note the
data below does not appear to account for all the memory
used by the squid process(es), which is also reported by
the squid-exporter container. Our node level stats show
that the 5.x squid is currently using up more than 28G,
while the 4.17 version is under 7G. Both instances were
setup to take traffic about 10 days ago.</p>
<p class="gmail-p2"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue";min-height:15px"><br>
</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue"">Thanks</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue"">Praveen</p>
<p class="gmail-p2"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue";min-height:15px"><br>
</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue"">Version 5.3</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue"">——————</p>
<p class="gmail-p2"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue";min-height:15px"><br>
</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue"">Memory accounted for:</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue""><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Total
accounted: <span class="gmail-Apple-converted-space">
</span>1215892 KB</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue""><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>memPoolAlloc
calls: 7529400915</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue""><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>memPoolFree
calls:<span class="gmail-Apple-converted-space"> </span>7624989161</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue"">File descriptor usage for squid:</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue""><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Maximum
number of file descriptors: <span
class="gmail-Apple-converted-space"> </span>31457280</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue""><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Largest
file desc currently in use: <span
class="gmail-Apple-converted-space"> </span>24962</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue""><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Number
of file desc currently in use: 4486</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue""><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Files
queued for open: <span
class="gmail-Apple-converted-space"> </span>0</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue""><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Available
number of file descriptors: 31452794</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue""><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Reserved
number of file descriptors:<span
class="gmail-Apple-converted-space"> </span>3000</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue""><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Store
Disk files open: <span
class="gmail-Apple-converted-space"> </span>0</p>
<p class="gmail-p2"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue";min-height:15px"><br>
</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue"">Version 4.17</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue"">——————</p>
<p class="gmail-p2"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue";min-height:15px"><br>
</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue"">Memory accounted for:</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue""><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Total
accounted:<span class="gmail-Apple-converted-space">
</span>76878 KB</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue""><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>memPoolAlloc
calls: 6217787434</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue""><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>memPoolFree
calls:<span class="gmail-Apple-converted-space"> </span>6301483686</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue"">File descriptor usage for squid:</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue""><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Maximum
number of file descriptors: <span
class="gmail-Apple-converted-space"> </span>31457280</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue""><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Largest
file desc currently in use: <span
class="gmail-Apple-converted-space"> </span>20184</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue""><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Number
of file desc currently in use: 4419</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue""><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Files
queued for open: <span
class="gmail-Apple-converted-space"> </span>0</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue""><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Available
number of file descriptors: 31452861</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue""><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Reserved
number of file descriptors:<span
class="gmail-Apple-converted-space"> </span>3000</p>
<p class="gmail-p1"
style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:13px;line-height:normal;font-family:"Helvetica
Neue""><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Store
Disk files open: <span
class="gmail-Apple-converted-space"> </span>0</p>
</div>
</div>
<br>
<fieldset class="moz-mime-attachment-header"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
squid-users mailing list
<a class="moz-txt-link-abbreviated moz-txt-link-freetext" href="mailto:squid-users@lists.squid-cache.org" moz-do-not-send="true">squid-users@lists.squid-cache.org</a>
<a class="moz-txt-link-freetext" href="http://lists.squid-cache.org/listinfo/squid-users" moz-do-not-send="true">http://lists.squid-cache.org/listinfo/squid-users</a>
</pre>
</blockquote>
<div id="DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2"><br>
<table style="border-top: 1px solid #D3D4DE;">
<tbody>
<tr>
<td style="width: 55px; padding-top: 18px;"><a
href="https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient"
target="_blank" moz-do-not-send="true"><img
src="https://ipmcdn.avast.com/images/icons/icon-envelope-tick-
round-orange-animated-no-repeat-v1.gif" alt=""
style="width: 46px; height: 29px;"
moz-do-not-send="true" width="46" height="29"></a></td>
<td style="width: 470px; padding-top: 17px; color:
#41424e; font-size: 13px; font-family: Arial, Helvetica,
sans-serif; line-height: 18px;">Bez virů. <a
href="https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient"
target="_blank" style="color: #4453ea;"
moz-do-not-send="true">www.avast.com</a> </td>
</tr>
</tbody>
</table>
<a href="#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2" width="1"
height="1" moz-do-not-send="true"> </a></div>
<br>
<fieldset class="moz-mime-attachment-header"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
squid-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:squid-users@lists.squid-cache.org">squid-users@lists.squid-cache.org</a>
<a class="moz-txt-link-freetext" href="http://lists.squid-cache.org/listinfo/squid-users">http://lists.squid-cache.org/listinfo/squid-users</a>
</pre>
</blockquote>
</body>
</html>