<div dir="ltr">Hi Alex/Amos,<div><br></div><div>Do you still need memory logs from version 5.3 after stopping traffic through the squid? We have disabled traffic to the 5.3 version squid about 6 hours ago and have not seen any memory being freed up since. This node has used up ~50G more memory compared with 4.17 squid taking similar traffic over the last 3+ weeks. I am collecting hourly memory logs on 5.3 after stopping traffic. Let me know and I can attach the log tomorrow morning.</div><div><br></div><div>Thanks</div><div>Praveen</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Dec 27, 2021 at 4:58 PM Praveen Ponakanti <<a href="mailto:pponakanti@roblox.com">pponakanti@roblox.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">I cant make any changes to our prod squids this week. I have a squid instance (5.3v) in a test env but could not reproduce the leak by starting & stopping traffic with a bulk http req generator (wrk). Was able to send 175k rps @ 20k concurrent sessions (each doing a get on a 1KB object) through the 30-worker squid. This initially caused a 3G increase in memory usage and then flattened out after stopping the requests. If I restart the bulk reqs, the memory usage only goes up ~0.5GB and then drops back down. Live traffic is probably exercising a different code path within squid's memory pools.</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Dec 27, 2021 at 2:26 AM Lukáš Loučanský <<a href="mailto:loucansky.lukas@kjj.cz" target="_blank">loucansky.lukas@kjj.cz</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p><font size="2" face="Helvetica, Arial, sans-serif">After one day
of running without clients my squid memory is stable</font></p>
<p><font size="2" face="Helvetica, Arial, sans-serif">29345
proxy 20 0 171348 122360 14732 S 0.0 0.7 0:25.96
(squid-1) --kid squid-1 -YC -f /etc/squid5/squid.conf<br>
29343 root 20 0 133712 79264 9284 S 0.0 0.5
0:00.00 /usr/sbin/squid -YC -f /etc/squid5/squid.conf<br>
</font></p>
<pre style="color:rgb(0,0,0);font-size:medium;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><font size="2" face="Helvetica, Arial, sans-serif">Storage Mem size: 3944 KB
Storage Mem capacity: 0.2% used, 99.8% free
Maximum Resident Size: 489440 KB
Page faults with physical i/o: 0
Memory accounted for:
Total accounted: 15741 KB
memPoolAlloc calls: 1061495
memPoolFree calls: 1071691
Total allocated 15741 kB
So this does not seem to be the problem...
</font>L
</pre>
<div>Dne 26.12.2021 v 10:02 Lukáš Loučanský
napsal(a):<br>
</div>
<blockquote type="cite">ok - as it
seems my squid quacked on low memory again today -
<br>
<br>
Dec 26 00:04:25 gw (squid-1): FATAL: Too many queued store_id
requests; see on-persistent-overload.#012 current master
transaction: master4629331
<br>
Dec 26 00:04:28 gw squid[15485]: Squid Parent: squid-1 process
15487 exited with status 1
<br>
Dec 26 00:04:28 gw squid[15485]: Squid Parent: (squid-1) process
28375 started
<br>
<br>
2021/12/26 00:01:20 kid1| helperOpenServers: Starting 5/64
'storeid_file_rewrite' processes
<br>
2021/12/26 00:01:20 kid1| ipcCreate: fork: (12) Cannot allocate
memory
<br>
2021/12/26 00:01:20 kid1| WARNING: Cannot run
'/lib/squid5/storeid_file_rewrite' process.
<br>
2021/12/26 00:01:20 kid1| ipcCreate: fork: (12) Cannot allocate
memory
<br>
<br>
I'm going to reroute my clients (which are on their days off
anyway) to direct connections and run it "dry" - on it's own. But
I'm not able to to test it before "lack of memory issues occur" -
because my clients are offline. So I'll watch squid for it's own
memory consuption. It's all I can do right now - my squid already
restarted and it's memory has been freed - so I think just now I
have no power to fill it up again :-]
<br>
<br>
L
<br>
<br>
Dne 26.12.2021 v 7:41 Amos Jeffries napsal(a):
<br>
<blockquote type="cite">
<br>
If possible can one of you run a Squid to get this behaviour,
then stop new clients connecting to it before lack of memory
issues occur and see if the memory usage disappears or reduces
after a 24-48hr wait.
<br>
<br>
A series of regular mempools report dumps from across the test
may help Alex or whoever works on the bug eliminate further
which cache and client related things are releasing properly.
<br>
<br>
<br>
Amos
<br>
<br>
_______________________________________________
<br>
squid-users mailing list
<br>
<a href="mailto:squid-users@lists.squid-cache.org" target="_blank">squid-users@lists.squid-cache.org</a>
<br>
<a href="http://lists.squid-cache.org/listinfo/squid-users" target="_blank">http://lists.squid-cache.org/listinfo/squid-users</a>
<br>
</blockquote>
<br>
</blockquote>
<div id="gmail-m_-6622557068709516458gmail-m_9217020348889694418DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2"><br> <table style="border-top:1px solid rgb(211,212,222)">
<tbody><tr>
<td style="width:55px;padding-top:18px"><a href="https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient" target="_blank"><img src="https://ipmcdn.avast.com/images/icons/icon-envelope-tick-round-orange-animated-no-repeat-v1.gif" alt="" width="46" height="29" style="width: 46px; height: 29px;"></a></td>
<td style="width:470px;padding-top:17px;color:rgb(65,66,78);font-size:13px;font-family:Arial,Helvetica,sans-serif;line-height:18px">Bez virů. <a href="https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient" style="color:rgb(68,83,234)" target="_blank">www.avast.com</a> </td>
</tr>
</tbody></table>
<a href="#m_-6622557068709516458_m_9217020348889694418_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2" width="1" height="1"> </a></div></div>
_______________________________________________<br>
squid-users mailing list<br>
<a href="mailto:squid-users@lists.squid-cache.org" target="_blank">squid-users@lists.squid-cache.org</a><br>
<a href="http://lists.squid-cache.org/listinfo/squid-users" rel="noreferrer" target="_blank">http://lists.squid-cache.org/listinfo/squid-users</a><br>
</blockquote></div>
</blockquote></div>