[squid-users] Significant memory leak with version 5.x (not with 4.17)

Thu Jan 6 07:50:10 UTC 2022

Hi Alex/Amos,

Do you still need memory logs from version 5.3 after stopping traffic
through the squid? We have disabled traffic to the 5.3 version squid about
6 hours ago and have not seen any memory being freed up since. This node
has used up ~50G more memory compared with 4.17 squid taking similar
traffic over the last 3+ weeks. I am collecting hourly memory logs on 5.3
after stopping traffic. Let me know and I can attach the
log tomorrow morning.

Thanks
Praveen

On Mon, Dec 27, 2021 at 4:58 PM Praveen Ponakanti <pponakanti at roblox.com>
wrote:

> I cant make any changes to our prod squids this week. I have a squid
> instance (5.3v) in a test env but could not reproduce the leak by starting
> & stopping traffic with a bulk http req generator (wrk). Was able to send
> 175k rps @ 20k concurrent sessions (each doing a get on a 1KB object)
> through the 30-worker squid. This initially caused a 3G increase in memory
> usage and then flattened out after stopping the requests. If I restart the
> bulk reqs, the memory usage only goes up ~0.5GB and then drops back down.
> Live traffic is probably exercising a different code path within squid's
> memory pools.
>
> On Mon, Dec 27, 2021 at 2:26 AM Lukáš Loučanský <loucansky.lukas at kjj.cz>
> wrote:
>
>> After one day of running without clients my squid memory is stable
>>
>> 29345 proxy     20   0  171348 122360  14732 S   0.0   0.7   0:25.96
>> (squid-1) --kid squid-1 -YC -f /etc/squid5/squid.conf
>> 29343 root      20   0  133712  79264   9284 S   0.0   0.5   0:00.00
>> /usr/sbin/squid -YC -f /etc/squid5/squid.conf
>>
>> Storage Mem size:	3944 KB
>> Storage Mem capacity:	 0.2% used, 99.8% free
>> 	
>> Maximum Resident Size: 489440 KB
>> Page faults with physical i/o: 0
>> Memory accounted for:
>> Total accounted:        15741 KB
>> memPoolAlloc calls:   1061495
>> memPoolFree calls:    1071691
>>
>> Total allocated 15741	kB
>>
>> So this does not seem to be the problem...
>> L
>>
>> Dne 26.12.2021 v 10:02 Lukáš Loučanský napsal(a):
>>
>> ok - as it seems my squid quacked on low memory again today -
>>
>> Dec 26 00:04:25 gw (squid-1): FATAL: Too many queued store_id requests;
>> see on-persistent-overload.#012    current master transaction:
>> master4629331
>> Dec 26 00:04:28 gw squid[15485]: Squid Parent: squid-1 process 15487
>> exited with status 1
>> Dec 26 00:04:28 gw squid[15485]: Squid Parent: (squid-1) process 28375
>> started
>>
>> 2021/12/26 00:01:20 kid1| helperOpenServers: Starting 5/64
>> 'storeid_file_rewrite' processes
>> 2021/12/26 00:01:20 kid1| ipcCreate: fork: (12) Cannot allocate memory
>> 2021/12/26 00:01:20 kid1| WARNING: Cannot run
>> '/lib/squid5/storeid_file_rewrite' process.
>> 2021/12/26 00:01:20 kid1| ipcCreate: fork: (12) Cannot allocate memory
>>
>> I'm going to reroute my clients (which are on their days off anyway) to
>> direct connections and run it "dry" - on it's own. But I'm not able to to
>> test it before "lack of memory issues occur" - because my clients are
>> offline. So I'll watch squid for it's own memory consuption. It's all I can
>> do right now - my squid already restarted and it's memory has been freed -
>> so I think just now I have no power to fill it up again :-]
>>
>> L
>>
>> Dne 26.12.2021 v 7:41 Amos Jeffries napsal(a):
>>
>>
>> If possible can one of you run a Squid to get this behaviour, then stop
>> new clients connecting to it before lack of memory issues occur and see if
>> the memory usage disappears or reduces after a 24-48hr wait.
>>
>> A series of regular mempools report dumps from across the test may help
>> Alex or whoever works on the bug eliminate further which cache and client
>> related things are releasing properly.
>>
>>
>> Amos
>>
>> _______________________________________________
>> squid-users mailing list
>> squid-users at lists.squid-cache.org
>> http://lists.squid-cache.org/listinfo/squid-users
>>
>>
>>
>>
>> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> Bez
>> virů. www.avast.com
>> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>> <#m_-6622557068709516458_m_9217020348889694418_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>> _______________________________________________
>> squid-users mailing list
>> squid-users at lists.squid-cache.org
>> http://lists.squid-cache.org/listinfo/squid-users
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20220105/aa5adf1b/attachment.htm>