[squid-users] Significant memory leak with version 5.x (not with 4.17)

Fri Jan 7 04:16:04 UTC 2022

On 1/6/22 2:50 AM, Praveen Ponakanti wrote:
> Hi Alex/Amos,
> 
> Do you still need memory logs from version 5.3 after stopping traffic
> through the squid?

I cannot answer for Amos who asked for those logs, but you may want to
try a fix posted at
https://bugs.squid-cache.org/show_bug.cgi?id=5132#c27

HTH,

Alex.

> We have disabled traffic to the 5.3 version squid
> about 6 hours ago and have not seen any memory being freed up since.
> This node has used up ~50G more memory compared with 4.17 squid taking
> similar traffic over the last 3+ weeks. I am collecting hourly memory
> logs on 5.3 after stopping traffic. Let me know and I can attach the
> log tomorrow morning.
> 
> Thanks
> Praveen
> 
> On Mon, Dec 27, 2021 at 4:58 PM Praveen Ponakanti <pponakanti at roblox.com
> <mailto:pponakanti at roblox.com>> wrote:
> 
>     I cant make any changes to our prod squids this week. I have a squid
>     instance (5.3v) in a test env but could not reproduce the leak by
>     starting & stopping traffic with a bulk http req generator (wrk).
>     Was able to send 175k rps @ 20k concurrent sessions (each doing a
>     get on a 1KB object) through the 30-worker squid. This initially
>     caused a 3G increase in memory usage and then flattened out after
>     stopping the requests. If I restart the bulk reqs, the memory usage
>     only goes up ~0.5GB and then drops back down. Live traffic is
>     probably exercising a different code path within squid's memory pools.
> 
>     On Mon, Dec 27, 2021 at 2:26 AM Lukáš Loučanský
>     <loucansky.lukas at kjj.cz <mailto:loucansky.lukas at kjj.cz>> wrote:
> 
>         After one day of running without clients my squid memory is stable
> 
>         29345 proxy     20   0  171348 122360  14732 S   0.0   0.7  
>         0:25.96 (squid-1) --kid squid-1 -YC -f /etc/squid5/squid.conf
>         29343 root      20   0  133712  79264   9284 S   0.0   0.5  
>         0:00.00 /usr/sbin/squid -YC -f /etc/squid5/squid.conf
> 
>         Storage Mem size: 3944 KB Storage Mem capacity: 0.2% used, 99.8%
>         free Maximum Resident Size: 489440 KB Page faults with physical
>         i/o: 0 Memory accounted for: Total accounted: 15741 KB
>         memPoolAlloc calls: 1061495 memPoolFree calls: 1071691 Total
>         allocated 15741 kB So this does not seem to be the problem... L
> 
>         Dne 26.12.2021 v 10:02 Lukáš Loučanský napsal(a):
>>         ok - as it seems my squid quacked on low memory again today -
>>
>>         Dec 26 00:04:25 gw (squid-1): FATAL: Too many queued store_id
>>         requests; see on-persistent-overload.#012    current master
>>         transaction: master4629331
>>         Dec 26 00:04:28 gw squid[15485]: Squid Parent: squid-1 process
>>         15487 exited with status 1
>>         Dec 26 00:04:28 gw squid[15485]: Squid Parent: (squid-1)
>>         process 28375 started
>>
>>         2021/12/26 00:01:20 kid1| helperOpenServers: Starting 5/64
>>         'storeid_file_rewrite' processes
>>         2021/12/26 00:01:20 kid1| ipcCreate: fork: (12) Cannot
>>         allocate memory
>>         2021/12/26 00:01:20 kid1| WARNING: Cannot run
>>         '/lib/squid5/storeid_file_rewrite' process.
>>         2021/12/26 00:01:20 kid1| ipcCreate: fork: (12) Cannot
>>         allocate memory
>>
>>         I'm going to reroute my clients (which are on their days off
>>         anyway) to direct connections and run it "dry" - on it's own.
>>         But I'm not able to to test it before "lack of memory issues
>>         occur" - because my clients are offline. So I'll watch squid
>>         for it's own memory consuption. It's all I can do right now -
>>         my squid already restarted and it's memory has been freed - so
>>         I think just now I have no power to fill it up again :-]
>>
>>         L
>>
>>         Dne 26.12.2021 v 7:41 Amos Jeffries napsal(a):
>>>
>>>         If possible can one of you run a Squid to get this behaviour,
>>>         then stop new clients connecting to it before lack of memory
>>>         issues occur and see if the memory usage disappears or
>>>         reduces after a 24-48hr wait.
>>>
>>>         A series of regular mempools report dumps from across the
>>>         test may help Alex or whoever works on the bug eliminate
>>>         further which cache and client related things are releasing
>>>         properly.
>>>
>>>
>>>         Amos
>>>
>>>         _______________________________________________
>>>         squid-users mailing list
>>>         squid-users at lists.squid-cache.org
>>>         <mailto:squid-users at lists.squid-cache.org>
>>>         http://lists.squid-cache.org/listinfo/squid-users
>>>         <http://lists.squid-cache.org/listinfo/squid-users>
>>
> 
>         <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>         	Bez virů. www.avast.com
>         <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
> 
> 
>         <#m_-6622557068709516458_m_9217020348889694418_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>         _______________________________________________
>         squid-users mailing list
>         squid-users at lists.squid-cache.org
>         <mailto:squid-users at lists.squid-cache.org>
>         http://lists.squid-cache.org/listinfo/squid-users
>         <http://lists.squid-cache.org/listinfo/squid-users>
> 
> 
> _______________________________________________
> squid-users mailing list
> squid-users at lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users
> 