[squid-users] Significant memory leak with version 5.x (not with 4.17)
Alex Rousskov
rousskov at measurement-factory.com
Fri Jan 7 16:54:27 UTC 2022
On 1/7/22 12:12 AM, Praveen Ponakanti wrote:
> Is there a build with the fix, or do you have some recommended steps
> to manually pull the source, patch the fix and then recompile?
Yes, applying the patch to official Squid sources and then bootstrapping
and building from patched sources is what folks using patches usually
have to do. Roughly speaking:
cd squid-x.y.z/
patch -p1 < ...
./bootstrap.sh
./configure
make
make check
sudo make install
The above steps usually require installing a few build-related tools
(and custom options like --prefix and --with-openssl). A capable
sysadmin should be able to get this done in most cases. It is possible
to avoid the bootstrapping step if the patch does not modify
bootstrapping-sensitive files and you can get a bootstrapped version of
the sources from http://www.squid-cache.org/Versions/
I am not aware of anybody providing ready-to-use builds that include the
proposed bug 5132 fix.
HTH,
Alex.
> On Thu, Jan 6, 2022 at 8:16 PM Alex Rousskov wrote:
>
> On 1/6/22 2:50 AM, Praveen Ponakanti wrote:
> > Hi Alex/Amos,
> >
> > Do you still need memory logs from version 5.3 after stopping traffic
> > through the squid?
>
> I cannot answer for Amos who asked for those logs, but you may want to
> try a fix posted at
> https://bugs.squid-cache.org/show_bug.cgi?id=5132#c27
>
>
> HTH,
>
> Alex.
>
>
> > We have disabled traffic to the 5.3 version squid
> > about 6 hours ago and have not seen any memory being freed up since.
> > This node has used up ~50G more memory compared with 4.17 squid taking
> > similar traffic over the last 3+ weeks. I am collecting hourly memory
> > logs on 5.3 after stopping traffic. Let me know and I can attach the
> > log tomorrow morning.
> >
> > Thanks
> > Praveen
> >
> > On Mon, Dec 27, 2021 at 4:58 PM Praveen Ponakanti
> <pponakanti at roblox.com <mailto:pponakanti at roblox.com>
> > <mailto:pponakanti at roblox.com <mailto:pponakanti at roblox.com>>> wrote:
> >
> > I cant make any changes to our prod squids this week. I have a
> squid
> > instance (5.3v) in a test env but could not reproduce the leak by
> > starting & stopping traffic with a bulk http req generator (wrk).
> > Was able to send 175k rps @ 20k concurrent sessions (each doing a
> > get on a 1KB object) through the 30-worker squid. This initially
> > caused a 3G increase in memory usage and then flattened out after
> > stopping the requests. If I restart the bulk reqs, the memory
> usage
> > only goes up ~0.5GB and then drops back down. Live traffic is
> > probably exercising a different code path within squid's
> memory pools.
> >
> > On Mon, Dec 27, 2021 at 2:26 AM Lukáš Loučanský
> > <loucansky.lukas at kjj.cz <mailto:loucansky.lukas at kjj.cz>
> <mailto:loucansky.lukas at kjj.cz <mailto:loucansky.lukas at kjj.cz>>> wrote:
> >
> > After one day of running without clients my squid memory
> is stable
> >
> > 29345 proxy 20 0 171348 122360 14732 S 0.0 0.7
> > 0:25.96 (squid-1) --kid squid-1 -YC -f /etc/squid5/squid.conf
> > 29343 root 20 0 133712 79264 9284 S 0.0 0.5
> > 0:00.00 /usr/sbin/squid -YC -f /etc/squid5/squid.conf
> >
> > Storage Mem size: 3944 KB Storage Mem capacity: 0.2% used,
> 99.8%
> > free Maximum Resident Size: 489440 KB Page faults with
> physical
> > i/o: 0 Memory accounted for: Total accounted: 15741 KB
> > memPoolAlloc calls: 1061495 memPoolFree calls: 1071691 Total
> > allocated 15741 kB So this does not seem to be the
> problem... L
> >
> > Dne 26.12.2021 v 10:02 Lukáš Loučanský napsal(a):
> >> ok - as it seems my squid quacked on low memory again today -
> >>
> >> Dec 26 00:04:25 gw (squid-1): FATAL: Too many queued store_id
> >> requests; see on-persistent-overload.#012 current master
> >> transaction: master4629331
> >> Dec 26 00:04:28 gw squid[15485]: Squid Parent: squid-1
> process
> >> 15487 exited with status 1
> >> Dec 26 00:04:28 gw squid[15485]: Squid Parent: (squid-1)
> >> process 28375 started
> >>
> >> 2021/12/26 00:01:20 kid1| helperOpenServers: Starting 5/64
> >> 'storeid_file_rewrite' processes
> >> 2021/12/26 00:01:20 kid1| ipcCreate: fork: (12) Cannot
> >> allocate memory
> >> 2021/12/26 00:01:20 kid1| WARNING: Cannot run
> >> '/lib/squid5/storeid_file_rewrite' process.
> >> 2021/12/26 00:01:20 kid1| ipcCreate: fork: (12) Cannot
> >> allocate memory
> >>
> >> I'm going to reroute my clients (which are on their days off
> >> anyway) to direct connections and run it "dry" - on it's own.
> >> But I'm not able to to test it before "lack of memory issues
> >> occur" - because my clients are offline. So I'll watch squid
> >> for it's own memory consuption. It's all I can do right now -
> >> my squid already restarted and it's memory has been freed
> - so
> >> I think just now I have no power to fill it up again :-]
> >>
> >> L
> >>
> >> Dne 26.12.2021 v 7:41 Amos Jeffries napsal(a):
> >>>
> >>> If possible can one of you run a Squid to get this
> behaviour,
> >>> then stop new clients connecting to it before lack of memory
> >>> issues occur and see if the memory usage disappears or
> >>> reduces after a 24-48hr wait.
> >>>
> >>> A series of regular mempools report dumps from across the
> >>> test may help Alex or whoever works on the bug eliminate
> >>> further which cache and client related things are releasing
> >>> properly.
> >>>
> >>>
> >>> Amos
> >>>
> >>> _______________________________________________
> >>> squid-users mailing list
> >>> squid-users at lists.squid-cache.org
> <mailto:squid-users at lists.squid-cache.org>
> >>> <mailto:squid-users at lists.squid-cache.org
> <mailto:squid-users at lists.squid-cache.org>>
> >>> http://lists.squid-cache.org/listinfo/squid-users
> <http://lists.squid-cache.org/listinfo/squid-users>
> >>> <http://lists.squid-cache.org/listinfo/squid-users
> <http://lists.squid-cache.org/listinfo/squid-users>>
> >>
> >
> >
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>>
> > Bez virů. www.avast.com <http://www.avast.com>
> >
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>>
> >
> >
> >
> <#m_-6622557068709516458_m_9217020348889694418_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
> > _______________________________________________
> > squid-users mailing list
> > squid-users at lists.squid-cache.org
> <mailto:squid-users at lists.squid-cache.org>
> > <mailto:squid-users at lists.squid-cache.org
> <mailto:squid-users at lists.squid-cache.org>>
> > http://lists.squid-cache.org/listinfo/squid-users
> <http://lists.squid-cache.org/listinfo/squid-users>
> > <http://lists.squid-cache.org/listinfo/squid-users
> <http://lists.squid-cache.org/listinfo/squid-users>>
> >
> >
> > _______________________________________________
> > squid-users mailing list
> > squid-users at lists.squid-cache.org
> <mailto:squid-users at lists.squid-cache.org>
> > http://lists.squid-cache.org/listinfo/squid-users
> <http://lists.squid-cache.org/listinfo/squid-users>
> >
>
More information about the squid-users
mailing list