[squid-users] leaking memory in squid 3.4.8 and 3.4.7.

[Amos Jeffries]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 1/10/2014 3:59 a.m., Steve Hill wrote:
> 
> On 30.09.14 15:13, Amos Jeffries wrote:
> 
>> IIRC the valgrind report is strangely at the end of mgr:info
>> rather than mgr:mem.
> 
> In that case the wiki is wrong. :) Anyway, it doesn't show up on
> either of them for me so I guess you might be right that it isn't
> "real" leaked memory.  (I still thought I'd see some indication
> that it was invoking the valgrind report though, even if it
> contained no data)
> 
>> With one small exception all the reports of memory "leaks" in 
>> 3.2/3.3/3.4 series are in fact memory over-allocations. The
>> difference is that over-allocated memory is still referenced by
>> some part of Squid, so it *does not* show up in a leak finder
>> like valgrind.
> 
> Do you have any advice for finding out what memory is still
> referenced in large amounts?  Since this is unaccounted memory, it
> presumably doesn't appear in mgr:mem (at least, I can't see
> anything obvious in there)?
> 
> I'm trying to figure out if there's a way of convincing valgrind to
> dump info about all the currently allocated memory while the
> program is still running - there would be a lot of legitimate stuff
> in the report, but hopefully a few hundred MB of memory that
> shouldn't be there would stick out like a sore thumb.

That would be lovely. If you can find it I'd like to know too :-)

A third option is using a ALL,9 cache.log tracer. We have begun
instrumenting class contstructors and destructors with debug
statements to record their lifetimes regardless of whether they are
accounted. A lot are not yet done though.
 The bzr repo contains a debug script at scripts/find-alive.pl which
correlates constructors to destructors in that log. It is useful for
scanning a cache.log while Squid is still running, crashed, or paused
in debugger to show what Jobs/requests etc are currently active.

> 
>> If you are using a 64-bit OS then the unaccounted numbers there
>> are bogus. They are based on mallinfo() lookup results which
>> suffer from 32-bit wrap issues. Use the OS command line to get a
>> memory report from top or similar instead.
> 
> In this case I don't believe the data is bogus.  I have 8 workers,
> which top shows as: 26249 squid     20   0  871m 782m 5348 S  0.7
> 10.0   8:44.81 squid 26245 squid     20   0  732m 644m 5344 S  0.3
> 8.3   8:00.77 squid 26244 squid     20   0  706m 617m 5348 S  1.0
> 7.9   7:42.29 squid 26250 squid     20   0  699m 613m 5348 S  3.6
> 7.9   4:43.49 squid 26246 squid     20   0  699m 612m 5348 S  2.3
> 7.9   6:12.78 squid 26251 squid     20   0  662m 576m 5348 S  0.7
> 7.4   5:45.11 squid 26248 squid     20   0  649m 564m 5348 S  2.3
> 7.2   9:22.91 squid 26247 squid     20   0  603m 518m 5348 S  1.3
> 6.6   4:45.47 squid
> 
> Adding the sizes in the "virt" column gives me 5621MB.  The
> combined RSS is 4926MB.  Process 26250 has just 2MB in swap, the
> rest have no swap at all.  I guess the difference between the RSS
> and Virt totals can probably be almost entirely accounted for by
> the executables.
> 
> mallinfo() says there is about 4897MB in the arena - close enough
> to sum of the RSS that I'm inclined to believe it.  Since no one
> process exceeds 2GB, mallinfo() is probably trustworthy in this
> case.
> 
> The accounted for memory is 440MB - potentially higher than I
> would expect for a Squid with caching disabled, but certainly low
> enough to not be an immediate concern.  But that gives us about
> 4.4GB of memory unaccounted for (by subtracting 440MB from the sum
> of the RSS as shown by top).
> 
> 4.4GB of unaccounted memory after running for just 6 hours is a 
> significant amount, and if left to their own devices Squid
> continues to eat all the memory, leaving the machine thrashing
> swap.
> 
> I count 531 sockets in netstat, so my guess is that it isn't just
> data associated with some open sockets: netstat -apn|egrep
> '26249|26245|26244|26250|26246|26251|26248|26247' -c 531
> 
> 

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (MingW32)

iQEcBAEBAgAGBQJUKsh9AAoJELJo5wb/XPRjUnAIALX5yfqY/XOVOcH90OXUree2
sHDJdCB5zlr4EO94qvichnkUm5bCMlg78dkPv8Nwyd43nG4SUUBNt4nh2dvZxYHn
8scHWFA1ZMsbl3k3dQ3eEHNCm+PfugvBo34ZokkMpSR8mLZhh9vqxXDhh9rwj6ez
K8mKnYMbNI/z/DOHhJxcM1NdnbYIyO84EUQfb9RGaJaKb8LDNLt4j6yCEeKa2Z8N
YcnPdA9VZZ3RCpImq0Py8U0kZbCPPBRs+cJAvUIgaAO080ZTrbF+bLZ+dHe4W7hA
JMxXUD4lfEy2HVja2+jfKBcJQwVCDwBK0tcu76bQ64WIrfuIIc/NyIOBwEhPT1I=
=cLQ3
-----END PGP SIGNATURE-----