[squid-users] Memory Leak Squid 3.4.9 on FreeBSD 10.0 x64

[Amos Jeffries]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 9/01/2015 8:10 a.m., Doug Sampson wrote:
>> Hi,
>> 
>> I have the similar problem on FreeBSD 10.1-STABLE #1 r275861
>> with squid-3.4.10. I also applied MEMPOOLS=1 when starting squid.
>> I experience the process slowing down and unacceptable
>> performance.
>> 
>> Squid is configured to use kerberos and ntlm authentication and
>> lap group authentication. other settings:
>> 

~14 MB for Squid process memory.

>> cache_replacement_policy heap LFUDA cache_mem 4096 MB

+ 4096 MB for RAM cache
+ 60 MB for RAM cache index

>> maximum_object_size 32 MB cache_dir diskd /usr/local/squid/cache
>> 32768 32 256

+ 540 MB for disk cache index

+ (600 * 16 * 0.5) MB for each active client connection state.
  ==> 4800 MB

  NP: modern web browsers open up to 8 parallel connections to load a
page ("happy eyeballs" makes that 16 TCP sockets) per client ==> ~8 MB
per active client.

Grand total:
  => 9.5 GB of RAM just for Squid.

.. then there is whatever memory the helper programs, other software
on the server and operating system all need.


>> 
>> I have seen the following errors in cache.log:
>> 
>> FATAL: Received Segment Violation...dying. FATAL: Received Bus
>> Error...dying.

need a backtrace to tell what the "Segment Violation" is about.

"Bus Error" is your CPU or RAM futzing up. The OS or hardware is
broken. Quite possibly this is the result of runnign out of RAM and
swap space.

>> 
>> after this the squid restarts. The system has 10GB of memory and
>> is working as internal cache for ~600 users.
>> 
>> Please point me in the right direction. I have no problem
>> running squid33-3.3.13 on FreeBSD 9.3-STABLE #0 r270210.
>> 
>> Thank you very much.
>> 
>> Regards,
>> 
>> lk
> 
> Man, I empathize with you. Have you tried running Squid 3.4.x on
> FreeBSD 9.3? Sometimes I wonder if it's FreeBSD 10.x that's causing
> the issue...
> 
> I tried the shell variable MEMPOOLS=1 and that quickly made the
> situation a lot worse. Swap space would get filled up very quickly
> and the system would slow down quickly before crashing.
> 

All MEMPOOLS=1 does is make Squid internally recycle the memory that
gets free'd. Re-using it for other objects of the same type that are
allocated right after its free'd. So new memory only gets allocated
once the recycling "pool" for an object type is empty, everything
allocated so far actively in use.

The main point of using MEMPOOLS=1 is a debugging aid to track down
leaks, by a) reduced OS allocator calls making frequent ones obvious,
and b) the mgr:mem report more clearly listing what objects are being
free'd (chunked pool contains recycled objects) and which are not
(pool always empty).



Steve Hill has come up with some patches that resolve memory issues
with the new 3.4 helper annotations feature when using Negotiate or
NTMLM auth helpers. For authenticated clients with long connection
times or high traffic volumes the state can accumulate up to quite
large amounts of memory. see the "Debugging slow proxy" thread of
earlier this week.

HTH
Amos

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (MingW32)

iQEcBAEBAgAGBQJUrysEAAoJELJo5wb/XPRjFFMH/RNhVzPWfFixDDarpHUFQMoE
6QLkaD5wwQfRp3xfBW/QfZ9vvu04zptFOvNr4766jGsh9RSGZdNZwgPc9pCADDrv
Xs0HCO1VDMxGoBjFfaS2XJ5DLX2zQdbYlZKb0yghxXSYzg0ZZEhmsKO8r1Exp8j7
KO95yP3z/8agmwXbU2sJ1esqRC7IfW2sF/DtU8cPTzUf0cKEyjGoCbVrNAN1qKUH
jLf1iw8sNr3xGwFG/WmQmKpYTsXInSp4GmwXgCSQ0T5DCLVDtWuvnXpobxR8iuSW
zIX/CDKTe58lfVP4EXdzJNBgiuEiVZaRccynbk2L8HRvJ8kt9Pk50P5tWXM5FMw=
=lv6G
-----END PGP SIGNATURE-----