[squid-users] SSL bump , high memory usage

[Steve Hill]

I think I've identified the bulk of the memory "leak" I've been tracking
down for the past few days.  As it turns out, it doesn't seem to be a
leak, but a problem with the SSL certificate caching.

The certificate cache is set by dynamic_cert_mem_cache_size and defaults
to 4MB.  Squid assumes an SSL context is 1KB, so we can cache up to 4096
certificates:

/// TODO: Replace on real size.
#define SSL_CTX_SIZE 1024

Unfortunately the assumed size isn't even close - it looks like an SSL
context usually weighs in at about 900KB!  So the default limit means
the cache can actually grow to around 3.6GB.  To make matters worse,
each worker gets its own cache, so in an 8-way SMP configuration, the
default "4MB" cache size limit actually ends up as around 30GB.


Possible fixes:
1. Stick with a static SSL_CTX_SIZE but use a more accurate estimate (I
suggest 1MB / context, based on my observations).
2. Calculate the context sizes correctly.
3. In the config, specify the maximum number of certificates to be
cached, rather than their total size.


In any case, as it stands the defaults are pretty much guaranteed to
cause a server to run out of memory.

How "heavy weight" is SSL context generation anyway?  i.e. if we expect
to generate 20 certificates a minute (which is what I see on a
production system in the first 5 minutes after startup), is that going
to place a significant load on the system, or is that ok?

the 900KB context size that I've observed seems pretty big to me.  Does
it sound reasonable, or is there some extra data being cached along with
the context that could have been freed earlier?


The instrumentation isn't good enough to be able to spot where this
memory usage originates without extensive debugging.  The memory isn't
allocated to memory pools, so mgr:mem doesn't show it, and
mgr:cached_ssl_cert relies on the incorrect estimate, so appears small
and inconsequential.  The processes can clearly be seen to grow rapidly,
but it all just shows up as unaccounted memory in mgr:info.  Secondly,
mgr:cached_ssl_cert doesn't appear to work with SMP workers.


My Squid process sizes are still larger than I would expect, so I'll
need to do more investigation, but reducing dynamic_cert_mem_cache_size
has stopped the rapid unbounded growth I have been seeing.

-- 

 - Steve Hill
   Technical Director
   Opendium Limited     http://www.opendium.com

Direct contacts:
   Instant messager: xmpp:steve at opendium.com
   Email:            steve at opendium.com
   Phone:            sip:steve at opendium.com

Sales / enquiries contacts:
   Email:            sales at opendium.com
   Phone:            +44-1792-825748 / sip:sales at opendium.com

Support contacts:
   Email:            support at opendium.com
   Phone:            +44-1792-824568 / sip:support at opendium.com