[squid-users] squid 3.5.23 memory usage

[Amos Jeffries]

On 20/01/2017 1:23 p.m., Ivan Larionov wrote:
> Hello.
> 
> I'm pretty sure this question has been asked multiple times already, but
> after reading everything I found I still can't figure out squid memory
> usage patterns.
> 
> We're currently trying to upgrade from squid 2.7 to squid 3.5 and memory
> usage on squid 3 is much much higher compared to squid 2 with the same
> configuration.

One thing to be aware of with this big step in versions is that 3.x has
a lot more things 64-bit enabled where 2.x was more 32-bit oriented. It
is minor in any one place, but does add up when dealing with large
numbers of objects.


> 
> What do I see:
> 
> squid running for several days with low traffic:
> 
> # top
>  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>  7367 squid     20   0 4780m 4.4g 5224 S  6.0 60.6 105:01.76 squid -N
> 
> So it uses 4.4GB resident memory. Ok, let's see important config options:
> 
> cache_mem 2298756 KB
> maximum_object_size_in_memory 8 KB
> memory_replacement_policy lru
> cache_replacement_policy lru
> 
> cache_dir aufs /mnt/services/squid/cache 445644 16 256
> 
> minimum_object_size 64 bytes # none-zero so we dont cache mistakes
> maximum_object_size 102400 KB
> 
> So we configured 2.2GB memory cache and 500GB disk cache. Disk cache is
> quite big but current usage is only 3GB:
> 
> # du -sh /mnt/services/squid/cache # cache_dir
> 3.0G  /mnt/services/squid/cache
> 
> Now I'm looking into this page
> http://wiki.squid-cache.org/SquidFaq/SquidMemory and see:
> 
> 14 MB of memory per 1 GB on disk for 64-bit Squid
> 

These wiki numbers are based on an average object size of 32KB.

By setting "maximum_object_size_in_memory 8 KB" you reduce that by 3x so
need to multiply the overhead per object by (3x more objects in same
space) for the cache_mem value to get a better estimate.

So,
 ... up to 100 MB of index for cache_mem
 ... up to 6 GB of index for cache_dir

Another difference is the buffers in Squid-3 are a bit bigger than those
used for Squid-2. Up to 256 KB per FD (Squid-2 stopped at 64KB).
 BUT, your pool details below show only the 16KB buffer being used much.
So I doubt it is client connections related.



> Which means disk cache should use ~50MB of RAM.
> 
> All these means we have ~2.2GB ram used for everything else except
> cache_mem and disk cache index.

No, the index is also in that 2.2 GB which is not being used by the
cache_mem.


> 
> Let's see top pools from mgr:mem:
> 
> Pool                  (KB)     %Tot
> mem_node              2298833  55.082
> Short Strings         622365   14.913
> HttpHeaderEntry       404531   9.693
> Long Strings          284520   6.817
> MemObject             182288   4.368
> HttpReply             155612   3.729
> StoreEntry            73965    1.772
> Medium Strings        71152    1.705
> cbdata MemBuf (12)    35573    0.852
> LRU policy node       30403    0.728
> MD5 digest            11380    0.273
> 16K Buffer            1056     0.025
> 
> These pools consume ~35% of total squid memory usage: Short Strings,
> HttpHeaderEntry, Long Strings, HttpReply. Looks suspicious. On squid 2 same
> pools use 10 times less memory.


The mem_node is the cache_mem space itself, plus active transactions data.

The StoreEntry is the index entry for each object (cache_dir, cache_mem
and in-transit).
The MemObject is the index entry for each in-memory object (cache_mem
and in-transit).
The HttpReply are those cached objects in parsed format.
The HttpHeaderEntry are all the headers in those reply objects.
The various Strings are the individual words/lines etc in those headers.

So we are under 1% values by the time we are done eliminating data
stored in cache_mem objects and active transaction data.


> 
> I found a bug which looks similar to our experience:
> http://bugs.squid-cache.org/show_bug.cgi?id=4084.
> 

Since you have configured your cache_mem to be 2.2 GB and total memory
usage is 4.4 GB the report saying 55% of memory is used for mem_node
looks fine to me. 50% of 4.4 GB is your 2.2 GB cache_mem setting, and
the extra 5% is probably active transactions and maybe some nodes for
the cache_dir data.

So I dont think it is the issue I mentioned in comments 8.
That said we have not fully identified what the bug problem was.


> I'm attaching our config, mgr:info, mgr:mem and some system info I
> collected.
> 
> Could someone say if this is normal and why it's so much different from
> squid 2?
> 

Well, tentatively yes. The Squid-3 numbers all looks reasonably
accurate. So there is no obvious sign of any problem from this one point
in time.

But if it worries you keep an eye on it for a week or so and see if
anything starts to skew. Graphs like Martin had in comment 5 on that bug
report would be a good indicator of whether there is a problem or just a
new "normal" level.

Amos