[squid-users] Performance issue /cache_dir / cache_mem

[Alex Rousskov]

On 11/8/18 8:46 AM, pacolo wrote:

> assertion failed: filemap.cc:50: "capacity_ <= (1 << 24)"

> I have noticed that this problem could be related to the maximum value of
> our cache_dir size, according to...
> 	https://bugs.squid-cache.org/show_bug.cgi?id=3566

As that bug discussion attempts to clarify, it is not the cache_dir size
as such, but the number of objects in that cache_dir. The latter is
limited by 16777216 objects (a hard-coded limit).


> What we need is to deploy 8 TB per server,

Divide 8 TB by your average disk-cached object size, then divide by
16777216 to find out the minimum number of cache_dirs per server.

For example, with 13KB average disk-cached object size, caching 8 TB on
disk requires 40 cache_dirs (8 * 1024 * 1024 * 1024 / 13 / 16777216).

See also: store_avg_object_size


> cache_dir aufs /cache 5242880 16 256

With default store_avg_object_size of 13KB, the above yields 412977624
objects, which is x24 more than the 16777216 limit.

I do not remember if Squid has a hard-coded maximum for the number of
cache_dirs. However, even if Squid does not, please note that several
algorithms iterate through all cache_dirs. Each iteration is relatively
fast (e.g., a complex hash lookup), but having lots of cache_dirs may
slow your Squid down because of these linear searches.

Alex.
P.S. If you move to SMP Squid, please note that you should not continue
to use aufs cache_dirs.