[squid-users] HDD/RAM Capacity vs store_avg_object_size

[Alex Rousskov]

On 07/12/2017 10:11 AM, Amos Jeffries wrote:
> On 12/07/17 22:31, bugreporter wrote:
>> Hi,
>>
>> Can anybody help me to confirm my understanding of the memory usage vs
>> the
>> persistent cache capacity? Below my understanding:
>>
>> According to http://wiki.squid-cache.org/SquidFaq/SquidMemory:
>>
>> 1- We need 14 MB of memory per 1 GB on disk for 64-bit Squid.The wiki is
>> there since I know squid (ie. i'm very old now). Is this information
>> still
>> valid?
> 
> Yes. It is a rough estimate based on the size of code objects used to
> store each request message - they have not changed in at least the past
> 10 years.

For the record, the StoreEntry object has changed, but I do not know how
much those (minor!) changes affect the rough estimate. It is likely that
they do not.


>> 2- Is this assumption based on the default value of 13 KB for
>> *store_avg_object_size*?
> 
> No.

Actually, the answer is probably "yes" or "yes, that or a similar mean
object size value".


> That avg object size is for the full object with payload.

... which is used to estimate how many cache_dir index entries Squid
will need to create for a cache_dir of a given size.


> The 10 or 14 MB is purely for the metadata necessary to index those
> cached objects. Which is the HTTP message header text plus a bunch of
> Squid code objects.

HTTP headers are not a part of the in-memory cache_dir index. StoreEntry
and LruNode (or equivalent) are pretty much the only structures we place
in the cache_dir index. A "bunch of Squid code objects" are not created
for that index (but get created during a cache hit and may remain in RAM
for some time after that).


> To get the metadata size add the per-object sizes (first number column)
> of HttpReply + MemObject + HttpHeaderEntry + all objects whose name
> starts with HttpHdr* + StoreEntry + all objects whose name starts with
> StoreMeta*.

AFAICT, StoreEntry and LruNode (or equivalent) are the only structures
created for the cache_dir index. All other structures are not relevant
in that scope.


> Though nowdays the 2^27 objects per cache_dir limitation is proving to
> be far more restrictive than the RAM index size. 

Agreed, but YMMV.


> So depending on your
> "Mean Object Size" you may find yourself limited to only using 100 GB or
> less of a TB HDD.

... unless you use multiple cache_dirs per HDD.

Alex.