[squid-users] Cache reference age for heap LRU/LFUDA and rock/aufs

[Alex Rousskov]

On 02/12/2018 04:25 PM, Ivan Larionov wrote:
> On Fri, Feb 9, 2018 at 7:50 PM, Alex Rousskov wrote:
> 
>     please note that rock
>     cache_dirs do not support/have/use a configurable replacement policy:
>     Each incoming object is assigned a slot based on its key hash. With
>     modern rock code, it is possible to remove that limitation IIRC, but
>     nobody have done that.

> So what does rock do when storage is full then?

Becoming full is not a special condition for rock cache_dirs. In other
words, a rock cache_dir does not do anything special when it becomes (or
is) full.

Roughly speaking, when a rock cache_dir needs to store a new entry, it
computes its starting location by hashing the entry URL and starts
storing the entry at that location, overwriting any cached entry that
used the same starting location. Same for the shared memory cache. See
Ipc::StoreMap::openForWriting().

Rock is meant/optimized for large caches. The probability of overwriting
a valuable entry in a large cache is small -- most cached entries are
not valuable because they will never be requested again.

Needless to say, there are cases where such crude approach is harmful
(e.g., two popular entries competing for the same unlucky location and
overwriting each other). As I said, modern SMP caching code may have
enough bells and whistles to support such cases better, but the required
changes are still far from trivial, and nobody has contributed or
sponsored that improvement (yet?).


>     If it is important to get this right, then I would not trust replacement
>     policy metadata with this: The corresponding code interfaces look
>     unreliable to me, and access counts/timestamps for a ufs-based cache_dir
>     are not updated across Squid restarts when the swap log is lost (at
>     least).

> It's actually fine, we never restart squid and if it restarted by any
> unexpected reason (host reboot, crash or w/e) we just replace the host.

FWIW, please note that my "unreliable" remark applies to Squids that are
never restarted. YMMV.


>     If there is no "age" ACL to use with the send_hit directive, then you
>     may need to add one.
> 
> I was thinking about similar solution but this is exactly why I wasn't
> able to use it – there seems to be no acl suitable for such task.

IMO, treating Squid feature set as a constant outside of your control
means ignoring 50% of Squid advantages (while suffering from 100% of its
drawbacks). One can add the missing pieces:

https://wiki.squid-cache.org/SquidFaq/AboutSquid#How_to_add_a_new_Squid_feature.2C_enhance.2C_of_fix_something.3F


HTH,

Alex.