[squid-users] refresh_pattern and same objects

[Amos Jeffries]

On 21/08/2015 8:36 p.m., Stakres wrote:
> Hi Amos,
> Is that possible to have a dedicated option with the Squid to keep objects
> in the cache if they're regulary used even if the time is expired ?
> Cleaning small expired files (<16kb) is not a problem but we must keep big
> files into the cache if often used.
> There are many "small" ISPs with 2, 4 or 8Mbps bandwidth and big files are a
> problem if they've to download a fresh copy every month (if max-day 1
> month).
> Doing a local copy is not a right way here because too many possible big
> objects and issues to manage them, we should have Squid able to do that.
> 
> I'm thinking something like that:
> save_big_file on/off
> save_big_file_min_size 128 MB
> save_big_file_max_time 1 years
> 
> Would it be something you could implement with Squid ?
> I'm sure it would work so fine 


Stale objects *do* stay in cache. I think that behaviour is pretty much
what happens with LRU in practice. To be removed it has to be *the*
least recently used object in cache when that watermark algorithm we are
fixing in the other thread runs.

Its just that if you are talking an object used literally once a month
and the smaller objects have a high churn rate resulting in a 1-week LRU
age. Then it would be stored for only that week before it met the
criteria for removal. If its touched at any time in that week its
LRU-age becomes 0 again and starts re-counting back towards a week while
the churn goes on with other objects.

The current policy/algorithm options are here:
 <http://www.squid-cache.org/Doc/config/cache_replacement_policy/>

new ones are possible of course with the usual wait/sponsor/DIY options
and disclaimers about feature requests.


Note that this is all per-cache_dir. What I recommend is a cache_dir
with some TB of HDD behind it for approx. GB sized objects.

That way they are *not* competeing with many small objects at high churn
rate. The cache_dir LRU age could be weeks/months/years if all the big
objects are reasonably low popularity.

Also, you can size the cache_dir capacities to avoid wastage in the
smaller cache_dir from the 2^25 object limitations vs the size of
objects stored there.

Amos