[squid-dev] The next step towards: StoreID and metalink.

[Alex Rousskov]

On 09/21/2015 01:28 PM, Amos Jeffries wrote:
> On 22/09/2015 6:19 a.m., Alex Rousskov wrote:
>> On 09/19/2015 05:28 PM, Eliezer Croitoru wrote:
>>
>>> There are things which can help an ECAP module to decide how to do
>>> things when handling metalinks related affairs.
>>> For example, a cache object lookup from within an ECAP module.
>>> So basically the logic of the ECAP module is 100% blind to the cache
>>> internals.
>>
>> FWIW, there have been requests to expose host cache state to eCAP
>> adapters. It can be easily done (via Cache-Lookup metadata) for the
>> current request [in pre-cache REQMOD]. Doing so will even be compatible
>> with ICAP services.
>>
>> Unfortunately, but most real use cases I know about actually need a true
>> "cache lookup using an arbitrary request and/or StoreID" functionality
>> and not just a "is the current request likely to be served from cache?"
>> answer. Supporting arbitrary lookups would require expanding eCAP API. I
>> would support such expansion, especially if we have several different
>> specific use cases the back the new API design up.

> What I have been thinking of for Metalinks was a StoreID helper that
> took the requested URI and matched it against an index/DB of it own to
> see if there was a previous ID for it.

Mapping a given request [URL] to Store ID(s) can be supported using
eCAP-supplied annotations without any eCAP API changes. An adapter can
simply return a "use these IDs" annotation with Store ID(s) that Squid
would recognize. That part requires straightforward Squid changes,
especially if only REQMOD support is needed.

My response to Eliezer question above was primarily about allowing an
eCAP adapter to ask the host application (such as Squid) whether an
arbitrary request [URI] is likely to be satisfied from its cache. That
part requires non-trivial eCAP API changes [and it would be good to have
more specific use cases to guide those changes].


> That could be paired with an eCAP RESPMOD adapter that scanned for
> metalink URIs and added records to the StoreID index/DB for them. So
> that future requests for those URIs got the now-cached content.

Sure.


> There are issues around invalidation to work out. Specifically when the
> cached object gets replaced and has different metalinks attached, how to
> find the obsolete ones. Doing a delete operation at speed may be
> problematic.


> If the index/DB could be Squids internal cache index, or a special case
> per-cache indexes like Transients but for the eCAP adapter. Then the
> StoreID helper may not be even necessary. But that might be a lot more work.


If we want tight integration between Squid cache and an adapter,
including things like notifications about purged cached entries, then we
probably want the Store to "adapt" its cache _operations_ (add, update,
delete cache entry) rather than HTTP messages. In other words, in
addition to icap_service and ecap_service, we would have a cache_service
(that can still use eCAP API but send cache operations instead of real
HTTP messages for "adaptation").


HTH,

Alex.