[squid-dev] The next step towards: StoreID and metalink.

[Alex Rousskov]

On 09/21/2015 03:40 PM, Eliezer Croitoru wrote:
> The first step now I am trying to grasp first what could be done in the
> current state of squid and ECAP without any changes.

> Currently the ECAP module can calculate hashes on the fly and then in
> the end of the transaction write the result to either a log file or a DB.
...
> There is another way to de-duplicate content for metalinks using cache
> objects planting\redirection.
...
> Which of the ideas is a more realistic one compared to changing squid
> and\or ECAP?

The checksum computation and 302 redirections (vie REQMOD request
satisfaction) that you have described above are as realistic as changing
Squid/eCAP. I do not think realism is the deciding factor here.

Alex.



> On 21/09/2015 23:10, Alex Rousskov wrote:
>> On 09/21/2015 01:28 PM, Amos Jeffries wrote:
>>> On 22/09/2015 6:19 a.m., Alex Rousskov wrote:
>>>> On 09/19/2015 05:28 PM, Eliezer Croitoru wrote:
>>>>
>>>>> There are things which can help an ECAP module to decide how to do
>>>>> things when handling metalinks related affairs.
>>>>> For example, a cache object lookup from within an ECAP module.
>>>>> So basically the logic of the ECAP module is 100% blind to the cache
>>>>> internals.
>>>>
>>>> FWIW, there have been requests to expose host cache state to eCAP
>>>> adapters. It can be easily done (via Cache-Lookup metadata) for the
>>>> current request [in pre-cache REQMOD]. Doing so will even be compatible
>>>> with ICAP services.
>>>>
>>>> Unfortunately, but most real use cases I know about actually need a
>>>> true
>>>> "cache lookup using an arbitrary request and/or StoreID" functionality
>>>> and not just a "is the current request likely to be served from cache?"
>>>> answer. Supporting arbitrary lookups would require expanding eCAP
>>>> API. I
>>>> would support such expansion, especially if we have several different
>>>> specific use cases the back the new API design up.
>>
>>> What I have been thinking of for Metalinks was a StoreID helper that
>>> took the requested URI and matched it against an index/DB of it own to
>>> see if there was a previous ID for it.
>>
>> Mapping a given request [URL] to Store ID(s) can be supported using
>> eCAP-supplied annotations without any eCAP API changes. An adapter can
>> simply return a "use these IDs" annotation with Store ID(s) that Squid
>> would recognize. That part requires straightforward Squid changes,
>> especially if only REQMOD support is needed.
>>
>> My response to Eliezer question above was primarily about allowing an
>> eCAP adapter to ask the host application (such as Squid) whether an
>> arbitrary request [URI] is likely to be satisfied from its cache. That
>> part requires non-trivial eCAP API changes [and it would be good to have
>> more specific use cases to guide those changes].
>>
>>
>>> That could be paired with an eCAP RESPMOD adapter that scanned for
>>> metalink URIs and added records to the StoreID index/DB for them. So
>>> that future requests for those URIs got the now-cached content.
>>
>> Sure.
>>
>>
>>> There are issues around invalidation to work out. Specifically when the
>>> cached object gets replaced and has different metalinks attached, how to
>>> find the obsolete ones. Doing a delete operation at speed may be
>>> problematic.
>>
>>
>>> If the index/DB could be Squids internal cache index, or a special case
>>> per-cache indexes like Transients but for the eCAP adapter. Then the
>>> StoreID helper may not be even necessary. But that might be a lot
>>> more work.
>>
>>
>> If we want tight integration between Squid cache and an adapter,
>> including things like notifications about purged cached entries, then we
>> probably want the Store to "adapt" its cache _operations_ (add, update,
>> delete cache entry) rather than HTTP messages. In other words, in
>> addition to icap_service and ecap_service, we would have a cache_service
>> (that can still use eCAP API but send cache operations instead of real
>> HTTP messages for "adaptation").
>>
>>
>> HTH,
>>
>> Alex.
>>
>> _______________________________________________
>> squid-dev mailing list
>> squid-dev at lists.squid-cache.org
>> http://lists.squid-cache.org/listinfo/squid-dev
>>
> 
> _______________________________________________
> squid-dev mailing list
> squid-dev at lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-dev