[squid-dev] [RFC] client header mangling

[Amos Jeffries]

On 10/06/2016 9:49 a.m., Eliezer Croitoru wrote:
> I am trying to understand so bear with me couple seconds.
> I have seen that there are pages\servers which doesn't state about the User-Agent in the Vary response while still taking it into account.
> 
> The caching side of the picture is storing an object which will never be served.
> The HIT ratio is a whole other story  of the picture.
> 
> Since I am not inside the code but I do try to understand, currently what happens?

Currently what happens is Squid uses the real client headers or
ICAP/eCAP adapted headers when looking up the cached objects variant.

The admin might be fixing the actual response using
request_header_access/replace to craft what the server sees. But that
does not help Squid with the variants. Altering the headers prior to
cache lookup is needed for that, which today means ICAP/eCAP are needed.

I'm proposing making he header alteratiosn affect input as well as output.


> How many lookups are done for\per a request? 

1 or 2 if it is a Vary response.

> Do we run an object lookup after the response headers was received from the server?

No.

> Can we predict a Vary object based on the request only?(I assume that it will be an estimated and not absolute certainty if at all)

No.

> 
> Also let say we have a 1k page ahead, would we want it to be fetched
> from disk\ram store rather then from the origin server after we told
> it we want the object?

This is not relevant. Size of the objects and where each would be placed
is not relevant to the problem.

The issue is that there are a huge, possibly infinite number of such
objects wasting filenum spaces/slots in the cache. There are only 2^25
object slots per cache location, so these huge sets of variants can
really cramp the storage even if they are only 1 bytes in size.


> 
> I am almost sure that lowering the disk and ram stored objects should
> be a goal by itself if we cannot "dig" them up from ram or disk later
> for any use.

Yes, but not relevant to the current decision.

The question here is whether we should allow pre-cache header mangling
by admin as a way to reduce number of objects count for Vary responses.

The alternative is requiring them to use ICAP or eCAP to do it.
Possibly asking someone to write an eCAP module.


> 
> A request_header_replace can work only for "generic" ones such as
> without a language preference such as "br" added to some requests by
> browser add-ons.

No. It can and will work for any header. Just requires the admin to know
what ones to modify and when. Which is still somewhat hard for unusual
headers.

Which is part of why I RFC'd it rather than going ahead and proposing a
patch.

> 
> Now a step further, I can write a tiny ICAP service that will
> "handle" common Vary headers from FireFox and other browsers to test
> how it affects caches in general.

I'd rather eCAP for this than ICAP. But if you think you can do it and
want to try then we can work on the details of what the code needs to do.

Amos