[squid-users] Squid 3.5.16 and vary loop objects (bug ?)

[Amos Jeffries]

On 8/04/2016 2:39 a.m., joe wrote:
> some of my post you miss reading or
> wen i ask for water  and i get water  same size same glas  but i was offered
> somthing else i was refuse that

In terms of the protocol, you cant refuse exactly. It is what was
responded with. The server/provider 'owns' the resource and is the
control authority about what should be provided to any request.


> and my friend ask for the same he should get same glas of water same size
> so what i was trying to enplane is
>  
> 
> i ask  for gzip  the the md5 vary calculate with string

Asking for gzip would be sending the header exact and full value of:
  Accept-Ecoding: gzip\r\n

... not the below multiple list of values:

> accept-encoding="gzip,%20deflate,%20sdch" 
> so the gzip dose not have %20  on front of it so we can use only that and

%20 is a space character when URL-encoded for adding to the URL hash key.

The client sending a request with that header value(s) is asking for
gzip OR deflate OR sdch encoding (in that order of preference). The
server decides whether one of those can be produced. And identity
encoding (unencoded) is guaranteed to always be acceptible.


> filter out those ,%20deflate,%20sdch
> it will be string accept-encoding="gzip" use  that in md5 calc only
> if other browser has in vary string accept-encoding="gzip,%20deflate" 
> without examle  %20sdch
> md5 calc use string accept-encoding="gzip,%20deflate"
> so we also filter out -- > ,%20deflate
> and keep only to use md5 with  string accept-encoding="gzip"
> 
> since  gzip dose not have %20 on front of it

What if the sdch was the type that server actually wanted to produce?
If you are using Google Docs sdch objects actually *are* more often than
not what gets sent. Its a type of patch/diff format for collaborative
tools updating an object in bits and pieces.

> 
> that will make a beter hit without any problem on all browser
> 

*Maybe*. You are free to do so if you like. We cannot do it in the
general case though for Squid has to work according to proper HTTP
requirements in networks other than yours (and there are several others
also wanting this same thing have configured their proxies to do it).


> as i says and i test the link i provide before   
> on firefox   first time after the cache ar empty  i get HIT  alwes just by
> using firefox
> until now perfect
> 
> but  wen i use chrome since it send and resive in very  string
> accept-encoding="gzip,%20deflate,%20sdch"
> the cached file ar deleted and resolt  = MISS  then it re creat new file
> with HIT alwes just by using chrome
> 
> if i return to firefox its  MISS  then file got purged from cache then its
> HIT again

Two things:

1) MISS does not mean any content was thrown away. It just means what
was already in the cache was not able to be used.
  - it can happen because of a max-age=0 or no-cache from the client.
Chrome and other Google products seem to like sending those. Very
unfriendly to caches.
  - it can happen if Vary exists and the particular object this request
needs is not one of the current set of variants in the cache.

2) You can have the appearance of stuff being "thrown away" if the Vary
marker object used to store the pattern actually was thrown away for
some reason. Without it there is no way to reach the variant objects
which are still in cache.
 - A MISS is needed in order to re-fetch a response with Vary pattern to
create the marker, and bingo all the variants that pattern describes
which were still in the cache can be fetched as HITs again.
 - BUT, if the Vary pattern changed they stay 'lost'.

This #2 is a common problem seen with Apache servers which emit
different Vary patterns depending on which modules were run on the request.


> 
> if other browser has  vary  with string
> accept-encoding="%20gzip,deflate,%20sdch"
> watch this      deflate   has no --->> %20 in front of it so md5 calc should
> use only deflate   not encluding the one with  %20   string
> this how i understand it   and i monitor those prossess i prove that is
> waste

Whitespace normalization (or lack of it rather) does cause wastage. The
only reason that is not done is nobody did it yet. Patches fixing that
are welcome.

> 
> so anoter example cause i dont know if im trying to explain it in right
> manner 
>  another browser get vary  string accept-encoding="%20gzip,%20deflate,sdch"
> as you see  sdch  its the one used  so we must use that filter out the rest 
> 
> variety with string confuse the correct match match lookup it it purge the
> same file just becaus somthing in vary has extra string and its not used
> 

A)
I know what you are trying to say. I'm trying to say why its not
necessarily such a great idea (for Squid). There are subtle things in
the protocol design that prevent it and/or would be broken by it.


B)
You are also focused on The Browser(s). Some of the smart guys in IETF
HTTP Working Group did some research a few months ago. Its a funny story
really:

* the Chrome guys did their research and found that almost all servers
responding to Chrome could be forced to send gzip. With all modern
browsers supporting gzip there are great bandwidth reduction benefits
waiting to be had. Therefore argued that gzip should be the one and only
output from server by default and middleware like Squid should
auto-encode to produce gzip only.

* the server guys did their own research and found that Browser traffic
was less than ~20% of HTTP total traffic (and declining as IoT
increases). Optimizing all networks for the Browsers causes big problems
for all that major portion of non-Browser traffic which consists of
tools using other encoding types (there are at least 8 now in active use
around the world, some of them much better than gzip).

 --> Oops.

C)
Vary as a header is kind of set in stone. We can adjust a few things.
But its really depended on by too many pieces of old software.

There is another header called Key being designed by IETF right now. It
does a better job to describe the variant patterns in a more detailed
way. Such as to say that any Accept-Encoding header containing 'gzip'
could get this reply, or any Accept-Encoding header containing 'deflate'
gets another.

 That new header is just very experimental right now and I haven't had
much time to make Squid use it yet.

Amos