[squid-users] Caching Google Chrome googlechromestandaloneenterprise64.msi

Mon Oct 24 08:59:10 UTC 2016

On Mon, 2016-10-24 at 19:03 +1300, Amos Jeffries wrote:
> On 24/10/2016 6:28 a.m., garryd at comnet.uz wrote:
> > 
> > On 2016-10-23 18:31, Amos Jeffries wrote:
> > > 
> > > On 23/10/2016 2:32 a.m., garryd wrote:
> > > > 
> > > > Since I started use Squid, it's configuration always RFC
> > > > compliant by
> > > > default, _but_ there were always knobs for users to make it
> > > > HTTP
> > > > violent. It was in hands of users to decide how to handle a web
> > > > resource. Now it is not always possible, and the topic is an
> > > > evidence.
> > > > For example, in terms of this topic, users can't violate this
> > > > RFC
> > > > statement [1]:
> > > > 
> > > >    A Vary field value of "*" signals that anything about the
> > > > request
> > > >    might play a role in selecting the response representation,
> > > > possibly
> > > >    including elements outside the message syntax (e.g., the
> > > > client's
> > > >    network address).  A recipient will not be able to determine
> > > > whether
> > > >    this response is appropriate for a later request without
> > > > forwarding
> > > >    the request to the origin server.  A proxy MUST NOT generate
> > > > a Vary
> > > >    field with a "*" value.
> > > > 
> > > > [1] https://tools.ietf.org/html/rfc7231#section-7.1.4
> > > 
> > > 
> > > Please name the option in any version of Squid which allowed
> > > Squid to
> > > cache those "Vary: *" responses.
> > > 
> > > No such option ever existed. For the 20+ years Vary has existed
> > > Squid
> > > has behaved in the same way it does today. For all that time you
> > > did not
> > > notice these responses.
> > 
> > You are absolutely right, but there were not such abuse vector in
> > the
> > past (at least in my practice). There were tools provided by devs
> > to
> > admins to protect against trending abuse cases.
> 
> What trend? There is exactly one mentioned URL that I'm aware of, the
> Chrome browser download URL. I've posted two reasons why Chrome uses
> the
> Vary:* header. Just opinions of mine, but formed after actual
> discussions with the Chrome developers some years back.
> 
> 
> [I very much dislike writing this. But you seem to have been sucked
> in
> and deserve to know the history.]
> 
> All the fuss that is going on AFAICS was started by Yuri. His comment
> history here and in bugzilla, and in private responses range from
> non-compromising "cache everything no matter what - do what I say,
> now!"
> (repeatedy in unrelated bugzilla reports), "f*ck the RFCs and anyone
> following them, just store everything I dont care about what happens"
> (this mornings post), to personal attacks against anyone who mentions
> the previous stance might have problems (all the "Squid developers
> believe/say/..." comments - none of which match what the team we have
> actually said to him or believe).
> 
> There is one other email address which changes its name occasionally
> and
> posts almost exactly the same words as Yuri's. So it looks to me as
> Yuri
> and some sock puppets performing a campaign to spread lies and FUD
> about
> Squid and hurt the people doing work on it.
> 
> Not exactly a good way to get people to do things for free. But it
> seems
> to have worked on getting you and a few others now doing the coding
> part
> for him at no cost, and I have now wasted time responding to you and
> thinking of a solution for it that might get accepted for merge.
> 
> 
> This particular topic is not the first to have such behaviour by
> Yuri.
> There have been other things where someone made a mistake (overlooked
> something) and all hell full of insults broke loose at them. And
> several
> other cases where missing features in Squid did not get instant
> obedience to quite blunt and insulting demands. Followed by weeks of
> insults until the bug was fixed by other people - then suddenly
> polite
> Yuri comes back overnight.
> 
> 
> As a developer, I personally decided not to write the requested code.
> Not in the way demanded. This seems to have upset Yuri who has taken
> to
> insulting me and the rest of the dev team as a whole. I'm not sure if
> he
> is trolling to intentionally cause the above mentioned effects, or
> really in need of medical assistance to deal with work related
> stress.
> 
> [/history]
> 
> 
> > 
> > So, the question arised,
> > what changed in Squid development policy?
> 
> In policy: Nothing I'm aware of in the past 10 years.
> 
> What changed on the Internet? a new bunch of RFCs came out, the
> server
> and clients Squid talks to all got updated to follow those documents
> more closely.
> 
> What changed in Squid? the dev team have been slowly adding the new
> abilities to Squid. One by one, its only ~90% (maybe less) compliant
> withe the MUST conditions, not even close to that on the SHOULDs,
> MAYs,
> and implied processing abilities.
> 
> 
> What do you think should happen to Squid when all the software it
> talks
> to speaks and expects what the RFCs say they should expect from
> recipients/Squid ?
> 
> Consider the extreme this leads towards: You could easily write a
> piece
> of software that took HTTP requests and sent back random data that
> looked vaguely like HTTP responses. But how useless that is when used
> as
> a proxy. So much for ignoring the HTTP specs.
> 
> 
> > 
> > Why there is no configuration
> > option like 'ignore_vary [acl]', so highly demanded by many users
> > in the
> > list?
> 
> AFAIK nobody has ever even proposed adding one until these past few
> weeks.
> 
> A proposals to ignore "Vary: *" has come up every few years, but when
> the proposer was made aware of the server expectations as stated in
> the
> RFC 2616 they went away, no attempt to go any further. So I assume
> that
> means it wasn't really a problem for them, not a serious one.
> 
> In RFC 2616 (or older) compliant web there is no benefit to caching
> these objects. The revalidate clause did not exist so the server can
> be
> expected to always produce a new one.
> So what is gained by caching it? a waste of space other HIT's could
> have
> used better. Negative benefits really, not even zero gain.
> 
> The middle sentence of that part of RFC 7231 has only existed for the
> past few years. Even I had overlooked it until today. Sorry, but I
> have
> been concentrating on getting Cache-Control going properly (no-cache
> and
> side effects are still a hot topic 4 years after it went in). Vary is
> much broken in other serious ways, '*' storage seems a minor issue to
> me.
> 
> Also be aware that Squid is still in the process of moving from
> HTTP/1.0
> behaviour to HTTP/1.1 revalidation. We (Eduard, Alex and myself) have
> not magically made everything work perfectly in one release. This is
> one
> of probably many cases where revalidation is potentially usable -
> just
> that nobody has coded it yet. Or cases which were previously
> forbidden
> and HTTPbis discussions (recent 7-ish years) shown that other vendors
> are widely okay with it so its been allowed now by the 723x RFC
> series.
> 
> 
> As maintainer its my responsibility to point out the issues to anyone
> even proposing a violation gets added. To make sure they are aware of
> the expectations the servers and clients may have of Squid.
> 
> All violation proposals get the same treatment (heck all patches get
> this same treatment too, I'm just a little more lenient on compliance
> increasing patches):
>  - make the proposer aware of the problems they will cause,
>  ** my posts in reply to this and other threads.
> 
>  - mention any alternatives (where available),
>  ** others have mentioned eCAP/ICAP as better suited to such
> violations.
> That is exactly what those interfaces are designed for.
>  ** re-reading the section you quoted above, the middle sentence adds
> a
> possibility I overlooked earlier. You now have a potential direction
> to
> go without even doing any violation. The audit process shows its
> worth
> right there :-).
> 
>  - and let them decide if it is worth the trouble to go forward.
>  ** so far mostly whats happened is a lot of complaints that "Squid
> dev
> team" are not providing things for free, on demand, to service a very
> abusive and extremist person. Or just outright abuse. Seems like the
> individual who started the discussions does not care about getting it
> to
> happen. You care more, and as you say below ...
> 
> 
> > 
> > Personally, I'm no affected by the Vary abuse, but I suppose there
> > will be increasing number of abuse cases in the future.
> 
> Me neither (for seeing the problem in my sysadmin roles), so for both
> of
> us and many others it seems (to me) not to be a major problem. Or at
> least not worth facing the consequences the proposed change risks
> causing.
> 
> We can guess at what the future holds. But until we get there we
> really
> dont know.
> 
> My view of things includes years of app developer queries sent to the
> IETF HTTPbis WG. There is an increasing number of applications that
> *need* proxies to follow the spec requirements. I am not seeing them
> much in the wild yet, but those applications are definitely around
> and
> breaking them could lead people to a lot of trouble.
> 
> 
> FWIW:  The current spec for HTTP/1.1 is extremely tollerant. There
> are a
> very few places where it comes to an absolute inviolate rule. Those
> are
> cases which have very clear use-case mentioned in the spec and
> problems
> being avoided mentioned as well.
> 
> For example the quoted section above about Vary. I overlooked that
> middle sentance that has been added since 2616. That creates the
> leeway
> we need to cache the object within compliance. We are thusly allowed
> to
> treat "Vary:*" as another of the must-revalidate caching cases, just
> like Cc:private or Authenticated responses.
> (From the use-cases that have been mentioned in IETF WG about this
> header, and cases people have been advised to use it I very much
> doubt
> any server will respond with 304 to such revalidation. But we might
> get
> lucky, or future servers might do so.)
> 
> NP: all previous discussions and proposals of Vary:* handling came up
> when 2616 was the spec to follow, and Squid did not do 1.1 much
> anyway -
> so using revalidation was not even on the horizon so to speak.
> 
> 
> Anyhow, back to me rant ;-P
> 
> The current specs were written by representatives standing for *all*
> the
> major browsers, *all* the major web servers, *all* the major
> language/library frameworks, almost all the major command-line tools,
> and many application developers as well.
> 
> So it is not written in isolation by some few "ivory tower types"
> Yuri
> would have you believe (yes some specs are - HTTP is not, or at least
> those types are outnumbered for HTTP). The RFC 723x set is explicitly
> a
> collection of details about how the current HTTP applications
> actually
> found around the world *do* talk to each other today and how best to
> exchange messages so the processing operations are understood at both
> ends of the network connection.
> 
> The HTTP RFC's are effectively a description of current best
> practice.
> Violating is only useful if you are fixing some broken application
> your
> particular proxy has to deal with. All other cases are just causing
> inefficiency and varying amounts of nasty depending on the violation.
> 
> This latter point is part of why we require a clear use-case before
> accepting violation patches - the need for it should be well thought
> out. People who choose to violate that type of spec without a good
> reason are just being stupid.
> 
> 
> The three persons (well, 2 use aliased email addresses - I suspect
> are
> actually one or two persons from the same region of the world)
> providing
> the most noise about Vary keep demanding it be provided for free.
> 
> 
> > 
> > One of your
> > answers confirmed my assumption regarding the question:
> > 
> > > 
> > >  - there is a very high risk of copy-and-paste sysadmin spreading
> > > the
> > > problems without realising what they are doing. Particularly
> > > since those
> > > proposing it are so vocal about how great it *seems* for them.
> > 
> 
> When someone else does the coding I act as both auditor and
> maintainer.
> The policies checked during audit do not go into good/bad judgements
> -
> just: code style, quality (no identifiable bugs) and a clear
> statement
> of the need behind the addition (for the merge repo comment to guide
> future maintenance of that code).
> 
> As maintainer I merge patches myself only if I'm happy to take on
> code
> maintenance for them, or supplier explicitly takes it on themselves
> (ie
> helper authors). Sometimes I get patches up to scratch in audit then
> let
> others merge, or advise they just be used as custom patches. In the
> latter case we at least know that patch is not full of obvious bug
> behaviours.
> 
> But note so far *nobody* has submitted patches to audit about this.
> It
> has all just been a rather heated discussion in various unrelated bug
> reports and some threads in this users list.
> 
> Amos
> 
> _______________________________________________
> squid-users mailing list
> squid-users at lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users

Hi Amos,

Thank you very much for so detailed explanation. I've made conclusions
from presented information. I deeply regret, that the topic took so
many time from you. I believe, information presented here will be
helpful for the community.

Nevertheless, the topic surfaced new details regarding the Vary and I
tried conditional requests on same URL (Google Chrome) from different
machines/IPs. Here results:

$ curl --head --header "If-Modified-Since: Thu, 22 Oct 2016 08:29:09
GMT" https://dl.google.com/linux/direct/google-chrome-stable_current_am
d64.deb
HTTP/1.1 304 Not Modified
Etag: "101395"
Server: downloads
Vary: *
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 1; mode=block
Date: Mon, 24 Oct 2016 08:53:44 GMT
Alt-Svc: quic=":443"; ma=2592000; v="36,35,34"

----

$ curl --head --header 'If-None-Match: "101395"' https://dl.google.com/
linux/direct/google-chrome-stable_current_amd64.deb 
HTTP/1.1 304 Not Modified
Etag: "101395"
Last-Modified: Thu, 20 Oct 2016 08:29:09 GMT
Server: downloads
Vary: *
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 1; mode=block
Date: Mon, 24 Oct 2016 08:54:18 GMT
Alt-Svc: quic=":443"; ma=2592000; v="36,35,34"

Garri