[squid-users] squid won't return cached even with refresh_pattern extra options override-lastmod override-expire ignore-reload ignore-no-store ignore-private store-stale
Eliezer Croitoru
ngtech1ltd at gmail.com
Sat Mar 27 13:25:59 UTC 2021
Hey Mirek,
This is not the first time this issue rises.
There are risks in implementing any solution for this *issue*.
I have implemented YouTube caching in the past using couple twisted techniques while leaving squid un-touched.
The desire to caching sometimes can overcome couple very big risks to the integrity if the date/content.
It is possible to use an ICAP service with a 206 response instead of 204 or 200 however I believe
that you wouldn’t need to cache any POST requests so a simple ICAP service would be sufficient.
I believe that It is preferred to leave squid sources un touched for such a purpose.
An example for such a twist is at:
* https://github.com/elico/squid-helpers/tree/master/squid_helpers/youtubetwist
* https://wiki.squid-cache.org/ConfigExamples/DynamicContent/Coordinator?highlight=%28cache_peer%29#Implementing_ICAP_solution
* https://ieeexplore.ieee.org/abstract/document/9072556
I wrote a public example of an ICAP server that was used to prove vulnerabilities in HTTP which is now used in the prove of HTTPS vulnerabilities.
Take a peek at:
* https://github.com/elico/bgu-icap-example
It's written in GoLang and works under pretty heavy loads.
Let me know if you need more help,
Eliezer
----
Eliezer Croitoru
Tech Support
Mobile: +972-5-28704261
Email: ngtech1ltd at gmail.com
Zoom: Coming soon
-----Original Message-----
From: squid-users <squid-users-bounces at lists.squid-cache.org> On Behalf Of Alex Rousskov
Sent: Friday, March 26, 2021 10:36 PM
To: squid-users at lists.squid-cache.org
Subject: Re: [squid-users] squid won't return cached even with refresh_pattern extra options override-lastmod override-expire ignore-reload ignore-no-store ignore-private store-stale
On 3/24/21 3:34 PM, Miroslaw Malinowski wrote:
> I thought about upper service but as is not required at the moment,
> introducing extra hop just to remove the header looks a bit like a
> hammer approach. I'll look into how easily I can amend the code as the
> other option is to introduce a proxy like a feature to the application,
> so either way, it is a code change. The only problem here is that it's
> an OPNSense squid service so I have to compile from source on BSD and
> then keep adding in manually each time they do the update.
At the risk of stating the obvious: If your feature is officially
accepted into Squid sources, then you would not have to keep adding it
manually (once the changes reach your Squid packaging source).
Alex.
> On Wed, Mar 24, 2021 at 7:11 PM Alex Rousskov wrote:
>
> On 3/24/21 2:49 PM, Miroslaw Malinowski wrote:
>
> > looking at the code and reading carefully your response, you're saying
> > there is no way you can do it with squid.
>
> With Squid, your options include:
>
> 1. Squid source code changes. Should not be too difficult and, IMO, a
> high-quality implementation would deserve official acceptance because it
> is a generally useful feature in line with existing control knobs.
> https://wiki.squid-cache.org/SquidFaq/AboutSquid#How_to_add_a_new_Squid_feature.2C_enhance.2C_of_fix_something.3F
>
> 2. An adaptation service that removes Cache-Control:no-cache from the
> response before Squid processes it:
> https://wiki.squid-cache.org/SquidFaq/ContentAdaptation
>
>
> HTH,
>
> Alex.
>
> > On Wed, Mar 24, 2021 at 6:28 PM Miroslaw Malinowski wrote:
> >
> > Hi,
> >
> > You've right yes it's revalidating as API server I'm
> requesting data
> > is setting Cache-Control: no-cache. My question is how I can force
> > squid to cache and not validate as I know it's safe to do so. As
> > I've explained earlier we are making the same request and
> receiving
> > the same response from 100+ server so as to reduce number of
> > requests to the external server we would like squid to cache the
> > response and issue a cached version.
> >
> > 2021/03/24 18:00:54.867 kid1| 22,3| refresh.cc(351) refreshCheck:
> > YES: Must revalidate stale object (origin set no-cache or private)
> >
> > Mirek
> >
> > On Wed, Mar 24, 2021 at 6:15 PM Alex Rousskov
> > <rousskov at measurement-factory.com
> <mailto:rousskov at measurement-factory.com>
> > <mailto:rousskov at measurement-factory.com
> <mailto:rousskov at measurement-factory.com>>> wrote:
> >
> > On 3/24/21 12:48 PM, Miroslaw Malinowski wrote:
> >
> > > Probably, me missing on something silly or it can't be done
> > but I don't
> > > know why but squid won't return the cached version even
> when I
> > turn all
> > > override options ON in refresh_pattern.
> >
> > AFAICT, no configuration options that can disable
> revalidation of
> > Cache-Control:no-cache responses. refresh_pattern does not
> have an
> > (equivalent of) "ignore-no-cache-in-responses" option.
> >
> > IIRC, older Squids were violating an HTTP MUST by
> forgetting to
> > revalidate Cache-Control:no-cache responses, but that was
> fixed
> > in [1].
> > Your Squid version has that fix.
> >
> > [1]
> >
> https://github.com/squid-cache/squid/commit/fa83b766a208b27abed8da4c9073cf8784cf10fa
> <https://github.com/squid-cache/squid/commit/fa83b766a208b27abed8da4c9073cf8784cf10fa>
> >
> <https://github.com/squid-cache/squid/commit/fa83b766a208b27abed8da4c9073cf8784cf10fa
> <https://github.com/squid-cache/squid/commit/fa83b766a208b27abed8da4c9073cf8784cf10fa>>
> >
> >
> > > With debug, I can see the rule is matched and the cache is
> > fresh but
> > > still in access.log is TCP_REFRESH_MODIFIED
> >
> > > 2021-03-24T15:04:34 squid .710 kid1| 11,3| http.cc(982)
> > > haveParsedReplyHeaders: decided: cache positively and share
> > because
> >
> > FYI: You are looking at cache.log lines logged _after_
> Squid has
> > already
> > decided to refresh the cached version. If you want to analyze
> > why Squid
> > decided to refresh the cached version, you should look
> _before_
> > Squid
> > logged the request to the server (and before any FwdState.cc
> > lines). I
> > have not checked the details, but I bet that your Squid
> revalidates
> > because of Cache-Control:no-cache in the response. Look for
> > "YES: Must
> > revalidate stale object".
> >
> >
> > HTH,
> >
> > Alex.
> >
> > > squid conf:
> > > refresh_pattern -i <URL> 4320 80% 129600 override-lastmod
> > > override-expire ignore-reload ignore-no-store ignore-private
> > store-stale
> > >
> > > curl headers:
> > > curl --insecure --verbose --request GET --url 'URL'
> >/dev/null
> > > * TCP_NODELAY set
> > > * ALPN, offering h2
> > > * ALPN, offering http/1.1
> > > * successfully set certificate verify locations:
> > > * CAfile: /etc/ssl/certs/ca-certificates.crt
> > > CApath: /etc/ssl/certs
> > > } [5 bytes data]
> > > * TLSv1.3 (OUT), TLS handshake, Client hello (1):
> > > } [512 bytes data]
> > > * TLSv1.3 (IN), TLS handshake, Server hello (2):
> > > { [122 bytes data]
> > > * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
> > > { [6 bytes data]
> > > * TLSv1.3 (IN), TLS handshake, Certificate (11):
> > > { [1956 bytes data]
> > > * TLSv1.3 (IN), TLS handshake, CERT verify (15):
> > > { [78 bytes data]
> > > * TLSv1.3 (IN), TLS handshake, Finished (20):
> > > { [52 bytes data]
> > > * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
> > > } [1 bytes data]
> > > * TLSv1.3 (OUT), TLS handshake, Finished (20):
> > > } [52 bytes data]
> > > * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
> > >
> > >> GET URL HTTP/1.1
> > >> Host: URL
> > >> User-Agent: curl/7.68.0
> > >> Accept: */*
> > >>
> > > { [5 bytes data]
> > > * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
> > > { [217 bytes data]
> > > * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
> > > { [217 bytes data]
> > > * old SSL session ID is stale, removing
> > > { [5 bytes data]
> > > * Mark bundle as not supporting multiuse
> > > < HTTP/1.1 200 OK
> > > < Cache-Control: no-cache
> > > < Content-Type: application/json
> > > < X-Cloud-Trace-Context: d3c27833b8b4312ce31a2dbae7e12fd0
> > > < Date: Wed, 24 Mar 2021 15:04:34 GMT
> > > < Server: Google Frontend
> > > < Content-Length: 7950
> > > < X-Cache: MISS from server
> > > < X-Cache-Lookup: HIT from server
> > > < Via: 1.1 server (squid/4.14)
> > > < Connection: keep-alive
> > >
> > > access log:
> > > 243 172.16.230.249 TCP_REFRESH_MODIFIED/200 8328 GET URL -
> > > ORIGINAL_DST/IP application/json
> > >
> > > cache log:
> > > 2021-03-24T15:04:34 squid .710 kid1| 11,3| http.cc(982)
> > > haveParsedReplyHeaders: decided: cache positively and share
> > because
> > > refresh check returned cacheable; HTTP status 200
> > e:=p2V/0x34868914670*3
> > > 2021-03-24T15:04:34 squid .710 kid1| 22,3|
> refresh.cc(470)
> > refreshCheck:
> > > returning FRESH_MIN_RULE
> > > 2021-03-24T15:04:34 squid .710 kid1| 22,3|
> refresh.cc(455)
> > refreshCheck:
> > > Object isn't stale..
> > > 2021-03-24T15:04:34 squid .710 kid1| 22,3|
> refresh.cc(327)
> > refreshCheck:
> > > Staleness = -1
> > > 2021-03-24T15:04:34 squid .710 kid1| 22,3|
> refresh.cc(199)
> > > refreshStaleness: FRESH: age (60 sec) is less than
> configured
> > minimum
> > > (259200 sec)
> > > 2021-03-24T15:04:34 squid .710 kid1| 22,3|
> refresh.cc(166)
> > > refreshStaleness: No explicit expiry given, using
> heuristics to
> > > determine freshness
> > > 2021-03-24T15:04:34 squid .710 kid1| 22,3|
> refresh.cc(307)
> > refreshCheck:
> > > entry->timestamp: Wed, 24 Mar 2021 15:04:34 GMT
> > > 2021-03-24T15:04:34 squid .710 kid1| 22,3|
> refresh.cc(305)
> > refreshCheck:
> > > check_time: Wed, 24 Mar 2021 15:05:34 GMT
> > > 2021-03-24T15:04:34 squid .710 kid1| 22,3|
> refresh.cc(303)
> > refreshCheck:
> > > age: 60
> > > 2021-03-24T15:04:34 squid .710 kid1| 22,3|
> refresh.cc(301)
> > refreshCheck:
> > > Matched 'URL 259200 80%% 7776000'
> > > 2021-03-24T15:04:34 squid .710 kid1| 22,3|
> refresh.cc(279)
> > refreshCheck:
> > > checking freshness of URI: https://URL <https://URL>
> <https://URL <https://URL>>
> > <https://URL <https://URL> <https://URL <https://URL>>>
> > >
> > >
> > > _______________________________________________
> > > squid-users mailing list
> > > squid-users at lists.squid-cache.org
> <mailto:squid-users at lists.squid-cache.org>
> > <mailto:squid-users at lists.squid-cache.org
> <mailto:squid-users at lists.squid-cache.org>>
> > > http://lists.squid-cache.org/listinfo/squid-users
> <http://lists.squid-cache.org/listinfo/squid-users>
> > <http://lists.squid-cache.org/listinfo/squid-users
> <http://lists.squid-cache.org/listinfo/squid-users>>
> > >
> >
>
_______________________________________________
squid-users mailing list
squid-users at lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users
More information about the squid-users
mailing list