<div dir="ltr">Hi, <div>Thanks for the info Alex. Patching code and building is a little beyond me tbh, especially as I would need this as a debian package to deploy to many machines. With that in mind I am trying the config workaround approach. Below is the config snippet I have added. I made the assumption that for the <i>refresh_pattern, max-stale=NN </i>config, the NN is in minutes as per the rest of that config directive.</div><div>I am testing this right now</div><div><br></div><div><span id="gmail-docs-internal-guid-143fc182-7fff-7778-41b8-604378795733"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="background-color:transparent;color:rgb(0,0,0);font-family:Consolas,sans-serif;font-size:8pt"># this should allow stale objects up to 1 year if allowed by Cache-Control </span>repsonse<span style="background-color:transparent;color:rgb(0,0,0);font-family:Consolas,sans-serif;font-size:8pt"> headers ...</span><br></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:8pt;font-family:Consolas,sans-serif;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;vertical-align:baseline"># ... setting both options just in case</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:8pt;font-family:Consolas,sans-serif;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;vertical-align:baseline">max_stale 525600 minutes</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:8pt;font-family:Consolas,sans-serif;color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;vertical-align:baseline">refresh_pattern . 0 20% 4320 max-stale=525600</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><br></p><p style="line-height:1.38;margin-top:0pt;margin-bottom:0pt">Thanks again for your help</p><p style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><br></p><p style="line-height:1.38;margin-top:0pt;margin-bottom:0pt">Robin</p><p style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><br></p><p style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><br></p></span></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, 8 Feb 2024 at 17:42, Alex Rousskov <<a href="mailto:rousskov@measurement-factory.com">rousskov@measurement-factory.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi Robin,<br>
<br>
AFAICT from the logs you have privately shared and your squid.conf <br>
that you have posted earlier, your Squid overwrites <br>
stale-if-error=31536000 in the response with "refresh_pattern <br>
max-stale=0" default. That 0 value is wrong. The correct value should be <br>
taken from max_stale directive that defaults to 1 week, not zero:<br>
<br>
refresh_pattern<br>
...<br>
max-stale=NN provide a maximum staleness factor. Squid won't<br>
serve objects more stale than this even if it failed to<br>
validate the object. Default: use the max_stale global limit.<br>
<br>
This wrong default is a Squid bug AFAICT. I posted an _untested_ fix as <br>
Squid PR 1664: <a href="https://github.com/squid-cache/squid/pull/1664" rel="noreferrer" target="_blank">https://github.com/squid-cache/squid/pull/1664</a><br>
<br>
If possible, please test the corresponding patch:<br>
<a href="https://github.com/squid-cache/squid/commit/571973589b5a46d458311f8b60dcb83032fd5cec.patch" rel="noreferrer" target="_blank">https://github.com/squid-cache/squid/commit/571973589b5a46d458311f8b60dcb83032fd5cec.patch</a><br>
<br>
AFAICT, you can also work around that bug by configuring an explicit <br>
refresh_pattern rule with an explicit max-stale option (see <br>
squid.conf.documented for examples). I have not tested that theory either.<br>
<br>
<br>
HTH,<br>
<br>
Alex.<br>
<br>
<br>
On 2024-02-07 13:45, Robin Carlisle wrote:<br>
> Hi,<br>
> <br>
> I have just started my enhanced logging journey and have a small snippet <br>
> below that might illuminate the issue ...<br>
> <br>
> /2024/02/07 17:06:39.212 kid1| 88,3| client_side_reply.cc(507) <br>
> handleIMSReply: origin replied with error 502, forwarding to client due <br>
> to fail_on_validation_err/<br>
> <br>
> A few lines below in the log it looks like squid sent :-<br>
> <br>
> /2024/02/07 17:06:39.212 kid1| 11,2| Stream.cc(280) sendStartOfMessage: <br>
> HTTP Client REPLY:<br>
> ---------<br>
> HTTP/1.1 502 Bad Gateway<br>
> Server: squid/5.7<br>
> Mime-Version: 1.0<br>
> Date: Wed, 07 Feb 2024 17:06:39 GMT<br>
> Content-Type: text/html;charset=utf-8<br>
> Content-Length: 3853<br>
> X-Squid-Error: ERR_READ_ERROR 0<br>
> Vary: Accept-Language<br>
> Content-Language: en<br>
> X-Cache: MISS from labs-maul-st-15<br>
> X-Cache-Lookup: HIT from labs-maul-st-15:3129<br>
> Via: 1.1 labs-maul-st-15 (squid/5.7)<br>
> Connection: close/<br>
> <br>
> <br>
> The rest of the logs are quite large and contain URLs I cannot put <br>
> here. The logs were generated with debug_options to ALL,3.<br>
> <br>
> Any ideas? Or should I generate more detailed logs and send them <br>
> privately?<br>
> <br>
> Thanks again,<br>
> <br>
> Robin<br>
> <br>
> <br>
> <br>
> <br>
> On Fri, 2 Feb 2024 at 11:20, Robin Carlisle <br>
> <<a href="mailto:robin.carlisle@framestore.com" target="_blank">robin.carlisle@framestore.com</a> <mailto:<a href="mailto:robin.carlisle@framestore.com" target="_blank">robin.carlisle@framestore.com</a>>> <br>
> wrote:<br>
> <br>
> Hi, thanks for your reply.<br>
> <br>
> I have been looking at :<br>
> <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control" rel="noreferrer" target="_blank">https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control</a> <<a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control" rel="noreferrer" target="_blank">https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control</a>><br>
> <br>
> /The stale-if-error response directive indicates that the cache can<br>
> reuse a stale response when an upstream server generates an error,<br>
> or when the error is generated locally. Here, an error is considered<br>
> any response with a status code of 500, 502, 503, or 504.<br>
> <br>
> Cache-Control: max-age=604800, stale-if-error=86400<br>
> In the example above, the response is fresh for 7 days (604800s).<br>
> Afterwards, it becomes stale, but can be used for an extra 1 day<br>
> (86400s) when an error is encountered.<br>
> <br>
> After the stale-if-error period passes, the client will receive any<br>
> error generated/<br>
> <br>
> Given what you have said and what the above docs say - I am still<br>
> confused as it looks like (in my test cases) the cached response can<br>
> be used for 3600 secs (this works), after which the cached response<br>
> can still be used for an additional 31536000 seconds on an error<br>
> (this doesnt work).<br>
> <br>
> I am going to dig into the error logging you suggested to see if I<br>
> can make sense of that - and will send on if I can't.<br>
> <br>
> Thanks v much for your help again,<br>
> <br>
> Robin<br>
> <br>
> <br>
> <br>
> <br>
> <br>
> On Thu, 1 Feb 2024 at 18:27, Alex Rousskov<br>
> <<a href="mailto:rousskov@measurement-factory.com" target="_blank">rousskov@measurement-factory.com</a><br>
> <mailto:<a href="mailto:rousskov@measurement-factory.com" target="_blank">rousskov@measurement-factory.com</a>>> wrote:<br>
> <br>
> On 2024-02-01 12:03, Robin Carlisle wrote:<br>
> > Hi, I am having trouble with stale-if-error response.<br>
> <br>
> If I am interpreting Squid code correctly, in primary use cases:<br>
> <br>
> * without a Cache-Control:stale-if-error=X in the original<br>
> response,<br>
> Squid sends a stale object if revalidation results in a 5xx error;<br>
> <br>
> * with a Cache-Control:stale-if-error=X and object age at most<br>
> X, Squid<br>
> sends a stale object if revalidation results in a 5xx error;<br>
> <br>
> * with a Cache-Control:stale-if-error=X and object age exceeding X,<br>
> Squid forwards the 5xx error response if revalidation results in<br>
> a 5xx<br>
> error;<br>
> <br>
> In other words, stale-if-error=X turns on a "fail on validation<br>
> errors"<br>
> behavior for stale objects older than X. It has no other effects.<br>
> <br>
> In your test case, the stale objects are much younger than<br>
> stale-if-error value (e.g., Age~=3601 vs. stale-if-error=31536000).<br>
> Thus, stale-if-error should have no relevant effect.<br>
> <br>
> Something else is probably preventing your Squid from serving<br>
> the stale<br>
> response when facing a 5xx error. I do not know what that<br>
> something is.<br>
> <br>
> I recommend sharing (privately if you need to protect sensitive<br>
> info) a<br>
> pointer to a compressed ALL,9 cache.log collected while<br>
> reproducing the<br>
> problem (using two transactions similar to the ones you have shared<br>
> below -- a successful stale hit and a problematic one):<br>
> <a href="https://wiki.squid-cache.org/SquidFaq/BugReporting#debugging-a-single-transaction" rel="noreferrer" target="_blank">https://wiki.squid-cache.org/SquidFaq/BugReporting#debugging-a-single-transaction</a> <<a href="https://wiki.squid-cache.org/SquidFaq/BugReporting#debugging-a-single-transaction" rel="noreferrer" target="_blank">https://wiki.squid-cache.org/SquidFaq/BugReporting#debugging-a-single-transaction</a>><br>
> <br>
> Alternatively, you can try to study cache.log yourself after<br>
> setting<br>
> debug_options to ALL,3. Searching for "refresh" and<br>
> "handleIMSReply" may<br>
> yield enough clues.<br>
> <br>
> <br>
> HTH,<br>
> <br>
> Alex.<br>
> <br>
> <br>
> <br>
> <br>
> > # /etc/squid/squid.conf :<br>
> ><br>
> > acl to_aws dstdomain .<a href="http://amazonaws.com" rel="noreferrer" target="_blank">amazonaws.com</a> <<a href="http://amazonaws.com" rel="noreferrer" target="_blank">http://amazonaws.com</a>><br>
> <<a href="http://amazonaws.com" rel="noreferrer" target="_blank">http://amazonaws.com</a> <<a href="http://amazonaws.com" rel="noreferrer" target="_blank">http://amazonaws.com</a>>><br>
> ><br>
> > acl from_local src localhost<br>
> ><br>
> > http_access allow to_aws<br>
> ><br>
> > http_access allow from_local<br>
> ><br>
> > cache allow all<br>
> ><br>
> > cache_dir ufs /var/cache/squid 1024 16 256<br>
> ><br>
> > http_port 3129 ssl-bump cert=/etc/squid/maul.pem<br>
> > generate-host-certificates=on dynamic_cert_mem_cache_size=4MB<br>
> ><br>
> > sslcrtd_program /usr/lib/squid/security_file_certgen -s<br>
> > /var/lib/squid/ssl_db -M 4MB<br>
> ><br>
> > acl step1 at_step SslBump1<br>
> ><br>
> > ssl_bump bump step1<br>
> ><br>
> > ssl_bump bump all<br>
> ><br>
> > sslproxy_cert_error deny all<br>
> ><br>
> > cache_store_log stdio:/var/log/squid/store.log<br>
> ><br>
> > logfile_rotate 0<br>
> ><br>
> > shutdown_lifetime 3 seconds<br>
> ><br>
> ><br>
> > # /usr/bin/proxy-test :<br>
> ><br>
> > #!/bin/bash<br>
> ><br>
> > curl --proxy <a href="http://localhost:3129" rel="noreferrer" target="_blank">http://localhost:3129</a> <<a href="http://localhost:3129" rel="noreferrer" target="_blank">http://localhost:3129</a>><br>
> <<a href="http://localhost:3129" rel="noreferrer" target="_blank">http://localhost:3129</a> <<a href="http://localhost:3129" rel="noreferrer" target="_blank">http://localhost:3129</a>>> \<br>
> ><br>
> > --cacert /etc/squid/stuff.pem \<br>
> ><br>
> > -v "<a href="https://stuff.amazonaws.com/api/v1/stuff/stuff.json" rel="noreferrer" target="_blank">https://stuff.amazonaws.com/api/v1/stuff/stuff.json</a><br>
> <<a href="https://stuff.amazonaws.com/api/v1/stuff/stuff.json" rel="noreferrer" target="_blank">https://stuff.amazonaws.com/api/v1/stuff/stuff.json</a>><br>
> > <<a href="https://stuff.amazonaws.com/api/v1/stuff/stuff.json" rel="noreferrer" target="_blank">https://stuff.amazonaws.com/api/v1/stuff/stuff.json</a><br>
> <<a href="https://stuff.amazonaws.com/api/v1/stuff/stuff.json" rel="noreferrer" target="_blank">https://stuff.amazonaws.com/api/v1/stuff/stuff.json</a>>>" \<br>
> ><br>
> > -H "Authorization: token MYTOKEN" \<br>
> ><br>
> > -H "Content-Type: application/json" \<br>
> ><br>
> > --output "/tmp/stuff.json"<br>
> ><br>
> ><br>
> ><br>
> > Tests ..........<br>
> ><br>
> ><br>
> > At this point in time the network cable is unattached. Squid<br>
> returns<br>
> > the cached object it got when the network was online earlier.<br>
> The Age of<br>
> > this object is just still under the max_age of 3600. <br>
> Previously I<br>
> > was using offline_mode but I found that it did not try to<br>
> revalidate<br>
> > from the origin after the object expired (defined via max-age<br>
> response).<br>
> > My understanding is that stale-if-error should work under my<br>
> > circumstances.<br>
> ><br>
> ><br>
> > # /var/log/squid/access.log<br>
> ><br>
> > 1706799404.440 6 127.0.0.1 NONE_NONE/200 0 CONNECT<br>
> > <a href="http://stuff.amazonaws.com:443" rel="noreferrer" target="_blank">stuff.amazonaws.com:443</a> <<a href="http://stuff.amazonaws.com:443" rel="noreferrer" target="_blank">http://stuff.amazonaws.com:443</a>><br>
> <<a href="http://stuff.amazonaws.com:443" rel="noreferrer" target="_blank">http://stuff.amazonaws.com:443</a><br>
> <<a href="http://stuff.amazonaws.com:443" rel="noreferrer" target="_blank">http://stuff.amazonaws.com:443</a>>> - HIER_NONE/- -<br>
> ><br>
> > 1706799404.440 0 127.0.0.1 TCP_MEM_HIT/200 20726 GET<br>
> > <a href="https://stuff.amazonaws.com/stuff.json" rel="noreferrer" target="_blank">https://stuff.amazonaws.com/stuff.json</a><br>
> <<a href="https://stuff.amazonaws.com/stuff.json" rel="noreferrer" target="_blank">https://stuff.amazonaws.com/stuff.json</a>><br>
> > <<a href="https://stuff.amazonaws.com/stuff.json" rel="noreferrer" target="_blank">https://stuff.amazonaws.com/stuff.json</a><br>
> <<a href="https://stuff.amazonaws.com/stuff.json" rel="noreferrer" target="_blank">https://stuff.amazonaws.com/stuff.json</a>>> - HIER_NONE/-<br>
> application/json<br>
> ><br>
> ><br>
> > # extract from /usr/bin/proxy-test<br>
> ><br>
> > < HTTP/1.1 200 OK<br>
> ><br>
> > < Date: Thu, 01 Feb 2024 13:57:11 GMT<br>
> ><br>
> > < Content-Type: application/json<br>
> ><br>
> > < Content-Length: 20134<br>
> ><br>
> > < x-amzn-RequestId: 3a2d3b26-df73-4b30-88cb-1a9268fa0df2<br>
> ><br>
> > < Last-Modified: 2024-02-01T13:00:45.000Z<br>
> ><br>
> > < Access-Control-Allow-Origin: *<br>
> ><br>
> > < x-amz-apigw-id: SdZwpG7qiYcERUQ=<br>
> ><br>
> > < Cache-Control: public, max-age=3600, stale-if-error=31536000<br>
> ><br>
> > < ETag: "cec102b43372840737ab773c2e77858b"<br>
> ><br>
> > < X-Amzn-Trace-Id: Root=1-65bba337-292be751134161b03555cdd6<br>
> ><br>
> > < Age: 3573<br>
> ><br>
> > < X-Cache: HIT from labs-maul-st-31<br>
> ><br>
> > < X-Cache-Lookup: HIT from labs-maul-st-31:3129<br>
> ><br>
> > < Via: 1.1 labs-maul-st-31 (squid/5.7)<br>
> ><br>
> > < Connection: keep-alive<br>
> ><br>
> ><br>
> ><br>
> ><br>
> > Below .. the curl script executes again. The Age has gone<br>
> over the<br>
> > max-age so squid attempted to refresh from the origin. The<br>
> machine is<br>
> > still offline so the refresh failed. I expected that the<br>
> > stale-if-error response would instruct squid to return the<br>
> cached object<br>
> > as a 200.<br>
> ><br>
> ><br>
> > # /var/log/squid/access.log<br>
> ><br>
> > 1706799434.464 5 127.0.0.1 NONE_NONE/200 0 CONNECT<br>
> > <a href="http://stuff.amazonaws.com:443" rel="noreferrer" target="_blank">stuff.amazonaws.com:443</a> <<a href="http://stuff.amazonaws.com:443" rel="noreferrer" target="_blank">http://stuff.amazonaws.com:443</a>><br>
> <<a href="http://stuff.amazonaws.com:443" rel="noreferrer" target="_blank">http://stuff.amazonaws.com:443</a><br>
> <<a href="http://stuff.amazonaws.com:443" rel="noreferrer" target="_blank">http://stuff.amazonaws.com:443</a>>> - HIER_NONE/- -<br>
> ><br>
> > 1706799434.464 0 127.0.0.1 TCP_REFRESH_FAIL_ERR/502 4235<br>
> GET<br>
> > <a href="https://stuff.amazonaws.com/stuff.json" rel="noreferrer" target="_blank">https://stuff.amazonaws.com/stuff.json</a><br>
> <<a href="https://stuff.amazonaws.com/stuff.json" rel="noreferrer" target="_blank">https://stuff.amazonaws.com/stuff.json</a>><br>
> > <<a href="https://stuff.amazonaws.com/stuff.json" rel="noreferrer" target="_blank">https://stuff.amazonaws.com/stuff.json</a><br>
> <<a href="https://stuff.amazonaws.com/stuff.json" rel="noreferrer" target="_blank">https://stuff.amazonaws.com/stuff.json</a>>> - HIER_NONE/- text/html<br>
> ><br>
> ><br>
> > # extract from /usr/bin/proxy-test<br>
> ><br>
> > < HTTP/1.1 502 Bad Gateway<br>
> ><br>
> > < Server: squid/5.7<br>
> ><br>
> > < Mime-Version: 1.0<br>
> ><br>
> > < Date: Thu, 01 Feb 2024 14:57:14 GMT<br>
> ><br>
> > < Content-Type: text/html;charset=utf-8<br>
> ><br>
> > < Content-Length: 3853<br>
> ><br>
> > < X-Squid-Error: ERR_READ_ERROR 0<br>
> ><br>
> > < Vary: Accept-Language<br>
> ><br>
> > < Content-Language: en<br>
> ><br>
> > < X-Cache: MISS from labs-maul-st-31<br>
> ><br>
> > < X-Cache-Lookup: HIT from labs-maul-st-31:3129<br>
> ><br>
> > < Via: 1.1 labs-maul-st-31 (squid/5.7)<br>
> ><br>
> > < Connection: close<br>
> ><br>
> ><br>
> ><br>
> > Hope someone can help me with this. All the best,<br>
> ><br>
> ><br>
> > Robin Carlisle<br>
> ><br>
> ><br>
> ><br>
> > _______________________________________________<br>
> > squid-users mailing list<br>
> > <a href="mailto:squid-users@lists.squid-cache.org" target="_blank">squid-users@lists.squid-cache.org</a><br>
> <mailto:<a href="mailto:squid-users@lists.squid-cache.org" target="_blank">squid-users@lists.squid-cache.org</a>><br>
> > <a href="https://lists.squid-cache.org/listinfo/squid-users" rel="noreferrer" target="_blank">https://lists.squid-cache.org/listinfo/squid-users</a><br>
> <<a href="https://lists.squid-cache.org/listinfo/squid-users" rel="noreferrer" target="_blank">https://lists.squid-cache.org/listinfo/squid-users</a>><br>
> <br>
> _______________________________________________<br>
> squid-users mailing list<br>
> <a href="mailto:squid-users@lists.squid-cache.org" target="_blank">squid-users@lists.squid-cache.org</a><br>
> <mailto:<a href="mailto:squid-users@lists.squid-cache.org" target="_blank">squid-users@lists.squid-cache.org</a>><br>
> <a href="https://lists.squid-cache.org/listinfo/squid-users" rel="noreferrer" target="_blank">https://lists.squid-cache.org/listinfo/squid-users</a><br>
> <<a href="https://lists.squid-cache.org/listinfo/squid-users" rel="noreferrer" target="_blank">https://lists.squid-cache.org/listinfo/squid-users</a>><br>
> <br>
<br>
</blockquote></div>