<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>Hi All,</div><div><br></div><div>I have two things to clarify:</div><div>1) In earlier email (snipped below), Amos told that is caching and scheduled to download - does it mean that we got the answer and do some override?</div><div>-------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- </div><div>> This is what I see when the download is in progress:<br>> <br>> KEY 44000000000000000902000000000000<br>> STORE_PENDING NOT_IN_MEMORY SWAPOUT_NONE PING_DONE<br>> RELEASE_REQUEST,DISPATCHED,PRIVATE,VALIDATED<br><br>So file stored in memory and scheduled for removal. </div><div>-------------------- ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------</div><div> <br></div><div>2) With more debug_options enabled, I see that it is not caching because the response is part of authenticated flow. Is there a way I can override this?</div><div>- Initially when this file download starts, it gets authorized (I think that is why I see HEAD request is sent to target which has Cache-Control: max-age=0) and then the subsequent GET requests (dont have any cache-control header) download the chunk using adjusted range and offset.</div><div>- But the subsequent GET reply seems set with auth flag, see the code below snipped from source (<a href="https://github.com/squid-cache/squid/blob/4f1c93a7a0d14eec223e199275ce570d840f71bc/src/http.cc">https://github.com/squid-cache/squid/blob/4f1c93a7a0d14eec223e199275ce570d840f71bc/src/http.cc</a>). </div><div><div> // RFC 2068, sec 14.9.4 - MUST NOT cache any response with Authentication UNLESS certain CC controls are present</div><div> // allow HTTP violations to IGNORE those controls (ie re-block caching Auth)</div><div> if (request && (request->flags.auth || request->flags.authSent)) {</div><div> if (!rep->cache_control)</div><div> return decision.make(ReuseDecision::reuseNot,</div><div> "authenticated and server reply missing Cache-Control");</div></div><div>- I tried adding override-expire in the cgi-bin refresh pattern, but that will override only for max-age in Cache-control but not relevant for auth flag.</div><div><br></div><div>Please find the logs below:</div><div>Please let me know if I am missing something.</div><div><br></div><div>Thanks,</div><div>Hari</div><div><br></div><div>Refresh pattern (URL:<a href="https://example.com/pcgi-bin/swdld/download.cgi?dwnld_code=xhM...">https://example.com/pcgi-bin/swdld/download.cgi?dwnld_code=xhM...</a>):</div><div><div>refresh_pattern ^ftp: 1440 20% 10080</div><div>refresh_pattern ^gopher: 1440 0% 1440</div><div>refresh_pattern -i (/cgi-bin/|\?) 0 0% 0</div><div>refresh_pattern . 0 20% 4320</div></div><div><br></div><div>Logs:</div><div>2018/09/10 09:57:24.864| 11,2| http.cc(723) processReplyHeader: HTTP Server RESPONSE:</div><div>---------</div><div>HTTP/1.1 206 Partial Content</div><div>Date: Mon, 10 Sep 2018 09:57:20 GMT^M</div><div>Server: Apache/2.2^M</div><div>Content-Disposition: attachment; filename=somefile.iso;^M</div><div>Last-Modified: Thu, 22 Mar 2018 02:11:11 GMT^M</div><div>ETag: "4ad8a61-1193b000-567f6d2466dc0"^M</div><div>Accept-Ranges: bytes^M</div><div>Content-Length: 294342471^M</div><div>Content-Range: bytes 549049-294891519/294891520^M</div><div>Keep-Alive: timeout=5, max=100^M</div><div>Connection: Keep-Alive^M</div><div>Content-Type: application/unknown^M</div><div>^M</div><div>----------</div><div>2018/09/10 09:57:24.864| 11,5| Client.cc(134) setVirginReply: 0x5560bf163d78 setting virgin reply to 0x5560bf162c20</div><div>2018/09/10 09:57:24.864| ctx: exit level 0</div><div>2018/09/10 09:57:24.864| 83,3| AccessCheck.cc(42) Start: adaptation off, skipping</div><div>2018/09/10 09:57:24.864| 11,5| Client.cc(969) adaptOrFinalizeReply: adaptationAccessCheckPending=0</div><div>2018/09/10 09:57:24.864| 11,5| Client.cc(152) setFinalReply: 0x5560bf163d78 setting final reply to 0x5560bf162c20</div><div>2018/09/10 09:57:24.864| 20,3| store.cc(1807) replaceHttpReply: StoreEntry::replaceHttpReply: <a href="https://example.com/pcgi-bin/swdld/download.cgi?dwnld_code=xhMnkw8Z-oECuFusb12luTTCm0rP8jZiRFu8gsXRtc">https://example.com/pcgi-bin/swdld/download.cgi?dwnld_code=xhMnkw8Z-oECuFusb12luTTCm0rP8jZiRFu8gsXRtc</a></div><div>2018/09/10 09:57:24.864| ctx: enter level 0: '<a href="https://example.com/pcgi-bin/swdld/download.cgi?dwnld_code=xhMnkw8Z-oECuFusb12luTTCm0rP8jZiRFu8gsXRtcoacGcTu6dv-dkLcT4lqtgvM70n8-ucJsj09lRYt_a0t7_M5">https://example.com/pcgi-bin/swdld/download.cgi?dwnld_code=xhMnkw8Z-oECuFusb12luTTCm0rP8jZiRFu8gsXRtcoacGcTu6dv-dkLcT4lqtgvM70n8-ucJsj09lRYt_a0t7_M5</a></div><div>2018/09/10 09:57:24.864| 11,3| http.cc(907) haveParsedReplyHeaders: HTTP CODE: 206</div><div>2018/09/10 09:57:24.864| 73,3| HttpRequest.cc(664) storeId: sent back effectiveRequestUrl: <a href="https://dl.cisco.com/pcgi-bin/swdld/download.cgi?dwnld_code=xhMnkw8Z-oECuFusb12luTTCm0rP8jZiRFu8gsXRtcoac">https://dl.cisco.com/pcgi-bin/swdld/download.cgi?dwnld_code=xhMnkw8Z-oECuFusb12luTTCm0rP8jZiRFu8gsXRtcoac</a></div><div>2018/09/10 09:57:24.864| 20,3| Controller.cc(386) peek: 76E544615E001DBF49EF0F94EE0A8F9A</div><div>2018/09/10 09:57:24.865| 20,4| Controller.cc(420) peek: cannot locate 76E544615E001DBF49EF0F94EE0A8F9A</div><div>2018/09/10 09:57:24.865| 20,3| store.cc(450) releaseRequest: 0 e:=p2IV/0x5560bf16ac60*3</div><div>2018/09/10 09:57:24.865| 20,3| store.cc(580) setPrivateKey: 01 e:=p2IV/0x5560bf16ac60*3</div><div>2018/09/10 09:57:24.865| 11,3| http.cc(982) haveParsedReplyHeaders: decided: do not cache and do not share because authenticated and server reply missing Cache-Control; HTTP status 206 e:=p2XIV/0x</div><div>2018/09/10 09:57:24.865| ctx: exit level 0</div><div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr">On Sat, Sep 8, 2018 at 8:13 PM Hariharan Sethuraman <<a href="mailto:srnhari@gmail.com">srnhari@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto">But the partial data is been continuously sending back to client. Squid didn't wait for the complete file to download</div><br><div class="gmail_quote"><div dir="ltr">On Sat, 8 Sep 2018, 20:08 Amos Jeffries, <<a href="mailto:squid3@treenet.co.nz" target="_blank">squid3@treenet.co.nz</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On 8/09/18 7:44 PM, Hariharan Sethuraman wrote:<br>
> And here is my HTTP request and reply:<br>
> 1. HEAD request to read image information<br>
> 2. GET request to download the image<br>
> <br>
<br>
Not quite. GET partial / Range request to fetch the content.<br>
<br>
Squid converted it into a full request for the backend server due to<br>
range_offset_limit -1. But that does mean Squid had to download ~240MB<br>
of data before anything starts being sent to the client.<br>
<br>
<br>
<br>
> (2)<br>
> 2018/09/08 07:28:39.155| 11,2| client_side.cc(1278) parseHttpRequest:<br>
> HTTP Client REQUEST:<br>
> ---------<br>
> GET /DcKbz9kqMQXK-zp95pv9LH11kjhTpxOJsJ-1FYEL4<br>
> Host: <a href="http://example.com:3129" rel="noreferrer noreferrer" target="_blank">example.com:3129</a> <<a href="http://example.com:3129" rel="noreferrer noreferrer" target="_blank">http://example.com:3129</a>>^M<br>
> Range: bytes=242819145-^M<br>
> User-Agent: curl/7.56.1^M<br>
> Accept: */*^M<br>
> <br>
<br>
<br>
> 2018/09/08 07:28:39.938| 11,2| http.cc(2261) sendRequest: HTTP Server<br>
> REQUEST:<br>
> ---------<br>
> GET /DcKbz9kqMQXK-zp95pv9LH11kjhTpxOJsJ-1FYEL4<br>
> User-Agent: curl/7.56.1^M<br>
> Accept: */*^M<br>
> Host: <a href="http://exampletarget.com" rel="noreferrer noreferrer" target="_blank">exampletarget.com</a> <<a href="http://exampletarget.com" rel="noreferrer noreferrer" target="_blank">http://exampletarget.com</a>>^M<br>
> Via: 1.1 jb7mgd (squid/4.2)^M<br>
> Surrogate-Capability: jb7mgd="Surrogate/1.0"^M<br>
> X-Forwarded-For: **.**.**.**^M<br>
> Cache-Control: max-age=0^M<br>
> Connection: keep-alive^M<br>
> <br>
> 2018/09/08 07:28:44.359| 11,2| http.cc(723) processReplyHeader: HTTP<br>
> Server RESPONSE:<br>
> ---------<br>
> HTTP/1.1 200 OK<br>
> Date: Sat, 08 Sep 2018 07:28:40 GMT^M<br>
> Server: Apache/2.2^M<br>
> Content-Disposition: attachment; filename=somefile.iso;^M<br>
> Last-Modified: Fri, 20 Jul 2018 18:23:10 GMT^M<br>
> ETag: "4a54c59-11653800-571726350bf80"^M<br>
> Accept-Ranges: bytes^M<br>
> Content-Length: 291846144^M<br>
> Keep-Alive: timeout=5, max=100^M<br>
> Connection: Keep-Alive^M<br>
> Content-Type: application/unknown^M<br>
> <br>
> 2018/09/08 07:28:44.361| 11,2| Stream.cc(267) sendStartOfMessage: HTTP<br>
> Client REPLY:<br>
> ---------<br>
> HTTP/1.1 206 Partial Content^M<br>
> Date: Sat, 08 Sep 2018 07:28:40 GMT^M<br>
> Server: Apache/2.2^M<br>
> Content-Disposition: attachment; filename=somefile.iso;^M<br>
> Last-Modified: Fri, 20 Jul 2018 18:23:10 GMT^M<br>
> ETag: "4a54c59-11653800-571726350bf80"^M<br>
> Accept-Ranges: bytes^M<br>
> Content-Type: application/unknown^M<br>
> X-Cache: MISS from jb7mgd^M<br>
> X-Cache-Lookup: MISS from jb7mgd:3128^M<br>
> Via: 1.1 jb7mgd (squid/4.2)^M<br>
> Connection: keep-alive^M<br>
> Content-Range: bytes 242819145-291846143/291846144^M<br>
> Content-Length: 49026999^M<br>
> <br>
> Thanks,<br>
> Hari<br>
> <br>
> On Sat, Sep 8, 2018 at 9:48 AM Hariharan Sethuraman wrote:<br>
> <br>
> Hi Amos,<br>
> <br>
> This is what I see when the download is in progress:<br>
> <br>
> KEY 44000000000000000902000000000000<br>
> STORE_PENDING NOT_IN_MEMORY SWAPOUT_NONE PING_DONE<br>
> RELEASE_REQUEST,DISPATCHED,PRIVATE,VALIDATED<br>
<br>
So file stored in memory and scheduled for removal.<br>
<br>
> LV:1536379799 LU:1536379801 LM:1532110990 EX:-1<br>
> 4 locks, 1 clients, 1 refs<br>
> Swap Dir -1, File 0XFFFFFFFF<br>
> GET<br>
> <a href="https://example.com/DhAskLOUpvRG2oeR_f_FxYTyLVHIN5esRF-LXOUKwkwyT0TOf6xO-AUm3KaM" rel="noreferrer noreferrer" target="_blank">https://example.com/DhAskLOUpvRG2oeR_f_FxYTyLVHIN5esRF-LXOUKwkwyT0TOf6xO-AUm3KaM</a><br>
> inmem_lo: 99225582<br>
> inmem_hi: 99324372<br>
> swapout: 0 bytes queued<br>
> <br>
<br>
Amos<br>
<br>
</blockquote></div>
</blockquote></div></div></div></div></div>