[squid-users] Squid 4.2 : caching is not working
Hariharan Sethuraman
srnhari at gmail.com
Mon Sep 10 12:18:55 UTC 2018
Hi All,
I have two things to clarify:
1) In earlier email (snipped below), Amos told that is caching and
scheduled to download - does it mean that we got the answer and do some
override?
--------------------
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> This is what I see when the download is in progress:
>
> KEY 44000000000000000902000000000000
> STORE_PENDING NOT_IN_MEMORY SWAPOUT_NONE PING_DONE
> RELEASE_REQUEST,DISPATCHED,PRIVATE,VALIDATED
So file stored in memory and scheduled for removal.
--------------------
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
2) With more debug_options enabled, I see that it is not caching because
the response is part of authenticated flow. Is there a way I can override
this?
- Initially when this file download starts, it gets authorized (I think
that is why I see HEAD request is sent to target which has Cache-Control:
max-age=0) and then the subsequent GET requests (dont have any
cache-control header) download the chunk using adjusted range and offset.
- But the subsequent GET reply seems set with auth flag, see the code
below snipped from source (
https://github.com/squid-cache/squid/blob/4f1c93a7a0d14eec223e199275ce570d840f71bc/src/http.cc
).
// RFC 2068, sec 14.9.4 - MUST NOT cache any response with
Authentication UNLESS certain CC controls are present
// allow HTTP violations to IGNORE those controls (ie re-block caching
Auth)
if (request && (request->flags.auth || request->flags.authSent)) {
if (!rep->cache_control)
return decision.make(ReuseDecision::reuseNot,
"authenticated and server reply missing
Cache-Control");
- I tried adding override-expire in the cgi-bin refresh pattern, but that
will override only for max-age in Cache-control but not relevant for auth
flag.
Please find the logs below:
Please let me know if I am missing something.
Thanks,
Hari
Refresh pattern (URL:
https://example.com/pcgi-bin/swdld/download.cgi?dwnld_code=xhM...):
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
refresh_pattern . 0 20% 4320
Logs:
2018/09/10 09:57:24.864| 11,2| http.cc(723) processReplyHeader: HTTP Server
RESPONSE:
---------
HTTP/1.1 206 Partial Content
Date: Mon, 10 Sep 2018 09:57:20 GMT^M
Server: Apache/2.2^M
Content-Disposition: attachment; filename=somefile.iso;^M
Last-Modified: Thu, 22 Mar 2018 02:11:11 GMT^M
ETag: "4ad8a61-1193b000-567f6d2466dc0"^M
Accept-Ranges: bytes^M
Content-Length: 294342471^M
Content-Range: bytes 549049-294891519/294891520^M
Keep-Alive: timeout=5, max=100^M
Connection: Keep-Alive^M
Content-Type: application/unknown^M
^M
----------
2018/09/10 09:57:24.864| 11,5| Client.cc(134) setVirginReply:
0x5560bf163d78 setting virgin reply to 0x5560bf162c20
2018/09/10 09:57:24.864| ctx: exit level 0
2018/09/10 09:57:24.864| 83,3| AccessCheck.cc(42) Start: adaptation off,
skipping
2018/09/10 09:57:24.864| 11,5| Client.cc(969) adaptOrFinalizeReply:
adaptationAccessCheckPending=0
2018/09/10 09:57:24.864| 11,5| Client.cc(152) setFinalReply: 0x5560bf163d78
setting final reply to 0x5560bf162c20
2018/09/10 09:57:24.864| 20,3| store.cc(1807) replaceHttpReply:
StoreEntry::replaceHttpReply:
https://example.com/pcgi-bin/swdld/download.cgi?dwnld_code=xhMnkw8Z-oECuFusb12luTTCm0rP8jZiRFu8gsXRtc
2018/09/10 09:57:24.864| ctx: enter level 0: '
https://example.com/pcgi-bin/swdld/download.cgi?dwnld_code=xhMnkw8Z-oECuFusb12luTTCm0rP8jZiRFu8gsXRtcoacGcTu6dv-dkLcT4lqtgvM70n8-ucJsj09lRYt_a0t7_M5
2018/09/10 09:57:24.864| 11,3| http.cc(907) haveParsedReplyHeaders: HTTP
CODE: 206
2018/09/10 09:57:24.864| 73,3| HttpRequest.cc(664) storeId: sent back
effectiveRequestUrl:
https://dl.cisco.com/pcgi-bin/swdld/download.cgi?dwnld_code=xhMnkw8Z-oECuFusb12luTTCm0rP8jZiRFu8gsXRtcoac
2018/09/10 09:57:24.864| 20,3| Controller.cc(386) peek:
76E544615E001DBF49EF0F94EE0A8F9A
2018/09/10 09:57:24.865| 20,4| Controller.cc(420) peek: cannot locate
76E544615E001DBF49EF0F94EE0A8F9A
2018/09/10 09:57:24.865| 20,3| store.cc(450) releaseRequest: 0
e:=p2IV/0x5560bf16ac60*3
2018/09/10 09:57:24.865| 20,3| store.cc(580) setPrivateKey: 01
e:=p2IV/0x5560bf16ac60*3
2018/09/10 09:57:24.865| 11,3| http.cc(982) haveParsedReplyHeaders:
decided: do not cache and do not share because authenticated and server
reply missing Cache-Control; HTTP status 206 e:=p2XIV/0x
2018/09/10 09:57:24.865| ctx: exit level 0
On Sat, Sep 8, 2018 at 8:13 PM Hariharan Sethuraman <srnhari at gmail.com>
wrote:
> But the partial data is been continuously sending back to client. Squid
> didn't wait for the complete file to download
>
> On Sat, 8 Sep 2018, 20:08 Amos Jeffries, <squid3 at treenet.co.nz> wrote:
>
>> On 8/09/18 7:44 PM, Hariharan Sethuraman wrote:
>> > And here is my HTTP request and reply:
>> > 1. HEAD request to read image information
>> > 2. GET request to download the image
>> >
>>
>> Not quite. GET partial / Range request to fetch the content.
>>
>> Squid converted it into a full request for the backend server due to
>> range_offset_limit -1. But that does mean Squid had to download ~240MB
>> of data before anything starts being sent to the client.
>>
>>
>>
>> > (2)
>> > 2018/09/08 07:28:39.155| 11,2| client_side.cc(1278) parseHttpRequest:
>> > HTTP Client REQUEST:
>> > ---------
>> > GET /DcKbz9kqMQXK-zp95pv9LH11kjhTpxOJsJ-1FYEL4
>> > Host: example.com:3129 <http://example.com:3129>^M
>> > Range: bytes=242819145-^M
>> > User-Agent: curl/7.56.1^M
>> > Accept: */*^M
>> >
>>
>>
>> > 2018/09/08 07:28:39.938| 11,2| http.cc(2261) sendRequest: HTTP Server
>> > REQUEST:
>> > ---------
>> > GET /DcKbz9kqMQXK-zp95pv9LH11kjhTpxOJsJ-1FYEL4
>> > User-Agent: curl/7.56.1^M
>> > Accept: */*^M
>> > Host: exampletarget.com <http://exampletarget.com>^M
>> > Via: 1.1 jb7mgd (squid/4.2)^M
>> > Surrogate-Capability: jb7mgd="Surrogate/1.0"^M
>> > X-Forwarded-For: **.**.**.**^M
>> > Cache-Control: max-age=0^M
>> > Connection: keep-alive^M
>> >
>> > 2018/09/08 07:28:44.359| 11,2| http.cc(723) processReplyHeader: HTTP
>> > Server RESPONSE:
>> > ---------
>> > HTTP/1.1 200 OK
>> > Date: Sat, 08 Sep 2018 07:28:40 GMT^M
>> > Server: Apache/2.2^M
>> > Content-Disposition: attachment; filename=somefile.iso;^M
>> > Last-Modified: Fri, 20 Jul 2018 18:23:10 GMT^M
>> > ETag: "4a54c59-11653800-571726350bf80"^M
>> > Accept-Ranges: bytes^M
>> > Content-Length: 291846144^M
>> > Keep-Alive: timeout=5, max=100^M
>> > Connection: Keep-Alive^M
>> > Content-Type: application/unknown^M
>> >
>> > 2018/09/08 07:28:44.361| 11,2| Stream.cc(267) sendStartOfMessage: HTTP
>> > Client REPLY:
>> > ---------
>> > HTTP/1.1 206 Partial Content^M
>> > Date: Sat, 08 Sep 2018 07:28:40 GMT^M
>> > Server: Apache/2.2^M
>> > Content-Disposition: attachment; filename=somefile.iso;^M
>> > Last-Modified: Fri, 20 Jul 2018 18:23:10 GMT^M
>> > ETag: "4a54c59-11653800-571726350bf80"^M
>> > Accept-Ranges: bytes^M
>> > Content-Type: application/unknown^M
>> > X-Cache: MISS from jb7mgd^M
>> > X-Cache-Lookup: MISS from jb7mgd:3128^M
>> > Via: 1.1 jb7mgd (squid/4.2)^M
>> > Connection: keep-alive^M
>> > Content-Range: bytes 242819145-291846143/291846144^M
>> > Content-Length: 49026999^M
>> >
>> > Thanks,
>> > Hari
>> >
>> > On Sat, Sep 8, 2018 at 9:48 AM Hariharan Sethuraman wrote:
>> >
>> > Hi Amos,
>> >
>> > This is what I see when the download is in progress:
>> >
>> > KEY 44000000000000000902000000000000
>> > STORE_PENDING NOT_IN_MEMORY SWAPOUT_NONE PING_DONE
>> > RELEASE_REQUEST,DISPATCHED,PRIVATE,VALIDATED
>>
>> So file stored in memory and scheduled for removal.
>>
>> > LV:1536379799 LU:1536379801 LM:1532110990 EX:-1
>> > 4 locks, 1 clients, 1 refs
>> > Swap Dir -1, File 0XFFFFFFFF
>> > GET
>> >
>> https://example.com/DhAskLOUpvRG2oeR_f_FxYTyLVHIN5esRF-LXOUKwkwyT0TOf6xO-AUm3KaM
>> > inmem_lo: 99225582
>> > inmem_hi: 99324372
>> > swapout: 0 bytes queued
>> >
>>
>> Amos
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20180910/9219a282/attachment.html>
More information about the squid-users
mailing list