<div dir="ltr">Hey Amos,<div><br></div><div>I have implemented your patch at</div><div><br></div><div>and added following to my squid.conf</div><div><div>archive_mode allow all</div></div><div><br></div><div>and my refresh pattern is,</div><div><div>refresh_pattern <a href="http://dl-ssl.google.com/.*\.(deb|zip|tar|rpm)">dl-ssl.google.com/.*\.(deb|zip|tar|rpm)</a> 129600 100% 129600 ignore-reload ignore-no-store override-expire override-lastmod ignor$</div></div><div><br></div><div>but i am still not able to cache it, can you tell from below output what would be the problem ? Do i need to configure anything extra ?</div><div><br></div><div>here is the debug output for the same,</div><div>------------------------------------------------------------------------------------------------</div><div><br></div><div><div>2016/10/05 15:46:25.319 kid1| 5,2| TcpAcceptor.cc(220) doAccept: New connection on FD 14</div><div>2016/10/05 15:46:25.319 kid1| 5,2| TcpAcceptor.cc(295) acceptNext: connection on local=[::]:3128 remote=[::] FD 14 flags=9</div><div>2016/10/05 15:46:25.319 kid1| 11,2| client_side.cc(2346) parseHttpRequest: HTTP Client local=<a href="http://192.168.1.1:3128">192.168.1.1:3128</a> remote=<a href="http://192.168.1.76:51236">192.168.1.76:51236</a> FD 12 flags=1</div><div>2016/10/05 15:46:25.319 kid1| 11,2| client_side.cc(2347) parseHttpRequest: HTTP Client REQUEST:</div><div>---------</div><div>GET <a href="http://dl-ssl.google.com/dl/linux/direct/mod-pagespeed-beta_current_i386.deb">http://dl-ssl.google.com/dl/linux/direct/mod-pagespeed-beta_current_i386.deb</a> HTTP/1.1</div><div>Host: <a href="http://dl-ssl.google.com">dl-ssl.google.com</a></div><div>User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:49.0) Gecko/20100101 Firefox/49.0</div><div>Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8</div><div>Accept-Language: en-US,en;q=0.5</div><div>Accept-Encoding: gzip, deflate</div><div>Cookie: NID=88=109tS20j8Ec0EQb5HzuNnbwtsl4sK64aakVRn-2qOe91Zv4e3st9lfyik8qQe7d12J4xBDCmdKMwiXY98a2dj4mOitaP4AbJV6fD7o9YKTxE7MziEkNCJ45GiDszPM8wXca5cuYK_gE4QVrU52VqzSa1IzmHbh_7XKsvYuDCSsgIMZaC8d4Fp01vrAU8dHPXGopVpBIxgpHwAjPv8NvLFM3e4y-um5A8umQ-GCFmpaaLd1_1jyafkNLTj-9Ix4hfsw; SID=1ANPj1-lw03bKfunZfrmk8ZsjEcTl5AiLgwzgtzki8MZ3JuvGyYgiP7LRJ05U1HQWbf76g.; HSID=AUu5M-p2Rw1uDb2_0; APISID=ss4uEw9eIOgmsZXv/ARs9Vws4Es_o_sfVX</div><div>Connection: keep-alive</div><div>Upgrade-Insecure-Requests: 1</div><div><br></div><div><br></div><div>----------</div><div>2016/10/05 15:46:25.320 kid1| 85,2| client_side_request.cc(744) clientAccessCheckDone: The request GET <a href="http://dl-ssl.google.com/dl/linux/direct/mod-pagespeed-beta_current_i386.deb">http://dl-ssl.google.com/dl/linux/direct/mod-pagespeed-beta_current_i386.deb</a> is ALLOWED; last ACL checked: CONNECT</div><div>2016/10/05 15:46:25.320 kid1| 85,2| client_side_request.cc(720) clientAccessCheck2: No adapted_http_access configuration. default: ALLOW</div><div>2016/10/05 15:46:25.320 kid1| 85,2| client_side_request.cc(744) clientAccessCheckDone: The request GET <a href="http://dl-ssl.google.com/dl/linux/direct/mod-pagespeed-beta_current_i386.deb">http://dl-ssl.google.com/dl/linux/direct/mod-pagespeed-beta_current_i386.deb</a> is ALLOWED; last ACL checked: CONNECT</div><div>2016/10/05 15:46:25.320 kid1| 17,2| FwdState.cc(133) FwdState: Forwarding client request local=<a href="http://192.168.1.1:3128">192.168.1.1:3128</a> remote=<a href="http://192.168.1.76:51236">192.168.1.76:51236</a> FD 12 flags=1, url=<a href="http://dl-ssl.google.com/dl/linux/direct/mod-pagespeed-beta_current_i386.deb">http://dl-ssl.google.com/dl/linux/direct/mod-pagespeed-beta_current_i386.deb</a></div><div>2016/10/05 15:46:25.320 kid1| 44,2| peer_select.cc(258) peerSelectDnsPaths: Find IP destination for: <a href="http://dl-ssl.google.com/dl/linux/direct/mod-pagespeed-beta_current_i386.deb">http://dl-ssl.google.com/dl/linux/direct/mod-pagespeed-beta_current_i386.deb</a>' via <a href="http://dl-ssl.google.com">dl-ssl.google.com</a></div><div>2016/10/05 15:46:25.417 kid1| 44,2| peer_select.cc(280) peerSelectDnsPaths: Found sources for '<a href="http://dl-ssl.google.com/dl/linux/direct/mod-pagespeed-beta_current_i386.deb">http://dl-ssl.google.com/dl/linux/direct/mod-pagespeed-beta_current_i386.deb</a>'</div><div>2016/10/05 15:46:25.417 kid1| 44,2| peer_select.cc(281) peerSelectDnsPaths: always_direct = ALLOWED</div><div>2016/10/05 15:46:25.417 kid1| 44,2| peer_select.cc(282) peerSelectDnsPaths: never_direct = DENIED</div><div>2016/10/05 15:46:25.417 kid1| 44,2| peer_select.cc(286) peerSelectDnsPaths: DIRECT = local=[::] remote=[2404:6800:4008:c02::be]:80 flags=1</div><div>2016/10/05 15:46:25.417 kid1| 44,2| peer_select.cc(286) peerSelectDnsPaths: DIRECT = local=0.0.0.0 remote=<a href="http://74.125.23.136:80">74.125.23.136:80</a> flags=1</div><div>2016/10/05 15:46:25.417 kid1| 44,2| peer_select.cc(286) peerSelectDnsPaths: DIRECT = local=0.0.0.0 remote=<a href="http://74.125.23.93:80">74.125.23.93:80</a> flags=1</div><div>2016/10/05 15:46:25.417 kid1| 44,2| peer_select.cc(286) peerSelectDnsPaths: DIRECT = local=0.0.0.0 remote=<a href="http://74.125.23.91:80">74.125.23.91:80</a> flags=1</div><div>2016/10/05 15:46:25.418 kid1| 44,2| peer_select.cc(286) peerSelectDnsPaths: DIRECT = local=0.0.0.0 remote=<a href="http://74.125.23.190:80">74.125.23.190:80</a> flags=1</div><div>2016/10/05 15:46:25.418 kid1| 44,2| peer_select.cc(295) peerSelectDnsPaths: timedout = 0</div><div>2016/10/05 15:46:25.418 kid1| 14,2| ipcache.cc(924) ipcacheMarkBadAddr: ipcacheMarkBadAddr: <a href="http://dl-ssl.google.com">dl-ssl.google.com</a> [2404:6800:4008:c02::be]:80</div><div>2016/10/05 15:46:25.567 kid1| 11,2| http.cc(2203) sendRequest: HTTP Server local=<a href="http://192.168.1.1:36674">192.168.1.1:36674</a> remote=<a href="http://74.125.23.136:80">74.125.23.136:80</a> FD 13 flags=1</div><div>2016/10/05 15:46:25.567 kid1| 11,2| http.cc(2204) sendRequest: HTTP Server REQUEST:</div><div>---------</div><div>GET /dl/linux/direct/mod-pagespeed-beta_current_i386.deb HTTP/1.1</div><div>User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:49.0) Gecko/20100101 Firefox/49.0</div><div>Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8</div><div>Accept-Language: en-US,en;q=0.5</div><div>Accept-Encoding: gzip, deflate</div><div>Cookie: NID=88=109tS20j8Ec0EQb5HzuNnbwtsl4sK64aakVRn-2qOe91Zv4e3st9lfyik8qQe7d12J4xBDCmdKMwiXY98a2dj4mOitaP4AbJV6fD7o9YKTxE7MziEkNCJ45GiDszPM8wXca5cuYK_gE4QVrU52VqzSa1IzmHbh_7XKsvYuDCSsgIMZaC8d4Fp01vrAU8dHPXGopVpBIxgpHwAjPv8NvLFM3e4y-um5A8umQ-GCFmpaaLd1_1jyafkNLTj-9Ix4hfsw; SID=1ANPj1-lw03bKfunZfrmk8ZsjEcTl5AiLgwzgtzki8MZ3JuvGyYgiP7LRJ05U1HQWbf76g.; HSID=AUu5M-p2Rw1uDb2_0; APISID=ss4uEw9eIOgmsZXv/ARs9Vws4Es_o_sfVX</div><div>Host: <a href="http://dl-ssl.google.com">dl-ssl.google.com</a></div><div>Cache-Control: max-age=7776000</div><div>Connection: keep-alive</div><div><br></div><div><br></div><div>----------</div><div>2016/10/05 15:46:25.780 kid1| ctx: enter level 0: '<a href="http://dl-ssl.google.com/dl/linux/direct/mod-pagespeed-beta_current_i386.deb">http://dl-ssl.google.com/dl/linux/direct/mod-pagespeed-beta_current_i386.deb</a>'</div><div>2016/10/05 15:46:25.780 kid1| 11,2| http.cc(717) processReplyHeader: HTTP Server local=<a href="http://192.168.1.1:36674">192.168.1.1:36674</a> remote=<a href="http://74.125.23.136:80">74.125.23.136:80</a> FD 13 flags=1</div><div>2016/10/05 15:46:25.780 kid1| 11,2| http.cc(718) processReplyHeader: HTTP Server REPLY:</div><div>---------</div><div>HTTP/1.1 200 OK</div><div>Accept-Ranges: bytes</div><div>Content-Length: 6662208</div><div>Content-Type: application/x-debian-package</div><div>Etag: "fa383"</div><div>Last-Modified: Thu, 15 Sep 2016 19:24:00 GMT</div><div>Server: downloads</div><div>Vary: *</div><div>X-Content-Type-Options: nosniff</div><div>X-Frame-Options: SAMEORIGIN</div><div>X-Xss-Protection: 1; mode=block</div><div>Date: Wed, 05 Oct 2016 10:16:25 GMT</div><div><br></div><div>!<arch></div><div>debian-binary 1473872866 0 0 100644 4 `</div><div>2.0</div><div>control.tar.gz 1473872866 0 0 100644 7806 `</div><div> �</div><div>----------</div><div>2016/10/05 15:46:25.780 kid1| ctx: exit level 0</div><div>2016/10/05 15:46:25.780 kid1| 20,2| store.cc(949) checkCachable: StoreEntry::checkCachable: NO: not cachable</div><div>2016/10/05 15:46:25.780 kid1| 20,2| store.cc(949) checkCachable: StoreEntry::checkCachable: NO: not cachable</div><div>2016/10/05 15:46:25.781 kid1| 88,2| client_side_reply.cc(2005) processReplyAccessResult: The reply for GET <a href="http://dl-ssl.google.com/dl/linux/direct/mod-pagespeed-beta_current_i386.deb">http://dl-ssl.google.com/dl/linux/direct/mod-pagespeed-beta_current_i386.deb</a> is ALLOWED, because it matched all</div><div>2016/10/05 15:46:25.781 kid1| 11,2| client_side.cc(1392) sendStartOfMessage: HTTP Client local=<a href="http://192.168.1.1:3128">192.168.1.1:3128</a> remote=<a href="http://192.168.1.76:51236">192.168.1.76:51236</a> FD 12 flags=1</div><div>2016/10/05 15:46:25.781 kid1| 11,2| client_side.cc(1393) sendStartOfMessage: HTTP Client REPLY:</div><div>---------</div><div>HTTP/1.1 200 OK</div><div>Accept-Ranges: bytes</div><div>Content-Length: 6662208</div><div>Content-Type: application/x-debian-package</div><div>ETag: "fa383"</div><div>Last-Modified: Thu, 15 Sep 2016 19:24:00 GMT</div><div>Server: downloads</div><div>Vary: *</div><div>X-Content-Type-Options: nosniff</div><div>X-Frame-Options: SAMEORIGIN</div><div>X-Xss-Protection: 1; mode=block</div><div>Date: Wed, 05 Oct 2016 10:16:25 GMT</div><div>Connection: keep-alive</div><div><br></div><div><br></div><div>----------</div><div>2016/10/05 15:46:25.781 kid1| 20,2| store.cc(949) checkCachable: StoreEntry::checkCachable: NO: not cachable</div><div>2016/10/05 15:46:25.781 kid1| 20,2| store.cc(949) checkCachable: StoreEntry::checkCachable: NO: not cachable</div><div>2016/10/05 15:46:25.781 kid1| 20,2| store.cc(949) checkCachable: StoreEntry::checkCachable: NO: not cachable</div><div>2016/10/05 15:46:25.782 kid1| 20,2| store.cc(949) checkCachable: StoreEntry::checkCachable: NO: not cachable</div></div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Oct 4, 2016 at 8:00 PM, Hardik Dangar <span dir="ltr"><<a href="mailto:hardikdangar+squid@gmail.com" target="_blank">hardikdangar+squid@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Wow, i couldn't think about that. google might need tracking data that could be the reason they have blindly put vary * header. oh Irony, company which talks to all of us on how to deliver content is trying to do such thing.<div><br></div><div>I have looked at your patch but how do i enable that ? do i need to write custom ACL ? i know i need to compile and reinstall after applying patch but what do i need to do exactly in squid.conf file as looking at your patch i am guessing i need to write archive acl or i am too naive to understand C code :)</div><div><br></div><div>Also </div><div><br></div><div><div style="font-size:12.8px">reply_header_replace is any good for this ?</div></div><div style="font-size:12.8px"><br></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Oct 4, 2016 at 7:47 PM, Amos Jeffries <span dir="ltr"><<a href="mailto:squid3@treenet.co.nz" target="_blank">squid3@treenet.co.nz</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span>On 5/10/2016 2:34 a.m., Hardik Dangar wrote:<br>
> Hey Amos,<br>
><br>
> We have about 50 clients which downloads same google chrome update every 2<br>
> or 3 days means 2.4 gb. although response says vary but requested file is<br>
> same and all is downloaded via apt update.<br>
><br>
> Is there any option just like ignore-no-store? I know i am asking for too<br>
> much but it seems very silly on google's part that they are sending very<br>
> header at a place where they shouldn't as no matter how you access those<br>
> url's you are only going to get those deb files.<br>
<br>
<br>
</span>Some things G does only make sense whan you ignore all the PR about<br>
wanting to make the web more efficient and consider it's a company whose<br>
income is derived by recording data about peoples habits and activities.<br>
Caching can hide that info from them.<br>
<span><br>
><br>
> can i hack squid source code to ignore very header ?<br>
><br>
<br>
</span>Google are explicitly saying the response changes. I suspect there is<br>
something involving Google account data being embeded in some of the<br>
downloads. For tracking, etc.<br>
<br>
<br>
If you are wanting to test it I have added a patch to<br>
<<a href="http://bugs.squid-cache.org/show_bug.cgi?id=4604" rel="noreferrer" target="_blank">http://bugs.squid-cache.org/s<wbr>how_bug.cgi?id=4604</a>> that should implement<br>
archival of responses where the ACLs match. It is completely untested by<br>
me beyond building, so YMMV.<br>
<span class="m_7025760255039291774HOEnZb"><font color="#888888"><br>
Amos<br>
<br>
</font></span></blockquote></div><br></div>
</div></div></blockquote></div><br></div>