<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link="#0563C1" vlink="#954F72"><div class=WordSection1><p class=MsoNormal>So, here is the problem: I want to cache the images on craigslist. The headers all look thoroughly cacheable, some browsers (I’m glairing at you Chrome) send with this thing that requests that they not be cachable, but craigslist replies anyway and says “sure thing! Cache that sucker!” and firefox doesn’t even do that. An example of URL: <a href="http://images.craigslist.org/00o0o_3fcu92TR5jB_600x450.jpg">http://images.craigslist.org/00o0o_3fcu92TR5jB_600x450.jpg</a><o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>The request headers look like:<o:p></o:p></p><p class=MsoNormal>Host: images.craigslist.org<o:p></o:p></p><p class=MsoNormal>User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:41.0) Gecko/20100101 Firefox/41.0<o:p></o:p></p><p class=MsoNormal>Accept: image/png,image/*;q=0.8,*/*;q=0.5<o:p></o:p></p><p class=MsoNormal>Accept-Language: en-US,en;q=0.5<o:p></o:p></p><p class=MsoNormal>Accept-Encoding: gzip, deflate<o:p></o:p></p><p class=MsoNormal>Referer: http://seattle.craigslist.org/oly/hvo/5288435732.html<o:p></o:p></p><p class=MsoNormal>Cookie: cl_tocmode=sss%3Agrid; cl_b=hlJExhZ55RGzNupTXAYJOAIcZ80; cl_def_lang=en; cl_def_hp=seattle<o:p></o:p></p><p class=MsoNormal>Connection: keep-alive<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>The response headers are:<o:p></o:p></p><p class=MsoNormal>Cache-Control: public, max-age=2592000 <span style='font-family:Wingdings'>ß</span> doesn’t that say “keep that a very long time”?<o:p></o:p></p><p class=MsoNormal>Content-Length: 49811<o:p></o:p></p><p class=MsoNormal>Content-Type: image/jpeg<o:p></o:p></p><p class=MsoNormal>Date: Tue, 27 Oct 2015 23:04:14 GMT<o:p></o:p></p><p class=MsoNormal>Last-Modified: Tue, 27 Oct 2015 23:04:14 GMT<o:p></o:p></p><p class=MsoNormal>Server: craigslist/0<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Access log says:<br>1445989120.714 265 192.168.2.56 TCP_MISS/200 50162 GET http://images.craigslist.org/00Y0Y_kMkjOhL1Lim_600x450.jpg - ORIGINAL_DST/208.82.236.227 image/jpeg<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>And Store Log says:<br>1445989120.714 RELEASE -1 FFFFFFFF 27C2B2CEC9ACCA05A31E80479E5F0E9C ? ? ? ? ?/? ?/? ? ?<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I started out with a configuration from here: <a href="http://wiki.sebeka.k12.mn.us/web_services:squid_update_cache">http://wiki.sebeka.k12.mn.us/web_services:squid_update_cache</a> but have made a lot of tweaks to it. In fact, I’ve dropped all the updates, all the rewrite, store id, and a lot of other stuff. I’ve set cache allow all (which, I suspect I can simply leave blank, but I don’t know) I’ve cut it down quite a bit, the one I am testing right now for example looks like this:<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>My squid.conf (which has been hacked mercilously trying stuff, admittedly) looks like this:<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><BEGIN SQUID.CONF ><o:p></o:p></p><p class=MsoNormal>acl localnet src 10.0.0.0/8 # RFC1918 possible internal network<o:p></o:p></p><p class=MsoNormal>acl localnet src 172.16.0.0/12 # RFC1918 possible internal network<o:p></o:p></p><p class=MsoNormal>acl localnet src 192.168.0.0/16 # RFC1918 possible internal network<o:p></o:p></p><p class=MsoNormal>acl localnet src fc00::/7 # RFC 4193 local private network range<o:p></o:p></p><p class=MsoNormal>acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged) machines<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>acl SSL_ports port 443<o:p></o:p></p><p class=MsoNormal>acl Safe_ports port 80 # http<o:p></o:p></p><p class=MsoNormal>acl Safe_ports port 21 # ftp<o:p></o:p></p><p class=MsoNormal>acl Safe_ports port 443 # https<o:p></o:p></p><p class=MsoNormal>acl Safe_ports port 70 # gopher<o:p></o:p></p><p class=MsoNormal>acl Safe_ports port 210 # wais<o:p></o:p></p><p class=MsoNormal>acl Safe_ports port 1025-65535 # unregistered ports<o:p></o:p></p><p class=MsoNormal>acl Safe_ports port 280 # http-mgmt<o:p></o:p></p><p class=MsoNormal>acl Safe_ports port 488 # gss-http<o:p></o:p></p><p class=MsoNormal>acl Safe_ports port 591 # filemaker<o:p></o:p></p><p class=MsoNormal>acl Safe_ports port 777 # multiling http<o:p></o:p></p><p class=MsoNormal>acl CONNECT method CONNECT<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>http_access allow localnet<o:p></o:p></p><p class=MsoNormal>http_access allow localhost<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal># And finally deny all other access to this proxy<o:p></o:p></p><p class=MsoNormal>http_access deny all<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>http_port 3128<o:p></o:p></p><p class=MsoNormal>http_port 3129 tproxy<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>cache_dir aufs /var/spool/squid/ 40000 32 256<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>cache_swap_low 90<o:p></o:p></p><p class=MsoNormal>cache_swap_high 95<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>dns_nameservers 8.8.8.8 8.8.4.4<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>cache allow all<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>maximum_object_size 8000 MB<o:p></o:p></p><p class=MsoNormal>range_offset_limit 8000 MB<o:p></o:p></p><p class=MsoNormal>quick_abort_min 512 KB<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>cache_store_log /var/log/squid/store.log<o:p></o:p></p><p class=MsoNormal>access_log daemon:/var/log/squid/access.log squid<o:p></o:p></p><p class=MsoNormal>cache_log /var/log/squid/cache.log<o:p></o:p></p><p class=MsoNormal>coredump_dir /var/spool/squid<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>max_open_disk_fds 8000<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>vary_ignore_expire on<o:p></o:p></p><p class=MsoNormal>request_entities on<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>refresh_pattern -i .*\.(gif|png|jpg|jpeg|ico|webp)$ 10080 100% 43200 ignore-no-store ignore-private ignore-reload store-stale<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>refresh_pattern ^ftp: 1440 20% 10080<o:p></o:p></p><p class=MsoNormal>refresh_pattern ^gopher: 1440 0% 1440<o:p></o:p></p><p class=MsoNormal>refresh_pattern -i .*\.index.(html|htm)$ 2880 40% 10080<o:p></o:p></p><p class=MsoNormal>refresh_pattern -i .*\.(html|htm|css|js)$ 120 40% 1440<o:p></o:p></p><p class=MsoNormal>refresh_pattern -i (/cgi-bin/|\?) 0 0% 0<o:p></o:p></p><p class=MsoNormal>refresh_pattern . 0 40% 40320<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>cache_mgr <my address><o:p></o:p></p><p class=MsoNormal>cache_effective_user proxy<o:p></o:p></p><p class=MsoNormal>cache_effective_group proxy<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><END SQUID.CONF><o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>There is a good deal of hacking that has gone into this configuration, and I accept that this will eventually be gutted and replaced with something less, broken. Where I am pulling my hair out is trying to figure out why things are cached and then not cached. That top refresh line (the one looking for jpg, gifs etc) has taken many forms, and I am getting inconsistent results. The above image will cache just fine, a couple times, but if I go back, clear the cache on the browser, close out, restart and reload, it releases the link and never again shall it cache. What is worse, it appears to get getting worse over time until it isn’t really picking up much of anything. What starts out as a few missed entries piles up into a huge list of cache misses over time.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Right now, I am running somewhere in the 0.1% hits rate, and I can only assume I have buckled something in all the compile and re-compiles, and reconfigurations. What started out as “gee, I wonder if I can cache updates” has turned into quite the rabbit hole!<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>So, big question, what debug level do I use to see this thing making decisions on whether to cache, and any tips anyone has about this would be appreciated. Thank you!<o:p></o:p></p></div></body></html>