<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Arial Rounded MT Bold";
panose-1:0 0 0 0 0 0 0 0 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";
color:black;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body bgcolor=white lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Hey Yuri,<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>The issue is not money alone…<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>To my understanding Squid is written in C++ and is very complex, due to this it requires more then basic level knowledge.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>However I can clearly say that it's not a big issue to use current squid APIs\Interfaces(ICAP\ECAP) to implement a solution which will act like the nginx "module".<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>I do not know how long it would take or how much it will cost since it requires time…<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>This time is required for:<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>- Learning\Relearning<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>- Identifying and predicting the different cases<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>- Basic testing for the different cases<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>- Implementing a basic structure<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>- Testing<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>- (In a loop and\or couple trees…)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>From my point of view compared to "ransom" or any similar idea, anyone that will write any piece of software to implement this specific idea should be able to take on his shoulders more then only this but more then just this.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>And just to illustrate, imagine that some nice guy pops into Boing or RedHat offices and will leave a DiskOnKey at the front desk with a note "This flash drive contains an idea that will bring you lots of money"(not saying the current idea itself is bad or wrong..).<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>What would these companies do? Will they put a team of engineers in a second?<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>I do believe that they are not "hot headed" enough to act in a second.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>I received a link couple years ago from Amos for an e-cap module:<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><a href="https://github.com/creamy/ecap-mongo">https://github.com/creamy/ecap-mongo</a><o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Which does couple very interesting things but, despite to the fact that I learned to program in C and C++ I couldn't understand and\or implement a Store API which could be used for\by squid.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>However I implemented this:<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Windows Updates a Caching Stub zone[ <a href="http://www1.ngtech.co.il/wpe/?page_id=301">http://www1.ngtech.co.il/wpe/?page_id=301</a> ]<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>And while implementing the idea one of the main things I noticed is that trying to "catch" all traffic into disk is the wrong way to define a goal.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Indeed it can be written to be done "automatically" but I will ask:<br>What is it worth to write everything into disk if you never read from disk more then 1% of the files content?<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>If you have a specific targeted site it's one thing but trying to catch them all is kind of like tying your feet with a rope to a door and then shove\slam the door to the other direction.<br>Imagine yourself how far and fast you will fly?<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Thanks,<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Eliezer<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><div><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Arial Rounded MT Bold","sans-serif";color:#1F497D'>----<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Arial Rounded MT Bold","sans-serif";color:#1F497D'><a href="http://ngtech.co.il/lmgtfy/"><span style='color:#0563C1'>Eliezer Croitoru</span></a><br>Linux System Administrator<br>Mobile: +972-5-28704261<br>Email: eliezer@ngtech.co.il<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><img border=0 width=183 height=69 id="Picture_x0020_1" src="cid:image002.png@01D1F12B.CFC13AD0"><o:p></o:p></span></p></div><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal><b><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:windowtext'>From:</span></b><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:windowtext'> squid-users [mailto:squid-users-bounces@lists.squid-cache.org] <b>On Behalf Of </b>Yuri Voinov<br><b>Sent:</b> Sunday, August 7, 2016 9:23 PM<br><b>To:</b> squid-users@lists.squid-cache.org<br><b>Subject:</b> Re: [squid-users] Range header is a hit ratio killer<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><br>-----BEGIN PGP SIGNED MESSAGE----- <br>Hash: SHA256 <br> <br>So,<br><br>the overall answer is "NO".<br><br>You can use Store-ID + collapsed forwarding functionality to achieve something your want. May be together, may be separate. Hard luck :)<br><br>But this is your own problem. No one will solve the problem without the infusion of large amounts of money to make it interesting.<br><br>:)<br><br><br>07.08.2016 20:12, Amos Jeffries пишет:<br>> On 6/08/2016 9:56 p.m., k simon<o:p></o:p></p><p class=MsoNormal> wrote:<br><br><o:p></o:p></p><p class=MsoNormal> >> Hi,list,<br><br><o:p></o:p></p><p class=MsoNormal> >> Code 206 is the most pain for our forwed proxy. Squid<o:p></o:p></p><p class=MsoNormal> use<br><br><o:p></o:p></p><p class=MsoNormal> >> “range_offset_limit” to process byte-range request. when<o:p></o:p></p><p class=MsoNormal> set it "none",<br><br><o:p></o:p></p><p class=MsoNormal> >> it has 2 wellknown issue:<br><br><o:p></o:p></p><p class=MsoNormal> >> 1. boost the traffic on the server side, we observed<o:p></o:p></p><p class=MsoNormal> it's amplified<br><br><o:p></o:p></p><p class=MsoNormal> >> 500% compared to clients side on our box.<br><br><o:p></o:p></p><p class=MsoNormal> ><br><br><o:p></o:p></p><p class=MsoNormal> > To which the answer currently is to see if enabling<o:p></o:p></p><p class=MsoNormal> collapsed_forwarding<br><br><o:p></o:p></p><p class=MsoNormal> > works okay for your needs.<br><br><o:p></o:p></p><p class=MsoNormal> ><br><br><o:p></o:p></p><p class=MsoNormal> >> 2. it's always failed on a lossy link, and squid<o:p></o:p></p><p class=MsoNormal> refetched it again and<br><br><o:p></o:p></p><p class=MsoNormal> >> again.<br><br><o:p></o:p></p><p class=MsoNormal> >> I've noticed that nginx have supported "byte-range<o:p></o:p></p><p class=MsoNormal> cacheing" since<br><br><o:p></o:p></p><p class=MsoNormal> >> 1.9.8 by Module ngx_http_slice_module officially.<br><br><o:p></o:p></p><p class=MsoNormal> >> (1.<br><br><o:p></o:p></p><p class=MsoNormal> >><o:p></o:p></p><p class=MsoNormal><a href="http://nginx.org/en/docs/http/ngx_http_slice_module.html?_ga=1.140845234.106894549.1470474534">http://nginx.org/en/docs/http/ngx_http_slice_module.html?_ga=1.140845234.106894549.1470474534</a><br><br><o:p></o:p></p><p class=MsoNormal> >><br><br><o:p></o:p></p><p class=MsoNormal> ><br><br><o:p></o:p></p><p class=MsoNormal> > So? what relevance does other software features have to Squid<o:p></o:p></p><p class=MsoNormal> behaviour?<br><br><o:p></o:p></p><p class=MsoNormal> ><br><br><o:p></o:p></p><p class=MsoNormal> > <o:p></o:p></p><p class=MsoNormal><a href="http://wiki.squid-cache.org/SquidFaq/AboutSquid#How_to_add_a_new_Squid_feature.2C_enhance.2C_of_fix_something.3F"><http://wiki.squid-cache.org/SquidFaq/AboutSquid#How_to_add_a_new_Squid_feature.2C_enhance.2C_of_fix_something.3F></a><br><br><o:p></o:p></p><p class=MsoNormal> ><br><br><o:p></o:p></p><p class=MsoNormal> > ... to be fair the storage code in Squid is a bit hairy in<o:p></o:p></p><p class=MsoNormal> places. So<br><br><o:p></o:p></p><p class=MsoNormal> > paying for it to be done is unlikely to be cheap. But still,<o:p></o:p></p><p class=MsoNormal> waiting<br><br><o:p></o:p></p><p class=MsoNormal> > wont fix the problem. We nearly go there in Squid-2.7, but<o:p></o:p></p><p class=MsoNormal> the<br><br><o:p></o:p></p><p class=MsoNormal> > experiment there is not able to completely port across to<o:p></o:p></p><p class=MsoNormal> Squid-3 and<br><br><o:p></o:p></p><p class=MsoNormal> > had some important problems anyway.<br><br><o:p></o:p></p><p class=MsoNormal> ><br><br><o:p></o:p></p><p class=MsoNormal> ><br><br><o:p></o:p></p><p class=MsoNormal> >> 2.<o:p></o:p></p><p class=MsoNormal> <a href="https://www.nginx.com/resources/admin-guide/content-caching/">https://www.nginx.com/resources/admin-guide/content-caching/</a> ).<br><br><o:p></o:p></p><p class=MsoNormal> >> The solution is not perfect, but it's really more<o:p></o:p></p><p class=MsoNormal> usable than<br><br><o:p></o:p></p><p class=MsoNormal> >> "range_offset_limit". The secret it's use a fixed size<o:p></o:p></p><p class=MsoNormal> object replaced<br><br><o:p></o:p></p><p class=MsoNormal> >> the whole file, and we can alter the request range offset<o:p></o:p></p><p class=MsoNormal> and passed it<br><br><o:p></o:p></p><p class=MsoNormal> >> to server;<br><br><o:p></o:p></p><p class=MsoNormal> ><br><br><o:p></o:p></p><p class=MsoNormal> > Ah, thats what range_offset_limit does today. Updates the<o:p></o:p></p><p class=MsoNormal> server request<br><br><o:p></o:p></p><p class=MsoNormal> > to say "deliver all of it" and stores the response in a file<o:p></o:p></p><p class=MsoNormal> the size of<br><br><o:p></o:p></p><p class=MsoNormal> > the whole expected response the server informs will be<o:p></o:p></p><p class=MsoNormal> arriving.<br><br><o:p></o:p></p><p class=MsoNormal> ><br><br><o:p></o:p></p><p class=MsoNormal> > The reason you are seeing that 500% increase in bandwidth is<o:p></o:p></p><p class=MsoNormal> that<br><br><o:p></o:p></p><p class=MsoNormal> > multiple Range requests arrive while the initial part of the<o:p></o:p></p><p class=MsoNormal> first<br><br><o:p></o:p></p><p class=MsoNormal> > response is still arriving back to Squid, so 5 of them get<o:p></o:p></p><p class=MsoNormal> sent through<br><br><o:p></o:p></p><p class=MsoNormal> > to the server. When that first one finishes, its object<o:p></o:p></p><p class=MsoNormal> becomes<br><br><o:p></o:p></p><p class=MsoNormal> > available for use as a HIT and followup Range requests get<o:p></o:p></p><p class=MsoNormal> bits of it<br><br><o:p></o:p></p><p class=MsoNormal> > (so you dont see 600% -> millions of % bandwidth<o:p></o:p></p><p class=MsoNormal> increase).<br><br><o:p></o:p></p><p class=MsoNormal> ><br><br><o:p></o:p></p><p class=MsoNormal> > collapsed_fowarding alters this by letting the first response<o:p></o:p></p><p class=MsoNormal> be used by<br><br><o:p></o:p></p><p class=MsoNormal> > other requests while it is still incomplete. But YMMV<o:p></o:p></p><p class=MsoNormal> regarding the<br><br><o:p></o:p></p><p class=MsoNormal> > savings and CF affects all traffic, so it may cause<o:p></o:p></p><p class=MsoNormal> behaviours you dont<br><br><o:p></o:p></p><p class=MsoNormal> > want on other types of request. Worth a try though.<br><br><o:p></o:p></p><p class=MsoNormal> ><br><br><o:p></o:p></p><p class=MsoNormal> ><br><br><o:p></o:p></p><p class=MsoNormal> >> perhaps forward the origin range offset and cache a part<o:p></o:p></p><p class=MsoNormal> of<br><br><o:p></o:p></p><p class=MsoNormal> >> the object with a range key is a better idea.<br><br><o:p></o:p></p><p class=MsoNormal> >> And squid should know how<br><br><o:p></o:p></p><p class=MsoNormal> >> to make up those object and process the request with<o:p></o:p></p><p class=MsoNormal> range header.<br><br><o:p></o:p></p><p class=MsoNormal> >> And with a fixed size object to cache it may benefits<o:p></o:p></p><p class=MsoNormal> to disk IO.<br><br><o:p></o:p></p><p class=MsoNormal> >> Sounds it's similar like big-rock db concept, though I've<o:p></o:p></p><p class=MsoNormal> not got<br><br><o:p></o:p></p><p class=MsoNormal> >> successed with rock on FreeBSD nor ubuntu box.<br><br><o:p></o:p></p><p class=MsoNormal> >> Does squid has some plan to support this method or have<o:p></o:p></p><p class=MsoNormal> another solution?<br><br><o:p></o:p></p><p class=MsoNormal> >><br><br><o:p></o:p></p><p class=MsoNormal> ><br><br><o:p></o:p></p><p class=MsoNormal> > squid is software. It doesn't have its own plans (at least I<o:p></o:p></p><p class=MsoNormal> hope not).<br><br><o:p></o:p></p><p class=MsoNormal> ><br><br><o:p></o:p></p><p class=MsoNormal> > I'm not aware of any plans specifically to add Range caching<o:p></o:p></p><p class=MsoNormal> any time<br><br><o:p></o:p></p><p class=MsoNormal> > soon. Ideas for how to do it get thrown around in squid-dev a<o:p></o:p></p><p class=MsoNormal> couple of<br><br><o:p></o:p></p><p class=MsoNormal> > times a year, so lots of ideas but so far nothing concrete<o:p></o:p></p><p class=MsoNormal> has come out<br><br><o:p></o:p></p><p class=MsoNormal> > of it. Yes rock and/or memory caches are looking like the<o:p></o:p></p><p class=MsoNormal> most easily<br><br><o:p></o:p></p><p class=MsoNormal> > adapted cache types to enable storing partial objects in,<o:p></o:p></p><p class=MsoNormal> someone still<br><br><o:p></o:p></p><p class=MsoNormal> > has to do the actual coding work though.<br><br><o:p></o:p></p><p class=MsoNormal> ><br><br><o:p></o:p></p><p class=MsoNormal> > Amos<br><br><o:p></o:p></p><p class=MsoNormal> ><br><br><o:p></o:p></p><p class=MsoNormal> > _______________________________________________<br><br><o:p></o:p></p><p class=MsoNormal> > squid-users mailing list<br><br><o:p></o:p></p><p class=MsoNormal> > <a href="mailto:squid-users@lists.squid-cache.org">squid-users@lists.squid-cache.org</a><br><br><o:p></o:p></p><p class=MsoNormal style='margin-bottom:12.0pt'> > <a href="http://lists.squid-cache.org/listinfo/squid-users">http://lists.squid-cache.org/listinfo/squid-users</a><br><br>-----BEGIN PGP SIGNATURE----- <br>Version: GnuPG v2 <br> <br>iQEcBAEBCAAGBQJXp3yAAAoJENNXIZxhPexGDEYH/AjM5hD4ahsF9IXsMo5PrWLE <br>nlKCloampXBur7o1PD2uOynU35ayV2wjJlBU9P9uQrux5lus0FCHt0yD/X9rEJ4J <br>E1AcShLUfK6ezMdSKnn5tylfTg1+4U08/hJz3+E9PGxA//peoPVovgHu/WuFXM4X <br>2NtSaVeib1O7QI2Dr0yRGZKKuJ9C2/YZqTAgjUa8rT2JZAsi5bzVU4WoaXwxklZD <br>OolueguA63dKo3n86oAR1W0jmPoXQeHvPuleYbGLUwWVBBlRJdCHpKfVnh/D+ni4 <br>w0hR0MRBZzS8dVNQTrTCCzuFoIH1VjHhYKKZaIQEgPpo7AsvtABNTxLM7MJmqRY= <br>=UX3s <br>-----END PGP SIGNATURE----- <o:p></o:p></p></div></body></html>