<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii">
<META NAME="Generator" CONTENT="MS Exchange Server version rmj.rmm.rup.rpr">
<TITLE>RE: [squid-users] Windows Updates a Caching Stub zone, A windows updates store.</TITLE>
</HEAD>
<BODY>
<!-- Converted from text/rtf format -->
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">Hey Omid,</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="he"></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">Indeed my preference is that if you can ask ask and I will try to give you couple more details on the service and the subject.</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">Windows updates are considered very static since they have Last-Modified and Date headers. </FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">Else then that, they support to validate and invalidate requests based on these times.</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">This what makes it so simple to store them.</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">Now the main issue(which you probably already know) is that clients are requesting partial content from a full static object.</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">The full file or object is a resource and when you have the full object most web services can serve the partial content.</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">Technically if the client software uses static ranges when accessing the a resource it would be very simple to "map" a range request into a specific object "ID" but the issue is that there are scenarios which the client ask for multiple ranges in the same request and everything get a bit complicated.</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="he"></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">From a cache(squid) point of view when a client runs a "fetch" operation he also populate the cache.</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">This is the most "preferred" way of handling cache population since it relies on a need which is somehow considered as required.</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">Now when you look at it, in many cases it's a bit silly and can be considered in a way simple minded when you are talking about GB's of static content.</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">When I look at MS updates I see lots of "Cache-Control: public,max-age=172800" in responses and it might be based on the assumption that the object is predicted to be a part of an update "torrent" of about 48 hours.</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">The real world is far from this header and caches needs to be smarter in order to avoid re-population and re-deletion of existing content.</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">Now since MS updates would be probably used over and over again by real clients it's sometimes would be good to just store them for a period of time.</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">For example there aren’t many Windows XP out there under paid support but if clients are still updating then it's right to have them.</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="he"></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">Now what I did was simply wrote a simple web-service which is also a forward proxy that is based on another file system rather compared to the standard ones.</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">You have a storedata directory which can be changed in the command line.</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">You have a listening port, and you also have some level of debug info.</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">The store data have three important sub directories:</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">request/v1/</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">header/v1/</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">response/v1/</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="he"></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">Since it's a simple web service that relies on</FONT></SPAN><SPAN LANG="en-us"> <FONT FACE="Calibri">a specific</FONT></SPAN><SPAN LANG="en-us"><FONT FACE="Calibri"> file system structure it doesn't have the TOS and QOS features that</FONT></SPAN><SPAN LANG="en-us"> <FONT FACE="Calibri">are in</FONT></SPAN><SPAN LANG="en-us"> <FONT FACE="Calibri">a much lower level services have.</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">Since you have full control on the web-service and the response headers are reliable you can safely use some kind of Internal response headers and be sure that MS and their CDN network will not use these and will "harm" your statistics.</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">You will just need to use the concept which was mentioned in the HIT-MISS thread from 2012:</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">acl mshit rep_header X-Company-Ms-Cache HIT</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">clientside_tos 0x30 mshit</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="he"></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">And you can get wild with the name to verify that it will be 100% unique and will not collide with the CDN headers.</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">Also you can use another term then HIT for example "INTERNAL-CACHE-SERVER" would probably not be coming from the up-stream CDN.</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">Or even add a unique ID(#uuidgen) for this service that should never be mimicked.</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="he"></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">Since it's a web-service with static header files you will just need to use the perl script which I sent you in the last email to inject these headers into the response</FONT></SPAN><SPAN LANG="en-us"><FONT FACE="Calibri"> files</FONT></SPAN><SPAN LANG="en-us"><FONT FACE="Calibri">.</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">If the store-service is only piping the connection to the up-stream services</FONT></SPAN><SPAN LANG="en-us"> <FONT FACE="Calibri">the response</FONT></SPAN><SPAN LANG="en-us"> <FONT FACE="Calibri">will</FONT></SPAN><SPAN LANG="en-us"> <FONT FACE="Calibri">not include your customized embedded headers.</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">The response headers are at:</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">/var/storedata/header/v1/*</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">The bodies are at:</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">/var/storedata/</FONT></SPAN><SPAN LANG="en-us"><FONT FACE="Calibri">response</FONT></SPAN><SPAN LANG="en-us"><FONT FACE="Calibri">/v1/*</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">Just as a side note: t</FONT></SPAN><SPAN LANG="en-us"><FONT FACE="Calibri">his store service was designed for MS updates only and using it for other services</FONT></SPAN><SPAN LANG="en-us"> <FONT FACE="Calibri">is prohibited in the code level.</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">In the mean while I will look at the TOS\QOS options if at all.</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">Eliezer</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="he"></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="he"><FONT FACE="Arial"><SPAN DIR=RTL>----</SPAN></FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">Eliezer Croitoru</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">Linux System Administrator</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">Mobile: +972-5-28704261</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">Email: eliezer@ngtech.co.il</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="he"></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">-----Original Message-----</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">From: squid-users [<A HREF="mailto:squid-users-bounces@lists.squid-cache.org">mailto:squid-users-bounces@lists.squid-cache.org</A>] On Behalf Of Omid Kosari</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">Sent: Sunday, July 17, 2016 9:34 AM</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">To: squid-users@lists.squid-cache.org</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">Subject: Re: [squid-users] Windows Updates a Caching Stub zone, A windows updates store.</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="he"></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">Dear Eliezer,</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="he"></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">Thanks for reply .</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="he"></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">1. I am trying to understand but with your description it would be simpler </FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="he"></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">2. I already use logical volumes . Silly question</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="he"></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">3. I don't want just a hit in log . I try to describe my need . Currently we</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">have exclude the cache hits (based on TOS value) from our customers reserved</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">bandwidth . For example you have 150Mbps internet link from our company and</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">we have limitation for you on our QOS routers . But we have excluded cache</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">hits from your 150M and you may have more than that if you are downloading</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">from our cache hits .</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="he"></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">qos_flows local-hit=0x90</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">qos_flows sibling-hit=0x90</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">qos_flows parent-hit=0x90</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="he"></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">But the hits from your code could not be counted . Even you may help me to</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">do that with linux iptables+squid it would be fine .</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="he"></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">Thanks</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="he"></SPAN></P>
<BR>
<P DIR=LTR><SPAN LANG="he"><FONT FACE="Arial"><SPAN DIR=RTL>--</SPAN></FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">View this message in context: <A HREF="http://squid-web-proxy-cache.1019090.n4.nabble.com/Windows-Updates-a-Caching-Stub-zone-A-windows-updates-store-tp4678454p4678530.html">http://squid-web-proxy-cache.1019090.n4.nabble.com/Windows-Updates-a-Caching-Stub-zone-A-windows-updates-store-tp4678454p4678530.html</A></FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">Sent from the Squid - Users mailing list archive at Nabble.com.</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="en-us"></SPAN><SPAN LANG="he"><FONT FACE="Arial"><SPAN DIR=RTL>_______________________________________________</SPAN></FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">squid-users mailing list</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri">squid-users@lists.squid-cache.org</FONT></SPAN></P>
<P DIR=LTR><SPAN LANG="en-us"><FONT FACE="Calibri"><A HREF="http://lists.squid-cache.org/listinfo/squid-users">http://lists.squid-cache.org/listinfo/squid-users</A></FONT></SPAN></P>
</BODY>
</HTML>