[squid-users] Windows Updates a Caching Stub zone, A windows updates store.

[Eliezer Croitoru]

Hey Omid,

Indeed my preference is that if you can ask ask and I will try to give you
couple more details on the service and the subject.
Windows updates are considered very static since they have Last-Modified and
Date headers. 
Else then that,  they support to validate and invalidate requests based on
these times.
This what makes it so simple to store them.
Now the main issue(which you probably already know) is that clients are
requesting partial content from a full static object.
The full file or object is a resource and when you have the full object most
web services can serve the partial content.
Technically if the client software uses static ranges when accessing the a
resource it would be very simple to "map" a range request into a specific
object "ID" but the issue is that there are scenarios which the client ask
for multiple ranges in the same request and everything get a bit
complicated.

>From a cache(squid) point of view when a client runs a "fetch"  operation he
also populate the cache.
This is the most "preferred" way of handling cache population since it
relies on a need which is somehow considered as required.
Now when you look at it, in many cases it's a bit silly and can be
considered in a way simple minded when you are talking about GB's of static
content.
When I look at MS updates I see lots of "Cache-Control:
public,max-age=172800" in responses and it might be based on the assumption
that the object is predicted to be a part of an update "torrent" of about 48
hours.
The real world is far from this header and caches needs to be smarter in
order to avoid re-population and re-deletion of existing content.
Now since MS updates would be probably used over and over again by real
clients it's sometimes would be good to just store them for a period of
time.
For example there aren't many Windows XP out there under paid support but if
clients are still updating then it's right to have them.

Now what I did was simply wrote a simple web-service which is also a forward
proxy that is based on another file system rather compared to the standard
ones.
You have a storedata directory which can be changed in the command line.
You have a listening port, and you also have some level of debug info.
The store data have three important sub directories:
request/v1/
header/v1/
response/v1/

Since it's a simple web service that relies on a specific file system
structure it doesn't have the TOS and QOS features that are in a much lower
level services have.
Since you have full control on the web-service and the response headers are
reliable you can safely use some kind of Internal response headers and be
sure that MS and their CDN network will not use these and will "harm" your
statistics.
You will just need to use the concept which was mentioned in the HIT-MISS
thread from 2012:
acl mshit rep_header X-Company-Ms-Cache HIT
clientside_tos 0x30 mshit

And you can get wild with the name to verify that it will be 100% unique and
will not collide with the CDN headers.
Also you can use another term then HIT for example "INTERNAL-CACHE-SERVER"
would probably not be coming from the up-stream CDN.
Or even add a unique ID(#uuidgen) for this service that should never be
mimicked.

Since it's a web-service with static header files you will just need to use
the perl script which I sent you in the last email to inject these headers
into the response files.
If the store-service is only piping the connection to the up-stream services
the response will not include your customized embedded headers.
The response headers are at:
/var/storedata/header/v1/*

The bodies are at:
/var/storedata/response/v1/*

Just as a side note: this store service was designed for MS updates only and
using it for other services is prohibited in the code level.

In the mean while I will look at the TOS\QOS options if at all.

Eliezer

----
Eliezer Croitoru
Linux System Administrator
Mobile: +972-5-28704261
Email: eliezer at ngtech.co.il


-----Original Message-----
From: squid-users [mailto:squid-users-bounces at lists.squid-cache.org] On
Behalf Of Omid Kosari
Sent: Sunday, July 17, 2016 9:34 AM
To: squid-users at lists.squid-cache.org
Subject: Re: [squid-users] Windows Updates a Caching Stub zone, A windows
updates store.

Dear Eliezer,

Thanks for reply .

1. I am trying to understand but with your description it would be simpler 

2. I already use logical volumes . Silly question

3. I don't want just a hit in log . I try to describe my need . Currently we
have exclude the cache hits (based on TOS value) from our customers reserved
bandwidth . For example you have 150Mbps internet link from our company and
we have limitation for you on our QOS routers . But we have excluded cache
hits from your 150M and you may have more than that if you are downloading
from our cache hits .

qos_flows local-hit=0x90
qos_flows sibling-hit=0x90
qos_flows parent-hit=0x90

But the hits from your code could not be counted . Even you may help me to
do that with linux iptables+squid it would be fine .

Thanks



--
View this message in context:
http://squid-web-proxy-cache.1019090.n4.nabble.com/Windows-Updates-a-Caching
-Stub-zone-A-windows-updates-store-tp4678454p4678530.html
Sent from the Squid - Users mailing list archive at Nabble.com.
_______________________________________________
squid-users mailing list
squid-users at lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20160717/d947d90c/attachment.html>