[squid-users] TCP_REFRESH_UNMODIFIEDs are really MISSes

[Victor Sudakov]

Alex Rousskov wrote:
> On 10/10/2017 07:50 PM, Victor Sudakov wrote:
> 
> > When updating several almost identical FreeBSD hosts via a
> > squid-3.5.27 proxy, I expect to see lots of HITs, because the patches
> > on the update server should be identical too.
> > 
> > However, I see lots of TCP_REFRESH_UNMODIFIED lines, and only
> > occasional HIT statuses.
> 
> 
> There are many kinds of "hits":
> 
> * If your primary concern is data transfer/bandwidth usage, then you
> should focus on so called byte hits. TCP_REFRESH_UNMODIFIED is
> essentially one of those because the origin server sends a tiny
> headers-only response to Squid.

My primary concern is bandwidth usage. You are saying that the
TCP_REFRESH_UNMODIFIED transaction means that the bulk of the data is
not transferred from Webserver to Squid, only the headers are fetched?

If you look at this line:

1507630625.515    448 212.73.124.12 TCP_REFRESH_UNMODIFIED/200 980593 GET http://update6.freebsd.org/to-10.4-RELEASE/i386/bp/1195d725efce75f0db6317f507d8a9514f35613c7466673a86db23100bd6fa77-c4150431df5b14b15546b3db5e30f2bd2c35d6d52812f057dc101fab20847324 - HIER_DIRECT/198.148.79.66 application/octet-stream

There cannot be 980593 bytes of headers? What kind of gigantic header
does it make?

[dd]

> 
> > The "fetch" HTTP client the freebsd-update utility uses is rather dumb
> > and does not send IMS requests (also see below). Could someone please
> > explain what is happening here?
> 
> If the client is not sending any conditional headers, then Squid decides
> that the previously cached response has expired and needs to be revalidated.
> 
> 
> > The HTTP replies from the squid cache to the HTTP client are
> > "X-Cache-Lookup: MISS from proxy2.sibptus.ru:8080", so these
> > TCP_REFRESH_UNMODIFIED statuses seem to be really MISSes and I'm not
> > saving traffic on these updates.
> 
> According to your pasted log and HTTP messages, the GET request URL that
> resulted an "X-Cache-Lookup: MISS" response header does not match any of
> the logged URLs; even the host part of the URL does not match. 

Of course, there are just unrelated examples. There are thousands of
fetches from update*.freebsd.org

> To be sure that you are comparing apples to apples, you would need
> to log the X-Cache-Lookup response header to access.log (or enable
> ALL,2 debugging to see headers and/or collect packet traces).

I don't think the freebsd-update cares about the X-Cache-Lookup
response, I'm just trying to save bandwidth and don't quite like the
result.

-- 
Victor Sudakov,  VAS4-RIPE, VAS47-RIPN
AS43859