[squid-users] TCP_REFRESH_UNMODIFIEDs are really MISSes

Wed Oct 11 06:08:22 UTC 2017

On 10/10/2017 07:50 PM, Victor Sudakov wrote:

> When updating several almost identical FreeBSD hosts via a
> squid-3.5.27 proxy, I expect to see lots of HITs, because the patches
> on the update server should be identical too.
> 
> However, I see lots of TCP_REFRESH_UNMODIFIED lines, and only
> occasional HIT statuses.

There are many kinds of "hits":

* If your primary concern is data transfer/bandwidth usage, then you
should focus on so called byte hits. TCP_REFRESH_UNMODIFIED is
essentially one of those because the origin server sends a tiny
headers-only response to Squid.

* If your primary concern is latency, then TCP_REFRESH_UNMODIFIED should
be treated as a miss because Squid did talk to the origin server.

You may find more relevant information and discussions if you search
around for pages containing phrases like "byte hit ratio" and "document
hit ratio".

> TCP_REFRESH_UNMODIFIED status means that there was an IMS request
> involved. But I don't understand who made this IMS request: client to
> squid or squid to the webserver.

If my quick reading of the code is correct, then the
TCP_REFRESH_UNMODIFIED status itself does not distinguish those three
cases: It could be Squid; it could be client; and it could be both. The
only thing we can say based on the status alone is that Squid decided
that the cached content was stale. Many factors can contribute to that
decision.

> The "fetch" HTTP client the freebsd-update utility uses is rather dumb
> and does not send IMS requests (also see below). Could someone please
> explain what is happening here?

If the client is not sending any conditional headers, then Squid decides
that the previously cached response has expired and needs to be revalidated.

> The HTTP replies from the squid cache to the HTTP client are
> "X-Cache-Lookup: MISS from proxy2.sibptus.ru:8080", so these
> TCP_REFRESH_UNMODIFIED statuses seem to be really MISSes and I'm not
> saving traffic on these updates.

According to your pasted log and HTTP messages, the GET request URL that
resulted an "X-Cache-Lookup: MISS" response header does not match any of
the logged URLs; even the host part of the URL does not match. To be
sure that you are comparing apples to apples, you would need to log the
X-Cache-Lookup response header to access.log (or enable ALL,2 debugging
to see headers and/or collect packet traces).

HTH,

Alex.

> 1507630624.673    389 212.73.124.12 TCP_REFRESH_UNMODIFIED/200 572 GET http://update6.freebsd.org/to-10.4-RELEASE/i386/bp/82a25f03517d9d65b3c0af22f77ed9cbe94d09f1af6f4cd835a93b5f191122dc-63836fc812fc7b03a2ad1d2df9c59dde3178e1eaf4d0e32342a8a04a7121ac6d - HIER_DIRECT/198.148.79.66 application/octet-stream
> 1507630625.067    394 212.73.124.12 TCP_REFRESH_UNMODIFIED/200 574 GET http://update6.freebsd.org/to-10.4-RELEASE/i386/bp/584713682ac3cc86fd2bfb7edbb9030aa48efbf027465ebcce705864310140af-0981c929f128ad067d3c8c577f9c84302523e24548b561edceee28a3357a5007 - HIER_DIRECT/198.148.79.66 application/octet-stream
> 1507630625.515    448 212.73.124.12 TCP_REFRESH_UNMODIFIED/200 980593 GET http://update6.freebsd.org/to-10.4-RELEASE/i386/bp/1195d725efce75f0db6317f507d8a9514f35613c7466673a86db23100bd6fa77-c4150431df5b14b15546b3db5e30f2bd2c35d6d52812f057dc101fab20847324 - HIER_DIRECT/198.148.79.66 application/octet-stream
> 1507630625.914    399 212.73.124.12 TCP_REFRESH_UNMODIFIED/200 532 GET http://update6.freebsd.org/to-10.4-RELEASE/i386/bp/aac782643cbfaad218bf72f037624af304d353f2a12fb6a5eeb117cdd6c9bc27-2f6df820f68859d2e86a8f45777a81a7186477048893456f891fd91dae19af3b - HIER_DIRECT/198.148.79.66 application/octet-stream
> 1507630625.948     33 212.73.124.12 TCP_HIT/200 980593 GET http://update6.freebsd.org/to-10.4-RELEASE/i386/bp/1195d725efce75f0db6317f507d8a9514f35613c7466673a86db23100bd6fa77-c4150431df5b14b15546b3db5e30f2bd2c35d6d52812f057dc101fab20847324 - HIER_NONE/- application/octet-stream
> 1507630626.354    406 212.73.124.12 TCP_REFRESH_UNMODIFIED/200 531 GET http://update6.freebsd.org/to-10.4-RELEASE/i386/bp/58764b03f019f1aa9e28e24ddb9e74abf323cfc3984bd899accf7b15bb2a6462-6d53fae808e6d79c6c251f6b1fe5e26fbd84e191f84900bcf01931d49855249f - HIER_DIRECT/198.148.79.66 application/octet-stream
> 1507630626.391     36 212.73.124.12 TCP_HIT/200 980593 GET http://update6.freebsd.org/to-10.4-RELEASE/i386/bp/1195d725efce75f0db6317f507d8a9514f35613c7466673a86db23100bd6fa77-c4150431df5b14b15546b3db5e30f2bd2c35d6d52812f057dc101fab20847324 - HIER_NONE/- application/octet-stream
> 1507630626.391      0 212.73.124.12 TCP_MEM_HIT/200 527 GET http://update6.freebsd.org/to-10.4-RELEASE/i386/bp/1af4cec823d8b1f42c3b8a58ec0da40bfc2f97efacd92e7758096eba081e08db-2ac8890502bc272d65f2b7dd73fa7def11fa4d3feb2005b82b58ba43747b57e4 - HIER_NONE/- application/octet-stream
> 1507630626.785    393 212.73.124.12 TCP_REFRESH_UNMODIFIED/200 697 GET http://update6.freebsd.org/to-10.4-RELEASE/i386/bp/6c0eae2981826e2033797003f53aedba9adb7e317c936f6923912e5df2627216-341717a9aa21a95b7f9f83e29ac3de1cac6b9d58700ece767e4ab7cd028105c9 - HIER_DIRECT/198.148.79.66 application/octet-stream
> 1507630627.184    399 212.73.124.12 TCP_REFRESH_UNMODIFIED/200 1001 GET http://update6.freebsd.org/to-10.4-RELEASE/i386/bp/6f486e6baa2d057fa16d3fa43e666917ab9d045de2e59a7d05ace758c7d03373-653636793fd95f406da85e1375acd38f31b6c39c404b18cd8e0a641a2deb6065 - HIER_DIRECT/198.148.79.66 application/octet-stream
> 1507630627.577    392 212.73.124.12 TCP_REFRESH_UNMODIFIED/200 674 GET http://update6.freebsd.org/to-10.4-RELEASE/i386/bp/3c78173d804b5299f24199a125b3c52145c7e7680c97debb051d823526ae0579-595c12cd31ef1c4d084c308b7278d28bbb844873da246d88fb02db90f90b88b9 - HIER_DIRECT/198.148.79.66 application/octet-stream
> 1507630627.967    389 212.73.124.12 TCP_REFRESH_UNMODIFIED/200 1193 GET http://update6.freebsd.org/to-10.4-RELEASE/i386/bp/cc4d2e70a2c12900362eba48bae5fa8388bfd8ed9975246578d5c68fd00fb9b3-74473c002b6e269ddb7b3c385cfff0f43b4fcbf1dcba7f36dac7c7918b19f8a9 - HIER_DIRECT/198.148.79.66 application/octet-stream
> 1507630628.357    391 212.73.124.12 TCP_REFRESH_UNMODIFIED/200 29745 GET http://update6.freebsd.org/to-10.4-RELEASE/i386/bp/2f0861149594653c2b91faec675aef008728c5ba13b352ca3877714a40bffbd6-e6b495c0d23d7eb5fe43bb07e5a8a71f3bad37844a0e8a121ef2178e5fd6ca4a - HIER_DIRECT/198.148.79.66 application/octet-stream
> 
> 
> GET http://update.FreeBSD.org/to-10.4-RELEASE/i386/bp/4b363ee1c1e681fe5a67dd7201062ee1977f1437ce9de7c7bc9c9b84839f10df-6dc4edca0bcab3f4b0ae59d1a2714aeb0df22fce043c8897079ab9914fdafbf4 HTTP/1.1
> Host: update.FreeBSD.org
> User-Agent: freebsd-update (upgrade, 10.3-RELEASE-p20)
> Connection: Keep-Alive
> 
> HTTP/1.1 200 OK
> Date: Wed, 11 Oct 2017 01:38:16 GMT
> Server: Apache/2.2.16 (FreeBSD) mod_ssl/2.2.16 OpenSSL/0.9.8n DAV/2
> Last-Modified: Sun, 01 Oct 2017 19:32:05 GMT
> ETag: "1f0c0de-951-55a81501e5740"
> Accept-Ranges: bytes
> Content-Length: 2385
> Content-Type: text/plain
> X-Cache: MISS from proxy2.sibptus.ru
> X-Cache-Lookup: MISS from proxy2.sibptus.ru:8080
> Via: 1.1 proxy2.sibptus.ru (squid/3.5.27)
> Connection: keep-alive
> 
> BSDIFF40.......
> 