[squid-users] Squid transparent not caching apt requests from deb.debian.org

[Amos Jeffries]

On 5/04/20 2:53 am, Alex Rousskov wrote:
> On 4/3/20 4:55 PM, zrm wrote:
>> On 4/3/20 16:34, Alex Rousskov wrote:
>>> On 4/3/20 4:26 PM, zrm wrote:
>>>> In the first case we get TCP_MISS every time because it isn't caching
>>>> the data, in the second case it's only the first time and after that we
>>>> get TCP_REFRESH_UNMODIFIED. But how and why is this happening?
> 
>> squid to apt:
>> ---------
>> X-Cache: MISS from tproxy
>> X-Cache-Lookup: MISS from tproxy:3130
> 
>> squid to wget:
>> ---------
>> X-Cache: MISS from tproxy
>> X-Cache-Lookup: MISS from tproxy:3130
> 
> The headers you have posted tell us that the object was not in Squid
> cache when apt and wget transactions started. Since wget was started
> after apt, we can speculate that apt transaction did not cache the
> object. This is consistent with your observations. I saw nothing in the
> posted headers that would explain the difference between apt and wget
> outcomes.
> 
> Unfortunately, you did not show the headers for the case where the
> object actually got cached by Squid. You can probably do that by
> repeating the wget transaction twice. I would also repeat the apt
> transaction twice after that. It would also be interesting to see the
> wget-apt and apt-wget sequences. In summary, I would do
> wget-wget-apt-apt-wget-wget. Sleep for 10+ seconds between each
> transaction to reduce chances of overlapping cache operations.
> 
> BTW, you probably do not need to make ALL,2 logs pretty -- we can figure
> out what happens based on Squid messages if you submit one transaction
> at a time and disclose transaction sequence. You can just post (a link
> to) raw (or sanitized) logs. Compress them if they are too big.
> 
> Before sharing the logs, please double check that the problem you want
> to address was reproduced during the test.
> 
> 
>> Last-Modified: Sat, 15 Jun 2019 17:46:35 GMT
>> ETag: "1389dc-58b605823fa6e"
>> Cache-Control: public, max-age=2592000
>> Content-Length: 1280476
>> Age: 4248100
> 
> FWIW: The object is 4'248'100 seconds old. The object max-age is
> 2'592'000 seconds. Your Squid is probably using an internal max-age of
> 259'200 seconds, so Squid will require cache hit revalidation during
> subsequent transactions after Squid caches the object (if it caches it).
> 
> 

One thing to notice as well is that the object delivered by the upstream
caches expired over 19 days before these tests took place:

> Cache-Control: public, max-age=2592000
> Age: 4248100

The request from Squid in both transactions was to receive content no
longer than 3 days old. The other caches ignored that requirement and
served old content from their storage, apparently without even checking
an origin whether it was stale.

Amos