[squid-users] Cannot purge items that are not upstream anymore

[Amos Jeffries]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 13/11/2014 1:55 a.m., Hussam Al-Tayeb wrote:
> On Thursday 13 November 2014 01:39:27 Amos Jeffries wrote:
>> On 13/11/2014 12:17 a.m., Hussam Al-Tayeb wrote:
>>> Hello. I have a problem with 'squidclient -m PURGE' and also
>>> the purge command. They won't purge urls from disk that are
>>> not available online anymore or redirect to other links.
>> 
>> PURGE was designed for use in HTTP/1.0. It does not handle
>> HTTP/1.1 negotiated content / variants at all well.
>> 
>>> For example, 
>>> http://static.firedrive.com/dynamic/previews/75/27577be2d6d86af20265734b64
>>>
>>> 
e8d563.jpg
>> which corresponds to /home/squid/00/BB/0000BBC0
>> 
>>> Even "purge -e "static.firedrive.com" -c /etc/squid/squid.conf
>>> -P 0x01" reads it but will not really remove it from disk. Are
>>> such files stuck on disk forever?
>> 
>> No. Cache is a temporary storage location. Think of it as a
>> buffer in the network.
>> 
>> They will exist only until a) the storage space is needed for 
>> something else, or b) the timestamp in their Expires: header, or
>> c) an origin server informs Squid they are no longer existing.
>> 
>> PURGE method was a way to fake (c).
>> 
>>> What would the correct way to clear them?
>> 
>> By obeying HTTP protocol. HTTP has built-in mechanisms for doing
>> that automatically.
>> 
>> Or you can just delete the disk file. Squid will auto-detect the 
>> removal at some point when it needs to use or delete the object.
>> Until then it will just under-count the amount of used disk.
>> 
>> Amos
>> 
>> _______________________________________________ squid-users
>> mailing list squid-users at lists.squid-cache.org 
>> http://lists.squid-cache.org/listinfo/squid-users
> 
> 
> head /home/squid/00/BB/0000BBC0 -n20
> 
> �u��(▒��,�|�S�| 
> �S��f7�P`Uhttp://static.firedrive.com/dynamic/previews/75/27577be2d6d86af20265734b64e8d563.jpg
>
> 
R"accept-encoding="gzip,%20deflate"HTTP/1.1 200 OK

... notice and remember the accept-encoding=* value in quotes above.
We shall get back to that later.

Now for a little tale of woe...

On this Date:

> Date: Tue, 26 Aug 2014 12:25:13 GMT

... the

> Server: cloudflare-nginx

... representing static.firedrive.com has expicitly with complete
authority state that the:

> Content-Type: image/jpeg

... otherwise known as:

> ETag: "5090c137-2efb"

... should remain in cache until:

> Expires: Fri, 23 Aug 2024 12:25:13 GMT

... should remain in cache until Aug 2024.

Furthermore that:

> Last-Modified: Wed, 31 Oct 2012 06:12:07 GMT Cache-Control: public,
> max-age=315360000

... there is no need for any recipient storing the object to even
check for validity again until Oct 2022.


> Access-Control-Allow-Origin: * Access-Control-Allow-Methods: GET,
> POST, OPTIONS Access-Control-Allow-Headers:
> Origin,Referer,Accept-Encoding,Accept- 
> Language,Accept,DNT,X-Mx-ReqToken,Keep-Alive,User-Agent,X-Requested-With,If-
>
> 
Modified-Since,Cache-Control,Content-Type,X-Forwarded-For,X-Forwarded-Proto
> CF-Cache-Status: HIT Vary: Accept-Encoding Accept-Ranges: bytes 
> CF-RAY: 160002c2e4730887-FRA Content-Length: 12027 Connection:
> Keep-Alive Set-Cookie:
> __cfduid=dfcf568ed46ef827243b3d6c342b3bdc41409055913424; 
> expires=Mon, 23-Dec-2019 23:50:00 GMT; path=/;
> domain=.firedrive.com; HttpOnly

... and that the Cookies it served up shall remain on the users
machine until Dec 2019. Thereafter this object shall be served without
any Cookie.

> 
> 
> The Expires header is in the past. 
> "http://static.firedrive.com/dynamic/previews/75/27577be2d6d86af20265734b64e8d563.jpg"
>
> 
Sending HTTP request ... done.
> HTTP/1.1 404 Not Found Server: squid Mime-Version: 1.0 Date: Wed,
> 12 Nov 2014 12:54:13 GMT Content-Length: 0 X-Cache: MISS from
> SERV1 X-Cache-Lookup: NONE from SERV1:3129 Via: 1.1 SERV1 (squid) 
> Connection: close
> 
> Is that why  squidclient -m PURGE -h 127.0.0.1 -p 3129 says not
> found in cache?

The Vary: header is why your PURGE is not doing anything. It means you
have to find and send in the exactly right value for Accept-Encoding,
in order for PURGE to find and erase the actual object.

The request *not* having an Accept-Encoding header is one of the
possible variations of values. In particular it is the one which is
foudn to already be absent from your cache by your PURGE command.

This is where that quoted string up top of the object file comes in.
What is cached on disk is the object for:
squidclient -m PURGE -p 3129 -H 'Accept-Encoding: gzip, deflate\n'

The purge tool has not been updated in some years and does not support
these types of variant commonly found in HTTP/1.1 traffic.

Amos

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (MingW32)

iQEcBAEBAgAGBQJUY18/AAoJELJo5wb/XPRjEOMH+wXFvkHoSGbZMLKMZHmGfk6m
8CtmBoICMAhHsQ34vKNNn5ENZU79MuaI2OeTyDXK/xekr0/WnRmyAL+iQYRD+/B+
D9L4TJxNznx/8io5AAJ1rTU0Ua81Ey3mCgB9af7zabUICNb2pnSFx53ju8PMc6WR
FIre5qbBT+eeSxY3IHlyp61eqcvKqf1yYOnutAXSDarodvlUsydspjEMfxBYjpUT
8zAQq7cCxDb2H2jamFFaQhh4x+AR5yOEQ8jRCIhQgP8M2E0Cm6lorAVpeVS9sTv3
82DKXKNN+NaLKtiAhMMBx4LVeXmjQU6farLSz2Xj94WFLWsI7uE7fTyDzX2AaAs=
=RPQc
-----END PGP SIGNATURE-----