[squid-users] Cannot purge items that are not upstream anymore

Wed Nov 12 14:43:04 UTC 2014

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 13/11/2014 3:04 a.m., Hussam Al-Tayeb wrote:
> On Thursday 13 November 2014 02:23:12 Amos Jeffries wrote:
>> On 13/11/2014 1:55 a.m., Hussam Al-Tayeb wrote:
>>> On Thursday 13 November 2014 01:39:27 Amos Jeffries wrote:
>>>> On 13/11/2014 12:17 a.m., Hussam Al-Tayeb wrote:
>>>>> Hello. I have a problem with 'squidclient -m PURGE' and
>>>>> also the purge command. They won't purge urls from disk
>>>>> that are not available online anymore or redirect to other
>>>>> links.
>>>> 
>>>> PURGE was designed for use in HTTP/1.0. It does not handle 
>>>> HTTP/1.1 negotiated content / variants at all well.
>>>> 
>>>>> For example, 
>>>>> http://static.firedrive.com/dynamic/previews/75/27577be2d6d86af20265734b
>>>>>
>>>>> 
64
>> 
>> e8d563.jpg
>> 
>>>> which corresponds to /home/squid/00/BB/0000BBC0
>>>> 
>>>>> Even "purge -e "static.firedrive.com" -c
>>>>> /etc/squid/squid.conf -P 0x01" reads it but will not really
>>>>> remove it from disk. Are such files stuck on disk forever?
>>>> 
>>>> No. Cache is a temporary storage location. Think of it as a 
>>>> buffer in the network.
>>>> 
>>>> They will exist only until a) the storage space is needed
>>>> for something else, or b) the timestamp in their Expires:
>>>> header, or c) an origin server informs Squid they are no
>>>> longer existing.
>>>> 
>>>> PURGE method was a way to fake (c).
>>>> 
>>>>> What would the correct way to clear them?
>>>> 
>>>> By obeying HTTP protocol. HTTP has built-in mechanisms for
>>>> doing that automatically.
>>>> 
>>>> Or you can just delete the disk file. Squid will auto-detect
>>>> the removal at some point when it needs to use or delete the
>>>> object. Until then it will just under-count the amount of
>>>> used disk.
>>>> 
>>>> Amos
>>>> 
>>>> _______________________________________________ squid-users 
>>>> mailing list squid-users at lists.squid-cache.org 
>>>> http://lists.squid-cache.org/listinfo/squid-users
>>> 
>>> head /home/squid/00/BB/0000BBC0 -n20
>>> 
>>> �u��(▒��,�|�S�| 
>>> �S��f7�P`Uhttp://static.firedrive.com/dynamic/previews/75/27577be2d6d86af2
>>>
>>> 
0265734b64e8d563.jpg
>> R"accept-encoding="gzip,%20deflate"HTTP/1.1 200 OK
>> 
>> ... notice and remember the accept-encoding=* value in quotes
>> above. We shall get back to that later.
>> 
>> Now for a little tale of woe...
>> 
>> On this Date:
>>> Date: Tue, 26 Aug 2014 12:25:13 GMT
>> 
>> ... the
>> 
>>> Server: cloudflare-nginx
>> 
>> ... representing static.firedrive.com has expicitly with
>> complete
>> 
>> authority state that the:
>>> Content-Type: image/jpeg
>> 
>> ... otherwise known as:
>>> ETag: "5090c137-2efb"
>> 
>> ... should remain in cache until:
>>> Expires: Fri, 23 Aug 2024 12:25:13 GMT
>> 
>> ... should remain in cache until Aug 2024.
>> 
>> Furthermore that:
>>> Last-Modified: Wed, 31 Oct 2012 06:12:07 GMT Cache-Control:
>>> public, max-age=315360000
>> 
>> ... there is no need for any recipient storing the object to
>> even check for validity again until Oct 2022.
>> 
>>> Access-Control-Allow-Origin: * Access-Control-Allow-Methods:
>>> GET, POST, OPTIONS Access-Control-Allow-Headers: 
>>> Origin,Referer,Accept-Encoding,Accept- 
>>> Language,Accept,DNT,X-Mx-ReqToken,Keep-Alive,User-Agent,X-Requested-With,I
>>>
>>> 
f-
>> Modified-Since,Cache-Control,Content-Type,X-Forwarded-For,X-Forwarded-Proto
>>
>>>
>> 
CF-Cache-Status: HIT Vary: Accept-Encoding Accept-Ranges: bytes
>>> CF-RAY: 160002c2e4730887-FRA Content-Length: 12027 Connection: 
>>> Keep-Alive Set-Cookie: 
>>> __cfduid=dfcf568ed46ef827243b3d6c342b3bdc41409055913424; 
>>> expires=Mon, 23-Dec-2019 23:50:00 GMT; path=/; 
>>> domain=.firedrive.com; HttpOnly
>> 
>> ... and that the Cookies it served up shall remain on the users 
>> machine until Dec 2019. Thereafter this object shall be served
>> without any Cookie.
>> 
>>> The Expires header is in the past. 
>>> "http://static.firedrive.com/dynamic/previews/75/27577be2d6d86af20265734b6
>>>
>>> 
4e8d563.jpg"
>> Sending HTTP request ... done.
>> 
>>> HTTP/1.1 404 Not Found Server: squid Mime-Version: 1.0 Date:
>>> Wed, 12 Nov 2014 12:54:13 GMT Content-Length: 0 X-Cache: MISS
>>> from SERV1 X-Cache-Lookup: NONE from SERV1:3129 Via: 1.1 SERV1
>>> (squid) Connection: close
>>> 
>>> Is that why  squidclient -m PURGE -h 127.0.0.1 -p 3129 says
>>> not found in cache?
>> 
>> The Vary: header is why your PURGE is not doing anything. It
>> means you have to find and send in the exactly right value for
>> Accept-Encoding, in order for PURGE to find and erase the actual
>> object.
>> 
>> The request *not* having an Accept-Encoding header is one of the 
>> possible variations of values. In particular it is the one which
>> is foudn to already be absent from your cache by your PURGE
>> command.
>> 
>> This is where that quoted string up top of the object file comes
>> in. What is cached on disk is the object for: squidclient -m
>> PURGE -p 3129 -H 'Accept-Encoding: gzip, deflate\n'
>> 
>> The purge tool has not been updated in some years and does not
>> support these types of variant commonly found in HTTP/1.1
>> traffic.
>> 
>> Amos
> 
> Ok, thank you for the explanation. seems wget and curl also say 
> "X-Cache: MISS from SERV1 X-Cache-Lookup: NONE from SERV1:3129"

Expected. It's hard to set exactly the right header contents with
those tools. Thus squidclient is recommended when manually doing
fine-grained testing.

> 
> So the current natural ways of purging the object is waiting till
> squid runs out of space and rotates old objects or reaching the
> Expire date of 2024, correct? Both options are ok. I just wanted to
> make sure squid is self cleaning even if things take time.

Yes, it is.

To be more precise Squid currently will revalidate objects more than 1
year old if things actually stay in cache that long. Most times the
traffic throughput is big enough to push objects out of cache after
only a month or two even when configuring huge cache_dir spaces.

Amos
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (MingW32)

iQEcBAEBAgAGBQJUY3H3AAoJELJo5wb/XPRjr3sH/15KC9dV0VlqTdS6b/QKcBOM
LhCcNOhod/qVibtX0j4W1R+FtPvW+ZZqrgl/d3+XA50PF65Bsp6UAG9Greq1Kwav
cqu1tkhV/aokLgxxi9YdUcphLqq15hGdEusx/+XwwH59sPDqR9+PpIbRnWyLwBG9
b7TGgZFjNXvK4/8CLXcOPgjm0L5ML4mo0DPf4AYPCKo6kRJ7/AWNKBt93aanQcFx
dA6UIjvdWf8T86uF5NJXSHQf4T75fTRiTAZSDIiN8Q8TogeEG1ys8IUa5XZi008d
nst0I4EopqZalbh2e4yQ1taCNpR0wjIRdbwXzQVu1WxytnyrfRql/1gunTYGq8A=
=vDUl
-----END PGP SIGNATURE-----