[squid-users] Cannot purge items that are not upstream anymore

[Hussam Al-Tayeb]

On Thursday 13 November 2014 02:23:12 Amos Jeffries wrote:
> On 13/11/2014 1:55 a.m., Hussam Al-Tayeb wrote:
> > On Thursday 13 November 2014 01:39:27 Amos Jeffries wrote:
> >> On 13/11/2014 12:17 a.m., Hussam Al-Tayeb wrote:
> >>> Hello. I have a problem with 'squidclient -m PURGE' and also
> >>> the purge command. They won't purge urls from disk that are
> >>> not available online anymore or redirect to other links.
> >> 
> >> PURGE was designed for use in HTTP/1.0. It does not handle
> >> HTTP/1.1 negotiated content / variants at all well.
> >> 
> >>> For example,
> >>> http://static.firedrive.com/dynamic/previews/75/27577be2d6d86af20265734b
> >>> 64
> 
> e8d563.jpg
> 
> >> which corresponds to /home/squid/00/BB/0000BBC0
> >> 
> >>> Even "purge -e "static.firedrive.com" -c /etc/squid/squid.conf
> >>> -P 0x01" reads it but will not really remove it from disk. Are
> >>> such files stuck on disk forever?
> >> 
> >> No. Cache is a temporary storage location. Think of it as a
> >> buffer in the network.
> >> 
> >> They will exist only until a) the storage space is needed for
> >> something else, or b) the timestamp in their Expires: header, or
> >> c) an origin server informs Squid they are no longer existing.
> >> 
> >> PURGE method was a way to fake (c).
> >> 
> >>> What would the correct way to clear them?
> >> 
> >> By obeying HTTP protocol. HTTP has built-in mechanisms for doing
> >> that automatically.
> >> 
> >> Or you can just delete the disk file. Squid will auto-detect the
> >> removal at some point when it needs to use or delete the object.
> >> Until then it will just under-count the amount of used disk.
> >> 
> >> Amos
> >> 
> >> _______________________________________________ squid-users
> >> mailing list squid-users at lists.squid-cache.org
> >> http://lists.squid-cache.org/listinfo/squid-users
> > 
> > head /home/squid/00/BB/0000BBC0 -n20
> > 
> > �u��(▒��,�|�S�|
> > �S��f7�P`Uhttp://static.firedrive.com/dynamic/previews/75/27577be2d6d86af2
> > 0265734b64e8d563.jpg
> R"accept-encoding="gzip,%20deflate"HTTP/1.1 200 OK
> 
> ... notice and remember the accept-encoding=* value in quotes above.
> We shall get back to that later.
> 
> Now for a little tale of woe...
> 
> On this Date:
> > Date: Tue, 26 Aug 2014 12:25:13 GMT
> 
> ... the
> 
> > Server: cloudflare-nginx
> 
> ... representing static.firedrive.com has expicitly with complete
> 
> authority state that the:
> > Content-Type: image/jpeg
> 
> ... otherwise known as:
> > ETag: "5090c137-2efb"
> 
> ... should remain in cache until:
> > Expires: Fri, 23 Aug 2024 12:25:13 GMT
> 
> ... should remain in cache until Aug 2024.
> 
> Furthermore that:
> > Last-Modified: Wed, 31 Oct 2012 06:12:07 GMT Cache-Control: public,
> > max-age=315360000
> 
> ... there is no need for any recipient storing the object to even
> check for validity again until Oct 2022.
> 
> > Access-Control-Allow-Origin: * Access-Control-Allow-Methods: GET,
> > POST, OPTIONS Access-Control-Allow-Headers:
> > Origin,Referer,Accept-Encoding,Accept-
> > Language,Accept,DNT,X-Mx-ReqToken,Keep-Alive,User-Agent,X-Requested-With,I
> > f-
> Modified-Since,Cache-Control,Content-Type,X-Forwarded-For,X-Forwarded-Proto
> 
> > CF-Cache-Status: HIT Vary: Accept-Encoding Accept-Ranges: bytes
> > CF-RAY: 160002c2e4730887-FRA Content-Length: 12027 Connection:
> > Keep-Alive Set-Cookie:
> > __cfduid=dfcf568ed46ef827243b3d6c342b3bdc41409055913424;
> > expires=Mon, 23-Dec-2019 23:50:00 GMT; path=/;
> > domain=.firedrive.com; HttpOnly
> 
> ... and that the Cookies it served up shall remain on the users
> machine until Dec 2019. Thereafter this object shall be served without
> any Cookie.
> 
> > The Expires header is in the past.
> > "http://static.firedrive.com/dynamic/previews/75/27577be2d6d86af20265734b6
> > 4e8d563.jpg"
> Sending HTTP request ... done.
> 
> > HTTP/1.1 404 Not Found Server: squid Mime-Version: 1.0 Date: Wed,
> > 12 Nov 2014 12:54:13 GMT Content-Length: 0 X-Cache: MISS from
> > SERV1 X-Cache-Lookup: NONE from SERV1:3129 Via: 1.1 SERV1 (squid)
> > Connection: close
> > 
> > Is that why  squidclient -m PURGE -h 127.0.0.1 -p 3129 says not
> > found in cache?
> 
> The Vary: header is why your PURGE is not doing anything. It means you
> have to find and send in the exactly right value for Accept-Encoding,
> in order for PURGE to find and erase the actual object.
> 
> The request *not* having an Accept-Encoding header is one of the
> possible variations of values. In particular it is the one which is
> foudn to already be absent from your cache by your PURGE command.
> 
> This is where that quoted string up top of the object file comes in.
> What is cached on disk is the object for:
> squidclient -m PURGE -p 3129 -H 'Accept-Encoding: gzip, deflate\n'
> 
> The purge tool has not been updated in some years and does not support
> these types of variant commonly found in HTTP/1.1 traffic.
> 
> Amos
> 
> _______________________________________________
> squid-users mailing list
> squid-users at lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users

Ok, thank you for the explanation. seems wget and curl also say 
"X-Cache: MISS from SERV1
X-Cache-Lookup: NONE from SERV1:3129"

So the current natural ways of purging the object is waiting till squid runs 
out of space and rotates old objects or reaching the Expire date of 2024, 
correct?
Both options are ok. I just wanted to make sure squid is self cleaning even if 
things take time.