[squid-users] Getting the full file content on a range request, but not on EVERY get ...

[Hans-Peter Jansen]

Hi Heiler,

On Donnerstag, 12. Mai 2016 13:28:00 Heiler Bemerguy wrote:
> Hi Pete, thanks for replying... let me see if I got it right..
> 
> Will I need to specify every url/domain I want it to act on ? I want
> squid to do it for every range-request downloads that should/would be
> cached (based on other rules, pattern_refreshs etc)

Yup, that's right. At least, that's the common approach to deal with CDNs.
I think, that disallowing range requests is too drastic to work fine on the 
long run, but let us know, if you get to satisfactory solution this way.

> It doesn't need to delay any downloads as long as it isn't a dupe of
> what's already being downloaded.....

You can set to delay to zero of course.

This is only one side of the issues with CDNs. The other, more problematic 
side of it is, that many server with different URLs provide the same files.
Every new address will result in a new download of otherwise identical 
content.
 
Here's an example of openSUSE:

#
# this file was generated by gen_openSUSE_dedups
# from http://mirrors.opensuse.org/list/all.html
# with timestamp Thu, 12 May 2016 05:30:18 +0200
#
[openSUSE]
match:
    # openSUSE Headquarter
    http\:\/\/[a-z0-9]+\.opensuse\.org\/(.*)
    # South Africa (za)
    http\:\/\/ftp\.up\.ac\.za\/mirrors\/opensuse\/opensuse\/(.*)
    # Bangladesh (bd)
    http\:\/\/mirror\.dhakacom\.com\/opensuse\/(.*)
    http\:\/\/mirrors\.ispros\.com\.bd\/opensuse\/(.*)
    # China (cn)
    http\:\/\/mirror\.bjtu\.edu\.cn\/opensuse\/(.*)
    http\:\/\/fundawang\.lcuc\.org\.cn\/opensuse\/(.*)
    http\:\/\/mirrors\.tuna\.tsinghua\.edu\.cn\/opensuse\/(.*)
    http\:\/\/mirrors\.skyshe\.cn\/opensuse\/(.*)
    http\:\/\/mirrors\.hust\.edu\.cn\/opensuse\/(.*)
    http\:\/\/c\.mirrors\.lanunion\.org\/opensuse\/(.*)
    http\:\/\/mirrors\.hustunique\.com\/opensuse\/(.*)
    http\:\/\/mirrors\.sohu\.com\/opensuse\/(.*)
    http\:\/\/mirrors\.ustc\.edu\.cn\/opensuse\/(.*)
    # Hong Kong (hk)
    http\:\/\/mirror\.rackspace\.hk\/openSUSE\/(.*)
    # Indonesia (id)
    http\:\/\/mirror\.linux\.or\.id\/linux\/opensuse\/(.*)
    http\:\/\/buaya\.klas\.or\.id\/opensuse\/(.*)
    http\:\/\/kartolo\.sby\.datautama\.net\.id\/openSUSE\/(.*)
    http\:\/\/opensuse\.idrepo\.or\.id\/opensuse\/(.*)
    http\:\/\/mirror\.unej\.ac\.id\/opensuse\/(.*)
    http\:\/\/download\.opensuse\.or\.id\/(.*)
    http\:\/\/repo\.ugm\.ac\.id\/opensuse\/(.*)
    http\:\/\/dl2\.foss\-id\.web\.id\/opensuse\/(.*)
    # Israel (il)
    http\:\/\/mirror\.isoc\.org\.il\/pub\/opensuse\/(.*)
	
	[...] -> this list contains about 180 entries

replace: http://download.opensuse.org.%(intdomain)s/\1
# fetch all redirected objects explicitly
fetch: true


This is, how CDNs work, but it's a nightmare for caching proxies.
In such scenarios squid_dedup comes to rescue.

Cheers,
Pete