[squid-users] Caching mirrored origin server

[jimc]

I'm using squid-4.4-2.1.x86_64 from OpenSuSE Tumbleweed.  My goal is
that when doing periodic software updates. each host in my department
will contact my proxy to obtain the new metadata and packages (SuSE has
a syntax for this); the proxy will download each file only once.  This
sounds like pretty standard Squid operation, but there's a gross botfly
in the ointment: the origin servers return 302 Found, each time
redirecting to a different mirror, and with "normal" configuration this
result is passed back to the client which makes a new connection (via
the proxy) to that mirror, but the retrieved file will likely never be
accessed again from that mirror.

Is there anyone who really knows what Squid can do and who has an idea
how to get Squid to follow the redirects on a cache miss, and to respond
to the original URL with the cached copy, whichever mirror it came from?

https://ma.ttwagner.com/lazy-distro-mirrors-with-squid/ by Matt Wagner
(2014-04-03) has some suggestions along this line.  He fixates on one
mirror per origin server, and uses the virtual host feature to
effectively do an internal redirect to that mirror.  I also tried that
solution for a while, but mirrors come and go, and curating the mirror
selection turned out to be a reliability problem, so I gave it up.

Software package RPMs are often large, and in my application it's
important to raise the maximum_object_size; the largest item I've seen
is 158Mb and I've set the limit to 300Mb.  And of course enough disc
cache space has to be allowed to hold the expected set of RPMs to be
used, e.g. when installing the OS on several new machines.  My
cache_dir has 5000 Mb.

-- 
James F. Carter   Email: jimc at jfcarter.net
Web: http://www.math.ucla.edu/~jimc (q.v. for PGP key)

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20190102/dbbc4e01/attachment.sig>