[squid-users] Fwd: Using Squid to redirect Steam CDNs using storeID_rewrite
squid3 at treenet.co.nz
Sun Nov 13 14:26:06 UTC 2016
On 13/11/2016 12:50 p.m., Stefan Wickstrom wrote:
> Hello all,
> Apologies for the possibly incorrect format/posting of this query; I am new
> to this mode of discussion in relation to software.
> I am attempting to use Squid, in combination with storeID rewrite, to
> redirect Steam CDN requests allowing multiple CDN requests to be served
> from the single Squid cache entry.
Well, firstly. Store-ID does not _redirect_ anything. It simply
de-duplicates cache objects by re-writing the location where certain
ones are stored, so it is different to where their URL would store it.
If a backend server needs to be contacted the one the clients was going
to will be contacted and the shared Store-ID location gets updated with
any new data that server produces.
> acl cacheDomain dstdomain .steampowered.com .edgesuite.net .steamstatic.com
> cache deny !cacheDomain
> store_id_program /usr/lib/squid/storeid_file_rewrite
> store_id_children 10 startup=3 idle=1 concurrency=0
> ^http.*steam.*\.com\/(.*) http://steamupdates.squid.internal/$1
I highly recommend that you make that pattern a LOT more targeted. The
presence of ".*" allows any URL that happens to include the word "steam"
and then ".com/" to have its final portion stored in your cache as "game
> The issue appears to be stemming from how squid and storeID_rewrite
> interact; currently if I test the storeid_rewrite.conf with the following
> echo http://valve314.steamcontent.com/depot/255711/chunk/
> a3f17a1be9c7861cbc56d1098b8ede146e114391? | /usr/lib/squid/storeid_file_rewrite
> I get in return:
> OK store-id=http://steamupdates.squid.internal/depot/255711/chunk/
NOTE: when your log contains URLs ending in '?' check that you have
turned off the strip_query_terms mechanism before debugging. Otherwise
that significant part of the URLs will be hidden from you, and your test
results may be different from expectations.
> This indicates that storeID rewrite is functioning and using my RegEx
> command to rewrite the URL into one that only contains the unique game and
> chunk IDs from the URL.
No, the new key contains anything that happened to follow the string
> The issue is when I test the entire system using
> the following process I see multiple entries into the squid cache for the
> specific game chunk ID:
> - Remove /var/log/squid/access.log to ensure no previous attempts will
> appear in test
> - Clear the cache through ipFire webUI
> - Restart Squid cache service through ipFire webUI
> - Download game through Steam interface
> - Verify Squid cached the download chunks by grepping through both
> /var/log/squid/access.log and Squid cache for specific game chunk IDs (this
> is a spot check at best)
> - Change Steam CDN location through Steam UI
> - Delete Steam game local content
> - Re-download Steam game
> - Verify Squid cache using game chunk ID
> The command used to grep against the Squid cache and it's results are as
> squidclient -h 192.168.1.1 cache_object://192.168.1.1 mgr:objects | grep
> GET http://valve608.steamcontent.com/depot/26502/chunk/
> GET http://valve313.steamcontent.com/depot/26502/chunk/
See the note above about strip_query_terms.
If a single byte of any of the query portion of those URLs is different
then your Store-ID keys for them will be likewise different. Since
*everything* following the ".com/" is part of the Store-ID key produced
by your pattern.
> This indicates that at some point in the process, Squid is generating a KEY
> entry for the chunk based off the original Steam CDN URL and NOT the
> RegEx'd URL storeID_rewrite is supposedly generating.
The mgr:objects report lists the client request that object was a
response to. <http://wiki.squid-cache.org/Features/CacheManager/Objects>
So the test above just indicates that objects exist which were stored
from those URLs. Client requests are of course not for the Store-ID
keys, but (one of) the actual URLs for that object.
> I have attempted to determine how Squid is generating it's KEY entries for
> the chunks it is storing, but have had no luck (basing attempts off this white
> paper entry <http://www.squid-cache.org/CacheDigest/cache-digest-v5.txt>)
That is about using digests used to inform other proxies about what is
possibly still in cache. Nothing to do with the storage itself. And
should not contain Store-ID keys (if it does that is a bug).
> At this point I've exhausted my limited knowledge of how Squid and storeID
> rewrite function and any assistance would be quite welcome; please let me
> know if there's any further info needed to try and crack this walnut open!
Start with disabling strip_query_terms. Then check the Store-ID keys
actually contain only the usefully different parts of the URL(s) input.
More information about the squid-users