[squid-users] Fwd: Using Squid to redirect Steam CDNs using storeID_rewrite

[Amos Jeffries]

On 13/11/2016 12:50 p.m., Stefan Wickstrom wrote:
> Hello all,
> Apologies for the possibly incorrect format/posting of this query; I am new
> to this mode of discussion in relation to software.
> 
> I am attempting to use Squid, in combination with storeID rewrite, to
> redirect Steam CDN requests allowing multiple CDN requests to be served
> from the single Squid cache entry.

Well, firstly. Store-ID does not _redirect_ anything. It simply
de-duplicates cache objects by re-writing the location where certain
ones are stored, so it is different to where their URL would store it.

If a backend server needs to be contacted the one the clients was going
to will be contacted and the shared Store-ID location gets updated with
any new data that server produces.

> 
> /var/ipfire/proxy/advanced/acls/include.acl
> #squid.conf
> acl cacheDomain dstdomain .steampowered.com .edgesuite.net .steamstatic.com
> .steamcontent.com
> cache deny !cacheDomain
> store_id_program /usr/lib/squid/storeid_file_rewrite
> /etc/squid/storeid_rewrite.conf
> store_id_children 10 startup=3 idle=1 concurrency=0
> 
> /etc/squid/storeid_rewrite.conf
> ^http.*steam.*\.com\/(.*) http://steamupdates.squid.internal/$1
> 

I highly recommend that you make that pattern a LOT more targeted. The
presence of ".*" allows any URL that happens to include the word "steam"
and then ".com/" to have its final portion stored in your cache as "game
content".


> The issue appears to be stemming from how squid and storeID_rewrite
> interact; currently if I test the storeid_rewrite.conf with the following
> command:
> echo http://valve314.steamcontent.com/depot/255711/chunk/
> a3f17a1be9c7861cbc56d1098b8ede146e114391? | /usr/lib/squid/storeid_file_rewrite
> /etc/squid/storeid_rewrite.conf
> I get in return:
> OK store-id=http://steamupdates.squid.internal/depot/255711/chunk/
> a3f17a1be9c7861cbc56d1098b8ede146e114391?
> 

NOTE: when your log contains URLs ending in '?' check that you have
turned off the strip_query_terms mechanism before debugging. Otherwise
that significant part of the URLs will be hidden from you, and your test
results may be different from expectations.
<http://www.squid-cache.org/Doc/config/strip_query_terms/>


> This indicates that storeID rewrite is functioning and using my RegEx
> command to rewrite the URL into one that only contains the unique game and
> chunk IDs from the URL.

No, the new key contains anything that happened to follow the string
".com/".

eg.
http://haha.example.net/steam/gotcha.com/depot/255711/chunk/a3f17a1be9c7861cbc56d1098b8ede146e114391?boo


> The issue is when I test the entire system using
> the following process I see multiple entries into the squid cache for the
> specific game chunk ID:
> 
>    - Remove /var/log/squid/access.log to ensure no previous attempts will
>    appear in test
>    - Clear the cache through ipFire webUI
>    - Restart Squid cache service through ipFire webUI
>    - Download game through Steam interface
>    - Verify Squid cached the download chunks by grepping through both
>    /var/log/squid/access.log and Squid cache for specific game chunk IDs (this
>    is a spot check at best)
>    - Change Steam CDN location through Steam UI
>    - Delete Steam game local content
>    - Re-download Steam game
>    - Verify Squid cache using game chunk ID
> 
> The command used to grep against the Squid cache and it's results are as
> follows:
> squidclient -h 192.168.1.1 cache_object://192.168.1.1 mgr:objects | grep
> f37f9405e2f38417a226eac378ac3982223d2966?
> GET http://valve608.steamcontent.com/depot/26502/chunk/
> f37f9405e2f38417a226eac378ac3982223d2966?
> GET http://valve313.steamcontent.com/depot/26502/chunk/
> f37f9405e2f38417a226eac378ac3982223d2966?
> 

See the note above about strip_query_terms.

If a single byte of any of the query portion of those URLs is different
then your Store-ID keys for them will be likewise different. Since
*everything* following the ".com/" is part of the Store-ID key produced
by your pattern.


> This indicates that at some point in the process, Squid is generating a KEY
> entry for the chunk based off the original Steam CDN URL and NOT the
> RegEx'd URL storeID_rewrite is supposedly generating.

The mgr:objects report lists the client request that object was a
response to. <http://wiki.squid-cache.org/Features/CacheManager/Objects>

So the test above just indicates that objects exist which were stored
from those URLs. Client requests are of course not for the Store-ID
keys, but (one of) the actual URLs for that object.


> I have attempted to determine how Squid is generating it's KEY entries for
> the chunks it is storing, but have had no luck (basing attempts off this white
> paper entry <http://www.squid-cache.org/CacheDigest/cache-digest-v5.txt>)

That is about using digests used to inform other proxies about what is
possibly still in cache. Nothing to do with the storage itself. And
should not contain Store-ID keys (if it does that is a bug).

> At this point I've exhausted my limited knowledge of how Squid and storeID
> rewrite function and any assistance would be quite welcome; please let me
> know if there's any further info needed to try and crack this walnut open!
> 

Start with disabling strip_query_terms. Then check the Store-ID keys
actually contain only the usefully different parts of the URL(s) input.

Amos