[squid-users] Fwd: Using Squid to redirect Steam CDNs using storeID_rewrite

[Stefan Wickstrom]

Hello all,
Apologies for the possibly incorrect format/posting of this query; I am new
to this mode of discussion in relation to software.

I am attempting to use Squid, in combination with storeID rewrite, to
redirect Steam CDN requests allowing multiple CDN requests to be served
from the single Squid cache entry.
Here's a breakdown of my current versions/configurations:

OS: IPFire 2.19 (x86_64) - core103
Kernel: Linux ipfire 3.14.65-ipfire #1
<https://github.com/squid-cache/squid/pull/1> SMP Tue Jun 14 06:21:39 GMT
2016 x86_64 GNU/Linux
Squid: 3.5.19

/var/ipfire/proxy/advanced/acls/include.acl
#squid.conf
acl cacheDomain dstdomain .steampowered.com .edgesuite.net .steamstatic.com
.steamcontent.com
cache deny !cacheDomain
store_id_program /usr/lib/squid/storeid_file_rewrite
/etc/squid/storeid_rewrite.conf
store_id_children 10 startup=3 idle=1 concurrency=0

/etc/squid/storeid_rewrite.conf
^http.*steam.*\.com\/(.*) http://steamupdates.squid.internal/$1

The issue appears to be stemming from how squid and storeID_rewrite
interact; currently if I test the storeid_rewrite.conf with the following
command:
echo http://valve314.steamcontent.com/depot/255711/chunk/
a3f17a1be9c7861cbc56d1098b8ede146e114391? | /usr/lib/squid/storeid_file_rewrite
/etc/squid/storeid_rewrite.conf
I get in return:
OK store-id=http://steamupdates.squid.internal/depot/255711/chunk/
a3f17a1be9c7861cbc56d1098b8ede146e114391?

This indicates that storeID rewrite is functioning and using my RegEx
command to rewrite the URL into one that only contains the unique game and
chunk IDs from the URL. The issue is when I test the entire system using
the following process I see multiple entries into the squid cache for the
specific game chunk ID:

   - Remove /var/log/squid/access.log to ensure no previous attempts will
   appear in test
   - Clear the cache through ipFire webUI
   - Restart Squid cache service through ipFire webUI
   - Download game through Steam interface
   - Verify Squid cached the download chunks by grepping through both
   /var/log/squid/access.log and Squid cache for specific game chunk IDs (this
   is a spot check at best)
   - Change Steam CDN location through Steam UI
   - Delete Steam game local content
   - Re-download Steam game
   - Verify Squid cache using game chunk ID

The command used to grep against the Squid cache and it's results are as
follows:
squidclient -h 192.168.1.1 cache_object://192.168.1.1 mgr:objects | grep
f37f9405e2f38417a226eac378ac3982223d2966?
GET http://valve608.steamcontent.com/depot/26502/chunk/
f37f9405e2f38417a226eac378ac3982223d2966?
GET http://valve313.steamcontent.com/depot/26502/chunk/
f37f9405e2f38417a226eac378ac3982223d2966?

This indicates that at some point in the process, Squid is generating a KEY
entry for the chunk based off the original Steam CDN URL and NOT the
RegEx'd URL storeID_rewrite is supposedly generating.
I have attempted to determine how Squid is generating it's KEY entries for
the chunks it is storing, but have had no luck (basing attempts off this white
paper entry <http://www.squid-cache.org/CacheDigest/cache-digest-v5.txt>)
At this point I've exhausted my limited knowledge of how Squid and storeID
rewrite function and any assistance would be quite welcome; please let me
know if there's any further info needed to try and crack this walnut open!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20161112/3529b155/attachment.html>