<div dir="ltr"><div class="gmail_quote"><div dir="ltr">Hello all,<br>Apologies for the possibly incorrect format/posting of this query; I am new to this mode of discussion in relation to software.<br><br><p style="box-sizing:border-box;margin-bottom:16px;color:rgb(51,51,51);font-family:-apple-system,blinkmacsystemfont,"segoe ui",helvetica,arial,sans-serif,"apple color emoji","segoe ui emoji","segoe ui symbol";font-size:14px;margin-top:0px">I am attempting to use Squid, in combination with storeID rewrite, to redirect Steam CDN requests allowing multiple CDN requests to be served from the single Squid cache entry.<br style="box-sizing:border-box">Here's a breakdown of my current versions/configurations:</p><blockquote style="box-sizing:border-box;margin:0px 0px 16px;padding:0px 1em;color:rgb(119,119,119);border-left:0.25em solid rgb(221,221,221);font-family:-apple-system,blinkmacsystemfont,"segoe ui",helvetica,arial,sans-serif,"apple color emoji","segoe ui emoji","segoe ui symbol";font-size:14px"><p style="box-sizing:border-box;margin-top:0px;margin-bottom:0px">OS: IPFire 2.19 (x86_64) - core103<br style="box-sizing:border-box">Kernel: Linux ipfire 3.14.65-ipfire <a href="https://github.com/squid-cache/squid/pull/1" class="m_1541224772401716373gmail-issue-link m_1541224772401716373gmail-js-issue-link" style="box-sizing:border-box;background-color:transparent;color:rgb(64,120,192);text-decoration:none" target="_blank">#1</a> SMP Tue Jun 14 06:21:39 GMT 2016 x86_64 GNU/Linux<br style="box-sizing:border-box">Squid: 3.5.19</p></blockquote><blockquote style="box-sizing:border-box;margin:0px 0px 16px;padding:0px 1em;color:rgb(119,119,119);border-left:0.25em solid rgb(221,221,221);font-family:-apple-system,blinkmacsystemfont,"segoe ui",helvetica,arial,sans-serif,"apple color emoji","segoe ui emoji","segoe ui symbol";font-size:14px"><p style="box-sizing:border-box;margin-top:0px;margin-bottom:0px">/var/ipfire/proxy/advanced/<wbr>acls/include.acl<br style="box-sizing:border-box"><code style="box-sizing:border-box;font-family:consolas,"liberation mono",menlo,courier,monospace;font-size:11.9px;padding:0.2em 0px;margin:0px;background-color:rgba(0,0,0,0.0392157);border-radius:3px">#squid.conf</code><br style="box-sizing:border-box"><code style="box-sizing:border-box;font-family:consolas,"liberation mono",menlo,courier,monospace;font-size:11.9px;padding:0.2em 0px;margin:0px;background-color:rgba(0,0,0,0.0392157);border-radius:3px">acl cacheDomain dstdomain .<a href="http://steampowered.com" target="_blank">steampowered.com</a> .<a href="http://edgesuite.net" target="_blank">edgesuite.net</a> .<a href="http://steamstatic.com" target="_blank">steamstatic.com</a> .<a href="http://steamcontent.com" target="_blank">steamcontent.com</a></code><br style="box-sizing:border-box"><code style="box-sizing:border-box;font-family:consolas,"liberation mono",menlo,courier,monospace;font-size:11.9px;padding:0.2em 0px;margin:0px;background-color:rgba(0,0,0,0.0392157);border-radius:3px">cache deny !cacheDomain</code><br style="box-sizing:border-box"><code style="box-sizing:border-box;font-family:consolas,"liberation mono",menlo,courier,monospace;font-size:11.9px;padding:0.2em 0px;margin:0px;background-color:rgba(0,0,0,0.0392157);border-radius:3px">store_id_program /usr/lib/squid/storeid_file_<wbr>rewrite /etc/squid/storeid_rewrite.<wbr>conf</code><br style="box-sizing:border-box"><code style="box-sizing:border-box;font-family:consolas,"liberation mono",menlo,courier,monospace;font-size:11.9px;padding:0.2em 0px;margin:0px;background-color:rgba(0,0,0,0.0392157);border-radius:3px">store_id_children 10 startup=3 idle=1 concurrency=0</code></p></blockquote><blockquote style="box-sizing:border-box;margin:0px 0px 16px;padding:0px 1em;color:rgb(119,119,119);border-left:0.25em solid rgb(221,221,221);font-family:-apple-system,blinkmacsystemfont,"segoe ui",helvetica,arial,sans-serif,"apple color emoji","segoe ui emoji","segoe ui symbol";font-size:14px"><p style="box-sizing:border-box;margin-top:0px;margin-bottom:0px">/etc/squid/storeid_rewrite.<wbr>conf<br style="box-sizing:border-box"><code style="box-sizing:border-box;font-family:consolas,"liberation mono",menlo,courier,monospace;font-size:11.9px;padding:0.2em 0px;margin:0px;background-color:rgba(0,0,0,0.0392157);border-radius:3px">^http.*steam.*\.com\/(.*) <a href="http://steamupdates.squid.internal/$1" target="_blank">http://steamupdates.squid.<wbr>internal/$1</a></code></p></blockquote><p style="box-sizing:border-box;margin-top:0px;margin-bottom:16px;color:rgb(51,51,51);font-family:-apple-system,blinkmacsystemfont,"segoe ui",helvetica,arial,sans-serif,"apple color emoji","segoe ui emoji","segoe ui symbol";font-size:14px">The issue appears to be stemming from how squid and storeID_rewrite interact; currently if I test the storeid_rewrite.conf with the following command:<br style="box-sizing:border-box"><code style="box-sizing:border-box;font-family:consolas,"liberation mono",menlo,courier,monospace;font-size:11.9px;padding:0.2em 0px;margin:0px;background-color:rgba(0,0,0,0.0392157);border-radius:3px">echo <a href="http://valve314.steamcontent.com/depot/255711/chunk/a3f17a1be9c7861cbc56d1098b8ede146e114391" target="_blank">http://valve314.steamcontent.<wbr>com/depot/255711/chunk/<wbr>a3f17a1be9c7861cbc56d1098b8ede<wbr>146e114391</a>? | /usr/lib/squid/storeid_file_<wbr>rewrite /etc/squid/storeid_rewrite.<wbr>conf</code><br style="box-sizing:border-box">I get in return:<br style="box-sizing:border-box"><code style="box-sizing:border-box;font-family:consolas,"liberation mono",menlo,courier,monospace;font-size:11.9px;padding:0.2em 0px;margin:0px;background-color:rgba(0,0,0,0.0392157);border-radius:3px">OK store-id=<a href="http://steamupdates.squid.internal/depot/255711/chunk/a3f17a1be9c7861cbc56d1098b8ede146e114391" target="_blank">http://steamupdates.<wbr>squid.internal/depot/255711/<wbr>chunk/<wbr>a3f17a1be9c7861cbc56d1098b8ede<wbr>146e114391</a>?</code></p><p style="box-sizing:border-box;margin-top:0px;margin-bottom:16px;color:rgb(51,51,51);font-family:-apple-system,blinkmacsystemfont,"segoe ui",helvetica,arial,sans-serif,"apple color emoji","segoe ui emoji","segoe ui symbol";font-size:14px">This indicates that storeID rewrite is functioning and using my RegEx command to rewrite the URL into one that only contains the unique game and chunk IDs from the URL. The issue is when I test the entire system using the following process I see multiple entries into the squid cache for the specific game chunk ID:</p><ul style="box-sizing:border-box;padding-left:2em;margin-top:0px;margin-bottom:16px;color:rgb(51,51,51);font-family:-apple-system,blinkmacsystemfont,"segoe ui",helvetica,arial,sans-serif,"apple color emoji","segoe ui emoji","segoe ui symbol";font-size:14px"><li style="box-sizing:border-box;margin-left:0px">Remove /var/log/squid/access.log to ensure no previous attempts will appear in test</li><li style="box-sizing:border-box;margin-top:0.25em;margin-left:0px">Clear the cache through ipFire webUI</li><li style="box-sizing:border-box;margin-top:0.25em;margin-left:0px">Restart Squid cache service through ipFire webUI</li><li style="box-sizing:border-box;margin-top:0.25em;margin-left:0px">Download game through Steam interface</li><li style="box-sizing:border-box;margin-top:0.25em;margin-left:0px">Verify Squid cached the download chunks by grepping through both /var/log/squid/access.log and Squid cache for specific game chunk IDs (this is a spot check at best)</li><li style="box-sizing:border-box;margin-top:0.25em;margin-left:0px">Change Steam CDN location through Steam UI</li><li style="box-sizing:border-box;margin-top:0.25em;margin-left:0px">Delete Steam game local content</li><li style="box-sizing:border-box;margin-top:0.25em;margin-left:0px">Re-download Steam game</li><li style="box-sizing:border-box;margin-top:0.25em;margin-left:0px">Verify Squid cache using game chunk ID</li></ul><p style="box-sizing:border-box;margin-top:0px;margin-bottom:16px;color:rgb(51,51,51);font-family:-apple-system,blinkmacsystemfont,"segoe ui",helvetica,arial,sans-serif,"apple color emoji","segoe ui emoji","segoe ui symbol";font-size:14px">The command used to grep against the Squid cache and it's results are as follows:<br style="box-sizing:border-box"><code style="box-sizing:border-box;font-family:consolas,"liberation mono",menlo,courier,monospace;font-size:11.9px;padding:0.2em 0px;margin:0px;background-color:rgba(0,0,0,0.0392157);border-radius:3px">squidclient -h 192.168.1.1 cache_object://<a href="http://192.168.1.1" target="_blank">192.168.1.1</a> mgr:objects | grep f37f9405e2f38417a226eac378ac39<wbr>82223d2966?</code><br style="box-sizing:border-box"><code style="box-sizing:border-box;font-family:consolas,"liberation mono",menlo,courier,monospace;font-size:11.9px;padding:0.2em 0px;margin:0px;background-color:rgba(0,0,0,0.0392157);border-radius:3px">GET <a href="http://valve608.steamcontent.com/depot/26502/chunk/f37f9405e2f38417a226eac378ac3982223d2966" target="_blank">http://valve608.steamcontent.<wbr>com/depot/26502/chunk/<wbr>f37f9405e2f38417a226eac378ac39<wbr>82223d2966</a>?</code><br style="box-sizing:border-box"><code style="box-sizing:border-box;font-family:consolas,"liberation mono",menlo,courier,monospace;font-size:11.9px;padding:0.2em 0px;margin:0px;background-color:rgba(0,0,0,0.0392157);border-radius:3px">GET <a href="http://valve313.steamcontent.com/depot/26502/chunk/f37f9405e2f38417a226eac378ac3982223d2966" target="_blank">http://valve313.steamcontent.<wbr>com/depot/26502/chunk/<wbr>f37f9405e2f38417a226eac378ac39<wbr>82223d2966</a>?</code></p><p style="box-sizing:border-box;margin-top:0px;color:rgb(51,51,51);font-family:-apple-system,blinkmacsystemfont,"segoe ui",helvetica,arial,sans-serif,"apple color emoji","segoe ui emoji","segoe ui symbol";font-size:14px;margin-bottom:0px">This indicates that at some point in the process, Squid is generating a KEY entry for the chunk based off the original Steam CDN URL and NOT the RegEx'd URL storeID_rewrite is supposedly generating.<br style="box-sizing:border-box">I have attempted to determine how Squid is generating it's KEY entries for the chunks it is storing, but have had no luck (basing attempts off this <a href="http://www.squid-cache.org/CacheDigest/cache-digest-v5.txt" style="box-sizing:border-box;background-color:transparent;color:rgb(64,120,192);text-decoration:none" target="_blank">white paper entry</a>)<br style="box-sizing:border-box">At this point I've exhausted my limited knowledge of how Squid and storeID rewrite function and any assistance would be quite welcome; please let me know if there's any further info needed to try and crack this walnut open!</p></div>
</div><br></div>