<HTML><HEAD>
<STYLE id=eMClientCss>BLOCKQUOTE.cite {
PADDING-LEFT: 10px; MARGIN-LEFT: 5px; BORDER-LEFT: #cccccc 1px solid; PADDING-RIGHT: 0px; MARGIN-RIGHT: 0px
}
BLOCKQUOTE.cite2 {
PADDING-TOP: 0px; PADDING-LEFT: 10px; MARGIN-LEFT: 5px; BORDER-LEFT: #cccccc 1px solid; MARGIN-TOP: 3px; PADDING-RIGHT: 0px; MARGIN-RIGHT: 0px
}
.plain PRE {
FONT-SIZE: 100%; FONT-FAMILY: monospace; FONT-WEIGHT: normal; FONT-STYLE: normal
}
.plain TT {
FONT-SIZE: 100%; FONT-FAMILY: monospace; FONT-WEIGHT: normal; FONT-STYLE: normal
}
A IMG {
BORDER-TOP: 0px; BORDER-RIGHT: 0px; BORDER-BOTTOM: 0px; BORDER-LEFT: 0px
}
#x5f5ca13579ea4269aa2f4def7365716f {
FONT-SIZE: 12pt; FONT-FAMILY: Calibri
}
#xbde894fa6437418ea710ca906527dfa0 {
FONT-SIZE: 12pt; FONT-FAMILY: Calibri
}
.plain PRE {
FONT-SIZE: 12pt; FONT-FAMILY: Calibri
}
.plain TT {
FONT-SIZE: 12pt; FONT-FAMILY: Calibri
}
BODY {
FONT-SIZE: 12pt; FONT-FAMILY: Calibri
}
</STYLE>
<STYLE>#x5f5ca13579ea4269aa2f4def7365716f BLOCKQUOTE.cite
{PADDING-LEFT: 10px; MARGIN-LEFT: 5px; BORDER-LEFT: #cccccc 1px solid; PADDING-RIGHT: 0px; MARGIN-RIGHT: 0px}
#x5f5ca13579ea4269aa2f4def7365716f BLOCKQUOTE.cite2
{PADDING-TOP: 0px; PADDING-LEFT: 10px; MARGIN-LEFT: 5px; BORDER-LEFT: #cccccc 1px solid; MARGIN-TOP: 3px; PADDING-RIGHT: 0px; MARGIN-RIGHT: 0px}
#x5f5ca13579ea4269aa2f4def7365716f .plain PRE, #x5f5ca13579ea4269aa2f4def7365716f .plain TT
{FONT-SIZE: 100%; FONT-FAMILY: monospace; FONT-WEIGHT: normal; FONT-STYLE: normal}
#x5f5ca13579ea4269aa2f4def7365716f A IMG
{BORDER-TOP: 0px; BORDER-RIGHT: 0px; BORDER-BOTTOM: 0px; BORDER-LEFT: 0px}
#x5f5ca13579ea4269aa2f4def7365716f .plain PRE, #x5f5ca13579ea4269aa2f4def7365716f .plain TT, #x5f5ca13579ea4269aa2f4def7365716f
{FONT-SIZE: 12pt; FONT-FAMILY: Calibri}
</STYLE>
</HEAD>
<BODY scroll=auto class>
<DIV><STRONG>Hi,</STRONG></DIV>
<DIV id=x5f5ca13579ea4269aa2f4def7365716f>
<DIV>I'm trying to deal with dynamic content to be cached by Squid 3.5 (i tried many other version of squid e.g 2.7, 3.1, 3.4). By Dynamic I mean the URL for the actual content is always change this results in the wastage of Cache storage and low hit rate. As per my understanding I have two challenges atm:</DIV>
<DIV> </DIV>
<DIV>1- Websites with dynamic URL for requested content (e.g filehippo, download.com etc etc)</DIV>
<DIV>2- Streaming web sites where the dynamic URL has 206 (partial content) tune.pk videos for e.g or windows updates (enabling range off set limit to -1 causes havoc on upstream to we kept it disable is there some way to control the behavior ?)</DIV>
<DIV> </DIV>
<DIV>If someone has successfully configured the above scenario please help me out as i dont have programming background to deal with this complexity.</DIV>
<DIV> </DIV>
<DIV>I tried using different store-ID helpers but no saving on upstream the content is still coming from origin. Below is the helper i have used:</DIV>
<DIV> </DIV>
<DIV><SPAN id=xbde894fa6437418ea710ca906527dfa0>
<DIV><STRONG><U><FONT size=2 face="Courier New">My Setup Summary:</FONT></U></STRONG></DIV>
<DIV><FONT size=2 face="Courier New">Centos 6.5</FONT></DIV>
<DIV><FONT size=2 face="Courier New">Tproxy </FONT></DIV>
<DIV><FONT size=2 face="Courier New">Single Ethernet</FONT></DIV>
<DIV><FONT size=2 face="Courier New">Squid v3.5.16 (yum installed from repo)</FONT></DIV>
<DIV><FONT size=2 face="Courier New"> </FONT></DIV>
<DIV><FONT size=2 face="Courier New">Squid Cache: Version 3.5.16<BR>Service Name: squid<BR>configure options: '--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu' '--target=x86_64-redhat-linux-gnu' '--program-prefix=' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64' '--libexecdir=/usr/libexec' '--sharedstatedir=/var/lib' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--verbose' '--exec_prefix=/usr' '--libexecdir=/usr/lib64/squid' '--localstatedir=/var' '--datadir=/usr/share/squid' '--sysconfdir=/etc/squid' '--with-logdir=$(localstatedir)/log/squid' '--with-pidfile=$(localstatedir)/run/squid.pid' '--disable-dependency-tracking' '--enable-follow-x-forwarded-for' '--enable-auth' '--enable-auth-basic=DB,LDAP,NCSA,NIS,PAM,POP3,RADIUS,SASL,SMB,getpwnam' '--enable-auth-ntlm=smb_lm,fake' '--enable-auth-digest=file,LDAP' '--enable-auth-negotiate=kerberos,wrapper' '--enable-external-acl-helpers=wbinfo_group,kerberos_ldap_group' '--enable-cache-digests' '--enable-cachemgr-hostname=localhost' '--enable-delay-pools' '--enable-epoll' '--enable-icap-client' '--enable-ident-lookups' '--enable-linux-netfilter' '--enable-removal-policies=heap,lru' '--enable-snmp' '--enable-storeio=aufs,diskd,ufs,rock' '--enable-wccpv2' '--enable-esi' '--enable-ssl-crtd' '--enable-icmp' '--with-aio' '--with-default-user=squid' '--with-filedescriptors=16384' '--with-dl' '--with-openssl' '--with-pthreads' '--with-included-ltdl' '--disable-arch-native' '--without-nettle' 'build_alias=x86_64-redhat-linux-gnu' 'host_alias=x86_64-redhat-linux-gnu' 'target_alias=x86_64-redhat-linux-gnu' 'CFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic' 'CXXFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -fPIC' 'PKG_CONFIG_PATH=:/usr/lib64/pkgconfig:/usr/share/pkgconfig' --enable-ltdl-convenience<BR></FONT></DIV>
<DIV><FONT size=2 face="Courier New"> </FONT></DIV></SPAN></DIV>
<DIV><FONT size=2 face="Courier New">-----------------------------------------------</FONT></DIV>
<DIV><FONT size=2 face="Courier New">Store ID Helper</FONT></DIV>
<DIV><FONT size=2 face="Courier New">===============================================</FONT></DIV>
<DIV><FONT size=2 face="Courier New">#!/usr/bin/perl<BR># Improved\Converted StoreID helper based on perl and not on ruby<BR># this helper dosn't have my logic about youtube and other StoreID<BR># but it still do the trick in most cases and for my understanding.<BR># if you do want to understand the script logic rather then just<BR># use it try to look at: </FONT><A href="http://www1.ngtech.co.il/paste/1016/"><FONT size=2 face="Courier New">http://www1.ngtech.co.il/paste/1016/</FONT></A><BR><FONT size=2 face="Courier New"># Eliezer Croitoru eliezer<at>ngtech.co.il</FONT></DIV>
<DIV><FONT size=2 face="Courier New"></FONT> </DIV>
<DIV><FONT size=2 face="Courier New">$|=1;<BR>while (<>) {<BR> chomp;<BR> @X = split;<BR> if (@X[0] =~ m/^(exit|quit|x|q)/) {<BR> print STDERR "quiting helper quietly\n";<BR> exit 0;<BR> }</FONT></DIV>
<DIV><FONT size=2 face="Courier New"></FONT> </DIV>
<DIV><BR><FONT size=2 face="Courier New">if ($X[0] =~ m/^http\:\/\/.*(youtube|google).*(videoplayback|liveplay).*/){<BR> @itag = m/[&?](itag=[0-9]*)/;<BR> @id = m/[&?](id=[^\&]*)/;<BR> @range = m/[&?](range=[^\&\s]*)/;<BR> @begin = m/[&?](begin=[^\&\s]*)/;<BR> @redirect = m/[&?](redirect_counter=[^\&]*)/;<BR> $out="</FONT><A href="http://video-srv.youtube.com.squid.internal/@id&@itag&@range@begin@redirect"><FONT size=2 face="Courier New">http://video-srv.youtube.com.squid.internal/@id&@itag&@range@begin@redirect</FONT></A><FONT size=2 face="Courier New">";</FONT></DIV>
<DIV><FONT size=2 face="Courier New"></FONT> </DIV>
<DIV><FONT size=2 face="Courier New">} elsif ($X[0] =~ m/^http\:\/\/.*(profile|photo|creative).*\.ak\.fbcdn\.net\/((h|)(profile|photos)-ak-)(snc|ash|prn)[0-9]?(.*)/) {<BR> $out="</FONT><A href="http://fbcdn.net.squid.internal/"><FONT size=2 face="Courier New">http://fbcdn.net.squid.internal/</FONT></A><FONT size=2 face="Courier New">" . $2 . "fb" . $6 ;</FONT></DIV>
<DIV><FONT size=2 face="Courier New"></FONT> </DIV>
<DIV><FONT size=2 face="Courier New">} elsif ($X[0] =~ m/^http:\/\/i[1-4]\.ytimg\.com\/(.*)/) {<BR> $out="</FONT><A href="http://ytimg.com.squid.internal/"><FONT size=2 face="Courier New">http://ytimg.com.squid.internal/</FONT></A><FONT size=2 face="Courier New">" . $1 ;</FONT></DIV>
<DIV><FONT size=2 face="Courier New"></FONT> </DIV>
<DIV><FONT size=2 face="Courier New">} elsif ($X[0] =~ m/^http:\/\/.*\.dl\.sourceforge\.net\/(.*)/) {<BR> $out="</FONT><A href="http://dl.sourceforge.net.squid.internal/"><FONT size=2 face="Courier New">http://dl.sourceforge.net.squid.internal/</FONT></A><FONT size=2 face="Courier New">" . $1 ;</FONT></DIV>
<DIV><FONT size=2 face="Courier New"></FONT> </DIV>
<DIV><FONT size=2 face="Courier New"> #Speedtest<BR>} elsif ($X[0] =~ m/^http\:\/\/.*\/speedtest\/(.*\.(jpg|txt)).*/) {<BR> $out="</FONT><A href="http://speedtest.squid.internal/"><FONT size=2 face="Courier New">http://speedtest.squid.internal/</FONT></A><FONT size=2 face="Courier New">" . $1 ;</FONT></DIV>
<DIV><FONT size=2 face="Courier New"></FONT> </DIV>
<DIV><FONT size=2 face="Courier New"> #BLOGSPOT<BR>} elsif ($X[0] =~ m/^http:\/\/[1-4]\.bp\.(blogspot\.com.*)/) {<BR> $out="</FONT><A href="http://blog-cdn"><FONT size=2 face="Courier New">http://blog-cdn</FONT></A><FONT size=2 face="Courier New">." . $1 ;</FONT></DIV>
<DIV><FONT size=2 face="Courier New"></FONT> </DIV>
<DIV><FONT size=2 face="Courier New"> #AVAST<BR>} elsif ($X[0] =~ m/^http:\/\/download[0-9]{3}.(avast.com.*)/) {<BR> $out="</FONT><A href="http://avast-cdn"><FONT size=2 face="Courier New">http://avast-cdn</FONT></A><FONT size=2 face="Courier New">." . $1 ;</FONT></DIV>
<DIV><FONT size=2 face="Courier New"></FONT> </DIV>
<DIV><FONT size=2 face="Courier New"> #AVAST<BR>} elsif ($X[0] =~ m/^http:\/\/[0-9]*\.[0-9]*\.[0-9]*\.[0-9]*\/(iavs.*)/) {<BR> $out="</FONT><A href="http://avast-cdn.avast.com/"><FONT size=2 face="Courier New">http://avast-cdn.avast.com/</FONT></A><FONT size=2 face="Courier New">" . $1 ;</FONT></DIV>
<DIV><FONT size=2 face="Courier New"></FONT> </DIV>
<DIV><FONT size=2 face="Courier New"> #KAV<BR>} elsif ($X[0] =~ m/^http:\/\/dnl-[0-9]{2}.(geo.kaspersky.com.*)/) {<BR> $out="</FONT><A href="http://kav-cdn"><FONT size=2 face="Courier New">http://kav-cdn</FONT></A><FONT size=2 face="Courier New">." . $1 ;</FONT></DIV>
<DIV><FONT size=2 face="Courier New"></FONT> </DIV>
<DIV><FONT size=2 face="Courier New"> #AVG<BR>} elsif ($X[0] =~ m/^http:\/\/update.avg.com/) {<BR> $out="</FONT><A href="http://avg-cdn"><FONT size=2 face="Courier New">http://avg-cdn</FONT></A><FONT size=2 face="Courier New">." . $1 ;</FONT></DIV>
<DIV><FONT size=2 face="Courier New"></FONT> </DIV>
<DIV><FONT size=2 face="Courier New"> #maps.google.com<BR>} elsif ($X[0] =~ m/^http:\/\/(cbk|mt|khm|mlt|tbn)[0-9]?(.google\.co(m|\.uk|\.id).*)/) {<BR> $out="http://" . $1 . $2 ;</FONT></DIV>
<DIV><FONT size=2 face="Courier New"></FONT> </DIV>
<DIV><FONT size=2 face="Courier New"> #gstatic and/or wikimapia<BR>} elsif ($X[0] =~ m/^http:\/\/([a-z])[0-9]?(\.gstatic\.com.*|\.wikimapia\.org.*)/) {<BR> $out="http://" . $1 . $2 ;</FONT></DIV>
<DIV><FONT size=2 face="Courier New"></FONT> </DIV>
<DIV><FONT size=2 face="Courier New"> #maps.google.com<BR>} elsif ($X[0] =~ m/^http:\/\/(khm|mt)[0-9]?(.google.com.*)/) {<BR> $out="http://" . $1 . $2 ;</FONT></DIV>
<DIV><FONT size=2 face="Courier New"></FONT> </DIV>
<DIV><FONT size=2 face="Courier New"> #Google<BR>} elsif ($X[0] =~ m/^http:\/\/www\.google-analytics\.com\/__utm\.gif\?.*/) {<BR> $out="</FONT><A href="http://www.google-analytics.com/__utm.gif\n"><FONT size=2 face="Courier New">http://www.google-analytics.com/__utm.gif\n</FONT></A><FONT size=2 face="Courier New">";</FONT></DIV>
<DIV><FONT size=2 face="Courier New"></FONT> </DIV>
<DIV><FONT size=2 face="Courier New">} elsif ($X[0] =~ m/^http:\/\/(www\.ziddu\.com.*\.[^\/]{3,4})\/(.*?)/) {<BR> $out="http://" . $1 ;</FONT></DIV>
<DIV><FONT size=2 face="Courier New"></FONT> </DIV>
<DIV><FONT size=2 face="Courier New"> #cdn, varialble 1st path<BR>} elsif (($X[0] =~ /filehippo/) && (m/^http:\/\/(.*?)\.(.*?)\/(.*?)\/(.*)\.([a-z0-9]{3,4})(\?.*)?/)) {<BR> @y = ($1,$2,$4,$5);<BR> $y[0] =~ s/[a-z0-9]{2,5}/cdn./;<BR> $out="http://" . $y[0] . $y[1] . "/" . $y[2] . "." . $y[3] ;</FONT></DIV>
<DIV><FONT size=2 face="Courier New"></FONT> </DIV>
<DIV><FONT size=2 face="Courier New"> #rapidshare<BR>} elsif (($X[0] =~ /rapidshare/) && (m/^http:\/\/(([A-Za-z]+[0-9-.]+)*?)([a-z]*\.[^\/]{3}\/[a-z]*\/[0-9]*)\/(.*?)\/([^\/\?\&]{4,})$/)) {<BR> $out="</FONT><A href="http://cdn"><FONT size=2 face="Courier New">http://cdn</FONT></A><FONT size=2 face="Courier New">." . $3 . "/squid.internal/" . $5 ;</FONT></DIV>
<DIV><FONT size=2 face="Courier New"></FONT> </DIV>
<DIV><FONT size=2 face="Courier New"> #for yimg.com video<BR>} elsif ($X[0] =~ m/^http:\/\/(.*yimg.com)\/\/(.*)\/([^\/\?\&]*\/[^\/\?\&]*\.[^\/\?\&]{3,4})(\?.*)?$/) {<BR> $out="</FONT><A href="http://cdn.yimg.com/"><FONT size=2 face="Courier New">http://cdn.yimg.com/</FONT></A><FONT size=2 face="Courier New">" . $3 ;</FONT></DIV>
<DIV><FONT size=2 face="Courier New"></FONT> </DIV>
<DIV><FONT size=2 face="Courier New"> #for yimg.com doubled<BR>} elsif ($X[0] =~ m/^http:\/\/(.*?)\.yimg\.com\/(.*?)\.yimg\.com\/(.*?)\?(.*)/) {<BR> $out="</FONT><A href="http://cdn.yimg.com/"><FONT size=2 face="Courier New">http://cdn.yimg.com/</FONT></A><FONT size=2 face="Courier New">" . $3 ;</FONT></DIV>
<DIV><FONT size=2 face="Courier New"></FONT> </DIV>
<DIV><FONT size=2 face="Courier New"> #for Filehippo files<BR>} elsif ($X[0] =~ m/^https?:\/\/.*\.(filehippo\.com)\/.*\/(.*[\.exe|zip|cab|msi|mru|mri|bz2|gzip|tgz|rar|pdf])/) {<BR> $out="</FONT><A href="http://filehippo.sqinternal/"><FONT size=2 face="Courier New">http://filehippo.sqinternal/</FONT></A><FONT size=2 face="Courier New">" . $1 . $2 ;</FONT></DIV>
<DIV><FONT size=2 face="Courier New"></FONT> </DIV>
<DIV><FONT size=2 face="Courier New"></FONT> </DIV>
<DIV><FONT size=2 face="Courier New"> #for yimg.com with &sig=<BR>} elsif ($X[0] =~ m/^http:\/\/([^\.]*)\.yimg\.com\/(.*)/) {<BR> @y = ($1,$2);<BR> $y[0] =~ s/[a-z]+([0-9]+)?/cdn/;<BR> $y[1] =~ s/&sig=.*//;<BR> $out="http://" . $y[0] . ".yimg.com/" . $y[1] ;</FONT></DIV>
<DIV><FONT size=2 face="Courier New"></FONT> </DIV>
<DIV><FONT size=2 face="Courier New">} else {<BR> $out="ERR";</FONT></DIV>
<DIV><FONT size=2 face="Courier New"></FONT> </DIV>
<DIV><FONT size=2 face="Courier New">}<BR>if ( $out =~ m/^http\:\/\/.*/) {<BR> print "OK store-id=$out\n" ;<BR>} else {<BR> print "ERR\n" ;<BR>}<BR>}<BR></FONT></DIV>
<DIV>--</DIV>
<DIV id=signature_old>
<DIV style="FONT-SIZE: 12pt; FONT-FAMILY: Calibri">
<DIV>Regards,</DIV>
<DIV>Faisal.</DIV></DIV></DIV>
<DIV> </DIV></DIV></BODY></HTML>