[squid-users] Experimental YouTube Caching helper\tool and SQUID 3.5.25 + 4.0.19 RPM's RELEASED
Eliezer Croitoru
eliezer at ngtech.co.il
Wed May 17 02:50:53 UTC 2017
Now I am encountering an issue:
I worked on the ICAP service and it works fine to find and extract the unique identifier for the googlevideo id prediction.
Now I have a working ICAP+StoreID helpers but from an unknown reason it seems that squid find's a match in the cache but still serves the content from the origin server.
What might be causing it?
I tried to run a basic debug but I need some help in the lookup, what to look for?
What might be causing it?
I am attaching "debug_options ALL,1 11,3" output of a request(I am removing the Accept-Enconding from the request, which doesn't seems to affect anything)
Also I am attaching store.log output which shows a release of the object just before it's being swaped out.
What debug_options might help in this scenario? I am kind of clueless since I have not touched the subject for a very long time.
## cache.log
2017/05/17 05:00:23.861 kid1| 11,3| http.cc(1061) persistentConnStatus: local=192.168.89.12:36708 remote=194.90.196.53:443 FD 37 flags=1 eof=0
2017/05/17 05:00:24.351 kid1| 11,2| client_side.cc(2364) parseHttpRequest: HTTP Client local=192.168.89.12:13128 remote=192.168.89.58:61588 FD 89 flags=1
2017/05/17 05:00:24.351 kid1| 11,2| client_side.cc(2365) parseHttpRequest: HTTP Client REQUEST:
---------
GET /videoplayback?sparams=clen%2Cdur%2Cei%2Cgir%2Cid%2Cinitcwndbps%2Cip%2Cipbits%2Citag%2Ckeepalive%2Clmt%2Cmime%2Cmm%2Cmn%2Cms%2Cmv%2Cpl%2Crequiressl%2Csource%2Cupn%2Cexpire&expire=1495008016&ipbits=0&requiressl=yes&keepalive=yes&mt=1494986305&upn=gDVDWK1Q9qM&mime=video%2Fmp4&initcwndbps=892500&pl=20&ei=sK4bWZ6AHMGsW7bJgcgP&clen=68830605&itag=135&gir=yes&mm=31&mn=sn-aigllndd&dur=3835.833&id=o-AOxPmibb2YStmY_Tqj-VR0X40QeAgJtRsH3sB1Ma2WO1&key=yt6&ip=84.95.212.160&lmt=1428485111698865&source=youtube&mv=m&ms=au&alr=yes&ratebypass=yes&signature=C6DF66F0AC390B367375FF487E7FC11CA01BC834.9392D0F984CBCAE83A1FB380BE826C93E8739747&cpn=NowGUcZPl2y4ArWs&c=WEB&cver=1.20170511&range=1029733-2546112&rn=9&rbuf=34495 HTTP/1.1
Host: r15---sn-aigllndd.googlevideo.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:53.0) Gecko/20100101 Firefox/53.0
Accept: */*
Accept-Language: en-US,he;q=0.7,en;q=0.3
Accept-Encoding: gzip, deflate, br
Referer: https://www.youtube.com/
Origin: https://www.youtube.com
Connection: keep-alive
----------
2017/05/17 05:00:24.358 kid1| 11,3| http.cc(2274) httpStart: GET http://ytgv.squid.internal/id=X727AYHiNLM&itag=135&range=1029733-2546112
2017/05/17 05:00:24.358 kid1| 11,2| http.cc(2230) sendRequest: HTTP Server local=192.168.89.12:54702 remote=209.85.230.84:443 FD 90 flags=1
2017/05/17 05:00:24.358 kid1| 11,2| http.cc(2231) sendRequest: HTTP Server REQUEST:
---------
GET /videoplayback?sparams=clen%2Cdur%2Cei%2Cgir%2Cid%2Cinitcwndbps%2Cip%2Cipbits%2Citag%2Ckeepalive%2Clmt%2Cmime%2Cmm%2Cmn%2Cms%2Cmv%2Cpl%2Crequiressl%2Csource%2Cupn%2Cexpire&expire=1495008016&ipbits=0&requiressl=yes&keepalive=yes&mt=1494986305&upn=gDVDWK1Q9qM&mime=video%2Fmp4&initcwndbps=892500&pl=20&ei=sK4bWZ6AHMGsW7bJgcgP&clen=68830605&itag=135&gir=yes&mm=31&mn=sn-aigllndd&dur=3835.833&id=o-AOxPmibb2YStmY_Tqj-VR0X40QeAgJtRsH3sB1Ma2WO1&key=yt6&ip=84.95.212.160&lmt=1428485111698865&source=youtube&mv=m&ms=au&alr=yes&ratebypass=yes&signature=C6DF66F0AC390B367375FF487E7FC11CA01BC834.9392D0F984CBCAE83A1FB380BE826C93E8739747&cpn=NowGUcZPl2y4ArWs&c=WEB&cver=1.20170511&range=1029733-2546112&rn=9&rbuf=34495 HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:53.0) Gecko/20100101 Firefox/53.0
Accept: */*
Accept-Language: en-US,he;q=0.7,en;q=0.3
Referer: https://www.youtube.com/
Origin: https://www.youtube.com
Host: r15---sn-aigllndd.googlevideo.com
Cache-Control: max-age=4794000
Connection: keep-alive
----------
2017/05/17 05:00:24.436 kid1| ctx: enter level 0: 'http://ytgv.squid.internal/id=X727AYHiNLM&itag=135&range=1029733-2546112'
2017/05/17 05:00:24.436 kid1| 11,3| http.cc(694) processReplyHeader: processReplyHeader: key 'C40AE216D4CE802B5FAE71A34B75A843'
2017/05/17 05:00:24.436 kid1| 11,2| http.cc(735) processReplyHeader: HTTP Server local=192.168.89.12:54702 remote=209.85.230.84:443 FD 90 flags=1
2017/05/17 05:00:24.436 kid1| 11,2| http.cc(736) processReplyHeader: HTTP Server REPLY:
---------
HTTP/1.1 200 OK
Last-Modified: Wed, 08 Apr 2015 09:25:11 GMT
Content-Type: video/mp4
Date: Wed, 17 May 2017 02:00:24 GMT
Expires: Wed, 17 May 2017 02:00:24 GMT
Cache-Control: private, max-age=21292
Accept-Ranges: bytes
Content-Length: 1516380
Connection: keep-alive
Alt-Svc: quic=":443"; ma=2592000
Access-Control-Allow-Origin: https://www.youtube.com
Access-Control-Allow-Credentials: true
Timing-Allow-Origin: https://www.youtube.com
Access-Control-Expose-Headers: Client-Protocol, Content-Length, Content-Type, X-Bandwidth-Est, X-Bandwidth-Est2, X-Bandwidth-App-Limited, X-Bandwidth-Est-App-Limited, X-Bandwidth-Est-Comp, X-Bandwidth-Avg, X-Head-Time-Sec, X-Head-Seqnum, X-Sequence-Num, X-Segment-Lmt, X-Walltime-Ms
X-Content-Type-Options: nosniff
Server: gvs 1.0
----------
2017/05/17 05:00:24.436 kid1| ctx: exit level 0
2017/05/17 05:00:24.436 kid1| 11,3| Client.cc(936) noteAdaptationAclCheckDone: no adapation needed
2017/05/17 05:00:24.436 kid1| ctx: enter level 0: 'http://ytgv.squid.internal/id=X727AYHiNLM&itag=135&range=1029733-2546112'
2017/05/17 05:00:24.436 kid1| 11,3| http.cc(905) haveParsedReplyHeaders: HTTP CODE: 200
2017/05/17 05:00:24.437 kid1| ctx: exit level 0
2017/05/17 05:00:24.437 kid1| 11,2| client_side.cc(1408) sendStartOfMessage: HTTP Client local=192.168.89.12:13128 remote=192.168.89.58:61588 FD 89 flags=1
2017/05/17 05:00:24.437 kid1| 11,2| client_side.cc(1409) sendStartOfMessage: HTTP Client REPLY:
---------
HTTP/1.1 200 OK
Last-Modified: Wed, 08 Apr 2015 09:25:11 GMT
Content-Type: video/mp4
Date: Wed, 17 May 2017 02:00:24 GMT
Expires: Wed, 17 May 2017 02:00:24 GMT
Cache-Control: private, max-age=21292
Accept-Ranges: bytes
Content-Length: 1516380
Alt-Svc: quic=":443"; ma=2592000
Access-Control-Allow-Origin: https://www.youtube.com
Access-Control-Allow-Credentials: true
Timing-Allow-Origin: https://www.youtube.com
Access-Control-Expose-Headers: Client-Protocol, Content-Length, Content-Type, X-Bandwidth-Est, X-Bandwidth-Est2, X-Bandwidth-App-Limited, X-Bandwidth-Est-App-Limited, X-Bandwidth-Est-Comp, X-Bandwidth-Avg, X-Head-Time-Sec, X-Head-Seqnum, X-Sequence-Num, X-Segment-Lmt, X-Walltime-Ms
X-Content-Type-Options: nosniff
Server: gvs 1.0
X-Cache: MISS from filter
X-Cache-Lookup: HIT from filter:3128
Connection: keep-alive
----------
## End of cache.log
## store.log
1494988764.096 RELEASE 00 000000FC 9F3457302C3C18CC9420F78989501F21 ? ? ? ? ?/? ?/? ? ?
1494988764.180 SWAPOUT 00 000001C8 9F3457302C3C18CC9420F78989501F21 200 1494988764 1425925658 1495009739 audio/webm 414553/414553 GET https://r15---sn-aigllndd.googlevideo.com/videoplayback?clen=76689610&source=youtube&sparams=clen%2Cdur%2Cei%2Cgir%2Cid%2Cinitcwndbps%2Cip%2Cipbits%2Citag%2Ckeepalive%2Clmt%2Cmime%2Cmm%2Cmn%2Cms%2Cmv%2Cpl%2Crequiressl%2Csource%2Cupn%2Cexpire&expire=1495010039&dur=3836.501&ei=l7YbWfjyEdPacseFsOgB&initcwndbps=847500&mt=1494988343&requiressl=yes&ip=84.95.212.160&ms=au&gir=yes&mv=m&pl=20&mime=audio%2Fwebm&id=o-AAyoF-88PGHEqhkC-hBvKEPjiIeUoDi6DohmCQ0WoGs8&mn=sn-aigllndd&mm=31&ipbits=0&itag=251&keepalive=yes&lmt=1425925658208642&key=yt6&upn=JGXkLnd5jwI&alr=yes&ratebypass=yes&signature=7A20E093615AC902AAE33D726AD3F9BE3934DFDD.BDCC662814E40739201EA9EE69D1058F7C74C7A1&cpn=ZWjhgUtxrr_va9Vr&c=WEB&cver=1.20170511&range=8837089-9251641&rn=32&rbuf=352381
1494988783.284 RELEASE 00 000000FD 0436BF4E1A398F3365103BE514907CD5 ? ? ? ? ?/? ?/? ? ?
1494988783.366 SWAPOUT 00 000001C9 0436BF4E1A398F3365103BE514907CD5 200 1494988783 1425925658 1495009739 audio/webm 413087/413087 GET https://r15---sn-aigllndd.googlevideo.com/videoplayback?clen=76689610&source=youtube&sparams=clen%2Cdur%2Cei%2Cgir%2Cid%2Cinitcwndbps%2Cip%2Cipbits%2Citag%2Ckeepalive%2Clmt%2Cmime%2Cmm%2Cmn%2Cms%2Cmv%2Cpl%2Crequiressl%2Csource%2Cupn%2Cexpire&expire=1495010039&dur=3836.501&ei=l7YbWfjyEdPacseFsOgB&initcwndbps=847500&mt=1494988343&requiressl=yes&ip=84.95.212.160&ms=au&gir=yes&mv=m&pl=20&mime=audio%2Fwebm&id=o-AAyoF-88PGHEqhkC-hBvKEPjiIeUoDi6DohmCQ0WoGs8&mn=sn-aigllndd&mm=31&ipbits=0&itag=251&keepalive=yes&lmt=1425925658208642&key=yt6&upn=JGXkLnd5jwI&alr=yes&ratebypass=yes&signature=7A20E093615AC902AAE33D726AD3F9BE3934DFDD.BDCC662814E40739201EA9EE69D1058F7C74C7A1&cpn=ZWjhgUtxrr_va9Vr&c=WEB&cver=1.20170511&range=9251642-9664728&rn=33&rbuf=372381
1494988939.087 RELEASE 00 000001EB 2257A59BCD5457AC526432A31D8C1DAE ? ? ? ? ?/? ?/? ? ?
1494988939.176 SWAPOUT 00 00000215 2257A59BCD5457AC526432A31D8C1DAE 200 1494988939 1425925658 1495010229 audio/webm 290838/290838 GET https://r15---sn-aigllndd.googlevideo.com/videoplayback?clen=76689610&upn=Ar8EsOEX6u8&source=youtube&expire=1495010529&lmt=1425925658208642&mime=audio%2Fwebm&gir=yes&requiressl=yes&keepalive=yes&id=o-AH3RVksarviuBTrz7-iVa695QDmwgtd25mI2pwtDvNDe&sparams=clen%2Cdur%2Cei%2Cgir%2Cid%2Cinitcwndbps%2Cip%2Cipbits%2Citag%2Ckeepalive%2Clmt%2Cmime%2Cmm%2Cmn%2Cms%2Cmv%2Cpl%2Crequiressl%2Csource%2Cupn%2Cexpire&initcwndbps=822500&ms=au&itag=251&mt=1494988830&dur=3836.501&mv=m&pl=20&mm=31&ipbits=0&mn=sn-aigllndd&key=yt6&ei=gbgbWdf_FtW9cseknvgM&ip=84.95.212.160&alr=yes&ratebypass=yes&signature=7C1C7F0389FF6102FAAFB4F4978E8A2BEAE64D40.596DE1F2DEAD3091CD0B49354E49A42BEE812362&cpn=Xy6NBwdfsknjfruc&c=WEB&cver=1.20170511&range=327790-618627&rn=8&rbuf=16213
1494988940.164 RELEASE 00 00000207 E306D0FF544DCD9B997C189364B9173D ? ? ? ? ?/? ?/? ? ?
1494988940.493 SWAPOUT 00 0000021B E306D0FF544DCD9B997C189364B9173D 200 1494988940 1428485111 1495010229 video/mp4 1516380/1516380 GET https://r15---sn-aigllndd.googlevideo.com/videoplayback?clen=68830605&upn=Ar8EsOEX6u8&source=youtube&expire=1495010529&lmt=1428485111698865&mime=video%2Fmp4&gir=yes&requiressl=yes&keepalive=yes&id=o-AH3RVksarviuBTrz7-iVa695QDmwgtd25mI2pwtDvNDe&sparams=clen%2Cdur%2Cei%2Cgir%2Cid%2Cinitcwndbps%2Cip%2Cipbits%2Citag%2Ckeepalive%2Clmt%2Cmime%2Cmm%2Cmn%2Cms%2Cmv%2Cpl%2Crequiressl%2Csource%2Cupn%2Cexpire&initcwndbps=822500&ms=au&itag=135&mt=1494988830&dur=3835.833&mv=m&pl=20&mm=31&ipbits=0&mn=sn-aigllndd&key=yt6&ei=gbgbWdf_FtW9cseknvgM&ip=84.95.212.160&alr=yes&ratebypass=yes&signature=B38EC7D16A460518D56AB4AEC2686F8417E4F66D.5C0A937164F499F9951C0AC2ED4E3C1655718B98&cpn=Xy6NBwdfsknjfruc&c=WEB&cver=1.20170511&range=1029733-2546112&rn=10&rbuf=38883
## End of store.log
Thanks,
Eliezer
----
Eliezer Croitoru
Linux System Administrator
Mobile: +972-5-28704261
Email: eliezer at ngtech.co.il
-----Original Message-----
From: squid-users [mailto:squid-users-bounces at lists.squid-cache.org] On Behalf Of Eliezer Croitoru
Sent: Tuesday, May 16, 2017 4:04 AM
To: squid-users at lists.squid-cache.org
Subject: [squid-users] Experimental YouTube Caching helper\tool and SQUID 3.5.25 + 4.0.19 RPM's RELEASED
Hey List,
I have been working a while ago on a tool(ICAP service) that will do the
next:
- Snatch YouTube main video pages(watch?...) on the fly
- Fetch them
- Parse them
- Find the key to predict the ID of the X.googlevideo.com/Y which will match the YouTube video
- Store the key with the correlated video ID in a redis DB for an hour
- Then a StoreID helper will analyze X.googlevideo.com/Y urls and will try to see if it finds the matching a YouTube video by the ID on the redis DB
The concept is kind of "divide and conquer" while later if I will have time there will be an option to combine these two things into one ICAP service that will help to cache YouTube videos and probably also Windows Updates using squid.
Currently the ICAP service code will not be published but the binaries and the resources which I have used to create the tool will be published.
(Something like the ingredients without the recipe due to this task being considered a caching master level and not just any novice) However my email is wide open for questions requests and tips for anyone that wishes to get answers, directions and guidance.
@Alex, What will encourage you to help with an improved patch to add ICAP X-StoreID (or else) response header capability that will help me to write a complete StoreID solution based on ICAP alone?
@Others, if you are interested in this solution publication please respond to this thread and CC to me.
Thanks,
Eliezer
* I have released Squid-Cache 3.5.25 and 4.0.19 RPMS but Life takes more of my time and as I get busy I didn't have enough time to publish the article like in the past.
The article topic is "ENCRYPTION, HOW FAR WILL YOU GO?" at [http://www1.ngtech.co.il/wpe/?p=430]
----
Eliezer Croitoru
Linux System Administrator
Mobile: +972-5-28704261
Email: eliezer at ngtech.co.il
_______________________________________________
squid-users mailing list
squid-users at lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users
More information about the squid-users
mailing list