[squid-users] youtube videos and squid

Mon Jul 17 20:48:56 UTC 2017

Hey Sergei,

Thanks for sharing.
This is the first time I have seen EMBY server and client and it seems pretty decent\good and contains many very nice features.

I have been working on a solution that will be hosted on a Linux Server which I named "YouTube-Store".
It's working and I wrote installation scripts for CentOS 7, Ubuntu 16.04, Debian 9.

The concept of the service is like this:
You have a main menu and admin menu.(more pages can be added to store categories with list of movie etc)
The main menu has a search bar which you can use to search the locally hosted videos by their YouTube ID or Their YouTube title.
The Server have a scheduled "fetcher" which downloads videos from a "queue" that is managed by the service admin.(or a script)
There is an API which you(The admin, or a script) can send to it one the next: YouTube link, youtu.be link, YouTube video ID(..watch?v=ID).
The queue is being managed from the web interface and you have the option to add videos to the queue, blacklist or remove from the queue.
( it's very simple to add\remove\disable\hide etc videos)

The goal is to have the option to host a local video "store" which is based on YouTube.
The missing piece in all of this is the IPAD\ANDROID app, but I believe that it's possible to web CMS system to overcome this "gap".
And one last thing is the option to redirect Squid-Cache clients requests to a YouTube video into the locally hosted version(if the video already downloaded).

This is the git repository of the project:
http://gogs.ngtech.co.il/elicro/youtube-store

which is in the beta testing phase.

Thanks For the progress update and also for sharing EMBY,
Eliezer

----
http://ngtech.co.il/lmgtfy/
Linux System Administrator
Mobile: +972-5-28704261
Email: eliezer at ngtech.co.il

From: Sergei G [mailto:sergeig.public at gmail.com] 
Sent: Monday, July 17, 2017 23:05
To: Eliezer Croitoru <eliezer at ngtech.co.il>
Cc: Amos Jeffries <squid3 at treenet.co.nz>; squid-users at lists.squid-cache.org
Subject: Re: [squid-users] youtube videos and squid

Thank you for saving me time and effort.

I ended up following advice of downloading youtube movies to home computer and serving from it.

My choice was EMBY server for windows https://emby.media/windows-server.html although a couple others were good options as well.

I bought Emby app for iPad and I use a download extension for Chrome to get 720p resolution videos.  The content is static, but my drive is big and I feel that this way I can approve some of the content.  It certainly costs my time go get content to Emby server.

I could try installing Emby on Raspberry PI 3, but I decided against it.  It was more work, the download workflow had a more complicated step of moving video from PC to RPI.  I was not sure if RPI could be responsive with its decoding.

The setup works.

On Fri, Jul 7, 2017 at 7:15 AM, Eliezer Croitoru <mailto:eliezer at ngtech.co.il> wrote:
Hey Amos, Sergei,

I managed to write an ICAP service which works with a ruby\golang StorID helper and a redis DB that can help with the caching of YouTube videos.
However despite to the fact that I am able to tag urls with StoreID  squid is still not responding with a cache HIT but fetching from the original sources.

There are couple obstacles in this field and this specific solution I wrote is designed for PC and not IPAD or Andorid Based devices.
There is an issue with Android 6+(7..) and IOS 10 based devices which for some reason do not cope with SSL traffic interception.
If you do have a tiny server and you can run a tiny http service with nginx\apache I would recommend you to download the videos and serve them locally using chrome or another browser which works.
I found it more useful then intercepting and caching also, kids are usually enjoying from a more "stable" stash of movies\videos then others.
They can watch the same video over and over and it would be fun for them.
The teenagers are having issue's watching the same video over and over so..
I can recommend on a nice wordpress theme which can fit a "VOD" site and a video downloader which you can use to download specific videos or playlists.

This solution would be something like "netflix on a stick" and is very effective with a raspberry pi 3 with some external USB HDD.

I would be happy to get any response to any of the ideas.

Thanks In Advance,
Eliezer

 * If you are willing to devote some time to debug the issue with the current helpers let me know and bump me if you think I missed an email and didn't responded(it happens when you have kids...)
----
Eliezer Croitoru
Linux System Administrator
Mobile: tel:%2B972-5-28704261
Email: mailto:eliezer at ngtech.co.il

-----Original Message-----
From: squid-users [mailto:mailto:squid-users-bounces at lists.squid-cache.org] On Behalf Of Amos Jeffries
Sent: Friday, July 7, 2017 11:32
To: mailto:squid-users at lists.squid-cache.org
Subject: Re: [squid-users] youtube videos and squid

On 07/07/17 15:40, Sergei G wrote:
> Hi guys,
>
> I have a very specific scenario in mind to use squid for.
>
> I have 2 kids (2.5 and 4 years old) that are watching iPads and really
> using available Comcast bandwidth.  It does not help that they sometimes
> just leave those iPads running.
>
> They tend to re-watch youtube videos (click on the same icon that they
> liked before).  And that makes me think that squid could help me with
> caching off youtube content.  Am I correct?

Possibly. Google have actively been making it more difficult every year
for quite a while.

These days it requires intercepting the YouTube HTTPS connections. That
is only possible if the clients are not using Chrome or other Google
apps to fetch the videos - otherwise you run up against the cert pinning
wall.

After that you need some extra helper software to track the YT video
fetching process and decipher what the actual video URL is from the mess
of session traffic. That is being kept a bit of a secret these days,
since every time G find out how it is being done they change the process
to make it more obtuse and harder to do :-(

Eliezer has been trying to get a helper for that going most recently.
There are also some other products I forget the name of right now
(videobooster maybe), but should be easy to find that cache YouTube content.

>
> If not then I have no reason to bother you anymore :)
>
> If squid could help me, then could you point me to a an example
> configuration that would work?
>
> As far as hardware I have 2 options:
>
> 1. I can install squid on a Raspberry PI 3, if package is readily
> available.  that's my preferred solution.
> 2. I have an old server hardware with more power than RPI 3, but I don't
> like to run it, because it is noisy.  It has FreeBSD 10 installed and I
> can upgrade it to latest FreeBSD (11?) and isntall squid application
> that way.
>
>
> Does squid run on RPI3?  FreeBSD?

Yes to both, and at the small scale you need the RPi3 should be able to
cope with it.

Amos
_______________________________________________
squid-users mailing list
mailto:squid-users at lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

_______________________________________________
squid-users mailing list
mailto:squid-users at lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users