[squid-users] Skype for Business behind a transparent squid (TProxy) HTTP/S

Sameh Onaissi sameh.onaissi at solcv.com
Wed Dec 7 20:11:06 UTC 2016


iptables is the same.. here is after I ran the script twice (with and without proxy)
http://pastebin.com/YFtbG6St


I have a script that bridges the two network cards, that uses nat, hence having both

I can send you all the scripts I run to set up squid and the bypasses so you can reproduce the situation.

thanks again!



[cid:2FD1C3AB-E45C-49F0-84AB-0F8AC658BD11 at routerb408e2.com]Piensa en el medio ambiente antes de imprimir este email.

On Dec 7, 2016, at 12:12 PM, Eliezer Croitoru <eliezer at ngtech.co.il<mailto:eliezer at ngtech.co.il>> wrote:

Are you sure this setup works?
You have both REDIRECT and TPROXY on the same machine so you need to bypass
for both of these.
Is this iptables-save snapshot after you ran the script?
Also for this to work you will need to use the updated version of the script
at:
https://gist.github.com/elico/a54c2c8f8e1a2407b42210896b960f4b

And run it twice... once with tproxy and the other without tproxy.
After you run the script share the snapshot of "iptables-save" and "ipset
list".
If it can be reproducible here when I will be using SKYPE I would be able to
test it but I believe it should work good enough to minimize the issue.
The above depends only on one thing: NTOP Skype networks identification.
Maybe I have mentioned but there is another script which I wrote that is
more simplified at:
https://gist.github.com/elico/e0faadf0cc63942c5aaade808a87deef

But can be adapted to be used against skype known domains.
The next level would be to use some kind of splice rules with an
external_acl helper.
The external_acl helper can receive information from squid about the request
SNI and to verify if the server only serves skype domains or others.
The above is a much more deeper level and I think that the first scripts
should be enough to resolve most of the issues.

Eliezer

----
http://ngtech.co.il/lmgtfy/
Linux System Administrator
Mobile: +972-5-28704261
Email: eliezer at ngtech.co.il


From: Sameh Onaissi [mailto:sameh.onaissi at solcv.com]
Sent: Wednesday, December 7, 2016 6:09 PM
To: Eliezer Croitoru <eliezer at ngtech.co.il>
Cc: squid-users at lists.squid-cache.org
Subject: Re: [squid-users] Skype for Business behind a transparent squid
(TProxy) HTTP/S

iptables-save:  http://pastebin.com/9JrVANtt

ipset list : http://pastebin.com/wtMtzaQe
http://pastebin.com/wtMtzaQe
http://pastebin.com/wtMtzaQe
pastebin.com

http://pastebin.com/9JrVANtt
http://pastebin.com/9JrVANtt
pastebin.com


  Piensa en el medio ambiente antes de imprimir este email.
________________________________________
From: Eliezer Croitoru <mailto:eliezer at ngtech.co.il>
Sent: Wednesday, December 7, 2016 10:58:18 AM
To: Sameh Onaissi
Cc: mailto:squid-users at lists.squid-cache.org
Subject: RE: [squid-users] Skype for Business behind a transparent squid
(TProxy) HTTP/S

Give us the "iptables-save" output and also "ipset list".
(or what ever was the command of ipset to dump the content of the list).
After this we can understand what is causing this issue.

Eliezer

----
http://ngtech.co.il/lmgtfy/
Linux System Administrator
Mobile: +972-5-28704261
Email: mailto:eliezer at ngtech.co.il


From: Sameh Onaissi [mailto:sameh.onaissi at solcv.com]
Sent: Wednesday, December 7, 2016 5:23 PM
To: Eliezer Croitoru <mailto:eliezer at ngtech.co.il>
Cc: mailto:squid-users at lists.squid-cache.org
Subject: Re: [squid-users] Skype for Business behind a transparent squid
(TProxy) HTTP/S

Still not working and I do not know what to do next.

access.log shows IPs and domains that are supposed to be bypassed already.

Any further instructions are hugely appreciated.



Piensa en el medio ambiente antes de imprimir este email.

On Dec 7, 2016, at 9:50 AM, Eliezer Croitoru <mailto:eliezer at ngtech.co.il>
wrote:

Was there any progress with the script and the issues?

Eliezer

----
http://ngtech.co.il/lmgtfy/
Linux System Administrator
Mobile: +972-5-28704261
Email: mailto:eliezer at ngtech.co.il


From: Sameh Onaissi [mailto:sameh.onaissi at solcv.com]
Sent: Wednesday, December 7, 2016 12:36 AM
To: Eliezer Croitoru <mailto:eliezer at ngtech.co.il>
Cc: 'Amos Jeffries' <mailto:squid3 at treenet.co.nz>;
mailto:squid-users at lists.squid-cache.org
Subject: Re: [squid-users] Skype for Business behind a transparent squid
(TProxy) HTTP/S


Hello Eliezer and thanks again.

I ran the script with the tproxy argument.

Tried to reconnect skype for business...

After about a 3 min wait, a pop up saying "Skype for Business couldn't find
a skype for business server"  and access log shows:

1481061269.006    400 10.0.0.38 TCP_MISS/200 1068 GET
http://lyncdiscover.solcv.com/? - ORIGINAL_DST/132.245.1.28
application/vnd.microsoft.rtc.autodiscover+xml
1481061269.270    667 10.0.0.38 TAG_NONE/200 0 CONNECT 132.245.1.28:443 -
HIER_NONE/- -
1481061269.270    665 10.0.0.38 TCP_TUNNEL/200 5568 CONNECT
lyncdiscover.solcv.com:443 - ORIGINAL_DST/132.245.1.28 -
1481061269.770    596 10.0.0.38 TAG_NONE/200 0 CONNECT 52.112.64.14:443 -
HIER_NONE/- -
1481061269.770    594 10.0.0.38 TCP_TUNNEL/200 6981 CONNECT
webdir0a.online.lync.com:443 - ORIGINAL_DST/52.112.64.14 -
1481061270.679    897 10.0.0.38 TAG_NONE/200 0 CONNECT 52.112.64.14:443 -
HIER_NONE/- -
1481061270.679    895 10.0.0.38 TCP_TUNNEL/200 7733 CONNECT
webdir0a.online.lync.com:443 - ORIGINAL_DST/52.112.64.14 -
1481061272.178    841 10.0.0.38 TAG_NONE/200 0 CONNECT 23.100.120.65:443 -
HIER_NONE/- -
1481061272.178    840 10.0.0.38 TCP_TUNNEL/200 20539 CONNECT
login.microsoftonline.com:443 - ORIGINAL_DST/23.100.120.65 -
1481061273.713    641 10.0.0.38 TAG_NONE/200 0 CONNECT 52.112.64.14:443 -
HIER_NONE/- -
1481061273.713    640 10.0.0.38 TCP_TUNNEL/200 8037 CONNECT
webdir0a.online.lync.com:443 - ORIGINAL_DST/52.112.64.14 -
1481061273.751   3054 10.0.0.38 TAG_NONE/200 0 CONNECT 52.112.64.14:443 -
HIER_NONE/- -
1481061273.751   3052 10.0.0.38 TCP_TUNNEL/200 24458 CONNECT
webdir0a.online.lync.com:443 - ORIGINAL_DST/52.112.64.14 -
1481061273.751   1544 10.0.0.38 TAG_NONE/200 0 CONNECT 52.112.64.14:443 -
HIER_NONE/- -
1481061273.751   1543 10.0.0.38 TCP_TUNNEL/200 11653 CONNECT
webdir0a.online.lync.com:443 - ORIGINAL_DST/52.112.64.14 -


so I added more ip ranges to the cidr-to-bypass.txt and ran the script again


1481063243.370    371 10.0.0.38 TCP_MISS/200 1068 GET
http://lyncdiscover.solcv.com/? - ORIGINAL_DST/134.170.113.210
application/vnd.microsoft.rtc.autodiscover+xml
1481063278.271  74233 10.0.0.38 TAG_NONE/200 0 CONNECT 104.208.31.113:443 -
HIER_NONE/- -
1481063278.271  74231 10.0.0.38 TCP_TUNNEL/200 6746 CONNECT
pipe.skype.com:443 - ORIGINAL_DST/104.208.31.113 -
1481063344.143  60720 10.0.0.38 TAG_NONE/200 0 CONNECT 104.208.31.113:443 -
HIER_NONE/- -
1481063344.143  60719 10.0.0.38 TCP_TUNNEL/200 6389 CONNECT
pipe.skype.com:443 - ORIGINAL_DST/104.208.31.113 -

a new set showed up...

what more can we do?

keep adding ip ranges?

thanks

  Piensa en el medio ambiente antes de imprimir este email.
________________________________________
From: Eliezer Croitoru <mailto:eliezer at ngtech.co.il>
Sent: Tuesday, December 6, 2016 4:36:56 PM
To: Sameh Onaissi
Cc: 'Amos Jeffries'; mailto:squid-users at lists.squid-cache.org
Subject: RE: [squid-users] Skype for Business behind a transparent squid
(TProxy) HTTP/S

Try the next script:
https://gist.github.com/elico/a54c2c8f8e1a2407b42210896b960f4b
https://gist.github.com/elico/a54c2c8f8e1a2407b42210896b960f4b
https://gist.github.com/elico/a54c2c8f8e1a2407b42210896b960f4b
gist.github.com
bypass squid interception for skype



It has two modes: regular and tproxy.
In your case you should run the script with:
$ bypass-skype-cidr.sh tproxy

The tproxy flag should do the trick for you.

Let me know if it works for you.

Eliezer

----
http://ngtech.co.il/lmgtfy/
Linux System Administrator
Mobile: +972-5-28704261
Email: mailto:eliezer at ngtech.co.il


From: Sameh Onaissi [mailto:sameh.onaissi at solcv.com]
Sent: Tuesday, December 6, 2016 9:24 PM
To: Eliezer Croitoru <mailto:eliezer at ngtech.co.il>
Cc: Amos Jeffries <mailto:squid3 at treenet.co.nz>;
mailto:squid-users at lists.squid-cache.org
Subject: Re: [squid-users] Skype for Business behind a transparent squid
(TProxy) HTTP/S

Yes please, I would appreciate help with that script.

As I aforementioned, totally new to all this



Piensa en el medio ambiente antes de imprimir este email.

On Dec 6, 2016, at 1:27 PM, Eliezer Croitoru <mailto:eliezer at ngtech.co.il>
wrote:

Now you can enhance the script by adding manually the ntop skype related
networks based on:
https://github.com/ntop/nDPI/blob/d9a2d9a6bd4d476d666d26cb713952760a975d92/s
rc/lib/ndpi_content_match.c.inc#L286

/*
Skype (Microsoft CDN)
157.56.135.64/26, 157.56.185.0/26, 157.56.52.0/26,
157.56.53.128/25, 157.56.198.0/26
157.60.0.0/16, 157.54.0.0/15
13.107.3.128/32
13.107.3.129/32
111.221.64.0 - 111.221.127.255
91.190.216.0/21 (AS198015 Skype Communications Sarl)
91.190.218.0/24
40.126.129.109/32
65.55.223.0/26
*/

If you need help scripting this let me know.

Eliezer

----
http://ngtech.co.il/lmgtfy/
Linux System Administrator
Mobile: +972-5-28704261
Email: mailto:eliezer at ngtech.co.il


From: Sameh Onaissi [mailto:sameh.onaissi at solcv.com]
Sent: Tuesday, December 6, 2016 7:29 PM
To: Eliezer Croitoru <mailto:eliezer at ngtech.co.il>
Cc: Amos Jeffries <mailto:squid3 at treenet.co.nz>;
mailto:squid-users at lists.squid-cache.org
Subject: Re: [squid-users] Skype for Business behind a transparent squid
(TProxy) HTTP/S

Hello,

OK, I added the ssl_bump slice on the skype domains text file
I installed ipset and ran the script.

Now access.log has much less skype related logs:

What is left is:
1481044996.398   3412 10.0.0.11 TAG_NONE/200 0 CONNECT 132.245.1.32:443 -
ORIGINAL_DST/132.245.1.32 -
1481044996.423      0 10.0.0.11 TAG_NONE/400 3998 REGISTER
sip:solcv.comSIP/2.0 - HIER_NONE/- text/html
1481045000.296    372 10.0.0.11 TAG_NONE/200 0 CONNECT 134.170.113.207:443 -
ORIGINAL_DST/134.170.113.207 -
1481045000.325      0 10.0.0.11 TAG_NONE/400 3998 REGISTER
sip:solcv.comSIP/2.0 - HIER_NONE/- text/html
1481045008.685   4259 10.0.0.11 TAG_NONE/200 0 CONNECT 134.170.113.207:443 -
ORIGINAL_DST/134.170.113.207 -
1481045008.726      0 10.0.0.11 TAG_NONE/400 3998 REGISTER
sip:solcv.comSIP/2.0 - HIER_NONE/- text/html


although http://solve.com is in the text file.

I ran whois on the first IP and got:

NetRange:       132.245.0.0 - 132.245.255.255
CIDR:           132.245.0.0/16
NetName:        MICROSOFT


Same with the 134.170. address. Can we slice that range?





Sameh Onaissi
Ingeniero de Soporte
Sol Cable Visión
Cel: 316-3023424
Email: mailto:sameh.onaissi at solcv.com



Piensa en el medio ambiente antes de imprimir este email.

On Dec 6, 2016, at 12:11 PM, Eliezer Croitoru <mailto:eliezer at ngtech.co.il>
wrote:

Hey,

Depends on your OS you will need to installthe  ipset package.
Try to run "apt-get install ipset".
And then run the script.

Eliezer

----
http://ngtech.co.il/lmgtfy/
Linux System Administrator
Mobile: +972-5-28704261
Email: mailto:eliezer at ngtech.co.il
<Untitled Attachment 1.jpg>

From: Sameh Onaissi [mailto:sameh.onaissi at solcv.com]
Sent: Tuesday, December 6, 2016 5:23 PM
To: Amos Jeffries <mailto:squid3 at treenet.co.nz>
Cc: Eliezer Croitoru <mailto:eliezer at ngtech.co.il>
Subject: Re: [squid-users] Skype for Business behind a transparent squid
(TProxy) HTTP/S

Amos, thanks for the reply.


This is getting more confusing.

I changed the script to: http://pastebin.com/jLgywstg

And I ran it, but I am getting errors:

sudo sh http://bypass.sh/ + iptables -t mangle -L PREROUTING + grep
bypasspool + [ 1 -ne 0 ] + iptables -t mangle -I PREROUTING -m set
--match-set bypasspool dst,src -j DIVERT iptables http://v1.6.0/ Set
bypasspool doesn't exist. Try `iptables -h' or 'iptables --help' for more
information. + ipset create bypasspool hash:ip http://bypass.sh/ 10:
http://bypass.sh/ ipset: not found + read item +
echohttp://lyncdiscover.solcv.com/ http://lyncdiscover.solcv.com/ + host -4
http://lyncdiscover.solcv.com/ + grep has address + awk {print $4} + xargs
-l1 ipset add bypasspool xargs: ipset: No such file or directory + read item
+ echo http://webdir0a.online.lync.com/http://webdir0a.online.lync.com/ +
host -4 http://webdir0a.online.lync.com/ + grep has address + awk {print $4}
+ xargs -l1 ipset add bypasspool xargs: ipset: No such file or directory

. this goes on the same for all the domains in the text file

My iptables is still <http://pastebin.com/SqpbmYQQ>

I did not quite understand what you meant by
You should test whether -m set or -m socket work faster and put that one
first. My change above places it at line 2 (after -m socket) assuming
your iptables script is still <http://pastebin.com/SqpbmYQQ>

should I incorporate the bypass script into my iptables.sh script? run
iptables first then bypass?



On a side note, would adding ssl_bump exceptions to squid.conf do it?
Something like:

acl skype_domains <path to file>
ssl_bump splice skype_domains
ssl_bump bump all



Again, thanks again for your help.



<Untitled Attachment 2.jpg> Piensa en el medio ambiente antes de imprimir
este email.

On Dec 6, 2016, at 9:50 AM, Amos Jeffries <mailto:squid3 at treenet.co.nz>
wrote:

On 7/12/2016 3:19 a.m., Sameh Onaissi wrote:
Hello,

I tried doing the changes to nat/REDIRECT in iptables.sh and I must have
messed up somewhere, so I am sticking with mangle/tproxy for now since squid
is working with them.

How can I change Eliezer's script to mangle/tproxy?
https://gist.github.com/elico/e0faadf0cc63942c5aaade808a87deef

Excuse my novice knowledge in iptables.

No worries.

You need to change where iptables attaches the 'bypasspool'. Both the
table/location (-t) and the jump/action (-j).

iptables -t mangle -L PREROUTING |grep bypasspool
if [ "$?" -ne "0" ];then

  iptables -t mangle -I 2 PREROUTING \
    -m set --match-set bypasspool dst,src \
    -j DIVERT

fi

You should test whether -m set or -m socket work faster and put that one
first. My change above places it at line 2 (after -m socket) assuming
your iptables script is still <http://pastebin.com/SqpbmYQQ>

(Your script should do that line adding, not Eliezers - so that you can
be sure the order is always correct).


BTW: you should use iptables-save / iptables-restore instead of a slow
script calling iptables "manually". Those other tools will ensure there
are no gaps in the firewall initialization for nasty traffic to sneak
through.

I am looking at access.log to collect all domains I see heading to skype for
business, as well as IPs. My question is, can I add the domains AND IPs into
the domains-to-bypass.txt that the above script uses?

IIRC you should be able to use domain as the parameter to ipset. But it
will resolve the domain immediately and only add those IPs that it finds
at that time into the pool. Any future changes, or a hidden set of IPs
that rotate in/out will not be listed.

Amos







-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20161207/b2468f9e/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Image 5-5-16 at 11.48 AM.jpg
Type: image/jpeg
Size: 4083 bytes
Desc: Image 5-5-16 at 11.48 AM.jpg
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20161207/b2468f9e/attachment-0001.jpg>


More information about the squid-users mailing list