[squid-users] Bridge/Tproxy: https dns
Pedro Correia Sardinha
correiasardinha at gmail.com
Tue Aug 18 16:54:11 UTC 2015
Thank you Amos for the review.
Following your advice... trying to use the pure transparent proxy (Tproxy)
but getting different behaviours with google domains in google chrome and
after adding peek there aren't still no names in https addresses.
First create ssl certs directory, just to check later.
/usr/lib64/squid/ssl_crtd -c -s /var/log/squid/lib/ssl_db
remove from squid.conf:
#tcp_outgoing_address
#always_direct allow ssl-bump_port
#ssl_bump none all
add to squid.conf:
ssl_bump splice localnet
ssl_bump peek all
ssl_bump splice all
I removed IPv6 restriction in kernel and in new squid compile without
--disable-ipv6 option. And I added some similiar IPv6 rules to Tproxy boot
script.
I continue to not generate-host-certificates(=off), I can check it in
ssl_db directory. But, just for testing, if I change
generate-host-certificates to on then there are certificates changes with
my self CA Autority notice in Google Chrome related to google sites, some
other sites tested get the certificate in right way. In IE and Firefox
there are no certificates issues.
A workaround for this issue in Google Chrome with websites in google
domains was create a specific acl for IP from google domains and splice
first like did with localnet:
acl google dst "/etc/squid/google.txt"
...
ssl_bump splice google
...
After those changes and setup ssl-bump with peek for most of sites (e.g.
facebook) but still no names in logs, just the IP in https navigation.
access.log:
1439907457.958 49 192.168.0.102 TCP_MISS/200 910 POST
http://ocsp.digicert.com/ - ORIGINAL_DST/93.184.220.29
application/ocsp-response
1439907466.798 59053 192.168.0.102 TCP_TUNNEL/200 3944 CONNECT
212.113.184.216:443 - ORIGINAL_DST/212.113.184.216 -
1439907472.813 58798 192.168.0.102 TCP_TUNNEL/200 8320 CONNECT
212.113.185.35:443 - ORIGINAL_DST/212.113.185.35 -
1439907472.817 59014 192.168.0.102 TCP_TUNNEL/200 5234 CONNECT
212.113.184.221:443 - ORIGINAL_DST/212.113.184.221 -
1439907490.935 65674 192.168.0.102 TCP_TUNNEL/200 4207 CONNECT
54.171.32.174:443 - ORIGINAL_DST/54.171.32.174 -
1439907527.998 115712 192.168.0.102 TCP_TUNNEL/200 5485 CONNECT
54.192.61.197:443 - ORIGINAL_DST/54.192.61.197 -
1439907545.804 122741 192.168.0.102 TCP_TUNNEL/200 5817 CONNECT
132.245.50.66:443 - ORIGINAL_DST/132.245.50.66 -
cache.log:
---------
HTTP/1.1 200 OK
Accept-Ranges: bytes
Cache-Control: max-age=499906
Content-Type: application/ocsp-response
Date: Tue, 18 Aug 2015 14:17:31 GMT
ETag: "55d2d1da-1d7"
Expires: Tue, 25 Aug 2015 02:17:31 GMT
Last-Modified: Tue, 18 Aug 2015 06:34:02 GMT
Server: ECS (mad/42F2)
X-Cache: HIT
Content-Length: 471
X-Cache: MISS from squidhead2.skywalker.local
Via: 1.1 squidhead2.skywalker.local (squid/3.5.7)
Connection: keep-alive
----------
2015/08/18 15:17:37.958 kid1| store.cc(955) checkCachable:
StoreEntry::checkCachable: NO: not cachable
2015/08/18 15:17:37.958 kid1| store.cc(955) checkCachable:
StoreEntry::checkCachable: NO: not cachable
2015/08/18 15:17:37.958 kid1| store.cc(955) checkCachable:
StoreEntry::checkCachable: NO: not cachable
2015/08/18 15:17:38.152 kid1| TcpAcceptor.cc(222) doAccept: New connection
on FD 15
2015/08/18 15:17:38.152 kid1| TcpAcceptor.cc(297) acceptNext: connection on
local=[::]:3130 remote=[::] FD 15 flags=25
2015/08/18 15:17:38.153 kid1| client_side.cc(3890)
httpsSslBumpAccessCheckDone: sslBump not needed for local=31.13.90.6:443
remote=192.168.0.102 FD 40 flags=17
2015/08/18 15:17:38.153 kid1| client_side.cc(2337) parseHttpRequest: HTTP
Client local=31.13.90.6:443 remote=192.168.0.102 FD 40 flags=17
2015/08/18 15:17:38.153 kid1| client_side.cc(2338) parseHttpRequest: HTTP
Client REQUEST:
---------
CONNECT 31.13.90.6:443 HTTP/1.1
Host: 31.13.90.6:443
----------
2015/08/18 15:17:38.153 kid1| client_side_request.cc(741)
clientAccessCheckDone: The request CONNECT 31.13.90.6:443 is ALLOWED; last
ACL checked: ssl-bump_port
2015/08/18 15:17:38.153 kid1| client_side_request.cc(717)
clientAccessCheck2: No adapted_http_access configuration. default: ALLOW
2015/08/18 15:17:38.153 kid1| client_side_request.cc(741)
clientAccessCheckDone: The request CONNECT 31.13.90.6:443 is ALLOWED; last
ACL checked: ssl-bump_port
2015/08/18 15:17:38.153 kid1| peer_select.cc(280) peerSelectDnsPaths: Found
sources for '31.13.90.6:443'
2015/08/18 15:17:38.153 kid1| peer_select.cc(281) peerSelectDnsPaths:
always_direct = DENIED
2015/08/18 15:17:38.153 kid1| peer_select.cc(282) peerSelectDnsPaths:
never_direct = DENIED
2015/08/18 15:17:38.153 kid1| peer_select.cc(288) peerSelectDnsPaths:
ORIGINAL_DST = local=192.168.0.102 remote=31.13.90.6:443 flags=25
2015/08/18 15:17:38.153 kid1| peer_select.cc(295) peerSelectDnsPaths:
timedout = 0
2015/08/18 15:17:38.156 kid1| TcpAcceptor.cc(222) doAccept: New connection
on FD 15
2015/08/18 15:17:38.156 kid1| TcpAcceptor.cc(297) acceptNext: connection on
local=[::]:3130 remote=[::] FD 15 flags=25
2015/08/18 15:17:38.156 kid1| client_side.cc(3890)
httpsSslBumpAccessCheckDone: sslBump not needed for local=31.13.90.6:443
remote=192.168.0.102 FD 42 flags=17
2015/08/18 15:17:38.156 kid1| client_side.cc(2337) parseHttpRequest: HTTP
Client local=31.13.90.6:443 remote=192.168.0.102 FD 42 flags=17
2015/08/18 15:17:38.156 kid1| client_side.cc(2338) parseHttpRequest: HTTP
Client REQUEST:
---------
CONNECT 31.13.90.6:443 HTTP/1.1
Host: 31.13.90.6:443
----------
2015/08/18 15:17:38.157 kid1| client_side_request.cc(741)
clientAccessCheckDone: The request CONNECT 31.13.90.6:443 is ALLOWED; last
ACL checked: ssl-bump_port
2015/08/18 15:17:38.157 kid1| client_side_request.cc(717)
clientAccessCheck2: No adapted_http_access configuration. default: ALLOW
2015/08/18 15:17:38.157 kid1| client_side_request.cc(741)
clientAccessCheckDone: The request CONNECT 31.13.90.6:443 is ALLOWED; last
ACL checked: ssl-bump_port
2015/08/18 15:17:38.157 kid1| peer_select.cc(280) peerSelectDnsPaths: Found
sources for '31.13.90.6:443'
2015/08/18 15:17:38.157 kid1| peer_select.cc(281) peerSelectDnsPaths:
always_direct = DENIED
2015/08/18 15:17:38.157 kid1| peer_select.cc(282) peerSelectDnsPaths:
never_direct = DENIED
2015/08/18 15:17:38.157 kid1| peer_select.cc(288) peerSelectDnsPaths:
ORIGINAL_DST = local=192.168.0.102 remote=31.13.90.6:443 flags=25
2015/08/18 15:17:38.157 kid1| peer_select.cc(295) peerSelectDnsPaths:
timedout = 0
2015/08/18 15:17:45.351 kid1| TcpAcceptor.cc(222) doAccept: New connection
on FD 15
2015/08/18 15:17:45.351 kid1| TcpAcceptor.cc(297) acceptNext: connection on
local=[::]:3130 remote=[::] FD 15 flags=25
2015/08/18 15:17:45.351 kid1| client_side.cc(3890)
httpsSslBumpAccessCheckDone: sslBump not needed for local=212.113.185.24:443
remote=192.168.0.102 FD 46 flags=17
2015/08/18 15:17:45.351 kid1| client_side.cc(2337) parseHttpRequest: HTTP
Client local=212.113.185.24:443 remote=192.168.0.102 FD 46 flags=17
2015/08/18 15:17:45.351 kid1| client_side.cc(2338) parseHttpRequest: HTTP
Client REQUEST:
---------
CONNECT 212.113.185.24:443 HTTP/1.1
Host: 212.113.185.24:443
----------
2015/08/18 15:17:45.351 kid1| client_side_request.cc(741)
clientAccessCheckDone: The request CONNECT 212.113.185.24:443 is ALLOWED;
last ACL checked: ssl-bump_port
2015/08/18 15:17:45.351 kid1| client_side_request.cc(717)
clientAccessCheck2: No adapted_http_access configuration. default: ALLOW
2015/08/18 15:17:45.351 kid1| client_side_request.cc(741)
clientAccessCheckDone: The request CONNECT 212.113.185.24:443 is ALLOWED;
last ACL checked: ssl-bump_port
2015/08/18 15:17:45.351 kid1| peer_select.cc(280) peerSelectDnsPaths: Found
sources for '212.113.185.24:443'
2015/08/18 15:17:45.351 kid1| peer_select.cc(281) peerSelectDnsPaths:
always_direct = DENIED
2015/08/18 15:17:45.351 kid1| peer_select.cc(282) peerSelectDnsPaths:
never_direct = DENIED
2015/08/18 15:17:45.351 kid1| peer_select.cc(288) peerSelectDnsPaths:
ORIGINAL_DST = local=192.168.0.102 remote=212.113.185.24:443 flags=25
2015/08/18 15:17:45.351 kid1| peer_select.cc(295) peerSelectDnsPaths:
timedout = 0
Can I ask what I missed?
Thank you for your time.
On Tue, Aug 18, 2015 at 6:30 AM, Amos Jeffries <squid3 at treenet.co.nz> wrote:
> On 18/08/2015 12:25 a.m., Pedro Correia Sardinha wrote:
> > Hello,
> >
> > I'm trying to setup a Squid server simple as possible just to review the
> > web use in office using the last stable version 3.5.7.
> >
>
> And you chose TPROXY with ssl-bump'ing. The two most complex features to
> setup. lol.
>
> > I setup the bridge with 2 NIC, br0 with IP 192.168.0.5 and I had disable
> > IPv6 on boot in my Slackware Current (Fri Aug 14 2015) server.
>
> Sigh. Ever heard of IPv6-over-IPv4, 6-in-4, 6to4, etc. ?
> All protocols designed to "fix" connectivity going through machines
> setup like yours.
>
> And why bother disabling a (BCP 177) mandatory part of the kernel?
> The correct way to handle unwanted traffic is to firewall it. Not to
> play around with kernel internals.
>
> <snip>
> > My squid.conf:
>
> > tcp_outgoing_address 85.138.204.43
>
> This is irrelevant with TPROXY. The client IP address is used instead.
>
> For the regular forward-proxy traffic on port 3128 the machines default
> IP will be used.
>
>
> > dns_v4_first on
> > pinger_enable off
> > http_port 3128
> > http_port 3129 tproxy
> > https_port 3130 ssl-bump tproxy generate-host-certificates=off
> > cert=/etc/squid/ssl/squid.pem cafile=/etc/squid/ssl/squid.pem
> > always_direct allow ssl-bump_port
> > ssl_bump none all
>
> You have configued Squid not to even look at the TLS details.
>
>
> > dns_nameservers 8.8.8.8 8.8.4.4
> > access_log daemon:/var/log/squid/access.log squid
> > cache deny all
> > pid_filename /var/run/squid/squid.pid
> > coredump_dir /var/log/squid/cache/squid
> > visible_hostname myservername.domain.local
> >
> > In general the configuration (squid.conf) it's working but has some
> > incomplete behaviors as shows in log files.
> >
> > access.log (I know this is Facebook but there are no dns resolusion in
> > https, just IP):
> > 1439811492.625 2377 192.168.0.102 TCP_TUNNEL/200 3574 CONNECT
> > 31.13.90.2:443 - ORIGINAL_DST/31.13.90.2 -
>
>
> What sort of resolution were you expecting?
>
> * The above log line is recording the TCP connection. TCP packets do not
> have any "domain name" fields that need resolving to IP addresses.
>
> * you also configured Squid not to look at the TLS details where it
> might have found an SNI entry with server domain name.
>
> The result is that Squid is working purely with TPROXY IP addresses and
> setting up a TCP tunnel to relay the traffic through.
>
>
> >
> > cache.log:
> > HTTP/1.1 200 OK
> > Accept-Ranges: bytes
> > Cache-Control: max-age=504747
> > Content-Type: application/ocsp-response
> > Date: Mon, 17 Aug 2015 11:38:03 GMT
> > ETag: "55d15943-1d7"
> > Expires: Sun, 23 Aug 2015 23:38:03 GMT
> > Last-Modified: Mon, 17 Aug 2015 03:47:15 GMT
> > Server: ECS (mad/439C)
> > X-Cache: HIT
> > Content-Length: 471
> > X-Cache: MISS from squidhead2.skywalker.local
> > Via: 1.1 squidhead2.skywalker.local (squid/3.5.7)
> > Connection: keep-alive
> > ----------
> > 2015/08/17 12:38:09.067 kid1| store.cc(955) checkCachable:
> > StoreEntry::checkCachable: NO: not cachable
> > 2015/08/17 12:38:09.067 kid1| store.cc(955) checkCachable:
> > StoreEntry::checkCachable: NO: not cachable
> > 2015/08/17 12:38:09.067 kid1| store.cc(955) checkCachable:
> > StoreEntry::checkCachable: NO: not cachable
>
>
> Above is all the end of some transaction that was started earlier. No
> useful details in the provided log snippet about it.
>
>
>
> This is where a transaction actually starts:
>
> > 2015/08/17 12:38:10.248 kid1| TcpAcceptor.cc(222) doAccept: New
> connection
> > on FD 12
> > 2015/08/17 12:38:10.248 kid1| TcpAcceptor.cc(297) acceptNext: connection
> on
> > local=0.0.0.0:3130 remote=[::] FD 12 flags=25
> > 2015/08/17 12:38:10.248 kid1| client_side.cc(3890)
>
> Notice how the connection local IP:port details change from port 3130 to
> port 443. Thats TPROXY working.
>
> > httpsSslBumpAccessCheckDone: sslBump not needed for local=31.13.90.2:443
> > remote=192.168.0.102 FD 50 flags=17
>
> This is the "ssl_bump none" action working (by not doing anything TLS
> related) exactly as you configured.
>
> Squid is now processing an internally generated CONNECT request
> representing the intercepted TPROXY connection in a way that can be
> logged and/or relayed to other proxies if it needs to.
>
> > 2015/08/17 12:38:10.248 kid1| client_side.cc(2337) parseHttpRequest: HTTP
> > Client local=31.13.90.2:443 remote=192.168.0.102 FD 50 flags=17
> > 2015/08/17 12:38:10.248 kid1| client_side.cc(2338) parseHttpRequest: HTTP
> > Client REQUEST:
> > ---------
> > CONNECT 31.13.90.2:443 HTTP/1.1
> > Host: 31.13.90.2:443
> > ---------
> > 2015/08/17 12:38:10.248 kid1| client_side_request.cc(741)
> > clientAccessCheckDone: The request CONNECT 31.13.90.2:443 is ALLOWED;
> last
> > ACL checked: localnet
> > 2015/08/17 12:38:10.248 kid1| client_side_request.cc(717)
> > clientAccessCheck2: No adapted_http_access configuration. default: ALLOW
> > 2015/08/17 12:38:10.248 kid1| client_side_request.cc(741)
> > clientAccessCheckDone: The request CONNECT 31.13.90.2:443 is ALLOWED;
> last
> > ACL checked: localnet
> > 2015/08/17 12:38:10.248 kid1| peer_select.cc(280) peerSelectDnsPaths:
> Found
> > sources for '31.13.90.2:443'
>
> The CONNECT request uses a raw-IP address provided by TPROXY. There is
> no name to resolve.
>
> > 2015/08/17 12:38:10.248 kid1| peer_select.cc(281) peerSelectDnsPaths:
> > always_direct = ALLOWED
> > 2015/08/17 12:38:10.248 kid1| peer_select.cc(282) peerSelectDnsPaths:
> > never_direct = DENIED
> > 2015/08/17 12:38:10.248 kid1| peer_select.cc(288) peerSelectDnsPaths:
> > ORIGINAL_DST = local=192.168.0.102 remote=31.13.90.2:443 flags=25
> > 2015/08/17 12:38:10.248 kid1| peer_select.cc(295) peerSelectDnsPaths:
> > timedout = 0
>
> ... stuff happens for 2 seconds...
>
> > 2015/08/17 12:38:12.621 kid1| client_side.cc(815) swanSong: local=
> > 31.13.90.2:443 remote=192.168.0.102 flags=17
> > 2015/08/17 12:38:12.625 kid1| client_side.cc(815) swanSong: local=
> > 31.13.90.2:443 remote=192.168.0.102 flags=17
>
> Then the connection closes.
>
> Looks perfectly normal and expected behaviour to me considering what you
> configured.
>
> >
> >
> > The logs with http (port 80) has the name resolution of navigation.
> >
> > I disabled pinger because give some error:
> <snip>
> > 2015/08/17 12:49:55| FATAL: pinger: Unable to open any ICMP sockets.
>
> Okay. Not a big problem. The pinger helper needs its suid bit set, which
> is not working on all system installations yet. Disabling it is fine.
>
>
> > Do I have to setup local DNS server? the internal DNS of squid can't
> handle
> > https in Tproxy?
> > What's missing to have name resolution in https traffic as its showed in
> > http traffic?
>
> Whats missing is the SSL-bumping part. HTTPS works differently to HTTP.
>
> The URL domain name and all the rest of the HTTP message is encrypted
> *in full*. There is simply no client HTTP message involved if you don't
> decrypt.
>
> As I mentioned above what you are seeing in the log is a Squid-generated
> CONNECT message. Its the HTTP representation of the intercepted TCP SYN
> packet and contains purely raw-IP:port details.
>
> TLS does have an SNI record which is sent by browsers un-encrypted that
> can be used as domain for some things. BUT, that requires "ssl_bump
> peek" action at minimum, and has no guarantee of actually being present.
>
> SNI is also still a new feature, and is not used for these fake CONNECT
> requests anyway (since they represent the TCP SYN). So you wont see any
> log difference in 3.5 even if you do let your Squid use it for ACLs.
>
> Amos
>
> _______________________________________________
> squid-users mailing list
> squid-users at lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20150818/2caa9cdf/attachment-0001.html>
More information about the squid-users
mailing list