<div dir="ltr"><div><div>Thank you Amos for the review.</div><div><br></div><div>Following your advice... trying to use the pure transparent proxy (Tproxy) but getting different behaviours with google domains in google chrome and after adding peek there aren't still no names in https addresses.</div><div><br></div><div>First create ssl certs directory, just to check later.</div><div>/usr/lib64/squid/ssl_crtd -c -s /var/log/squid/lib/ssl_db</div><div><br></div><div>remove from squid.conf:</div><div>#tcp_outgoing_address</div><div>#always_direct allow ssl-bump_port</div><div>#ssl_bump none all</div><div><br></div><div>add to squid.conf:</div><div>ssl_bump splice localnet</div><div>ssl_bump peek all</div><div>ssl_bump splice all</div><div><br></div><div>I removed IPv6 restriction in kernel and in new squid compile without --disable-ipv6 option. And I added some similiar IPv6 rules to Tproxy boot script.</div><div><br></div><div>I continue to not generate-host-certificates(=off), I can check it in ssl_db directory. But, just for testing, if I change generate-host-certificates to on then there are certificates changes with my self CA Autority notice in Google Chrome related to google sites, some other sites tested get the certificate in right way. In IE and Firefox there are no certificates issues.</div><div>A workaround for this issue in Google Chrome with websites in google domains was create a specific acl for IP from google domains and splice first like did with localnet:</div><div>acl google dst "/etc/squid/google.txt"<br></div><div>...</div><div>ssl_bump splice google<br></div><div>...</div><div><br></div><div>After those changes and setup ssl-bump with peek for most of sites (e.g. facebook) but still no names in logs, just the IP in https navigation. </div><div><br></div></div><div>access.log:</div><div><div>1439907457.958 49 192.168.0.102 TCP_MISS/200 910 POST <a href="http://ocsp.digicert.com/">http://ocsp.digicert.com/</a> - ORIGINAL_DST/<a href="http://93.184.220.29">93.184.220.29</a> application/ocsp-response</div><div>1439907466.798 59053 192.168.0.102 TCP_TUNNEL/200 3944 CONNECT <a href="http://212.113.184.216:443">212.113.184.216:443</a> - ORIGINAL_DST/<a href="http://212.113.184.216">212.113.184.216</a> -</div><div>1439907472.813 58798 192.168.0.102 TCP_TUNNEL/200 8320 CONNECT <a href="http://212.113.185.35:443">212.113.185.35:443</a> - ORIGINAL_DST/<a href="http://212.113.185.35">212.113.185.35</a> -</div><div>1439907472.817 59014 192.168.0.102 TCP_TUNNEL/200 5234 CONNECT <a href="http://212.113.184.221:443">212.113.184.221:443</a> - ORIGINAL_DST/<a href="http://212.113.184.221">212.113.184.221</a> -</div><div>1439907490.935 65674 192.168.0.102 TCP_TUNNEL/200 4207 CONNECT <a href="http://54.171.32.174:443">54.171.32.174:443</a> - ORIGINAL_DST/<a href="http://54.171.32.174">54.171.32.174</a> -</div><div>1439907527.998 115712 192.168.0.102 TCP_TUNNEL/200 5485 CONNECT <a href="http://54.192.61.197:443">54.192.61.197:443</a> - ORIGINAL_DST/<a href="http://54.192.61.197">54.192.61.197</a> -</div><div>1439907545.804 122741 192.168.0.102 TCP_TUNNEL/200 5817 CONNECT <a href="http://132.245.50.66:443">132.245.50.66:443</a> - ORIGINAL_DST/<a href="http://132.245.50.66">132.245.50.66</a> -</div></div><div><br></div><div><br></div><div>cache.log:</div><div><div>---------</div><div>HTTP/1.1 200 OK</div><div>Accept-Ranges: bytes</div><div>Cache-Control: max-age=499906</div><div>Content-Type: application/ocsp-response</div><div>Date: Tue, 18 Aug 2015 14:17:31 GMT</div><div>ETag: "55d2d1da-1d7"</div><div>Expires: Tue, 25 Aug 2015 02:17:31 GMT</div><div>Last-Modified: Tue, 18 Aug 2015 06:34:02 GMT</div><div>Server: ECS (mad/42F2)</div><div>X-Cache: HIT</div><div>Content-Length: 471</div><div>X-Cache: MISS from squidhead2.skywalker.local</div><div>Via: 1.1 squidhead2.skywalker.local (squid/3.5.7)</div><div>Connection: keep-alive</div><div>----------</div><div>2015/08/18 15:17:37.958 kid1| store.cc(955) checkCachable: StoreEntry::checkCachable: NO: not cachable</div><div>2015/08/18 15:17:37.958 kid1| store.cc(955) checkCachable: StoreEntry::checkCachable: NO: not cachable</div><div>2015/08/18 15:17:37.958 kid1| store.cc(955) checkCachable: StoreEntry::checkCachable: NO: not cachable</div><div>2015/08/18 15:17:38.152 kid1| TcpAcceptor.cc(222) doAccept: New connection on FD 15</div><div>2015/08/18 15:17:38.152 kid1| TcpAcceptor.cc(297) acceptNext: connection on local=[::]:3130 remote=[::] FD 15 flags=25</div><div>2015/08/18 15:17:38.153 kid1| client_side.cc(3890) httpsSslBumpAccessCheckDone: sslBump not needed for local=<a href="http://31.13.90.6:443">31.13.90.6:443</a> remote=192.168.0.102 FD 40 flags=17</div><div>2015/08/18 15:17:38.153 kid1| client_side.cc(2337) parseHttpRequest: HTTP Client local=<a href="http://31.13.90.6:443">31.13.90.6:443</a> remote=192.168.0.102 FD 40 flags=17</div><div>2015/08/18 15:17:38.153 kid1| client_side.cc(2338) parseHttpRequest: HTTP Client REQUEST:</div><div>---------</div><div>CONNECT <a href="http://31.13.90.6:443">31.13.90.6:443</a> HTTP/1.1</div><div>Host: <a href="http://31.13.90.6:443">31.13.90.6:443</a></div><div>----------</div><div>2015/08/18 15:17:38.153 kid1| client_side_request.cc(741) clientAccessCheckDone: The request CONNECT <a href="http://31.13.90.6:443">31.13.90.6:443</a> is ALLOWED; last ACL checked: ssl-bump_port</div><div>2015/08/18 15:17:38.153 kid1| client_side_request.cc(717) clientAccessCheck2: No adapted_http_access configuration. default: ALLOW</div><div>2015/08/18 15:17:38.153 kid1| client_side_request.cc(741) clientAccessCheckDone: The request CONNECT <a href="http://31.13.90.6:443">31.13.90.6:443</a> is ALLOWED; last ACL checked: ssl-bump_port</div><div>2015/08/18 15:17:38.153 kid1| peer_select.cc(280) peerSelectDnsPaths: Found sources for '<a href="http://31.13.90.6:443">31.13.90.6:443</a>'</div><div>2015/08/18 15:17:38.153 kid1| peer_select.cc(281) peerSelectDnsPaths: always_direct = DENIED</div><div>2015/08/18 15:17:38.153 kid1| peer_select.cc(282) peerSelectDnsPaths: never_direct = DENIED</div><div>2015/08/18 15:17:38.153 kid1| peer_select.cc(288) peerSelectDnsPaths: ORIGINAL_DST = local=192.168.0.102 remote=<a href="http://31.13.90.6:443">31.13.90.6:443</a> flags=25</div><div>2015/08/18 15:17:38.153 kid1| peer_select.cc(295) peerSelectDnsPaths: timedout = 0</div><div>2015/08/18 15:17:38.156 kid1| TcpAcceptor.cc(222) doAccept: New connection on FD 15</div><div>2015/08/18 15:17:38.156 kid1| TcpAcceptor.cc(297) acceptNext: connection on local=[::]:3130 remote=[::] FD 15 flags=25</div><div>2015/08/18 15:17:38.156 kid1| client_side.cc(3890) httpsSslBumpAccessCheckDone: sslBump not needed for local=<a href="http://31.13.90.6:443">31.13.90.6:443</a> remote=192.168.0.102 FD 42 flags=17</div><div>2015/08/18 15:17:38.156 kid1| client_side.cc(2337) parseHttpRequest: HTTP Client local=<a href="http://31.13.90.6:443">31.13.90.6:443</a> remote=192.168.0.102 FD 42 flags=17</div><div>2015/08/18 15:17:38.156 kid1| client_side.cc(2338) parseHttpRequest: HTTP Client REQUEST:</div><div>---------</div><div>CONNECT <a href="http://31.13.90.6:443">31.13.90.6:443</a> HTTP/1.1</div><div>Host: <a href="http://31.13.90.6:443">31.13.90.6:443</a></div><div>----------</div><div>2015/08/18 15:17:38.157 kid1| client_side_request.cc(741) clientAccessCheckDone: The request CONNECT <a href="http://31.13.90.6:443">31.13.90.6:443</a> is ALLOWED; last ACL checked: ssl-bump_port</div><div>2015/08/18 15:17:38.157 kid1| client_side_request.cc(717) clientAccessCheck2: No adapted_http_access configuration. default: ALLOW</div><div>2015/08/18 15:17:38.157 kid1| client_side_request.cc(741) clientAccessCheckDone: The request CONNECT <a href="http://31.13.90.6:443">31.13.90.6:443</a> is ALLOWED; last ACL checked: ssl-bump_port</div><div>2015/08/18 15:17:38.157 kid1| peer_select.cc(280) peerSelectDnsPaths: Found sources for '<a href="http://31.13.90.6:443">31.13.90.6:443</a>'</div><div>2015/08/18 15:17:38.157 kid1| peer_select.cc(281) peerSelectDnsPaths: always_direct = DENIED</div><div>2015/08/18 15:17:38.157 kid1| peer_select.cc(282) peerSelectDnsPaths: never_direct = DENIED</div><div>2015/08/18 15:17:38.157 kid1| peer_select.cc(288) peerSelectDnsPaths: ORIGINAL_DST = local=192.168.0.102 remote=<a href="http://31.13.90.6:443">31.13.90.6:443</a> flags=25</div><div>2015/08/18 15:17:38.157 kid1| peer_select.cc(295) peerSelectDnsPaths: timedout = 0</div><div>2015/08/18 15:17:45.351 kid1| TcpAcceptor.cc(222) doAccept: New connection on FD 15</div><div>2015/08/18 15:17:45.351 kid1| TcpAcceptor.cc(297) acceptNext: connection on local=[::]:3130 remote=[::] FD 15 flags=25</div><div>2015/08/18 15:17:45.351 kid1| client_side.cc(3890) httpsSslBumpAccessCheckDone: sslBump not needed for local=<a href="http://212.113.185.24:443">212.113.185.24:443</a> remote=192.168.0.102 FD 46 flags=17</div><div>2015/08/18 15:17:45.351 kid1| client_side.cc(2337) parseHttpRequest: HTTP Client local=<a href="http://212.113.185.24:443">212.113.185.24:443</a> remote=192.168.0.102 FD 46 flags=17</div><div>2015/08/18 15:17:45.351 kid1| client_side.cc(2338) parseHttpRequest: HTTP Client REQUEST:</div><div>---------</div><div>CONNECT <a href="http://212.113.185.24:443">212.113.185.24:443</a> HTTP/1.1</div><div>Host: <a href="http://212.113.185.24:443">212.113.185.24:443</a></div><div>----------</div><div>2015/08/18 15:17:45.351 kid1| client_side_request.cc(741) clientAccessCheckDone: The request CONNECT <a href="http://212.113.185.24:443">212.113.185.24:443</a> is ALLOWED; last ACL checked: ssl-bump_port</div><div>2015/08/18 15:17:45.351 kid1| client_side_request.cc(717) clientAccessCheck2: No adapted_http_access configuration. default: ALLOW</div><div>2015/08/18 15:17:45.351 kid1| client_side_request.cc(741) clientAccessCheckDone: The request CONNECT <a href="http://212.113.185.24:443">212.113.185.24:443</a> is ALLOWED; last ACL checked: ssl-bump_port</div><div>2015/08/18 15:17:45.351 kid1| peer_select.cc(280) peerSelectDnsPaths: Found sources for '<a href="http://212.113.185.24:443">212.113.185.24:443</a>'</div><div>2015/08/18 15:17:45.351 kid1| peer_select.cc(281) peerSelectDnsPaths: always_direct = DENIED</div><div>2015/08/18 15:17:45.351 kid1| peer_select.cc(282) peerSelectDnsPaths: never_direct = DENIED</div><div>2015/08/18 15:17:45.351 kid1| peer_select.cc(288) peerSelectDnsPaths: ORIGINAL_DST = local=192.168.0.102 remote=<a href="http://212.113.185.24:443">212.113.185.24:443</a> flags=25</div><div>2015/08/18 15:17:45.351 kid1| peer_select.cc(295) peerSelectDnsPaths: timedout = 0</div></div><div><br></div><div><span id="result_box" class="short_text" lang="en"><span class="hps">Can</span> <span class="hps">I</span> <span class="hps">ask what</span> <span class="hps">I missed?</span></span><br></div><div>Thank you for your time.</div><div><br></div><div><br></div><div class="gmail_extra">
<br><div class="gmail_quote">On Tue, Aug 18, 2015 at 6:30 AM, Amos Jeffries <span dir="ltr"><<a href="mailto:squid3@treenet.co.nz" target="_blank">squid3@treenet.co.nz</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On 18/08/2015 12:25 a.m., Pedro Correia Sardinha wrote:<br>
> Hello,<br>
><br>
> I'm trying to setup a Squid server simple as possible just to review the<br>
> web use in office using the last stable version 3.5.7.<br>
><br>
<br>
</span>And you chose TPROXY with ssl-bump'ing. The two most complex features to<br>
setup. lol.<br>
<span class=""><br>
> I setup the bridge with 2 NIC, br0 with IP 192.168.0.5 and I had disable<br>
> IPv6 on boot in my Slackware Current (Fri Aug 14 2015) server.<br>
<br>
</span>Sigh. Ever heard of IPv6-over-IPv4, 6-in-4, 6to4, etc. ?<br>
All protocols designed to "fix" connectivity going through machines<br>
setup like yours.<br>
<br>
And why bother disabling a (BCP 177) mandatory part of the kernel?<br>
The correct way to handle unwanted traffic is to firewall it. Not to<br>
play around with kernel internals.<br>
<br>
<snip><br>
> My squid.conf:<br>
<br>
> tcp_outgoing_address 85.138.204.43<br>
<br>
This is irrelevant with TPROXY. The client IP address is used instead.<br>
<br>
For the regular forward-proxy traffic on port 3128 the machines default<br>
IP will be used.<br>
<span class=""><br>
<br>
> dns_v4_first on<br>
> pinger_enable off<br>
> http_port 3128<br>
> http_port 3129 tproxy<br>
> https_port 3130 ssl-bump tproxy generate-host-certificates=off<br>
> cert=/etc/squid/ssl/squid.pem cafile=/etc/squid/ssl/squid.pem<br>
> always_direct allow ssl-bump_port<br>
> ssl_bump none all<br>
<br>
</span>You have configued Squid not to even look at the TLS details.<br>
<span class=""><br>
<br>
> dns_nameservers 8.8.8.8 8.8.4.4<br>
> access_log daemon:/var/log/squid/access.log squid<br>
> cache deny all<br>
> pid_filename /var/run/squid/squid.pid<br>
> coredump_dir /var/log/squid/cache/squid<br>
> visible_hostname myservername.domain.local<br>
><br>
> In general the configuration (squid.conf) it's working but has some<br>
> incomplete behaviors as shows in log files.<br>
><br>
> access.log (I know this is Facebook but there are no dns resolusion in<br>
> https, just IP):<br>
> 1439811492.625 2377 192.168.0.102 TCP_TUNNEL/200 3574 CONNECT<br>
> <a href="http://31.13.90.2:443" rel="noreferrer" target="_blank">31.13.90.2:443</a> - ORIGINAL_DST/<a href="http://31.13.90.2" rel="noreferrer" target="_blank">31.13.90.2</a> -<br>
<br>
<br>
</span>What sort of resolution were you expecting?<br>
<br>
* The above log line is recording the TCP connection. TCP packets do not<br>
have any "domain name" fields that need resolving to IP addresses.<br>
<br>
* you also configured Squid not to look at the TLS details where it<br>
might have found an SNI entry with server domain name.<br>
<br>
The result is that Squid is working purely with TPROXY IP addresses and<br>
setting up a TCP tunnel to relay the traffic through.<br>
<span class=""><br>
<br>
><br>
> cache.log:<br>
> HTTP/1.1 200 OK<br>
> Accept-Ranges: bytes<br>
> Cache-Control: max-age=504747<br>
> Content-Type: application/ocsp-response<br>
> Date: Mon, 17 Aug 2015 11:38:03 GMT<br>
> ETag: "55d15943-1d7"<br>
> Expires: Sun, 23 Aug 2015 23:38:03 GMT<br>
> Last-Modified: Mon, 17 Aug 2015 03:47:15 GMT<br>
> Server: ECS (mad/439C)<br>
> X-Cache: HIT<br>
> Content-Length: 471<br>
> X-Cache: MISS from squidhead2.skywalker.local<br>
> Via: 1.1 squidhead2.skywalker.local (squid/3.5.7)<br>
> Connection: keep-alive<br>
> ----------<br>
> 2015/08/17 12:38:09.067 kid1| store.cc(955) checkCachable:<br>
> StoreEntry::checkCachable: NO: not cachable<br>
> 2015/08/17 12:38:09.067 kid1| store.cc(955) checkCachable:<br>
> StoreEntry::checkCachable: NO: not cachable<br>
> 2015/08/17 12:38:09.067 kid1| store.cc(955) checkCachable:<br>
> StoreEntry::checkCachable: NO: not cachable<br>
<br>
<br>
</span>Above is all the end of some transaction that was started earlier. No<br>
useful details in the provided log snippet about it.<br>
<br>
<br>
<br>
This is where a transaction actually starts:<br>
<span class=""><br>
> 2015/08/17 12:38:10.248 kid1| TcpAcceptor.cc(222) doAccept: New connection<br>
> on FD 12<br>
> 2015/08/17 12:38:10.248 kid1| TcpAcceptor.cc(297) acceptNext: connection on<br>
> local=<a href="http://0.0.0.0:3130" rel="noreferrer" target="_blank">0.0.0.0:3130</a> remote=[::] FD 12 flags=25<br>
> 2015/08/17 12:38:10.248 kid1| client_side.cc(3890)<br>
<br>
</span>Notice how the connection local IP:port details change from port 3130 to<br>
port 443. Thats TPROXY working.<br>
<span class=""><br>
> httpsSslBumpAccessCheckDone: sslBump not needed for local=<a href="http://31.13.90.2:443" rel="noreferrer" target="_blank">31.13.90.2:443</a><br>
> remote=192.168.0.102 FD 50 flags=17<br>
<br>
</span>This is the "ssl_bump none" action working (by not doing anything TLS<br>
related) exactly as you configured.<br>
<br>
Squid is now processing an internally generated CONNECT request<br>
representing the intercepted TPROXY connection in a way that can be<br>
logged and/or relayed to other proxies if it needs to.<br>
<span class=""><br>
> 2015/08/17 12:38:10.248 kid1| client_side.cc(2337) parseHttpRequest: HTTP<br>
> Client local=<a href="http://31.13.90.2:443" rel="noreferrer" target="_blank">31.13.90.2:443</a> remote=192.168.0.102 FD 50 flags=17<br>
> 2015/08/17 12:38:10.248 kid1| client_side.cc(2338) parseHttpRequest: HTTP<br>
> Client REQUEST:<br>
> ---------<br>
> CONNECT <a href="http://31.13.90.2:443" rel="noreferrer" target="_blank">31.13.90.2:443</a> HTTP/1.1<br>
> Host: <a href="http://31.13.90.2:443" rel="noreferrer" target="_blank">31.13.90.2:443</a><br>
> ---------<br>
> 2015/08/17 12:38:10.248 kid1| client_side_request.cc(741)<br>
> clientAccessCheckDone: The request CONNECT <a href="http://31.13.90.2:443" rel="noreferrer" target="_blank">31.13.90.2:443</a> is ALLOWED; last<br>
> ACL checked: localnet<br>
> 2015/08/17 12:38:10.248 kid1| client_side_request.cc(717)<br>
> clientAccessCheck2: No adapted_http_access configuration. default: ALLOW<br>
> 2015/08/17 12:38:10.248 kid1| client_side_request.cc(741)<br>
> clientAccessCheckDone: The request CONNECT <a href="http://31.13.90.2:443" rel="noreferrer" target="_blank">31.13.90.2:443</a> is ALLOWED; last<br>
> ACL checked: localnet<br>
> 2015/08/17 12:38:10.248 kid1| peer_select.cc(280) peerSelectDnsPaths: Found<br>
> sources for '<a href="http://31.13.90.2:443" rel="noreferrer" target="_blank">31.13.90.2:443</a>'<br>
<br>
</span>The CONNECT request uses a raw-IP address provided by TPROXY. There is<br>
no name to resolve.<br>
<span class=""><br>
> 2015/08/17 12:38:10.248 kid1| peer_select.cc(281) peerSelectDnsPaths:<br>
> always_direct = ALLOWED<br>
> 2015/08/17 12:38:10.248 kid1| peer_select.cc(282) peerSelectDnsPaths:<br>
> never_direct = DENIED<br>
> 2015/08/17 12:38:10.248 kid1| peer_select.cc(288) peerSelectDnsPaths:<br>
> ORIGINAL_DST = local=192.168.0.102 remote=<a href="http://31.13.90.2:443" rel="noreferrer" target="_blank">31.13.90.2:443</a> flags=25<br>
> 2015/08/17 12:38:10.248 kid1| peer_select.cc(295) peerSelectDnsPaths:<br>
> timedout = 0<br>
<br>
</span>... stuff happens for 2 seconds...<br>
<span class=""><br>
> 2015/08/17 12:38:12.621 kid1| client_side.cc(815) swanSong: local=<br>
> <a href="http://31.13.90.2:443" rel="noreferrer" target="_blank">31.13.90.2:443</a> remote=192.168.0.102 flags=17<br>
> 2015/08/17 12:38:12.625 kid1| client_side.cc(815) swanSong: local=<br>
> <a href="http://31.13.90.2:443" rel="noreferrer" target="_blank">31.13.90.2:443</a> remote=192.168.0.102 flags=17<br>
<br>
</span>Then the connection closes.<br>
<br>
Looks perfectly normal and expected behaviour to me considering what you<br>
configured.<br>
<span class=""><br>
><br>
><br>
> The logs with http (port 80) has the name resolution of navigation.<br>
><br>
> I disabled pinger because give some error:<br>
</span><snip><br>
<span class="">> 2015/08/17 12:49:55| FATAL: pinger: Unable to open any ICMP sockets.<br>
<br>
</span>Okay. Not a big problem. The pinger helper needs its suid bit set, which<br>
is not working on all system installations yet. Disabling it is fine.<br>
<span class=""><br>
<br>
> Do I have to setup local DNS server? the internal DNS of squid can't handle<br>
> https in Tproxy?<br>
> What's missing to have name resolution in https traffic as its showed in<br>
> http traffic?<br>
<br>
</span>Whats missing is the SSL-bumping part. HTTPS works differently to HTTP.<br>
<br>
The URL domain name and all the rest of the HTTP message is encrypted<br>
*in full*. There is simply no client HTTP message involved if you don't<br>
decrypt.<br>
<br>
As I mentioned above what you are seeing in the log is a Squid-generated<br>
CONNECT message. Its the HTTP representation of the intercepted TCP SYN<br>
packet and contains purely raw-IP:port details.<br>
<br>
TLS does have an SNI record which is sent by browsers un-encrypted that<br>
can be used as domain for some things. BUT, that requires "ssl_bump<br>
peek" action at minimum, and has no guarantee of actually being present.<br>
<br>
SNI is also still a new feature, and is not used for these fake CONNECT<br>
requests anyway (since they represent the TCP SYN). So you wont see any<br>
log difference in 3.5 even if you do let your Squid use it for ACLs.<br>
<br>
Amos<br>
<br>
_______________________________________________<br>
squid-users mailing list<br>
<a href="mailto:squid-users@lists.squid-cache.org">squid-users@lists.squid-cache.org</a><br>
<a href="http://lists.squid-cache.org/listinfo/squid-users" rel="noreferrer" target="_blank">http://lists.squid-cache.org/listinfo/squid-users</a><br>
</blockquote></div><br></div></div>