<div dir="ltr"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br><br><br>
Message: 5<br>
Date: Wed, 22 Jan 2020 18:36:37 +1300<br>
From: Amos Jeffries <<a href="mailto:squid3@treenet.co.nz" target="_blank">squid3@treenet.co.nz</a>><br>
To: <a href="mailto:squid-users@lists.squid-cache.org" target="_blank">squid-users@lists.squid-cache.org</a><br>
Subject: Re: [squid-users] Issues with TLS inspection Intercept Mode.<br>
Message-ID: <<a href="mailto:2fa147c4-e499-9244-d2d1-c52438e2f4f4@treenet.co.nz" target="_blank">2fa147c4-e499-9244-d2d1-c52438e2f4f4@treenet.co.nz</a>><br>
Content-Type: text/plain; charset=utf-8<br>
<br>
On 22/01/20 7:39 am, aashutosh kalyankar wrote:<br>
> The problem I am seeing is the intercept port initiates HTTP connection<br>
> to self-IP instead of the web server IP it gets from the DNS request.<br>
<br>
Neither of those IPs should be used.<br>
<br>
Self-IP indicates that the traffic is being delivered to the proxy port<br>
as explicit proxy traffic. Or that the NAT system is broken. We usually<br>
get this complaint from people who are testing their proxy by sending<br>
traffic directly to the proxy port.<br>
<br>
DNS is only involved to validate the HTTP Host header against the IP<br>
address to forbid caching dangerous contents.<br>
<br>
<br>
> Filtered Tcpdump<br>
> screenshot @ <a href="https://drive.google.com/open?id=0ByReiwdSAAY_VXBPTjF1M3dYTnBTTnhFVnRocXFveUlNSlNj" rel="noreferrer" target="_blank">https://drive.google.com/open?id=0ByReiwdSAAY_VXBPTjF1M3dYTnBTTnhFVnRocXFveUlNSlNj</a><br>
> <br>
<br>
Screenshots are rarely useful. You are logging level 11,3 to cache.log,<br>
so there should be full HTTP message traces with related connection<br>
details and flow direction (client vs server). That would be more useful.<br>
<br>
<br>
What I do see in the image is several "GET http://" lines. That<br>
absolute-form URL syntax is for explicit proxy traffic. Traffic<br>
intercepted from port 80 or 443 would use origin-form URLs.<br>
- This reinforces the idea that you are probably testing the proxy<br>
wrong - eg direct curl requests to the proxy?<br>
<br>
<br>
<br>
> Server IP: Eth0: IP: <a href="http://172.22.22.148/26" rel="noreferrer" target="_blank">172.22.22.148/26</a> (Same<br>
> eth0 interface reaches the internet gateway).<br>
> Configurations for <br>
> 1) Nat table: <br>
> Chain PREROUTING(policy ACCEPT 23 packets, 1632 bytes)<br>
> num pkts bytes target prot opt in out source <br>
> destination <br>
> 1 66 3960 REDIRECT tcp -- eth0 * <a href="http://0.0.0.0/0" rel="noreferrer" target="_blank">0.0.0.0/0</a><br>
> <a href="http://0.0.0.0/0" rel="noreferrer" target="_blank">0.0.0.0/0</a> tcp dpt:80 /*<br>
> Redirect http traffic eth0:80 to eth0:3128 */ redir ports 3128<br>
> <br>
<br>
You are missing the PREROUTING rule which would ACCEPT traffic outbound<br>
from the proxy to port 80.<br>
<br></blockquote><div>As suggested, taking reference from <a href="https://wiki.squid-cache.org/ConfigExamples/Intercept/LinuxRedirect">https://wiki.squid-cache.org/ConfigExamples/Intercept/LinuxRedirect</a>, I added the following rules. The only additional part is the Separation of HTTP and HTTPS traffic rules.<br></div><div><font face="trebuchet ms, sans-serif"><b>NAT TABLE</b></font></div><div><font size="1" face="monospace"><b> Chain PREROUTING</b></font></div><div><font size="1" face="monospace"> 1 0 0 ACCEPT tcp -- * * 172.22.22.148 <a href="http://0.0.0.0/0">0.0.0.0/0</a> tcp dpt:80 /* Accept http traffic at <a href="http://172.22.22.148:80">172.22.22.148:80</a> */</font></div><div><font size="1" face="monospace"><br></font></div><font size="1" face="monospace">2 79 4740 REDIRECT tcp -- * * <a href="http://0.0.0.0/0">0.0.0.0/0</a> <a href="http://0.0.0.0/0">0.0.0.0/0</a> tcp dpt:80 /* Redirect eth0:80 to eth0:3129 */ redir ports 3129</font></div><div class="gmail_quote"><font size="1" face="monospace"><br>3 0 0 ACCEPT tcp -- * * 172.22.22.148 <a href="http://0.0.0.0/0">0.0.0.0/0</a> tcp dpt:443 /* Accept https traffic at <a href="http://172.22.22.148:443">172.22.22.148:443</a> */</font></div><div class="gmail_quote"><font size="1" face="monospace"><br>4 565 33900 REDIRECT tcp -- * * <a href="http://0.0.0.0/0">0.0.0.0/0</a> <a href="http://0.0.0.0/0">0.0.0.0/0</a> tcp dpt:443 /* Redirect eth0:443 to eth0:3130 */ redir ports 3130</font></div><div class="gmail_quote"><font face="monospace" size="1"><b>Chain POSTROUTING<br></b></font></div><div class="gmail_quote"><font face="monospace" size="1">1 365K 23M MASQUERADE all -- * * <a href="http://0.0.0.0/0">0.0.0.0/0</a> <a href="http://0.0.0.0/0">0.0.0.0/0</a> /* Allows NAT To happen */</font><br></div><div class="gmail_quote"><font face="monospace" size="1"><br></font></div><div class="gmail_quote"><font face="trebuchet ms, sans-serif"><b>Mangle Table </b></font></div><div class="gmail_quote"><font size="1"><b>Chain PREROUTING<font face="monospace"><br></font></b></font></div><div class="gmail_quote"><font face="monospace" size="1">1 0 0 DROP tcp -- * * <a href="http://0.0.0.0/0">0.0.0.0/0</a> <a href="http://0.0.0.0/0">0.0.0.0/0</a> tcp dpt:3129<br>2 0 0 DROP tcp -- * * <a href="http://0.0.0.0/0">0.0.0.0/0</a> <a href="http://0.0.0.0/0">0.0.0.0/0</a> tcp dpt:3130</font><br></div><div class="gmail_quote"><font face="monospace" size="1"><br></font><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> This config page details what you need for REDIRECT interception on<br>
iptables:<br>
<<a href="https://wiki.squid-cache.org/ConfigExamples/Intercept/LinuxRedirect" rel="noreferrer" target="_blank">https://wiki.squid-cache.org/ConfigExamples/Intercept/LinuxRedirect</a>><br>
<br>
<br>
Note that it does not place the intercept flag on port 3128. That is<br>
because Squid will generate URLs with its hostname and that port for<br>
clients to use directly when needed (error page contents, manager<br>
reports, etc).<br>
<br></blockquote><font face="monospace" size="1"><div class="gmail_quote"><font face="monospace" size="1">Using ports 3129 for http and 3130 for https. </font></div>http_port <a href="http://172.22.22.148:3129">172.22.22.148:3129</a> intercept<br>https_port <a href="http://172.22.22.148:3130">172.22.22.148:3130</a> intercept ssl-bump cert=/etc/squid/ssl_certs/myCA.pem generate-host-certificates=on dynamic_cert_mem_cache_size=4MB</font><br><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
<br>
> Please let me know if I am missing some conf or the next steps I should<br>
> try to get this running.<br>
><br>
<br>
Firstly, add that missing NAT rule.<br></blockquote><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Secondly, make sure that your tests are accurately emulating how clients<br>
would "use" the proxy. That means making connections from a test machine<br>
directly to the Internet and seeing if the routing and NAT delivers the<br>
traffic to Squid properly.<br></blockquote><div><br></div><div>I am using a chromebook to test. In the configuration section of the wireless network there is an option to add proxy hostname and proxy port based on protocols. </div><div>Http proxy : proxy-tls 80</div><div>HTTPS proxy: proxy-tls 443</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
- Use cache.log to view the traffic coming into the proxy. It will be<br>
request messages with a prefix line indicating "Client HTTP request".<br>
Make sure that prefix line says the remote Internet IP address and port<br>
80/443 you were testing with.<br>
- If you want confirm that access.log has a transaction entry for the<br>
URL you tested with ORIGINAL_DST and the server IP.<br>
<br></blockquote><div>Sample cache.log for a test I did for <a href="http://neverssl.com">neverssl.com</a></div><div><br></div><font face="monospace" size="1">2020/01/22 17:08:30.236 kid1| 11,2| client_side.cc(2346) parseHttpRequest: HTTP Client local=<a href="http://172.22.22.148:80">172.22.22.148:80</a> remote=<a href="http://172.22.22.151:34728">172.22.22.151:34728</a> FD 12 flags=33<br>2020/01/22 17:08:30.236 kid1| 11,2| client_side.cc(2347) parseHttpRequest: HTTP Client REQUEST:<br>---------<br>GET <a href="http://neverssl.com/">http://neverssl.com/</a> HTTP/1.1<br>Host: <a href="http://neverssl.com">neverssl.com</a><br>Proxy-Connection: keep-alive<br>DNT: 1<br>Upgrade-Insecure-Requests: 1<br>User-Agent: Mozilla/5.0 (X11; CrOS x86_64 12499.12.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.26 Safari/537.36<br>Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3<br>Accept-Encoding: gzip, deflate<br>Accept-Language: en-US,en;q=0.9,en-GB;q=0.8,ar;q=0.7<br><br><br>----------<br>2020/01/22 17:08:30.249 kid1| 20,3| store.cc(774) storeCreatePureEntry: storeCreateEntry: '<a href="http://neverssl.com/">http://neverssl.com/</a>'<br>2020/01/22 17:08:30.250 kid1| 20,3| MemObject.cc(97) MemObject: new MemObject 0x7f3055a51990<br>2020/01/22 17:08:30.250 kid1| 20,3| store.cc(499) setReleaseFlag: StoreEntry::setReleaseFlag: '[null_store_key]'<br>2020/01/22 17:08:30.250 kid1| 20,3| store_key_md5.cc(89) storeKeyPrivate: storeKeyPrivate: GET <a href="http://neverssl.com/">http://neverssl.com/</a><br>2020/01/22 17:08:30.250 kid1| 20,3| store.cc(447) hashInsert: StoreEntry::hashInsert: Inserting Entry e:=XI/0x7f3055a51910*0 key '17D405610D6C945224B64DBB3180E9CB'<br>2020/01/22 17:08:30.250 kid1| 20,3| store.cc(483) lock: storeCreateEntry locked key 17D405610D6C945224B64DBB3180E9CB e:=XIV/0x7f3055a51910*1<br>2020/01/22 17:08:30.250 kid1| 20,3| store.cc(483) lock: clientReplyContext::setReplyToStoreEntry locked key 17D405610D6C945224B64DBB3180E9CB e:=XIV/0x7f3055a51910*2<br>2020/01/22 17:08:30.258 kid1| 20,3| store.cc(483) lock: StoreEntry::storeErrorResponse locked key 17D405610D6C945224B64DBB3180E9CB e:=XIV/0x7f3055a51910*3<br>2020/01/22 17:08:30.258 kid1| 20,3| store.cc(1862) replaceHttpReply: StoreEntry::replaceHttpReply: <a href="http://neverssl.com/">http://neverssl.com/</a><br>2020/01/22 17:08:30.258 kid1| 20,2| store.cc(949) checkCachable: StoreEntry::checkCachable: NO: not cachable<br>2020/01/22 17:08:30.258 kid1| 20,3| store.cc(1048) complete: storeComplete: '17D405610D6C945224B64DBB3180E9CB'<br>2020/01/22 17:08:30.258 kid1| 20,3| store.cc(1337) validLength: storeEntryValidLength: Checking '17D405610D6C945224B64DBB3180E9CB'<br>2020/01/22 17:08:30.258 kid1| 20,2| store.cc(949) checkCachable: StoreEntry::checkCachable: NO: not cachable<br>2020/01/22 17:08:30.258 kid1| 20,3| store.cc(521) unlock: StoreEntry::storeErrorResponse unlocking key 17D405610D6C945224B64DBB3180E9CB e:=sXINV/0x7f3055a51910*3<br>2020/01/22 17:08:30.259 kid1| 20,3| store.cc(483) lock: store_client::copy locked key 17D405610D6C945224B64DBB3180E9CB e:=sXINV/0x7f3055a51910*3<br>2020/01/22 17:08:30.259 kid1| 20,3| store.cc(483) lock: ClientHttpRequest::loggingEntry locked key 17D405610D6C945224B64DBB3180E9CB e:=sXINV/0x7f3055a51910*4<br>2020/01/22 17:08:30.259 kid1| 11,2| client_side.cc(1392) sendStartOfMessage: HTTP Client local=<a href="http://172.22.22.148:80">172.22.22.148:80</a> remote=<a href="http://172.22.22.151:34728">172.22.22.151:34728</a> FD 12 flags=33<br>2020/01/22 17:08:30.259 kid1| 11,2| client_side.cc(1393) sendStartOfMessage: HTTP Client REPLY:<br>---------<br>HTTP/1.1 403 Forbidden<br>Server: squid/3.5.19<br>Mime-Version: 1.0<br>Date: Thu, 23 Jan 2020 01:08:30 GMT<br>Content-Type: text/html;charset=utf-8<br>Content-Length: 3861<br>X-Squid-Error: ERR_ACCESS_DENIED 0<br>Vary: Accept-Language<br>Content-Language: en-us<br>X-Cache: MISS from proxy-tls<br>X-Cache-Lookup: NONE from proxy-tls:8080<br>Via: 1.1 proxy-tls (squid/3.5.19)<br>Connection: keep-alive</font><br><br>complete file:<a href="https://pastebin.com/iBvy9naz">https://pastebin.com/iBvy9naz</a><br><div>---------- </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
<br>
Thirdly, some squid.conf improvements for other problems you have either<br>
not noticed or not encountered yet:<br>
<br>
> 3) Squid.conf<br>
> <br>
...<br>
> acl my_machine src <a href="http://172.22.22.0/24" rel="noreferrer" target="_blank">172.22.22.0/24</a><br>
> http_access allow my_machine<br>
> acl ap_clients src <a href="http://172.16.10.0/24" rel="noreferrer" target="_blank">172.16.10.0/24</a><br>
> acl local_clients src <a href="http://172.18.10.0/24" rel="noreferrer" target="_blank">172.18.10.0/24</a><br>
> acl localnet src <a href="http://10.0.0.0/8" rel="noreferrer" target="_blank">10.0.0.0/8</a> <br>
...<br>
> http_access allow localnet<br>
> http_access allow ap_clients<br>
> http_access allow local_clients<br>
<br>
It looks sub-optimal to have these as separate ACLs. May as well use<br>
config comments to document what ranges are which clients and put them<br>
all in localnet (that is what it is for).<br>
<br>
eg:<br>
# AP clients<br>
acl localnet src <a href="http://172.16.10.0/24" rel="noreferrer" target="_blank">172.16.10.0/24</a><br>
# Local clients<br>
acl localnet src <a href="http://172.18.10.0/24" rel="noreferrer" target="_blank">172.18.10.0/24</a><br>
# LAN<br>
acl localnet src <a href="http://10.0.0.0/8" rel="noreferrer" target="_blank">10.0.0.0/8</a><br>
<br>
http_access allow localnet<br></blockquote><div><br></div><div>Updated as suggested:) </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
<br>
<br>
> acl purge method PURGE<br>
> http_access deny purge<br>
<br>
PURGE is largely obsoleted (by HTCP CLR feature) and requires Squid to<br>
perform a relatively large amount of processing per-request. Unless you<br>
have tools that use it specifically it is best to remove all mention of<br>
it from the config file to let Squid avoid all that work. If any mention<br>
remains Squid will auto-activate the cache indexing work.<br>
<br>
If you do have tools using it. Then it is past time to consider/plan<br>
moving those tools and Squid to using HTCP CLR instead.<br>
<br>
<br>
> http_access deny !Safe_ports<br>
> http_access deny CONNECT !SSL_ports<br>
<br>
... this is where all your custom http_access rules are supposed to be.<br>
The Safe_ports and SSL_Ports lines above are DoS and hijack protections.<br></blockquote><div> </div><div>IIUC, These are not required to be here so I commented out those lines. </div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
* DoS protections need to be first to do their job of minimizing the<br>
CPU impact effectively. It is counter productive to have allow's earlier<br>
than here.<br>
<br>
* It is risky to let clients hijack the proxy and Squid cannot handle<br>
native traffic to or from those ports anyway. If you need a specific<br>
port available<br>
<br>
<br>
<br>
> http_reply_access allow all<br>
<br>
Above is the default reply handling. No need to specify.<br>
<br>
> http_access deny all<br>
<br>
> http_port 8080<br>
> http_port <a href="http://172.22.22.148:3128" rel="noreferrer" target="_blank">172.22.22.148:3128</a> intercept<br>
> https_port <a href="http://172.22.22.148:3129" rel="noreferrer" target="_blank">172.22.22.148:3129</a> intercept ssl-bump cert=/etc/squid/ssl_certs/myCA.pem generate-host-certificates=on dynamic_cert_mem_cache_size=4MB<br>
> sslcrtd_program /usr/lib/squid/ssl_crtd -s /var/lib/ssl_db -M 4MB<br>
<br>
You do need at least one non-intercept port for the direct-to-proxy<br>
communications that are needed occasionally. Port 3128 is reserved for<br>
that. It is best to pick a random other port number for the intercepted<br>
traffic to arrive at and leave 3128 for the normal proxy traffic.<br>
<br>
<br>
<br>> acl step1 at_step SslBump1<br>> ssl_bump peek step1<br>> ssl_bump bump all<br>
<br>
NP: while this "works" bumping without any details about the server<br>
(obtained at step2) can cause a lot of connection compatibility problems<br>
and undesirable security side effects. Prefer to stare at step2 and<br>
bump at step 3. Do this step2 bump only as a last-resort (ie restrict it<br>
to sites that are broken and cannot be worked around in other ways).<br></blockquote><div> </div><div>As suggested, updated it to</div><div><font face="monospace">acl step1 at_step SslBump1<br>acl step2 at_step SslBump2<br>acl step3 at_step SslBump3<br><br>ssl_bump peek step1 all<br>ssl_bump stare step2<br>ssl_bump bump step3</font><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
<br>
> host_verify_strict off<br>
<br>
The above is the default, no need to specify.<br>
<br>
> acl oyster-vpn-test dstdomain .<a href="http://oyster-vpn-test.com" rel="noreferrer" target="_blank">oyster-vpn-test.com</a><br>
> cache deny oyster-vpn-test<br>
> visible_hostname <a href="http://togo.mtv.corp.google.com" rel="noreferrer" target="_blank">togo.mtv.corp.google.com</a><br>
<br>
Er ... AFAIK Google does not use that domain for their machine<br>
hostnames. You sure you want to advertise your network as *.<a href="http://google.com" rel="noreferrer" target="_blank">google.com</a> ?<br>
<br>
Also, you only need a visible_hostname line if Squid is unable to pull<br>
the machines hostname from the OS configuration.<br>
<br>
<br>
Amos<br>
<br></blockquote><div>Thanks for your time and help Amos. </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
------------------------------<br>
<br>
Subject: Digest Footer<br>
<br>
_______________________________________________<br>
squid-users mailing list<br>
<a href="mailto:squid-users@lists.squid-cache.org" target="_blank">squid-users@lists.squid-cache.org</a><br>
<a href="http://lists.squid-cache.org/listinfo/squid-users" rel="noreferrer" target="_blank">http://lists.squid-cache.org/listinfo/squid-users</a><br>
<br>
<br>
------------------------------<br>
<br>
End of squid-users Digest, Vol 65, Issue 30<br>
*******************************************<br>
</blockquote></div></div>