[squid-users] Issues with TLS inspection-2
aashutosh kalyankar
aashutosh.xyz at gmail.com
Thu Jan 23 02:11:55 UTC 2020
>
>
>
>
> Message: 5
> Date: Wed, 22 Jan 2020 18:36:37 +1300
> From: Amos Jeffries <squid3 at treenet.co.nz>
> To: squid-users at lists.squid-cache.org
> Subject: Re: [squid-users] Issues with TLS inspection Intercept Mode.
> Message-ID: <2fa147c4-e499-9244-d2d1-c52438e2f4f4 at treenet.co.nz>
> Content-Type: text/plain; charset=utf-8
>
> On 22/01/20 7:39 am, aashutosh kalyankar wrote:
> > The problem I am seeing is the intercept port initiates HTTP connection
> > to self-IP instead of the web server IP it gets from the DNS request.
>
> Neither of those IPs should be used.
>
> Self-IP indicates that the traffic is being delivered to the proxy port
> as explicit proxy traffic. Or that the NAT system is broken. We usually
> get this complaint from people who are testing their proxy by sending
> traffic directly to the proxy port.
>
> DNS is only involved to validate the HTTP Host header against the IP
> address to forbid caching dangerous contents.
>
>
> > Filtered Tcpdump
> > screenshot @
> https://drive.google.com/open?id=0ByReiwdSAAY_VXBPTjF1M3dYTnBTTnhFVnRocXFveUlNSlNj
> >
>
> Screenshots are rarely useful. You are logging level 11,3 to cache.log,
> so there should be full HTTP message traces with related connection
> details and flow direction (client vs server). That would be more useful.
>
>
> What I do see in the image is several "GET http://" lines. That
> absolute-form URL syntax is for explicit proxy traffic. Traffic
> intercepted from port 80 or 443 would use origin-form URLs.
> - This reinforces the idea that you are probably testing the proxy
> wrong - eg direct curl requests to the proxy?
>
>
>
> > Server IP: Eth0: IP: 172.22.22.148/26 (Same
> > eth0 interface reaches the internet gateway).
> > Configurations for
> > 1) Nat table:
> > Chain PREROUTING(policy ACCEPT 23 packets, 1632 bytes)
> > num pkts bytes target prot opt in out source
> > destination
> > 1 66 3960 REDIRECT tcp -- eth0 * 0.0.0.0/0
> > 0.0.0.0/0 tcp dpt:80 /*
> > Redirect http traffic eth0:80 to eth0:3128 */ redir ports 3128
> >
>
> You are missing the PREROUTING rule which would ACCEPT traffic outbound
> from the proxy to port 80.
>
> As suggested, taking reference from
https://wiki.squid-cache.org/ConfigExamples/Intercept/LinuxRedirect, I
added the following rules. The only additional part is the Separation of
HTTP and HTTPS traffic rules.
*NAT TABLE*
* Chain PREROUTING*
1 0 0 ACCEPT tcp -- * * 172.22.22.148
0.0.0.0/0 tcp dpt:80 /* Accept http traffic at 172.22.22.148:80
*/
2 79 4740 REDIRECT tcp -- * * 0.0.0.0/0
0.0.0.0/0 tcp dpt:80 /* Redirect eth0:80 to eth0:3129 */ redir
ports 3129
3 0 0 ACCEPT tcp -- * * 172.22.22.148
0.0.0.0/0 tcp dpt:443 /* Accept https traffic at
172.22.22.148:443 */
4 565 33900 REDIRECT tcp -- * * 0.0.0.0/0
0.0.0.0/0 tcp dpt:443 /* Redirect eth0:443 to eth0:3130 */ redir
ports 3130
*Chain POSTROUTING*
1 365K 23M MASQUERADE all -- * * 0.0.0.0/0
0.0.0.0/0 /* Allows NAT To happen */
*Mangle Table *
*Chain PREROUTING*
1 0 0 DROP tcp -- * * 0.0.0.0/0
0.0.0.0/0 tcp dpt:3129
2 0 0 DROP tcp -- * * 0.0.0.0/0
0.0.0.0/0 tcp dpt:3130
This config page details what you need for REDIRECT interception on
> iptables:
> <https://wiki.squid-cache.org/ConfigExamples/Intercept/LinuxRedirect>
>
>
> Note that it does not place the intercept flag on port 3128. That is
> because Squid will generate URLs with its hostname and that port for
> clients to use directly when needed (error page contents, manager
> reports, etc).
>
> Using ports 3129 for http and 3130 for https.
http_port 172.22.22.148:3129 intercept
https_port 172.22.22.148:3130 intercept ssl-bump
cert=/etc/squid/ssl_certs/myCA.pem generate-host-certificates=on
dynamic_cert_mem_cache_size=4MB
>
>
> > Please let me know if I am missing some conf or the next steps I should
> > try to get this running.
> >
>
> Firstly, add that missing NAT rule.
>
> Secondly, make sure that your tests are accurately emulating how clients
> would "use" the proxy. That means making connections from a test machine
> directly to the Internet and seeing if the routing and NAT delivers the
> traffic to Squid properly.
>
I am using a chromebook to test. In the configuration section of the
wireless network there is an option to add proxy hostname and proxy port
based on protocols.
Http proxy : proxy-tls 80
HTTPS proxy: proxy-tls 443
> - Use cache.log to view the traffic coming into the proxy. It will be
> request messages with a prefix line indicating "Client HTTP request".
> Make sure that prefix line says the remote Internet IP address and port
> 80/443 you were testing with.
> - If you want confirm that access.log has a transaction entry for the
> URL you tested with ORIGINAL_DST and the server IP.
>
> Sample cache.log for a test I did for neverssl.com
2020/01/22 17:08:30.236 kid1| 11,2| client_side.cc(2346) parseHttpRequest:
HTTP Client local=172.22.22.148:80 remote=172.22.22.151:34728 FD 12 flags=33
2020/01/22 17:08:30.236 kid1| 11,2| client_side.cc(2347) parseHttpRequest:
HTTP Client REQUEST:
---------
GET http://neverssl.com/ HTTP/1.1
Host: neverssl.com
Proxy-Connection: keep-alive
DNT: 1
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (X11; CrOS x86_64 12499.12.0) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/78.0.3904.26 Safari/537.36
Accept:
text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9,en-GB;q=0.8,ar;q=0.7
----------
2020/01/22 17:08:30.249 kid1| 20,3| store.cc(774) storeCreatePureEntry:
storeCreateEntry: 'http://neverssl.com/'
2020/01/22 17:08:30.250 kid1| 20,3| MemObject.cc(97) MemObject: new
MemObject 0x7f3055a51990
2020/01/22 17:08:30.250 kid1| 20,3| store.cc(499) setReleaseFlag:
StoreEntry::setReleaseFlag: '[null_store_key]'
2020/01/22 17:08:30.250 kid1| 20,3| store_key_md5.cc(89) storeKeyPrivate:
storeKeyPrivate: GET http://neverssl.com/
2020/01/22 17:08:30.250 kid1| 20,3| store.cc(447) hashInsert:
StoreEntry::hashInsert: Inserting Entry e:=XI/0x7f3055a51910*0 key
'17D405610D6C945224B64DBB3180E9CB'
2020/01/22 17:08:30.250 kid1| 20,3| store.cc(483) lock: storeCreateEntry
locked key 17D405610D6C945224B64DBB3180E9CB e:=XIV/0x7f3055a51910*1
2020/01/22 17:08:30.250 kid1| 20,3| store.cc(483) lock:
clientReplyContext::setReplyToStoreEntry locked key
17D405610D6C945224B64DBB3180E9CB e:=XIV/0x7f3055a51910*2
2020/01/22 17:08:30.258 kid1| 20,3| store.cc(483) lock:
StoreEntry::storeErrorResponse locked key 17D405610D6C945224B64DBB3180E9CB
e:=XIV/0x7f3055a51910*3
2020/01/22 17:08:30.258 kid1| 20,3| store.cc(1862) replaceHttpReply:
StoreEntry::replaceHttpReply: http://neverssl.com/
2020/01/22 17:08:30.258 kid1| 20,2| store.cc(949) checkCachable:
StoreEntry::checkCachable: NO: not cachable
2020/01/22 17:08:30.258 kid1| 20,3| store.cc(1048) complete: storeComplete:
'17D405610D6C945224B64DBB3180E9CB'
2020/01/22 17:08:30.258 kid1| 20,3| store.cc(1337) validLength:
storeEntryValidLength: Checking '17D405610D6C945224B64DBB3180E9CB'
2020/01/22 17:08:30.258 kid1| 20,2| store.cc(949) checkCachable:
StoreEntry::checkCachable: NO: not cachable
2020/01/22 17:08:30.258 kid1| 20,3| store.cc(521) unlock:
StoreEntry::storeErrorResponse unlocking key
17D405610D6C945224B64DBB3180E9CB e:=sXINV/0x7f3055a51910*3
2020/01/22 17:08:30.259 kid1| 20,3| store.cc(483) lock: store_client::copy
locked key 17D405610D6C945224B64DBB3180E9CB e:=sXINV/0x7f3055a51910*3
2020/01/22 17:08:30.259 kid1| 20,3| store.cc(483) lock:
ClientHttpRequest::loggingEntry locked key 17D405610D6C945224B64DBB3180E9CB
e:=sXINV/0x7f3055a51910*4
2020/01/22 17:08:30.259 kid1| 11,2| client_side.cc(1392)
sendStartOfMessage: HTTP Client local=172.22.22.148:80 remote=
172.22.22.151:34728 FD 12 flags=33
2020/01/22 17:08:30.259 kid1| 11,2| client_side.cc(1393)
sendStartOfMessage: HTTP Client REPLY:
---------
HTTP/1.1 403 Forbidden
Server: squid/3.5.19
Mime-Version: 1.0
Date: Thu, 23 Jan 2020 01:08:30 GMT
Content-Type: text/html;charset=utf-8
Content-Length: 3861
X-Squid-Error: ERR_ACCESS_DENIED 0
Vary: Accept-Language
Content-Language: en-us
X-Cache: MISS from proxy-tls
X-Cache-Lookup: NONE from proxy-tls:8080
Via: 1.1 proxy-tls (squid/3.5.19)
Connection: keep-alive
complete file:https://pastebin.com/iBvy9naz
----------
>
>
> Thirdly, some squid.conf improvements for other problems you have either
> not noticed or not encountered yet:
>
> > 3) Squid.conf
> >
> ...
> > acl my_machine src 172.22.22.0/24
> > http_access allow my_machine
> > acl ap_clients src 172.16.10.0/24
> > acl local_clients src 172.18.10.0/24
> > acl localnet src 10.0.0.0/8
> ...
> > http_access allow localnet
> > http_access allow ap_clients
> > http_access allow local_clients
>
> It looks sub-optimal to have these as separate ACLs. May as well use
> config comments to document what ranges are which clients and put them
> all in localnet (that is what it is for).
>
> eg:
> # AP clients
> acl localnet src 172.16.10.0/24
> # Local clients
> acl localnet src 172.18.10.0/24
> # LAN
> acl localnet src 10.0.0.0/8
>
> http_access allow localnet
>
Updated as suggested:)
>
>
>
> > acl purge method PURGE
> > http_access deny purge
>
> PURGE is largely obsoleted (by HTCP CLR feature) and requires Squid to
> perform a relatively large amount of processing per-request. Unless you
> have tools that use it specifically it is best to remove all mention of
> it from the config file to let Squid avoid all that work. If any mention
> remains Squid will auto-activate the cache indexing work.
>
> If you do have tools using it. Then it is past time to consider/plan
> moving those tools and Squid to using HTCP CLR instead.
>
>
> > http_access deny !Safe_ports
> > http_access deny CONNECT !SSL_ports
>
> ... this is where all your custom http_access rules are supposed to be.
> The Safe_ports and SSL_Ports lines above are DoS and hijack protections.
>
IIUC, These are not required to be here so I commented out those lines.
> * DoS protections need to be first to do their job of minimizing the
> CPU impact effectively. It is counter productive to have allow's earlier
> than here.
>
> * It is risky to let clients hijack the proxy and Squid cannot handle
> native traffic to or from those ports anyway. If you need a specific
> port available
>
>
>
> > http_reply_access allow all
>
> Above is the default reply handling. No need to specify.
>
> > http_access deny all
>
> > http_port 8080
> > http_port 172.22.22.148:3128 intercept
> > https_port 172.22.22.148:3129 intercept ssl-bump
> cert=/etc/squid/ssl_certs/myCA.pem generate-host-certificates=on
> dynamic_cert_mem_cache_size=4MB
> > sslcrtd_program /usr/lib/squid/ssl_crtd -s /var/lib/ssl_db -M 4MB
>
> You do need at least one non-intercept port for the direct-to-proxy
> communications that are needed occasionally. Port 3128 is reserved for
> that. It is best to pick a random other port number for the intercepted
> traffic to arrive at and leave 3128 for the normal proxy traffic.
>
>
>
> > acl step1 at_step SslBump1
> > ssl_bump peek step1
> > ssl_bump bump all
>
> NP: while this "works" bumping without any details about the server
> (obtained at step2) can cause a lot of connection compatibility problems
> and undesirable security side effects. Prefer to stare at step2 and
> bump at step 3. Do this step2 bump only as a last-resort (ie restrict it
> to sites that are broken and cannot be worked around in other ways).
>
As suggested, updated it to
acl step1 at_step SslBump1
acl step2 at_step SslBump2
acl step3 at_step SslBump3
ssl_bump peek step1 all
ssl_bump stare step2
ssl_bump bump step3
>
>
> > host_verify_strict off
>
> The above is the default, no need to specify.
>
> > acl oyster-vpn-test dstdomain .oyster-vpn-test.com
> > cache deny oyster-vpn-test
> > visible_hostname togo.mtv.corp.google.com
>
> Er ... AFAIK Google does not use that domain for their machine
> hostnames. You sure you want to advertise your network as *.google.com ?
>
> Also, you only need a visible_hostname line if Squid is unable to pull
> the machines hostname from the OS configuration.
>
>
> Amos
>
> Thanks for your time and help Amos.
>
> ------------------------------
>
> Subject: Digest Footer
>
> _______________________________________________
> squid-users mailing list
> squid-users at lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users
>
>
> ------------------------------
>
> End of squid-users Digest, Vol 65, Issue 30
> *******************************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20200122/e96b17b7/attachment-0001.html>
More information about the squid-users
mailing list