[squid-users] Squid proxy server to log HTTPS traffic

[Amos Jeffries]

On 02/08/18 19:05, Helen Rai wrote:
> Now what I want to do is log all the web traffic that is requested from
> those devices which are using WiFi.
> I have attached the configuration of squid and Mikrotik which I had done
> for HTTP. (Please note that when I had done this for HTTP, the squid
> wasn't configured with --enable-SSL or --enable-crtd)
> 

Yes, those options are required to handle the TLS which is part of
HTTPS. HTTP is not encrypted. Or rather: --with-openssl is required

Since you are rebuilding it with these extra options I recommend using
the Squid-4 code from the Debian Sid repository if you can. It contains
much better handling for HTTPS traffic which will be useful later to
avoid non-HTTPS traffic arriving on port 443.


> I had referred some online sites to do for HTTPS but none of them
> worked. Please help if anyone knows how to log all HTTPS request.
> 
> I have attached document regarding the rules applied in Mikrotik and
> squid3 configuration for HTTP.
> 

Okay. So first thing to do is fix some mistakes in your squid.conf

* "transparent" has long ago been replaced by "intercept"

 http_port 3127 intercept

* remove incomplete / broken config line

 #http_access deny

* remove the first two of these repeated lines:

 http_access allow localhost
 ...
 http_access allow localhost
 http_access allow localhost
 ...

* stop using "localhost" as the ACL name to represent Department-A.
Localhost is a reserved term in networking with a very specific meaning
(different to what you are using it for) which is also baked into Squid
default configuration settings.

Instead use an ACL which names the thing being matched. For example:

 acl Department-A src 10.0.7.0/24


* move your custom setting down below the default security http_access
rules. Those are the deny lines referring to SSL_ports and Safe_ports.

Your http_access lines should now look nice and simple, like this:


 http_access deny !Safe_ports
 http_access deny CONNECT !SSL_ports

 acl test dstdomain "/etc/squid3/acl/test"
 http_access deny test

 acl Department-A src 10.0.7.0/24
 http_access allow Department-A

 http_access deny all


*** Your HTTP traffic should still be fully operational and doing what
you want after these changes. If not something is broken we need to fix
before continuing.


Second, add the pieces necessary for handling port 443 traffic to
squid.conf.

You *will* need to generate a self-signed CA certificate with the
signing-cert (CA) and and HTTP server properties to use SSL-Bump features.
  But that will only need to be installed on clients machine *if* you go
as far as decrypting the HTTPS traffic or producing errors for clients.
This is where Squid-4 handles TLS far more cleanly than Squid-3.5.


 # A port to receive port 443 traffic from Mikrotik
 https_port 3129 intercept ssl-bump cert=/etc/squid3/your-CA.pem

 ... note the 's' in that directive name. A typo there breaks HTTPS in
confusing ways.


 # SSL-Bump rules to peek at details for logging, then relay the traffic
 acl step1 at_step SslBump1
 ssl_bump peek step1
 ssl_bump splice all

 # Squid-4 can pass any non-HTTPS through or omit the below line to
deliver error pages signed by the your-CA.pem certificate.

  on_unsupported_protocol tunnel


Thirdly, you need to add an entry to the Raspberry Pi NAT rules. Same as
for the port 80 NAT rule that should also exist, but for port 443.

 ** at this point you should be able to test the Raspberry PI is working
for HTTPS by setting a test machine to have the R-Pi IP address as its
gateway and requesting any https:// URL in a Browser, or wget, curl,
squidclient etc.


Lastly, when Squid is happily running with all that add to your Mikrotik
config a second line identical to the first one in your attachment -
except that it ends with "dst-port=443".

 As soon as that line is active in your router the HTTPS traffic should
start arriving and being logged at the Squid.



Please be aware of the following details:

* Do expect to see at least half of HTTPS traffic being CONNECT with
raw-IPs and 0 bytes transferred. That is from the first step of
SSL-Bump'ing. They should be followed by another CONNECT with the TLS
SNI domain name and correct size of data transferred (one-way, to client).


* The SNI is only accurate for honest clients. It can be forged and the
real HTTPS messages be using another domain name hosted on the same
server/CDN network.
 However, to be properly sure that the traffic inside the TLS is going
to the domain the SNI claimed you will have to fully decrypt the traffic
(bump action) and that means installing your CA cert on all clients
machines. Plus configuring dynamic cert generation on the https_port line.
 Start with getting the above non-decrypt setup working before going
near bumping.


Amos