[squid-users] Transparent HTTPs proxy with Squid 3.5

Tue Dec 13 01:17:47 UTC 2016

On 13/12/2016 5:11 a.m., Fomo Dong wrote:
> Hi all,
> 
> For couple of days I'm trying to figure out how to get a transparent HTTPs
> proxy to work with Squid. What I'm trying to achieve is a proxy that
> accepts internet traffic from ports 80 & 443, routes them through Squid to
> Privoxy and finally through Tor and returns back the data. So essentially I
> want to "automatically" revert some traffic through Tor without the user
> needing to add a proxy to their connection.
> 
> I know how to setup the Privoxy and Tor part, but I'm struggling with the
> Squid & IP tables configuration.

The first thing to be aware of is that Squid obeys the HTTPS requirement
that traffic received on TLS connection also goes out one. So your
Privoxy must be capable of receiving TLS connections from Squid.

If Privoxy cannot do TLS like that you could have Squid do the privacy
filtering. But then Tor would face the same requirement.

Second thing I want to make clear is that a *transparent* proxy is the
opposite of anonyizing proxy. A transparent proxy hides *itself* while
_revealing_ the client.  An anonymous proxy reveals itself, while hiding
the client(s). They are almost direct opposites in behaviour.

Anyhow, what you meant by the word "transparent" turns out to actually
be "intercepting". I hope that clarifies why 'tproxy' is definitely not
what you want to do here.

> Here is my setup
> 
> Download latest version
> 
> curl -O http://www.squid-cache.org/Versions/v3/3.5/squid-3.5.22.tar.gz
> && tar zxvf squid-3.5.22.tar.gz && cd squid-3.5.22
> 
> Install all needed packages
> 
> apt install devscripts build-essential openssl libssl-dev fakeroot
> libcppunit-dev libsasl2-dev cdbs ccze libfile-readbackwards-perl
> libcap2 libcap-dev libcap2-dev libnetfilter-conntrack-dev htop ccze
> sysv-rc-conf -y
> 
> Configure the build and make and install
> 
> ./configure \
<snip>
> --enable-err-languages=English \
> --enable-default-err-language=English \
<snip>

Those two saying "English" are not right. Squid uses ISO codes for
languages now. You can remove the above without affecting the build.

> 
> Allow ip4 forwarding
> 
> echo -e "net.ipv4.ip_forward = 1\nnet.ipv4.conf.default.rp_filter =
> 0\nnet.ipv4.conf.all.rp_filter = 0\nnet.ipv4.conf.eth0.rp_filter =
> 0\n" >> /etc/sysctl.conf
> 
> Generate certificates
> 
> mkdir /etc/squid/ssl_certs && cd /etc/squid/ssl_certs
> openssl genrsa -out squid.key 2048
> openssl req -new -key squid.key -out squid.csr -nodes
> openssl x509 -req -days 3652 -in squid.csr -signkey squid.key -out squid.crt
> cat squid.crt squid.key > squid.pem
> 
> Generate certificate cache
> 
> mkdir /var/lib/squid && chown -R proxy:proxy /var/lib/squid/
> /usr/lib/squid/ssl_crtd -c -s /var/lib/squid/ssl_db
> 
> Change ownership and rights to folders
> 
> mkdir -p /var/spool/squid
> 
> chown -R proxy:proxy /etc/squid/squid.conf | chown -R proxy:proxy
> /usr/lib/squid | chown -R proxy:proxy /var/lib/squid/ssl_db/ | chown
> -R proxy:proxy /var/spool/squid | chown -R proxy:proxy /var/log/squid
> | chmod 777 /var/spool/squid | chmod 777 /var/log/squid  | chmod 755
> /var/lib/squid/ssl_db/certs | chown proxy:proxy /var/log/squid/
> 

Er, I suggest you use '&&' or ';' between those instead of '|'.
chmod/chown do not take each others output as input.

Squid will be reading its config files as 'root', not as 'proxy'.

And you might want to rethink allowing world-writeable and
world-readable privileges on those directories. Particularly the TLS
certs one. 640 should be enough access for most logs, and 600 for the
certs area.

NP: if this is on a Debian/Ubuntu machine you should install the
squid/squid3 package to setup all the permissions and directories
properly. Then build your custom Squid so that the binary replaces the
default package one.

> Change configuration (bellow) and initialize the cache
> 
> squid -f /etc/squid/squid.conf -z
> 
> Redirect ports 80 and 443
> 
> iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-ports 3128
> iptables -t nat -A PREROUTING -p tcp --dport 443 -j REDIRECT --to-ports 3129
> 

Good start, there are a few more rules to add though. This config
example covers it all:
 <http://wiki.squid-cache.org/ConfigExamples/Intercept/LinuxRedirect>

> My actual squid configuration
> 
> acl localnet src all

No. localnet should only have your LAN subnets. There is a built-in ACL
'all' for use when you want the entire Internet to match.

> 
> acl SSL_ports port 443
> acl Safe_ports port 80          # http
> acl Safe_ports port 21          # ftp
> acl Safe_ports port 443         # https
> acl Safe_ports port 70          # gopher
> acl Safe_ports port 210         # wais
> acl Safe_ports port 1025-65535  # unregistered ports
> acl Safe_ports port 280         # http-mgmt
> acl Safe_ports port 488         # gss-http
> acl Safe_ports port 591         # filemaker
> acl Safe_ports port 777         # multiling http
> acl CONNECT method CONNECT
> 

You should at this position in your config add back in the default
security protection lines:

 http_access deny !Safe_ports
 http_access deny CONNECT !SSL_ports

> never_direct allow all
> always_direct allow all
> 

Remove the always_direct line. The never_direct should not be necessary,
but could help later.

You need to setup a cache_peer line pointing Squid at the Privoxy
listening address and port.
 * that line will also need at least the 'ssl' option for the HTTPS
traffic to be sent there.
 * Privoxy will need to be configured to receive TLS traffic. If it can
receive proxy<->proxy connections over TLS that would be great,
otherwise the cache_peer line might need the 'originserver' option to
send it port-443 / HTTPS formatted data.

More on cache_peer can be found at:
 <http://www.squid-cache.org/Versions/v3/3.5/cfgman/cache_peer.html>

> # Only allow cachemgr access from localhost
> http_access allow localhost manager
> http_access deny manager
> 
> http_access allow localnet
> http_access allow localhost
> 

You are missing this line:

  http_access deny all

> debug_options ALL,2
> 
> visible_hostname squid

This is supposed to be a FQDN. It gets used in URLs for icons and such
things that Squid serves up from its port 3128 ... er 3127 below.

With this config file the clients will be told to fetch things from URLs
starting "http://squid:3127/squid-internal-static/" ...

> 
> # stop squid taking forever to restart.
> shutdown_lifetime 3
> # for clients with a configured proxy.
> http_port 3127
> # for clients who are sent here via iptables ... REDIRECT.
> http_port 3128 tproxy
> # for https clients who are sent here via iptables ... REDIRECT
> https_port 3129 tproxy ssl-bump generate-host-certificates=on
> dynamic_cert_mem_cache_size=4MB cert=/etc/squid/ssl_certs/squid.pem

Replace 'tproxy' with 'intercept'. NAT is just an intercept of the
traffic, no transparency.

> 
> sslcrtd_program /usr/lib/squid/ssl_crtd -s /var/lib/squid/ssl_db -M
> 4MB sslcrtd_children 8 startup=1 idle=1
> 
> # acl step1 at_step SslBump1
> # ssl_bump peek step1
> # ssl_bump bump all
> 
> ssl_bump server-first all

Two things wrong here:

1) server-first mode requires that Squid upstream be the origin server
for the HTTPS website. That is because Squid mimics the server
connection certificate. If that connection is not the origin server the
cert Squid presents to the client is almost guaranteed to be invalid and
breaks HTTPS.

The configuration you are planning will only work with the very unsafe
client-first bumping style. Where Squid generates a completely false
certificate, but has some chance of working if the client doesn't
validate it too thoroughly.

2) The server-first/client-first config actions are deprecated. For
Squid-3.5 use the peek/splice/bump/terminate actions.

You want to start with the below rules and build any special handing
from there:

 acl step1 at_step BumpStep1
 ssl_bump stare step1
 ssl_bump bump all

> sslproxy_cert_error allow all
> sslproxy_flags DONT_VERIFY_PEER
> 

Those can go. never_direct prevents Squid -> origin connections existing
so the above dont work. Which is lucky, that could be nasty.

> via off
> forwarded_for off

'forwarded_for transparent' would be better for your requirements. Maybe
'delete' if you want to also prevent clients using the header.

> 
> request_header_access From deny all
> request_header_access Server deny all
> request_header_access WWW-Authenticate deny all

Server and WWW-Authenticate are not a request headers. You should be
able to remove those lines.

> request_header_access Link deny all
> request_header_access Cache-Control deny all

That Cache-Control line will seriously break HTTP for the clients.

> request_header_access Proxy-Connection deny all

Squid is already erasing the obsolete Proxy-Connection headers. You can
remove the above line.

> request_header_access X-Cache deny all
> request_header_access X-Cache-Lookup deny all
> request_header_access Via deny all

The via off line is already doing that. You can remove.

> request_header_access X-Forwarded-For deny all

The forwarded_for line should already do that. You can remove.

> request_header_access Pragma deny all

FYI: Like Cache-Control, doing this to Pragma will cause breakage in
HTTP. But since it is not an HTTP/1.1 header that will be minimal.

> request_header_access Keep-Alive deny all

I suggest adding this to remove all custom headers from the request:

 request_header_access Other deny all

Squid also has a reply_header_access directive for anonymizing response
headers. Though hiding server details from the client is less important.

 reply_header_access Set-Cookie deny all
 reply_header_access Set-Cookie2 deny all
 reply_header_access Other deny all

You may find that you dont actually need Privoxy after doing all this
request filtering in squid.conf.

> 
> cache_dir ufs /var/spool/squid 1024 16 256
> coredump_dir /var/cache/squid
> 
> refresh_pattern ^ftp:           1440    20%     10080
> refresh_pattern ^gopher:        1440    0%      1440
> refresh_pattern -i (/cgi-bin/|\?) 0     0%      0
> refresh_pattern .               0       20%     4320
> 
> ------------------------------
> 
> You can notice how benevolent I'm with the settings for Squid. It's only
> for testing.

If you are referring to the missing security rules and the disabled TLS
validation. That is bad for testing as well as production use.

You need to test the real config that will be used in production and fix
the problems encountered with that. Disabling things for testing hides
problems they cause when enabled, and even introduces false problems
that wont occur in production.

Amos