[squid-users] Transparent HTTPs proxy with Squid 3.5
Fomo Dong
fomodong at gmail.com
Mon Dec 12 16:11:03 UTC 2016
Hi all,
For couple of days I'm trying to figure out how to get a transparent HTTPs
proxy to work with Squid. What I'm trying to achieve is a proxy that
accepts internet traffic from ports 80 & 443, routes them through Squid to
Privoxy and finally through Tor and returns back the data. So essentially I
want to "automatically" revert some traffic through Tor without the user
needing to add a proxy to their connection.
I know how to setup the Privoxy and Tor part, but I'm struggling with the
Squid & IP tables configuration.
Here is my setup
Download latest version
curl -O http://www.squid-cache.org/Versions/v3/3.5/squid-3.5.22.tar.gz
&& tar zxvf squid-3.5.22.tar.gz && cd squid-3.5.22
Install all needed packages
apt install devscripts build-essential openssl libssl-dev fakeroot
libcppunit-dev libsasl2-dev cdbs ccze libfile-readbackwards-perl
libcap2 libcap-dev libcap2-dev libnetfilter-conntrack-dev htop ccze
sysv-rc-conf -y
Configure the build and make and install
./configure \
CHOST="x86_64-pc-linux-gnu" \
CFLAGS="-march=core2 -O2 -pipe" \
CXXFLAGS="${CFLAGS}" \
--build=x86_64-linux-gnu \
--prefix=/usr \
--exec-prefix=/usr \
--bindir=/usr/bin \
--sbindir=/usr/sbin \
--libdir=/usr/lib \
--sharedstatedir=/usr/com \
--includedir=/usr/include \
--localstatedir=/var \
--libexecdir=/usr/lib/squid \
--srcdir=. \
--datadir=/usr/share/squid \
--sysconfdir=/etc/squid \
--infodir=/usr/share/info \
--mandir=/usr/share/man \
--x-includes=/usr/include \
--x-libraries=/usr/lib \
--with-default-user=proxy \
--with-logdir=/var/log/squid \
--with-pidfile=/var/run/squid.pid \
--enable-err-languages=English \
--enable-default-err-language=English \
--enable-storeio=ufs,aufs,diskd \
--enable-linux-netfilter \
--enable-removal-policies=lru,heap \
--enable-gnuregex \
--enable-follow-x-forwarded-for \
--enable-x-accelerator-vary \
--enable-zph-qos \
--enable-delay-pools \
--enable-snmp \
--enable-underscores \
--with-openssl \
--enable-ssl-crtd \
--enable-http-violations \
--enable-async-io=24 \
--enable-storeid-rewrite-helpers \
--with-large-files \
--with-libcap \
--with-netfilter-conntrack \
--with-included-ltdl \
--with-maxfd=65536 \
--with-filedescriptors=65536 \
--with-pthreads \
--without-gnutls \
--without-mit-krb5 \
--without-heimdal-krb5 \
--without-gnugss \
--disable-icap-client \
--disable-wccp \
--disable-wccpv2 \
--disable-dependency-tracking \
--disable-auth --disable-epoll \
--disable-ident-lookups \
--disable-icmp
Allow ip4 forwarding
echo -e "net.ipv4.ip_forward = 1\nnet.ipv4.conf.default.rp_filter =
0\nnet.ipv4.conf.all.rp_filter = 0\nnet.ipv4.conf.eth0.rp_filter =
0\n" >> /etc/sysctl.conf
Generate certificates
mkdir /etc/squid/ssl_certs && cd /etc/squid/ssl_certs
openssl genrsa -out squid.key 2048
openssl req -new -key squid.key -out squid.csr -nodes
openssl x509 -req -days 3652 -in squid.csr -signkey squid.key -out squid.crt
cat squid.crt squid.key > squid.pem
Generate certificate cache
mkdir /var/lib/squid && chown -R proxy:proxy /var/lib/squid/
/usr/lib/squid/ssl_crtd -c -s /var/lib/squid/ssl_db
Change ownership and rights to folders
mkdir -p /var/spool/squid
chown -R proxy:proxy /etc/squid/squid.conf | chown -R proxy:proxy
/usr/lib/squid | chown -R proxy:proxy /var/lib/squid/ssl_db/ | chown
-R proxy:proxy /var/spool/squid | chown -R proxy:proxy /var/log/squid
| chmod 777 /var/spool/squid | chmod 777 /var/log/squid | chmod 755
/var/lib/squid/ssl_db/certs | chown proxy:proxy /var/log/squid/
Change configuration (bellow) and initialize the cache
squid -f /etc/squid/squid.conf -z
Redirect ports 80 and 443
iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-ports 3128
iptables -t nat -A PREROUTING -p tcp --dport 443 -j REDIRECT --to-ports 3129
My actual squid configuration
acl localnet src all
acl SSL_ports port 443
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 # https
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl CONNECT method CONNECT
never_direct allow all
always_direct allow all
# Only allow cachemgr access from localhost
http_access allow localhost manager
http_access deny manager
http_access allow localnet
http_access allow localhost
debug_options ALL,2
visible_hostname squid
# stop squid taking forever to restart.
shutdown_lifetime 3
# for clients with a configured proxy.
http_port 3127
# for clients who are sent here via iptables ... REDIRECT.
http_port 3128 tproxy
# for https clients who are sent here via iptables ... REDIRECT
https_port 3129 tproxy ssl-bump generate-host-certificates=on
dynamic_cert_mem_cache_size=4MB cert=/etc/squid/ssl_certs/squid.pem
sslcrtd_program /usr/lib/squid/ssl_crtd -s /var/lib/squid/ssl_db -M
4MB sslcrtd_children 8 startup=1 idle=1
# acl step1 at_step SslBump1
# ssl_bump peek step1
# ssl_bump bump all
ssl_bump server-first all
sslproxy_cert_error allow all
sslproxy_flags DONT_VERIFY_PEER
via off
forwarded_for off
request_header_access From deny all
request_header_access Server deny all
request_header_access WWW-Authenticate deny all
request_header_access Link deny all
request_header_access Cache-Control deny all
request_header_access Proxy-Connection deny all
request_header_access X-Cache deny all
request_header_access X-Cache-Lookup deny all
request_header_access Via deny all
request_header_access X-Forwarded-For deny all
request_header_access Pragma deny all
request_header_access Keep-Alive deny all
cache_dir ufs /var/spool/squid 1024 16 256
coredump_dir /var/cache/squid
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
refresh_pattern . 0 20% 4320
------------------------------
You can notice how benevolent I'm with the settings for Squid. It's only
for testing.
So where I got now is that nor intercept nor tproxy works. If I use accel
for the non-HTTPS traffic it works, but nothing else. If I use it as it is,
the result is that it will end up hanging for the client's timeout period
and then timeout.
Here is an example. I changed in /etc/hosts the IP for httpbin.org and
redirected it through the squid box.
❯ curl -vk https://httpbin.org/ip
* Trying *******...
* Connected to httpbin.org (*******) port 443 (#0)
* TLS 1.2 connection using TLS_RSA_WITH_AES_256_GCM_SHA384
* Server certificate: ******
* Server certificate: Universe
> GET /ip HTTP/1.1
> Host: httpbin.org
> User-Agent: curl/7.49.1
> Accept: */*
>
< HTTP/1.1 503 Service Unavailable
< Server: squid/3.5.22
< Mime-Version: 1.0
< Date: Mon, 05 Dec 2016 05:43:50 GMT
< Content-Type: text/html;charset=utf-8
< Content-Length: 3498
< X-Squid-Error: ERR_CONNECT_FAIL 110
< Vary: Accept-Language
< Content-Language: en
< X-Cache: MISS from pipik
< Connection: close
On the squid side
2016/12/05 05:42:50.362 kid1| 5,2| TcpAcceptor.cc(220) doAccept: New
connection on FD 28
2016/12/05 05:42:50.362 kid1| 5,2| TcpAcceptor.cc(295) acceptNext:
connection on local=[::]:3129 remote=[::] FD 28 flags=25
2016/12/05 05:42:50.363 kid1| 33,2| client_side.cc(3911)
httpsSslBumpAccessCheckDone: sslBump needed for local=*******:3129
remote=############# FD 11 flags=17 method 3
2016/12/05 05:42:50.363 kid1| 11,2| client_side.cc(2347)
parseHttpRequest: HTTP Client local=*******:3129 remote=#############
FD 11 flags=17
2016/12/05 05:42:50.363 kid1| 11,2| client_side.cc(2348)
parseHttpRequest: HTTP Client REQUEST:
---------
CONNECT *******:3129 HTTP/1.1
Host: *******:3129
----------
2016/12/05 05:42:50.363 kid1| 85,2| client_side_request.cc(744)
clientAccessCheckDone: The request CONNECT *******:3129 is ALLOWED;
last ACL checked: localnet
2016/12/05 05:42:50.363 kid1| 85,2| client_side_request.cc(720)
clientAccessCheck2: No adapted_http_access configuration. default:
ALLOW
2016/12/05 05:42:50.363 kid1| 85,2| client_side_request.cc(744)
clientAccessCheckDone: The request CONNECT *******:3129 is ALLOWED;
last ACL checked: localnet
2016/12/05 05:42:50.378 kid1| 83,2| client_side.cc(4284)
clientPeekAndSpliceSSL: SSL_accept failed.
2016/12/05 05:42:50.378 kid1| 83,2| client_side.cc(4284)
clientPeekAndSpliceSSL: SSL_accept failed.
2016/12/05 05:42:50.378 kid1| 83,2| client_side.cc(4284)
clientPeekAndSpliceSSL: SSL_accept failed.
2016/12/05 05:42:50.378 kid1| 83,2| client_side.cc(4284)
clientPeekAndSpliceSSL: SSL_accept failed.
2016/12/05 05:42:50.378 kid1| 83,2| client_side.cc(4284)
clientPeekAndSpliceSSL: SSL_accept failed.
2016/12/05 05:42:50.378 kid1| 83,2| client_side.cc(4284)
clientPeekAndSpliceSSL: SSL_accept failed.
2016/12/05 05:42:50.378 kid1| 83,2| client_side.cc(4284)
clientPeekAndSpliceSSL: SSL_accept failed.
2016/12/05 05:42:50.378 kid1| 83,2| client_side.cc(4284)
clientPeekAndSpliceSSL: SSL_accept failed.
2016/12/05 05:42:50.378 kid1| 83,2| client_side.cc(4284)
clientPeekAndSpliceSSL: SSL_accept failed.
2016/12/05 05:42:50.378 kid1| 83,2| client_side.cc(4284)
clientPeekAndSpliceSSL: SSL_accept failed.
2016/12/05 05:42:50.378 kid1| 83,2| client_side.cc(4284)
clientPeekAndSpliceSSL: SSL_accept failed.
2016/12/05 05:42:50.378 kid1| 83,2| client_side.cc(4284)
clientPeekAndSpliceSSL: SSL_accept failed.
2016/12/05 05:42:50.378 kid1| 83,2| client_side.cc(4284)
clientPeekAndSpliceSSL: SSL_accept failed.
2016/12/05 05:42:50.379 kid1| 83,2| client_side.cc(4284)
clientPeekAndSpliceSSL: SSL_accept failed.
2016/12/05 05:42:50.379 kid1| 83,2| client_side.cc(4284)
clientPeekAndSpliceSSL: SSL_accept failed.
2016/12/05 05:42:50.379 kid1| 83,2| client_side.cc(4284)
clientPeekAndSpliceSSL: SSL_accept failed.
2016/12/05 05:42:50.379 kid1| 83,2| client_side.cc(4284)
clientPeekAndSpliceSSL: SSL_accept failed.
2016/12/05 05:42:50.379 kid1| 17,2| FwdState.cc(133) FwdState:
Forwarding client request local=*******:3129 remote=############# FD
11 flags=17, url=*******:3129
2016/12/05 05:42:50.379 kid1| 44,2| peer_select.cc(280)
peerSelectDnsPaths: Found sources for '*******:3129'
2016/12/05 05:42:50.379 kid1| 44,2| peer_select.cc(281)
peerSelectDnsPaths: always_direct = ALLOWED
2016/12/05 05:42:50.379 kid1| 44,2| peer_select.cc(282)
peerSelectDnsPaths: never_direct = DUNNO
2016/12/05 05:42:50.379 kid1| 44,2| peer_select.cc(288)
peerSelectDnsPaths: ORIGINAL_DST = local=#############
remote=*******:3129 flags=25
2016/12/05 05:42:50.379 kid1| 44,2| peer_select.cc(295)
peerSelectDnsPaths: timedout = 0
2016/12/05 05:43:50.645 kid1| 4,2| errorpage.cc(1261) BuildContent: No
existing error page language negotiated for ERR_CONNECT_FAIL. Using
default error file.
2016/12/05 05:43:50.645 kid1| 20,2| store.cc(980) checkCachable:
StoreEntry::checkCachable: NO: not cachable
2016/12/05 05:43:50.645 kid1| 20,2| store.cc(980) checkCachable:
StoreEntry::checkCachable: NO: not cachable
2016/12/05 05:43:50.845 kid1| 83,2| client_side.cc(3811)
clientNegotiateSSL: clientNegotiateSSL: New session 0x29dda60 on FD 11
(#############:59117)
2016/12/05 05:43:50.943 kid1| 11,2| client_side.cc(2347)
parseHttpRequest: HTTP Client local=*******:3129 remote=#############
FD 11 flags=17
2016/12/05 05:43:50.944 kid1| 11,2| client_side.cc(2348)
parseHttpRequest: HTTP Client REQUEST:
---------
GET /ip HTTP/1.1
Host: httpbin.org
User-Agent: curl/7.49.1
Accept: */*
----------
2016/12/05 05:43:50.944 kid1| 33,2| QosConfig.cc(145) doTosLocalMiss:
QOS: Preserving TOS on miss, TOS=0
2016/12/05 05:43:50.944 kid1| 33,2| client_side_reply.cc(1534)
buildReplyHeader: clientBuildReplyHeader: Connection Keep-Alive not
requested by admin or client
2016/12/05 05:43:50.944 kid1| 88,2| client_side_reply.cc(2051)
processReplyAccessResult: The reply for GET https://httpbin.org/ip is
ALLOWED, because it matched (access_log
daemon:/var/log/squid/access.log line)
2016/12/05 05:43:50.944 kid1| 11,2| client_side.cc(1393)
sendStartOfMessage: HTTP Client local=*******:3129
remote=############# FD 11 flags=17
2016/12/05 05:43:50.944 kid1| 11,2| client_side.cc(1394)
sendStartOfMessage: HTTP Client REPLY:
---------
HTTP/1.1 503 Service Unavailable
Server: squid/3.5.22
Mime-Version: 1.0
Date: Mon, 05 Dec 2016 05:43:50 GMT
Content-Type: text/html;charset=utf-8
Content-Length: 3498
X-Squid-Error: ERR_CONNECT_FAIL 110
Vary: Accept-Language
Content-Language: en
X-Cache: MISS from squid
Connection: close
----------
2016/12/05 05:43:50.944 kid1| 33,2| client_side.cc(817) swanSong:
local=*******:3129 remote=############# flags=17
2016/12/05 05:43:50.944 kid1| 20,2| store.cc(980) checkCachable:
StoreEntry::checkCachable: NO: not cachable
2016/12/05 05:43:50.944 kid1| 20,2| store.cc(980) checkCachable:
StoreEntry::checkCachable: NO: not cachable
I tried so many different configurations that I'm already lost in what does
work and what doesn't. I'm probably not understanding the connection
between iptables and squid properly, but no matter what I read I always end
up here.
I appreciate any suggestions.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20161212/5eff0caf/attachment-0001.html>
More information about the squid-users
mailing list