[squid-users] HTTPS woes

Thu Apr 20 13:39:10 UTC 2017

After two and a bit weeks on this I finally have the Raspberry Pi working as a transparent proxy server utilising Diladele to provide web filtering. I'm going to trial it all for the next few weeks to ensure that it's stable but so far the results have been positive and its working with HTTP and HTTPS across Windows, IOS and Android devices. 

I wanted to say a big thank you to everyone who has responded to my many messages.I'm sure there will be more to come but I wouldn't have got this far without your help so thank you very much. 

FYI the following steps have been necessary:

HTTPS Squid on Raspberry Pi 3:
1. The stretch repositories are required to build squid 3.5 and should be enabled
2. after running apt-get update you should downgrade to openssl v1.0 (from v1.1) to avoid build failures
3. You must disable ecap functionality to avoid build failures, I couldn't get squid 3.5.23 to build with ecap regardless of the version of libecap I used.
4. download the 3.5.23 source from stretch and follow a guide online to configure, make, and install the packages with ssl and ssl_crtd enabled (careful with the flags if you're following a guide for an older version of squid as the syntax changed)
5. follow a guide online to install / configure squid 3.5 - specifically creating the cache folders and setting up ssl_crtd and the ssl cache
6. download the mozilla ca certs bundle (https://curl.haxx.se/ca/cacert.pem or google) which are required for HTTPS to work 
7. ensure sslproxy_session_cache_size is disabled (example config below). Squid will not load on boot with this setting enabled.

8. check permissions across your squid installation (specifically cache, ssl_crtd and cerificate cache/locations) to ensure the proxy:proxy account has access
9. be careful of the runtime directories which are used. The default location on Rpi is /squid3 but this approach will move everything in /squid so be sure that you use the right one in your config
10. Ensure you generate your self-signed CA certificate/key with SHA-256 (as a minimum) to avoid cert failures in the browser. 
11. Bear in mind that your CA certificate will need to be installed/trusted on any device that you wish to use HTTPS on the network

My Config:

acl SSL_ports port 443 
acl Safe_ports port 80        # http 
acl Safe_ports port 21        # ftp 
acl Safe_ports port 443        # https 
acl Safe_ports port 70        # gopher 
acl Safe_ports port 210        # wais 
acl Safe_ports port 1025-65535    # unregistered ports 
acl Safe_ports port 280        # http-mgmt 
acl Safe_ports port 488        # gss-http 
acl Safe_ports port 591        # filemaker 
acl Safe_ports port 777        # multiling http 
acl CONNECT method CONNECT 

http_access deny !Safe_ports 
http_access deny CONNECT !SSL_ports 
http_access allow all 

http_port 3130 
http_port 3128 intercept 
https_port 3129 intercept ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=4MB cert=/etc/squid3/ssl_cert/squid.crt key=/etc/squid3/ssl_cert/squid.key options=NO_SSLv3 

acl step1 at_step SslBump1 
ssl_bump peek step1 
ssl_bump bump all 
sslproxy_options NO_SSLv2,NO_SSLv3,SINGLE_DH_USE 
sslproxy_cipher EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+RC4:EECDH:EDH+aRSA:!RC4:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS 
sslproxy_cafile /etc/squid/ssl_cert/mozcacert.pem 

sslproxy_session_cache_size 0 
sslcrtd_program /usr/lib/squid/ssl_crtd -s /var/spool/squid_ssldb -M 4MB 
sslcrtd_children 8 startup=1 idle=1 

coredump_dir /var/spool/squid 

# Add any of your own refresh_pattern entries above these. 
refresh_pattern ^ftp:        1440    20%    10080 
refresh_pattern ^gopher:    1440    0%    1440 
refresh_pattern -i (/cgi-bin/|\?) 0    0%    0 
refresh_pattern .        0    20%    4320 

cache_dir ufs /cache 400 16 256 

----------------

It's worth noting that I could not get udhcpd to start on boot with the Raspberry Pi (which seemed to be the recommended DHCP server online) and had to switch to ISC to get DHCP to work. Bind works fine though and the Diladele filter also installed without a hitch so it's only really DHCP that can trip you up. 

Hope this helps someone

Olly

oliver at lennox-it.uk
lennox-it.uk
tel: 07900 648 252

________________________________
From: Alex Rousskov <rousskov at measurement-factory.com>
To: "'squid-users at squid-cache. org'" <squid-users at squid-cache.org> 
Cc: Olly Lennox <oliver at lennox-it.uk>
Sent: Thursday, 20 April 2017, 1:21
Subject: Re: [squid-users] HTTPS woes

On 04/19/2017 05:35 PM, Olly Lennox wrote:

> I can confirm that disabling the ssl sesison cache seems to have resolved the issue.

Great!

> I found another post which references this patch to resolve the issue:
> http://www.squid-cache.org/Versions/v4/changesets/squid-4-13984.patch

I am not sure that patch is related to any issues I have talked about.
What "another post" did you find?

> I check and the /dev/shm directory does exist with 777 permissions so
> from what I can see the OS should support it. I'm out of my depth
> here so maybe there is more to it but I can't see why squid couldn't
> write to this location.

Forget about my "OS environment is not compatible" theory (at least for
now). I now see that Squid is failing while trying to _open_ that memory
segment as opposed to failing while _creating_ it.

Did Squid try to create it? Set debug_options to "ALL,3 54,9" and search
for "shm_" and "ssl_session_cache" in cache.log for more clues.

Alex.

> ________________________________
> From: Alex Rousskov <rousskov at measurement-factory.com>
> To: "'squid-users at squid-cache. org'" <squid-users at squid-cache.org> 
> Cc: Olly Lennox <oliver at lennox-it.uk>
> Sent: Thursday, 20 April 2017, 0:13
> Subject: Re: [squid-users] HTTPS woes
> 
> 
> 
> On 04/19/2017 04:48 PM, Olly Lennox wrote:
> 
>> After further investigation the problem is something to do with permissions related to ssl_crtd.
> 
> No, it is not (or at least not yet).
> 
> 
>> I can run squid as root but using the default account (proxy?) it
>> won't run and is giving this error in cache.log:
> 
>> 2017/04/19 23:43:54 kid1| helperOpenServers: Starting 1/8 'ssl_crtd' processes
>> FATAL: Ipc::Mem::Segment::open failed to shm_open(/squid-ssl_session_cache.shm): (2) No such file or directory
> 
> The FATAL line is unrelated to the ssl_crtd line above it (this is one
> of several problems with FATAL error handling in Squid).
> 
> 
>> I've checked the file and folder permissions across all aspects of
>> squid and everything I can see is owned by proxy:proxy so not sure
>> where it is failing.
> 
> Squid is failing when trying to open a shared memory segment used for
> storing SSL sessions. This probably means two things:
> 
> 1. Your OS environment is not compatible with Squid shared memory needs
> (e.g., missing /dev/shm/ or equivalent). More info at
> http://wiki.squid-cache.org/Features/SmpScale#Ipc::Mem::Segment::create_failed_to_shm_open.28....29:_.282.29_No_such_file_or_directory
> 
> 2. There is a bug in Squid: Squid should not create shared memory
> segments when running in non-SMP mode. Please consider reporting this
> bug if it has not been reported already. At the expense of losing SSL
> session resumption capabilities, you should be able to work around this
> bug by disabling the session cache:
> http://www.squid-cache.org/Doc/config/sslproxy_session_cache_size/
> 
> 
> HTH,
> 
> Alex.
> 
> 
> 
>> acl SSL_ports port 443 
>> acl Safe_ports port 80        # http 
>> acl Safe_ports port 21        # ftp 
>> acl Safe_ports port 443        # https 
>> acl Safe_ports port 70        # gopher 
>> acl Safe_ports port 210        # wais 
>> acl Safe_ports port 1025-65535    # unregistered ports 
>> acl Safe_ports port 280        # http-mgmt 
>> acl Safe_ports port 488        # gss-http 
>> acl Safe_ports port 591        # filemaker 
>> acl Safe_ports port 777        # multiling http 
>> acl CONNECT method CONNECT 
>>
>> http_access deny !Safe_ports 
>> http_access deny CONNECT !SSL_ports 
>> http_access allow all 
>>
>> http_port 3130 
>>
>> http_port 3128 intercept 
>> https_port 3129 intercept ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=4MB cert=/etc/squid3/ssl_cert/squid.crt key=/etc/squid3/ssl_cert/squid.key options=NO_SSLv3 dhparams=/etc/squid3/ssl_cert/dhparam.pem 
>>
>> acl step1 at_step SslBump1 
>> ssl_bump peek step1 
>> ssl_bump bump all 
>> sslproxy_options NO_SSLv2,NO_SSLv3,SINGLE_DH_USE 
>> sslproxy_cipher EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+RC4:EECDH:EDH+aRSA:!RC4:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS 
>> sslproxy_cafile /etc/squid/ssl_cert/mozcacert.pem 
>>
>> sslcrtd_program /usr/lib/squid/ssl_crtd -s /var/spool/squid_ssldb -M 4MB 
>> sslcrtd_children 8 startup=1 idle=1 
>>
>> coredump_dir /var/spool/squid 
>>
>> # Add any of your own refresh_pattern entries above these. 
>> refresh_pattern ^ftp:        1440    20%    10080 
>> refresh_pattern ^gopher:    1440    0%    1440 
>> refresh_pattern -i (/cgi-bin/|\?) 0    0%    0 
>> refresh_pattern .        0    20%    4320 
>>
>> cache_dir ufs /cache 400 16 256 