[squid-users] network problems with squid ssl-bump

[Amos Jeffries]

On 18/07/19 7:52 pm, Ashley wrote:
> Hi all,
> 
> I have a couple of questions about squid 3.5. My company has set up a squid
> proxy with sslbump functionality. There are more than 300 people in my
> company and we are all intensive users of internet.
> 

The TLS environment is a very volatile place this past decade. Almost
every Squid release has some significant changes you will need for
SSL-Bump to work nicely or sometimes to work at all.

Due to this our official advice is to follow the latest release if you
need SSL-Bump. Especially if you encounter any problems, try the latest
and see if the problem disappears. Today that means Squid-4 (current
stable), or possibly even Squid-5 (cutting-edge, test well before use).

Squid-3.5.28 as bundled by some OS distributors can be reasonable *if*
it meets your needs. Which apparently it does not, and if you are
building your own Squid definitely build the latest version out.



> After we implemented our squid web proxy (with multiple instances) , we had
> problems with our network. The internet becomes very slow and some of the
> web pages are time out. We checked the cache.log and some error logs are
> shown as follows.
> 
> 2019/07/18 14:27:48 kid4| ERROR: Disconnecting from a helper that overflowed
> 32768-byte Squid input buffer: ssl_crtd #Hlpr33

That is a problem. The SSL-Bump cert generator cannot deliver the
intended fake cert back to Squid for delivery to a client.

Take a close look at the certs which are being mimic'd by this helper
and ensure they are less than 30KB in size. The 32KB buffer needs to
hold the cert plus a bunch of helper protocol values used by Squid.

If you have not done something that would make the cert be huge (eg a
server with lots of domain in its SubjectName). Then it may be a bug in
that helper, try newer Squid version.


The below look like side effects of the above. When no cert is able to
generate, negotiation will break.


> 019/07/18 14:59:34 kid6| Error negotiating SSL on FD 146: error:14077410:SSL
> routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure (1/-1/0)
> 2019/07/18 14:59:36 kid3| Error negotiating SSL on FD 153:
> error:00000000:lib(0):func(0):reason(0) (5/0/0)
> 2019/07/18 14:59:41 kid5| fqdncacheParse: No PTR record for '50.57.251.150'
> 2019/07/18 14:59:41 kid5| fqdncacheParse: No PTR record for '50.57.251.150'
> 2019/07/18 14:59:41 kid1| Failed to connect to nameserver 10.0.0.254 using
> TCP.
> 2019/07/18 14:59:41 kid5| Error negotiating SSL on FD 82: error:14077438:SSL
> routines:SSL23_GET_SERVER_HELLO:tlsv1 alert internal error (1/-1/0)
> 2019/07/18 14:59:43 kid5| Error negotiating SSL on FD 94: error:14077410:SSL
> routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure (1/-1/0)
> 2019/07/18 14:59:43 kid6| fqdncacheParse: No PTR record for
> '207.246.147.249'
> 2019/07/18 14:59:43 kid6| fqdncacheParse: No PTR record for
> '207.246.147.249'
> 2019/07/18 14:59:49 kid2| Error negotiating SSL on FD 64:
> error:00000000:lib(0):func(0):reason(0) (5/-1/54)
> 
> 
> Our squid.conf is as follows.
> 

...
> 
> cache_effective_user www
> 

Be Very Careful. That user account is a standard service name for web
servers. Squid is not a web server and should not be given access to
web-server related system resources. Likewise web servers should not be
given access to Squid internal state data (eg the SSL-Bump generated
security keys, cache contents, and shared memory sockets).

...
> 
> /usr/local/eaclhelper/CSQR_MyPortFWHelper.php
> external_acl_type myPortFWFilter children-max=6 ttl=1 %SRC
> /usr/local/eaclhelper/CSQR_MyPortFWHelper
> acl myPortFW external myPortFWFilter
> deny_info info_myPortFW myPortFW
> http_access deny myPortFW
> 
> external_acl_type myAclFilter children-idle=3 children-startup=6
> children-max=64 %SRC %SRCPORT %DST %PORT %URI
> /usr/local/eaclhelper/CSQR_MyAclHelper
> acl myAcl external myAclFilter
> deny_info info_myAcl myAcl
> http_access deny myAcl
> 
> http_access allow localnet
> http_access allow localhost
> 
> http_access deny all
> 
> workers 6
> sslproxy_session_cache_size 0
> 
> if ${process_number} = 7
> 
> http_port 127.0.0.1:3127 
> 
> else
> 
> http_port 127.0.0.${process_number}:3128
> https_port 127.0.0.${process_number}:3129 intercept ssl-bump
> cert=/usr/local/squid/ssl_cert/myCA.pem generate-host-certificates=on
> dynamic_cert_mem_cache_size=4MB
> 
... <snip unnecessary config> ...

> 
> endif # endif for squid coordinator
> 
> # ssl_bump ACL
> acl broken_dstdom dstdomain
> "/home/www/htdocs/snims/lurk/sslTP_default_conf/no_sslbump_dstdom.default"
> acl broken_dst dst
> "/home/www/htdocs/snims/lurk/sslTP_default_conf/no_sslbump_dst.default"
> acl bump_dstdom dstdomain
> "/home/www/htdocs/snims/lurk/sslTP_default_conf/sslbump_dstdom.default"
> acl bump_dst dst
> "/home/www/htdocs/snims/lurk/sslTP_default_conf/sslbump_dst.default"
> 
> always_direct allow all

Please remove. You do not have cache_peer. The above was only ever
needed to workaround a very short-lived bug back in Squid-3.1.



> # the following two options are unsafe and not always necessary:
> sslproxy_cert_error allow all
> sslproxy_flags DONT_VERIFY_PEER
> 

Please do not use DONT_VERIFY_PEER. All it does is prevent you (the
admin) from seeing what problems are happening.

The "allow all" for errors is also a bad idea. I suggest removing both
of these, then solving the error(s) that show up. yes you will see
errors happening, the idea is to look into them and see whether it is an
attack or network issue to fix before they become massive problems like
you are having now.


> # sslcrtd
> sslcrtd_program /usr/local/squid/libexec/ssl_crtd -s /var/squid/ssl_db -M
> 4MB
> sslcrtd_children 32 startup=5 idle=3
> 
> # ecap_adapter
> ecap_enable off
> adaptation_send_client_ip on
> loadable_modules /usr/local/ecap_adapter/lib/ecap_adapter_passthru.so
> ecap_service eReqmod reqmod_precache 0
> ecap://e-cap.org/ecap/services/sample/passthru 
> ecap_service eRespmod respmod_precache 0
> ecap://e-cap.org/ecap/services/sample/passthru 
> 
> adaptation_service_set reqFilter eReqmod
> adaptation_service_set respFilter eRespmod
> adaptation_access respFilter allow all
> adaptation_access reqFilter allow all
> 
...
> 
> ## sslBump ssl server name ACL
> acl monitorSites ssl::server_name
> "/home/www/htdocs/snims/lurk/sslTP_default_conf/monitor_sslserver.default"
> acl monitorSites_define ssl::server_name
> "/home/www/htdocs/snims/lurk/sslTP_default_conf/monitor_sslserver.define"
> acl monitorSites_security ssl::server_name
> "/home/www/htdocs/snims/lurk/sslTP_default_conf/monitor_sslserver.security"
> acl monitorSites_security_IP dst
> "/home/www/htdocs/snims/lurk/sslTP_default_conf/monitor_dst.security"
> acl brokenSites ssl::server_name
> "/home/www/htdocs/snims/lurk/sslTP_default_conf/no_sslbump_sslserver.default"
> acl brokenSites_define ssl::server_name
> "/home/www/htdocs/snims/lurk/sslTP_default_conf/no_sslbump_sslserver.define"
> 
> ## sslBump acl setting for squid 3.5 
> acl step1 at_step SslBump1
> acl step2 at_step SslBump2
> acl step3 at_step SslBump3

NP: you are not using the step2 or step3 ACLs. They can be removed.

> ssl_bump peek step1 all

The "all" on the above line is pointless.

> 
> ssl_bump splice broken_dstdom
> ssl_bump splice broken_dst
> ssl_bump splice brokenSites_define
> 

After the above line anything matching brokenSites_define will be
spliced. OR is a server which _cannot_ be spliced.

So you can remove the redundant "!brokenSites_define" from the below lines.

Since you are following these below lines with a "splice all" line you
can greatly simplify the ACL processing time by adding this line right here:
 ssl_bump splice brokenSites

.. then removing all the "!brokenSites" below.


> ssl_bump bump monitorSites !brokenSites !brokenSites_define
> ssl_bump bump monitorSites_define !brokenSites !brokenSites_define
> ssl_bump bump monitorSites_security !brokenSites !brokenSites_define
> ssl_bump bump monitorSites_security_IP !brokenSites !brokenSites_define
> 
> ssl_bump splice all
> 
> ## sslproxy cert setting for squid 3.5
> sslproxy_options NO_SSLv2,NO_SSLv3,SINGLE_DH_USE,SINGLE_ECDH_USE
> sslproxy_cipher
> EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+RC4:EECDH:EDH+aRSA:HIGH:!RC4:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS
> sslproxy_cert_adapt setValidAfter all
> 
> Can anyone advise what’s wrong with our settings? Or is there any limitation
> for sslbump in terms of concurrent connections? Any suggestions and advice
> will be highly appreciated.

Squid-3 SSL-Bump is limited to OpenSSL 1.0 capabilities. The bump action
is limited to the union of client capabilities, server capabilities,
Squid OpenSSL 1.0 capabilities, and the configured sslproxy_* restrictions.


Amos