[squid-users] Large memory leak with ssl_peek (now partly understood)

[Steve Hill]

I've been suffering from a significant memory leak on multiple servers 
running Squid 3.5 for months, but was unable to reproduce it in a test 
environment.  I've now figured out how to reproduce it and have done 
some investigation:

When using TPROXY, Squid generates fake "CONNECT 192.0.2.1:443" 
requests, using the IP address that the client connected to.  At 
ssl_bump step 1, we peek and Squid generates another fake "CONNECT 
example.com:443" request containing the SNI from the client's SSL handshake.

At ssl_bump step 2 we splice the connection and Squid does verification 
to make sure that example.com does actually resolve to 192.0.2.1.  If it 
doesn't, Squid is supposed to reject the connection in 
ClientRequestContext::hostHeaderVerifyFailed() to prevent clients from 
manipulating the SNI to bypass ACLs.

Unfortunately, when verification fails, rather than actually dropping 
the client's connection, Squid just leaves the client hanging. 
Eventually the client (hopefully) times out and drops the connection 
itself, but the associated ClientRequestContext is never destroyed.

This is testable by repeatedly executing:
openssl s_client -connect 17.252.76.30:443 -servername 
courier.push.apple.com

That is a traffic pattern that we see in the real world and is now 
clearly what is triggering the leak: Apple devices make connections to 
addresses within the 17.0.0.0/8 network with an SNI of 
"courier.push.apple.com".  courier.push.apple.com resolves to a CNAME 
pointing to courier-push-apple.com.akadns.net, but 
courier-push-apple.com.akadns.net doesn't exist.  Since Squid can't 
verify the connection, it won't allow it and after 30 seconds the client 
times out.  Each Apple device keeps retrying the connection, leaking a 
ClientRequestContext each time, and before long we've leaked several 
gigabytes of memory (on some networks I'm seeing 16GB or more of leaked 
RAM over 24 hours!).

Unfortunately I'm a bit lost in the Squid code and can't quite figure 
out how to gracefully terminate the connection and destroy the context.

-- 
  - Steve Hill
    Technical Director
    Opendium Limited     http://www.opendium.com

Sales / enquiries:
    Email:            sales at opendium.com
    Phone:            +44-1792-824568 / sip:sales at opendium.com

Support:
    Email:            support at opendium.com
    Phone:            +44-1792-825748 / sip:support at opendium.com