[squid-users] file descriptors leak

[Amos Jeffries]

On 22/11/2015 4:10 p.m., André Janna wrote:
> I'm running Squid 3.5.10 on Debian Jessie and after some hours of execution
> it runs out of file descriptors.
> Squid is listening on port 3125, 3126 and 3127.
> Port 3126 is used for intercepting, via iptables redirect, https
> connections mostly from mobile devices like smartphones. On this port is
> active ssl-bump but I'm not decrypting https traffic, only "peek" to get
> https server host name.
> Port 3125 is used for intercepting http connections of the same mobile
> devices whose https traffic is intercepted on port 3126.
> Port 3127 is used for clients configured to use a proxy.
> Leaked file descriptors are all related to connection on port 3126 (https
> intercept ssl-bump).
> A sample output of lsof command gives:
> 
>     squid     32490            proxy   12u    
> IPv6            4065613       0t0        TCP
> 172.16.10.22:3126->192.168.93.113:55815 (CLOSE_WAIT)
>     squid     32490            proxy   14u    
> IPv6            4097822       0t0        TCP
> 172.16.10.22:3126->192.168.90.207:52288 (ESTABLISHED)
>     ...
> 
> where 172.16.10.22 is an IP address of my Squid installation and
> 192.168.x.x are mobile devices.
> Is seems that this condition is triggered by "local IP does not match any
> domain IP" error logged by Squid in cache.log, but I'm not sure if all
> stuck connections are caused by this kind of error.
> For the 2 connections of the sample above the related cache.log errors are:
> 
>     2015/11/21 12:57:51.229 kid1| SECURITY ALERT: Host header forgery
> detected on local=23.0.163.57:443 remote=192.168.93.113:55815 FD 12
> flags=33 (local IP does not match any domain IP)
>     2015/11/21 13:59:44.230 kid1| SECURITY ALERT: Host header forgery
> detected on local=198.144.127.162:443 remote=192.168.90.207:52288 FD 14
> flags=33 (local IP does not match any domain IP)
> 
> "lsof" sample output was taken more that 10 hours after Squid logged these
> errors and it shows that Squid is still holding connections open, using a
> lot of file descriptors.

In HTTP connections stay open and get used for many requests.

You are however assuming that the connection actually contains HTTP.
There is no guarantee of that without bumping the decrypt and parsing
the content inside. There are a number of non-HTTP/1.1 protocols that
use port 443 to bypass proxy and firewall security.

CONNECT requests with tunnels can be particularly long lived, mobiles
and their applications stay active for weeks on end with few outward
signs of what is happening inside the encrypted tunnel. The only way to
be sure the connection is finished with is when one of the client or
server remote endpoints closes it.

* The port 55815 connection _was_ closed sometimes within 15 mintes of
the lsof being run.

* The port 52288 connection is still being used.

Given the timespan between those messages and the lsof, it is also
possible they were closed and reopened in between. If you have a lot of
ports in active use, then re-use of closed ones becomes more likely.
Though I suspect it is just persistence doing what it is designed to do.
You will need a more detailed trace of the entire time period to know.


PS. don't confuse file descriptors with ports. There as an absolute
maximum of 64K ports per IP on each device. But sockets FD can reach
millions, if you run out of FD just configure more to be allowed.

Amos