[squid-users] file descriptors leak

[Amos Jeffries]

On 27/11/2015 7:36 a.m., André Janna wrote:
> 
> Assinatura
> Em 24/11/2015 00:54, Amos Jeffries escreveu:
>> FYI: unless you have a specific need for 3.5 you should be fine with
>> the 3.4 squid3 package that is available for Jesse from Debian
>> backports. The alternative is going the other way and upgrading right
>> to the latest 3.5 snapshot (and/or 4.0 snapshot) to see if it is one
>> of the CONNECT or TLS issues we have fixed recently. 
> I'm using version 3.5 because 3.4 doesn't have ssl::server_name acl.
> Debian package is not built with openssl because of licensing issues so
> I rebuilt Debian testing 3.5 source package on Debian Jessie.
> This Squid installation is in production now and cannot be easily
> migrated. But I'll perform another installation for testing in the near
> future.
> 
>> Neither. So it is time to move away from lsof and start using packet
>> capture to get a full-body packet trace to find out what exact packets
>> are happening on at least one affected TCP connection.
>>
>> If possible identifying one of these connections from its SYN onwards
>> would be great, but if not then a 20min period of activity on an
>> existing one might still how more hints.
>>
> I did a test using a Windows laptop client with IP address 192.168.64.4,
> connected via wifi.
> I browsed a few https sites until triggering Squid "local IP does not
> match any domain IP" error.
> This error appeared when I was trying to open Yahoo home page. Browser
> redirected to https://br.yahoo.com/?p=us but page remained blank.
> Please note that this error appears randomly: opening the same site in
> another browser tab succeeded.
> 
> cache.log:
> 2015/11/26 13:54:45.471 kid1| SECURITY ALERT: Host header forgery
> detected on local=206.190.56.191:443 remote=192.168.64.4:58887 FD 17244
> flags=33 (local IP does not match any domain IP)
> 
> After a couple of minutes this connection disappeared from Windows
> netstat command output. Afterward I powered off Windows laptop.
> 
> Tcpdump on Squid box:
> 13:54:45.410907 IP 192.168.64.4.58887 > 206.190.56.191.443: Flags [S],
> seq 1831867, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK],
> length 0
> 13:54:45.411000 IP 206.190.56.191.443 > 192.168.64.4.58887: Flags [S.],
> seq 3695298276, ack 1831868, win 29200, options [mss
> 1460,nop,nop,sackOK,nop,wscale 7], length 0


client 192.168.64.4:58887 connects.

> 13:54:45.411630 IP 192.168.64.4.58887 > 206.190.56.191.443: Flags [.],
> ack 1, win 256, length 0
> 13:54:45.412490 IP 192.168.64.4.58887 > 206.190.56.191.443: Flags [P.],
> seq 1:185, ack 1, win 256, length 184
> 13:54:45.412573 IP 206.190.56.191.443 > 192.168.64.4.58887: Flags [.],
> ack 185, win 237, length 0

client sends 184 bytes of data.

> 13:54:55.439709 IP 192.168.64.4.58887 > 206.190.56.191.443: Flags [.],
> seq 184:185, ack 1, win 256, length 1
> 13:54:55.439761 IP 206.190.56.191.443 > 192.168.64.4.58887: Flags [.],
> ack 185, win 237, options [nop,nop,sack 1 {184:185}], length 0
> 13:55:05.439965 IP 192.168.64.4.58887 > 206.190.56.191.443: Flags [.],
> seq 184:185, ack 1, win 256, length 1
> 13:55:05.440022 IP 206.190.56.191.443 > 192.168.64.4.58887: Flags [.],
> ack 185, win 237, options [nop,nop,sack 1 {184:185}], length 0
> 13:55:15.445667 IP 192.168.64.4.58887 > 206.190.56.191.443: Flags [.],
> seq 184:185, ack 1, win 256, length 1
> 13:55:15.445737 IP 206.190.56.191.443 > 192.168.64.4.58887: Flags [.],
> ack 185, win 237, options [nop,nop,sack 1 {184:185}], length 0
> 13:55:25.447281 IP 192.168.64.4.58887 > 206.190.56.191.443: Flags [.],
> seq 184:185, ack 1, win 256, length 1
> 13:55:25.447351 IP 206.190.56.191.443 > 192.168.64.4.58887: Flags [.],
> ack 185, win 237, options [nop,nop,sack 1 {184:185}], length 0
> 13:55:35.494936 IP 192.168.64.4.58887 > 206.190.56.191.443: Flags [.],
> seq 184:185, ack 1, win 256, length 1
> 13:55:35.495005 IP 206.190.56.191.443 > 192.168.64.4.58887: Flags [.],
> ack 185, win 237, options [nop,nop,sack 1 {184:185}], length 0
> 13:55:45.491694 IP 192.168.64.4.58887 > 206.190.56.191.443: Flags [.],
> seq 184:185, ack 1, win 256, length 1
> 13:55:45.491761 IP 206.190.56.191.443 > 192.168.64.4.58887: Flags [.],
> ack 185, win 237, options [nop,nop,sack 1 {184:185}], length 0
> 13:55:55.492158 IP 192.168.64.4.58887 > 206.190.56.191.443: Flags [.],
> seq 184:185, ack 1, win 256, length 1
> 13:55:55.492208 IP 206.190.56.191.443 > 192.168.64.4.58887: Flags [.],
> ack 185, win 237, options [nop,nop,sack 1 {184:185}], length 0

client sends 1 byte of data - in 7x packets.

The recipient sends an ACK each and every time, but the client just
keeps repeating itself.

> 14:01:58.242748 IP 192.168.64.4.58887 > 206.190.56.191.443: Flags [F.],
> seq 185, ack 1, win 256, length 0
> 14:01:58.279916 IP 206.190.56.191.443 > 192.168.64.4.58887: Flags [.],
> ack 186, win 237, length 0

... until the client changes and sends a FIN.

> 
> Netstat output on Squid box:
> # date; netstat -tno | grep 192.168.64.4
> Thu Nov 26 13:59:40 BRST 2015
> tcp6       1      0 172.16.10.22:3126       192.168.64.4:58887
> CLOSE_WAIT  off (0.00/0/0)
> 
> And after 2 hours and a half netstat output is still the same:
> # date; netstat -tno | grep 192.168.64.4
> Thu Nov 26 16:32:37 BRST 2015
> tcp6       1      0 172.16.10.22:3126       192.168.64.4:58887
> CLOSE_WAIT  off (0.00/0/0)
> 
> Squid is still using the file descriptor.
> # date; lsof -n | grep 192.168.64.4
> Thu Nov 26 16:33:10 BRST 2015
> squid     29137            proxy *244u     IPv6 13127035      0t0       
> TCP 172.16.10.22:3126->192.168.64.4:58887 (CLOSE_WAIT)
> 


So, the first place to look is not Squid I think. But why at least 6 of
those ACK packets did not make it back to the client. That needs
resolving first to esure that the TCP level is operating correctly.

Only then if the problem remains looking at Squid, the use of port 443
indicates it is the crypto process is possibly waiting for something and
not closing the port on a 0-byte read(2) operation.

Amos