[squid-users] TCP out of memory

[Amos Jeffries]

On 09/01/18 22:40, Vieri wrote:
> 
> ________________________________
> From: Amos Jeffries <squid3 at treenet.co.nz>
>>
>> I have only taken a brief look, but so far it looks like the problematic
> 
>> sockets are not participating in any ICAP activity.
> 
> Do you see that from the cache.log, or from ":filedescriptors"?

I went through the cache.log looking for the FD numbers mentioned to 
correlate with the filedescriptors report lines. The first dozen I found 
all seemed to be either in some clearly described state (not that 
"127.0.0.1" description).

My deeper look was going to make a script to do the reverse check. For 
each entry in the filedescriptors, seeing if there was any ICAP mentions 
and print where in the cache.log to look.


> 
> 
> Both Nread and Nwrite seem to be well over 0.
> 

That I think is really odd for sockets to port 1344 which are not having 
any ICAP protocol activity happening.


>> That implies they > are possibly TCP connections which never complete their opening
>> sequence, or at least the result of connection attempts does not make it
>> back to the ICAP code somehow.
> 
> 
> ICAP and Squid are both on localhost. I'd like to find out why this is happening.
> 


That is why I specifically asked for 93,* trace. The 93,* traces should 
not contains any of the HTTP traffic to Squid just the ICAP.

If you can get an 11,* trace as well separately to see if any of the 
stuck port 1344 FDs are using HTTP. There should be none. But if there 
are that could well be the problem on those ones.


> 
> I believe I already posted a tcpdump trace of the ICAP traffic, but I don't know if you had a chance to take a look at it. I had a quick look, but I'm not familiar with the ICAP protocol. In any case, I probably would see a lot of OPTIONS, REQMOD, RESPMOD methods, but I don't know if I would clearly detect initial TCP issues.
> 

You mean the trace in your "TCP out of memory" thread last month?
What I'm seeing in there is connections being used for ICAP, then the 
ICAP service requesting keep-alive which Squid honors. Nothing obviously 
broken.


> 
> Anyway, here's a dumb question. Can't Squid "tell" when a TCP connection to an ICAP server has never completed correctly after x timeout, and close it down/reset it?
> I'm using default values in squid.conf for the following:
> connect_timeout
> icap_connect_timeout
> peer_connect_timeout
> 
> The docs say:
> #  TAG: icap_connect_timeout
> #       This parameter specifies how long to wait for the TCP connect to
> #       the requested ICAP server to complete before giving up and either
> #       terminating the HTTP transaction or bypassing the failure.
> 
> 
> BTW I guess it's just a typo error because instead of an "HTTP transaction" I should read "ICAP transaction", right?

Nope. If ICAP fails for any reason and is mandatory (bypass=0) the HTTP 
transaction which is trying to use it gets broken. So the HTTP request 
(transaction) needs to be aborted with an error.
That does not necessarily mean the TCP connection used by the HTTP 
though, just the request/reply transaction.


> 
> Anyway, I have "bypass=0" for the ICAP service so I guess it should honor connect_timeout.
> The default is connect_timeout 1 minute.
> 

No, icap_connect_timeout is the only one that should be relevant on 
connections to ICAP services.

connect_timeout is for HTP origin servers, and peer_connect_timeout is 
for cache_peers.


> 
> With a 1-minute timeout it may be hard to see the sockets close when there's plenty of traffic, but I think I should see a substantial drop of open sockets when traffic is low (eg. at night). However, I don't see it.
> 

If you mean the *connect_timeout, that only affects the time Squid waits 
for the outgoing TCP SYN/SYN+ACK process to complete. Persistent and 
other connections already opened have other timeouts applied depending 
on what they are being used for.

The *idle_pconn_timeout directives are more likely to result in closed 
connections for overnight etc. cases if clients do not actively end them 
anyway.


Amos