[squid-users] squid and netdata causes squid to drop SYN?

[Amos Jeffries]

On 22/01/20 8:54 pm, Amish wrote:
> 
> On 22/01/20 12:10 pm, Amos Jeffries wrote:
>> On 22/01/20 6:55 pm, Amish wrote:
>>
>>>> It appears that it runs a query on "counters". But I dont know if that
>>>> is counted as a "heavy" query or not.
>> It is one of the light ones. So if that were all that is going on I
>> would not be expecting a problem.
>>
>> ...
>>
>> Your mention of intercepting traffic to port 3128 makes me wonder if the
>> netdata auto-detect is trying to use that port.
>>   If that is happening there would be some visible effects:
>>   A) if you have firewall protection DROP'ing direct connections to the
>> intercept port. That would show up exactly as SYN with no SYN+ACK on any
>> auto-detect probes the tool used to that port. (This is one reason I am
>> so vocal about people not using port 3128 to intercept).
>>
>>   B) if you are missing that mandatory firewall protection, the tool may
>> be triggering forwarding loops. Which use many more FD than it should
>> (up to all of them) with the visible effects being clients not able to
>> connect around peak times and SYN being dropped when limits are hit
>> (either the loop limit, or interception port protection.
>>   If the tool is smart enough to detect the error state and move to
>> another port for a while that might explain the intermittent nature.
> 
> Connection from 127.0.0.1 (loopback interface) are not blocked. (DROPped)

It should be. *No* traffic can be allowed to connect directly to a Squid
'intercept' port. It is exclusively for use by NAT<->Squid communication.

Anything that does connect directly will start a loop which can consume
up to all of the resources on the server (TCP sockets, TCP ports, NAT
table entries, file descriptors, memory, CPU ... whichever exhausts
first). Squid uses Via header and some other details to try to detect
loops before they become a problem and forbid ASAP. But there may be
ways around the detection - since Via is often disabled and/or stripped
from traffic.

> 
> Also, if it was due to DROP then it should not work from beginning but
> here it works for a while (few hours) and then suddenly starts blocking.
> 

The netdata tool at least has multiple different ways to connect to the
proxy. It has some logic to auto-detect which work and which don't. One
of those ways is port 3128 and another is port 8080 - which on your
proxy are the intercept and explicit proxy ports respectively. It may
try a working port some executions and then a non-working port next
execution.


> Also I grepped whole cache.log for word "descriptor", it showed me no
> error. Only line I saw is that 16384 descriptors are available (when
> squid starts)

That is not what you should be looking at anyway. Look at the HTTP
messages arriving, where they are coming from and what happens to them.

> 
> I will try to reproduce the error when I get some time. Currently I have
> disabled netdata's squid module and things are stable.
> 
> Thank you very much for your response,

Aye, hope this info helps for that troubleshooting.

Amos