[squid-users] Excessive TCP memory usage
Eliezer Croitoru
eliezer at ngtech.co.il
Tue Jun 14 10:21:32 UTC 2016
Hey,
Steps to reproduce are not exactly everything since squid works fine in many other scenarios.
I do not know this specific system but if you are talking about 1-4k open connections it should not be a big problem for many servers.
The issue in hands is a bit different.
Have you tried tuning the ipv4\net using sysctl to see if it affects anything?
What I can offer is to build a tiny ICAP service that will use a 204 on every request and then moves on.
If the same happens with the dummy service it's probably a very bad scenario and if not then we can try to think
if there is something unique about your setup.
I have not seen this issue in my current testing setup which includes 3.5.19 + ICAP url filtering service.
Eliezer
----
Eliezer Croitoru
Linux System Administrator
Mobile: +972-5-28704261
Email: eliezer at ngtech.co.il
-----Original Message-----
From: squid-users [mailto:squid-users-bounces at lists.squid-cache.org] On Behalf Of Deniz Eren
Sent: Tuesday, June 14, 2016 11:07 AM
To: squid-users at lists.squid-cache.org
Subject: Re: [squid-users] Excessive TCP memory usage
Little bump :)
I have posted bug report with steps to reproduce. The problem still exists and I am curious whether anyone else is having the same problem, too.
http://bugs.squid-cache.org/show_bug.cgi?id=4526
On Wed, May 25, 2016 at 1:18 PM, Deniz Eren <denizlist at denizeren.net> wrote:
> When I listen to connections between squid and icap using tcpdump I
> saw that after a while icap closes the connection but squid does not
> close, so connection stays in CLOSE_WAIT state:
>
> [root at test ~]# tcpdump -i any -n port 34693
> tcpdump: WARNING: Promiscuous mode not supported on the "any" device
> tcpdump: verbose output suppressed, use -v or -vv for full protocol
> decode listening on any, link-type LINUX_SLL (Linux cooked), capture
> size 96 bytes
> 13:07:31.802238 IP 127.0.0.1.icap > 127.0.0.1.34693: F
> 2207817997:2207817997(0) ack 710772005 win 395 <nop,nop,timestamp
> 104616992 104016968>
> 13:07:31.842186 IP 127.0.0.1.34693 > 127.0.0.1.icap: . ack 1 win 3186
> <nop,nop,timestamp 104617032 104616992>
>
> [root at test ~]# netstat -tulnap|grep 34693
> tcp 215688 0 127.0.0.1:34693 127.0.0.1:1344
> CLOSE_WAIT 19740/(squid-1)
>
> These CLOSE_WAIT connections do not timeout and stay until squid
> process is killed.
>
> 2016-05-25 10:37 GMT+03:00 Deniz Eren <denizlist at denizeren.net>:
>> 2016-05-24 21:47 GMT+03:00 Amos Jeffries <squid3 at treenet.co.nz>:
>>> On 25/05/2016 5:50 a.m., Deniz Eren wrote:
>>>> Hi,
>>>>
>>>> After upgrading to squid 3.5.16 I realized that squid started using
>>>> much of kernel's TCP memory.
>>>
>>> Upgrade from which version?
>>>
>> Upgrading from squid 3.1.14. I started using c-icap and ssl-bump.
>>
>>>>
>>>> When squid was running for a long time TCP memory usage is like below:
>>>> test at test:~$ cat /proc/net/sockstat
>>>> sockets: used *
>>>> TCP: inuse * orphan * tw * alloc * mem 200000
>>>> UDP: inuse * mem *
>>>> UDPLITE: inuse *
>>>> RAW: inuse *
>>>> FRAG: inuse * memory *
>>>>
>>>> When I restart squid the memory usage drops dramatically:
>>>
>>> Of course it does. By restarting you just erased all of the
>>> operational state for an unknown but large number of active network connections.
>>>
>> That's true but what I mean was squid's CLOSE_WAIT connections are
>> using too much memory and they are not timing out.
>>
>>> Whether many of those should have been still active or not is a
>>> different question. the answer to which depends on how you have your
>>> Squid configured, and what the traffic through it has been doing.
>>>
>>>
>>>> test at test:~$ cat /proc/net/sockstat
>>>> sockets: used *
>>>> TCP: inuse * orphan * tw * alloc * mem 10
>>>> UDP: inuse * mem *
>>>> UDPLITE: inuse *
>>>> RAW: inuse *
>>>> FRAG: inuse * memory *
>>>>
>>>
>>> The numbers you replaced with "*" are rather important for context.
>>>
>>>
>> Today again I saw the problem:
>>
>> test at test:~$ cat /proc/net/sockstat
>> sockets: used 1304
>> TCP: inuse 876 orphan 81 tw 17 alloc 906 mem 29726
>> UDP: inuse 17 mem 8
>> UDPLITE: inuse 0
>> RAW: inuse 1
>> FRAG: inuse 0 memory 0
>>
>>>> I'm using Squid 3.5.16.
>>>>
>>>
>>> Please upgrade to 3.5.19. Some important issues have been resolved.
>>> Some of them may be related to your TCP memory problem.
>>>
>>>
>> I have upgraded now and problem still exists.
>>
>>>> When I look with "netstat" and "ss" I see lots of CLOSE_WAIT
>>>> connections from squid to ICAP or from squid to upstream server.
>>>>
>>>> Do you have any idea about this problem?
>>>
>>> Memory use by the TCP system of your kernel has very little to do
>>> with Squid. Number of sockets in CLOSE_WAIT does have some relation
>>> to Squid or at least to how the traffic going through it is handled.
>>>
>>> If you have disabled persistent connections in squid.conf then lots
>>> of closed sockets and FD are to be expected.
>>>
>>> If you have persistent connections enabled, then fewer closures
>>> should happen. But some will so expectations depends on how high the
>>> traffic load is.
>>>
>> Persistent connection parameters are enabled in my conf, the problem
>> occurs especially with connections to c-icap service.
>>
>> My netstat output is like this:
>> netstat -tulnap|grep squid|grep CLOSE
>>
>> tcp 211742 0 127.0.0.1:55751 127.0.0.1:1344
>> CLOSE_WAIT 17076/(squid-1)
>> tcp 215700 0 127.0.0.1:55679 127.0.0.1:1344
>> CLOSE_WAIT 17076/(squid-1)
>> tcp 215704 0 127.0.0.1:55683 127.0.0.1:1344
>> CLOSE_WAIT 17076/(squid-1)
>> ...(hundreds)
>> Above ones are connections to c-icap service.
>>
>> netstat -tulnap|grep squid|grep CLOSE Active Internet connections
>> (servers and established)
>> Proto Recv-Q Send-Q Local Address Foreign Address
>> State PID/Program name
>> tcp 1 0 192.168.2.1:8443 192.168.6.180:45182
>> CLOSE_WAIT 15245/(squid-1)
>> tcp 1 0 192.168.2.1:8443 192.168.2.177:50020
>> CLOSE_WAIT 15245/(squid-1)
>> tcp 1 0 192.168.2.1:8443 192.168.2.172:60028
>> CLOSE_WAIT 15245/(squid-1)
>> tcp 1 0 192.168.2.1:8443 192.168.6.180:44049
>> CLOSE_WAIT 15245/(squid-1)
>> tcp 1 0 192.168.2.1:8443 192.168.6.180:55054
>> CLOSE_WAIT 15245/(squid-1)
>> tcp 1 0 192.168.2.1:8443 192.168.2.137:52177
>> CLOSE_WAIT 15245/(squid-1)
>> tcp 1 0 192.168.2.1:8443 192.168.6.180:43542
>> CLOSE_WAIT 15245/(squid-1)
>> tcp 1 0 192.168.2.1:8443 192.168.6.155:39489
>> CLOSE_WAIT 15245/(squid-1)
>> tcp 1 0 192.168.2.1:8443 192.168.0.147:38939
>> CLOSE_WAIT 15245/(squid-1)
>> tcp 1 0 192.168.2.1:8443 192.168.6.180:38754
>> CLOSE_WAIT 15245/(squid-1)
>> tcp 1 0 192.168.2.1:8443 192.168.0.164:39602
>> CLOSE_WAIT 15245/(squid-1)
>> tcp 1 0 192.168.2.1:8443 192.168.0.147:54114
>> CLOSE_WAIT 15245/(squid-1)
>> tcp 1 0 192.168.2.1:8443 192.168.6.180:57857
>> CLOSE_WAIT 15245/(squid-1)
>> tcp 1 0 192.168.2.1:8443 192.168.0.156:43482
>> CLOSE_WAIT 15245/(squid-1)
>> ...(about 50)
>> Above ones are connections from https port to client.
>>
>> As you can see recv-q for icap connections allocate more memory but
>> connections from https_port to upstream server connections allocate
>> only one byte.
>>
>> What can be done to close these unused connections?
>>
>> The problem in this thread seems similar:
>> http://www.squid-cache.org/mail-archive/squid-users/201301/0092.html
>>
>>> Amos
>>>
>>> _______________________________________________
>>> squid-users mailing list
>>> squid-users at lists.squid-cache.org
>>> http://lists.squid-cache.org/listinfo/squid-users
_______________________________________________
squid-users mailing list
squid-users at lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users
More information about the squid-users
mailing list