[squid-users] squid hanging in 100% steal
Marc
gaardiolor at gmail.com
Thu Jan 24 12:24:18 UTC 2019
Hi,
For some reason my squid sometimes hangs (after weeks of running
smoothly) in 100% steal, until I kill the proces and restart it, after
which the proces will again run stable for weeks.
It's running on a AWS EC2 instance, squid version:
squid-3.5.20-10.34.amzn1.x86_64 , see below for some debugging info.
Any idea what could be the problem here ? Thanks!
top:
[11:56:49][root at ip-172-31-9-138 ~]# top
top - 11:57:11 up 218 days, 17:36, 1 user, load average: 1.06, 1.17, 1.09
Tasks: 81 total, 2 running, 79 sleeping, 0 stopped, 0 zombie
Cpu(s): 4.5%us, 0.3%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 95.2%st
Mem: 501220k total, 405748k used, 95472k free, 65512k buffers
Swap: 0k total, 0k used, 0k free, 88948k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
29963 squid 20 0 290m 171m 7472 R 99.9 35.1 672:59.73 squid
1 root 20 0 19648 2480 2148 S 0.0 0.5 0:02.05 init
<snip>
vmstat:
[11:57:39][root at ip-172-31-9-138 ~]# vmstat 1
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 1 0 95408 65536 89052 0 0 0 4 1 1 0 0 99 0 0
1 0 0 95408 65536 89040 0 0 0 4 56 36 5 0 0 0 95
2 0 0 95408 65536 89040 0 0 0 0 54 18 5 0 0 0 95
1 0 0 95408 65536 89040 0 0 0 0 57 30 5 0 0 0 95
1 0 0 95408 65536 89040 0 0 0 4 52 25 5 0 0 0 95
3 0 0 95408 65536 89040 0 0 0 0 52 14 6 0 0 0 94
1 0 0 95408 65536 89040 0 0 0 0 50 26 4 0 0 0 96
2 0 0 95408 65536 89040 0 0 0 0 53 21 6 0 0 0 94
1 0 0 95408 65540 89036 0 0 0 12 62 38 5 0 0 0 95
2 0 0 95408 65540 89040 0 0 0 36 55 14 5 0 0 0 95
1 0 0 95408 65540 89040 0 0 0 0 51 34 5 0 0 0 95
gdb:
[11:55:07][root at ip-172-31-9-138 ~]# sudo gdb -n -batch -ex backtrace -pid 29963
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
0x00000000007bca52 in
CbcPointer<Comm::TcpAcceptor>::operator=(CbcPointer<Comm::TcpAcceptor>
const&) ()
#0 0x00000000007bca52 in
CbcPointer<Comm::TcpAcceptor>::operator=(CbcPointer<Comm::TcpAcceptor>
const&) ()
#1 0x00000000007bc3d4 in Comm::AcceptLimiter::kick() ()
#2 0x0000000000721867 in AsyncCall::make() ()
#3 0x00000000007259e2 in AsyncCallQueue::fireNext() ()
#4 0x0000000000725e20 in AsyncCallQueue::fire() ()
#5 0x00000000005b0089 in EventLoop::runOnce() ()
#6 0x00000000005b0178 in EventLoop::run() ()
#7 0x00000000006192cc in SquidMain(int, char**) ()
#8 0x0000000000514b3b in main ()
strace:
[11:52:51][root at ip-172-31-9-138 ~]# strace -t -s 8192 -f -p 29963
Process 29963 attached
11:53:00 accept(10, {sa_family=AF_INET6, sin6_port=htons(45756),
inet_pton(AF_INET6, "::ffff:<snip>", &sin6_addr), sin6_flowinfo=0,
sin6_scope_id=0}, [28]) = 16
11:53:00 getsockname(16, {sa_family=AF_INET6, sin6_port=htons(3128),
inet_pton(AF_INET6, "::ffff:<snip>", &sin6_addr), sin6_flowinfo=0,
sin6_scope_id=0}, [28]) = 0
11:53:00 fcntl(16, F_GETFD) = 0
11:53:00 fcntl(16, F_SETFD, FD_CLOEXEC) = 0
11:53:00 fcntl(16, F_GETFL) = 0x2 (flags O_RDWR)
11:53:00 fcntl(16, F_SETFL, O_RDWR|O_NONBLOCK) = 0
11:53:00 socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 23
11:53:00 ioctl(23, SIOCGARP, 0x7ffd21abeaa0) = -1 ENODEV (No such device)
11:53:00 ioctl(23, SIOCGIFCONF, {120, {{"lo", {AF_INET,
inet_addr("127.0.0.1")}}, {"eth0", {AF_INET, inet_addr("<snip>")}},
{"eth1", {AF_INET, inet_addr("<snip>")}}}}) = 0
11:53:00 ioctl(23, SIOCGARP, 0x7ffd21abeaa0) = -1 ENXIO (No such
device or address)
11:53:00 ioctl(23, SIOCGARP, 0x7ffd21abeaa0) = -1 ENXIO (No such
device or address)
11:53:00 close(23) = 0
11:53:00 epoll_ctl(5, EPOLL_CTL_DEL, 27, {0, {u32=27, u64=4294967323}}) = 0
11:53:00 close(27) = 0
11:53:03 accept(10, {sa_family=AF_INET6, sin6_port=htons(50050),
inet_pton(AF_INET6, "::ffff:<snip>", &sin6_addr), sin6_flowinfo=0,
sin6_scope_id=0}, [28]) = 23
11:53:03 getsockname(23, {sa_family=AF_INET6, sin6_port=htons(3128),
inet_pton(AF_INET6, "::ffff:<snip>", &sin6_addr), sin6_flowinfo=0,
sin6_scope_id=0}, [28]) = 0
11:53:03 fcntl(23, F_GETFD) = 0
11:53:03 fcntl(23, F_SETFD, FD_CLOEXEC) = 0
11:53:03 fcntl(23, F_GETFL) = 0x2 (flags O_RDWR)
11:53:03 fcntl(23, F_SETFL, O_RDWR|O_NONBLOCK) = 0
11:53:03 socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 25
11:53:03 ioctl(25, SIOCGARP, 0x7ffd21abeaa0) = -1 ENODEV (No such device)
11:53:03 ioctl(25, SIOCGIFCONF, {120, {{"lo", {AF_INET,
inet_addr("127.0.0.1")}}, {"eth0", {AF_INET, inet_addr("<snip>")}},
{"eth1", {AF_INET, inet_addr("<snip>")}}}}) = 0
11:53:03 ioctl(25, SIOCGARP, 0x7ffd21abeaa0) = -1 ENXIO (No such
device or address)
11:53:03 ioctl(25, SIOCGARP, 0x7ffd21abeaa0) = -1 ENXIO (No such
device or address)
11:53:03 close(25) = 0
11:53:03 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
11:53:03 write(9, "<snip> <snip> <snip> - - [24/Jan/2019:11:52:44
+0000] \"CONNECT <snip> HTTP/1.1\" 200 0 \"-\" \"Mozilla/5.0 (Windows
NT 10.0; Win64; x64; rv:64.0) Gecko/20100101 Firefox/64.0\"
TCP_TUNNEL:HIER_DIRECT\n", 223) = 223
11:53:03 epoll_ctl(5, EPOLL_CTL_DEL, 15, {0, {u32=15, u64=4294967311}}) = 0
11:53:03 close(15) = 0
^CProcess 29963 detached
More information about the squid-users
mailing list