[squid-users] Squid crashes with 3.5.1

Mon May 4 22:59:07 UTC 2015

Hi folks,

I am seeing a bunch of Squid crashes after moving recently to 3.5.1.
The stack trace for the various crashes is below, along with info I
have been able to gather for them. This is on a setup where we have
Squid running in conjunction with c-icap (both on the same box).The
steps that led to these crashes isn't known yet.

Squid runs in non-transparent mode. It proxies both HTTP and HTTPS connections.

A couple of the crashes look to have been caused in the path where
Squid interacts with c-icap. The c-icap version is somewhat old --
0.1.7 -- although it was working well with a version 3.3.13 Squid
until the recent move.

Do any of the fixes (listed below, where I have found references) help
here? Any other known issues that may be related?

Thanks.
Prashanth

pprabhu:~$ sudo egrep 'assertion failed' /var/log/elastica/gateway/cache.log
2015/03/31 19:56:09| assertion failed: Read.cc:69:
"fd_table[conn->fd].halfClosedReader != NULL"
2015/04/01 07:04:59| assertion failed: Read.cc:205: "params.data == data"
2015/04/01 07:37:31| assertion failed: Read.cc:69:
"fd_table[conn->fd].halfClosedReader != NULL"
2015/04/01 11:06:29| assertion failed: Read.cc:69:
"fd_table[conn->fd].halfClosedReader != NULL"
2015/04/07 18:23:54| assertion failed: comm.cc:557: "F->flags.open"
2015/04/08 05:01:57| assertion failed: Read.cc:69:
"fd_table[conn->fd].halfClosedReader != NULL"
2015/04/08 14:46:01| assertion failed: Read.cc:69:
"fd_table[conn->fd].halfClosedReader != NULL"
2015/04/08 18:04:13| assertion failed: Read.cc:205: "params.data ==
data" IMMEDIATELY.
2015/04/13 09:38:28| assertion failed: Read.cc:69:
"fd_table[conn->fd].halfClosedReader != NULL"
2015/04/20 23:34:46| assertion failed: Read.cc:69:
"fd_table[conn->fd].halfClosedReader != NULL"
2015/04/28 23:12:40| assertion failed: Read.cc:69:
"fd_table[conn->fd].halfClosedReader != NULL"
2015/04/28 23:46:24| assertion failed: Read.cc:205: "params.data == data"
2015/04/28 23:59:07| assertion failed: Read.cc:69:
"fd_table[conn->fd].halfClosedReader != NULL"
2015/04/29 18:06:01| assertion failed: Read.cc:69:
"fd_table[conn->fd].halfClosedReader != NULL"

2015/04/28 23:46:24| assertion failed: Read.cc:205: "params.data == data"
----
http://squid-web-proxy-cache.1019090.n4.nabble.com/assertion-failed-Read-cc-205-quot-params-data-data-quot-td4670624.html

pprabhu:~$ gdb /usr/local/sbin/squid core-squid-508-1430264784
GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://bugs.launchpad.net/gdb-linaro/>...
Reading symbols from /usr/local/sbin/squid...done.
[New LWP 508]

warning: Can't read pathname for load map: Input/output error.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/local/sbin/squid -N -f /etc/squid/squid.conf -D'.
Program terminated with signal 6, Aborted.
#0  0x00007fd235dba0d5 in raise () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0  0x00007fd235dba0d5 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007fd235dbd83b in abort () from /lib/x86_64-linux-gnu/libc.so.6
    callback=0x6109a0 <IdleConnList::Read(RefCount<Comm::Connection>
const&, char*, unsigned long, Comm::Flag, int, void*)>, data=0xe1b078)
    at Read.cc:205
#4  0x000000000060dc54 in IdleConnList::clearHandlers (this=0xe1b078,
conn=...) at pconn.cc:157
#5  0x00000000006111cf in IdleConnList::pop (this=0xe1b078) at pconn.cc:223
#6  0x0000000000804f13 in Adaptation::Icap::ServiceRep::getConnection
(this=0xe1ac88, retriableXact=<optimized out>,
    reused=@0x7fff42b82bef: false) at ServiceRep.cc:116
#7  0x0000000000810f7f in Adaptation::Icap::Xaction::openConnection
(this=0x1847a68) at Xaction.cc:125
#8  0x000000000081fbc0 in Adaptation::Icap::ModXact::start
(this=0x1847a68) at ModXact.cc:101
#9  0x00000000006f5a84 in JobDialer<AsyncJob>::dial (this=0x18f81e0,
call=...) at ../../src/base/AsyncJobCalls.h:174
#10 0x00000000006f2309 in AsyncCall::make (this=0x18f81b0) at AsyncCall.cc:40
#11 0x00000000006f61ec in AsyncCallQueue::fireNext (this=<optimized
out>) at AsyncCallQueue.cc:56
#12 0x00000000006f6540 in AsyncCallQueue::fire (this=0xdd8d80) at
AsyncCallQueue.cc:42
#13 0x0000000000593bbc in EventLoop::runOnce (this=0x7fff42b82dc0) at
EventLoop.cc:120
#14 0x0000000000593d70 in EventLoop::run (this=0x7fff42b82dc0) at
EventLoop.cc:82
#15 0x00000000005f7d83 in SquidMain (argc=<optimized out>,
argv=<optimized out>) at main.cc:1508
#16 0x000000000050a0db in SquidMainSafe (argv=<optimized out>,
argc=<optimized out>) at main.cc:1240
#17 main (argc=<optimized out>, argv=<optimized out>) at main.cc:1233
(gdb) fr 3
#3  0x0000000000796908 in comm_read_cancel (fd=40,
    callback=0x6109a0 <IdleConnList::Read(RefCount<Comm::Connection>
const&, char*, unsigned long, Comm::Flag, int, void*)>, data=0xe1b078)
    at Read.cc:205
205 Read.cc: No such file or directory.
----

2015/04/28 23:59:07| assertion failed: Read.cc:69:
"fd_table[conn->fd].halfClosedReader != NULL"
----
(gdb) bt
#0  0x00007f7ba4ad80d5 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f7ba4adb83b in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x000000000058174f in xassert (msg=0x899348 "g read handler on FD
", file=0x8991bf "nish", line=69) at debug.cc:544
#3  0x0000000000797410 in comm_read_base (conn=..., buf=0x124cbf0 "",
size=65535, callback=...) at Read.cc:69
#4  0x000000000080f1b9 in comm_read (callback=..., len=65535,
buf=0x124cbf0 "", conn=...) at ../../../src/comm/Read.h:58
#5  Adaptation::Icap::Xaction::scheduleRead (this=0x124bb78) at Xaction.cc:397
#6  0x000000000081ba78 in Adaptation::Icap::ModXact::readMore
(this=0x124bb78) at ModXact.cc:561
#7  0x0000000000823742 in
Adaptation::Icap::ModXact::handleCommConnected (this=0x124bb78) at
ModXact.cc:191
#8  0x000000000080ff2b in Adaptation::Icap::Xaction::noteCommConnected
(this=0x124bb78, io=...) at Xaction.cc:266
#9  0x0000000000813a4b in JobDialer<Adaptation::Icap::Xaction>::dial
(this=0x1d150a0, call=...) at ../../../src/base/AsyncJobCalls.h:174
#10 0x00000000006f2309 in AsyncCall::make (this=0x1d15070) at AsyncCall.cc:40
#11 0x00000000006f61ec in AsyncCallQueue::fireNext (this=<optimized
out>) at AsyncCallQueue.cc:56
#12 0x00000000006f6540 in AsyncCallQueue::fire (this=0xdd8a50) at
AsyncCallQueue.cc:42
#13 0x0000000000593bbc in EventLoop::runOnce (this=0x7fff37077350) at
EventLoop.cc:120
#14 0x0000000000593d70 in EventLoop::run (this=0x7fff37077350) at
EventLoop.cc:82
#15 0x00000000005f7d83 in SquidMain (argc=<optimized out>,
argv=<optimized out>) at main.cc:1508
#16 0x000000000050a0db in SquidMainSafe (argv=<optimized out>,
argc=<optimized out>) at main.cc:1240
#17 main (argc=<optimized out>, argv=<optimized out>) at main.cc:1233

http://bugs.squid-cache.org/show_bug.cgi?id=3775
-- Stacktrace is different. c-icap connection appears to be where the assert
   was tripped.

http://bugs.squid-cache.org/show_bug.cgi?id=4173
-- Dup of 3775
----

2015/04/07 18:23:54| assertion failed: comm.cc:557: "F->flags.open"
----
http://bugs.squid-cache.org/show_bug.cgi?id=3329
-- Directed here, via
   http://squid-web-proxy-cache.1019090.n4.nabble.com/assertion-failed-comm-cc-557-quot-F-gt-flags-open-quot-td4670788.html
----