[squid-users] Squid 4.3 assertion failed

[Julian Perconti]

> Hi Alex/Amos
> 
> Since yesterday squid is running via this method in a cron script:
> 
> trap "rm -f $$.gdb" 0
> cat <<EOF >$$.gdb
> handle SIGPIPE pass nostop noprint
> handle SIGTERM pass nostop noprint
> handle SIGUSR1 pass nostop noprint
> handle SIGHUP  pass
> handle SIGKILL pass
> handle SIGSEGV stop
> handle SIGABRT stop
> run -NYCd3
> backtrace
> quit
> EOF
> while sleep 2; do
>   gdb -x $$.gdb /path/to/squid 2>&1 | tee -a squid.out done

Hi all,

After 2/3 days  Squid has crashed:

First signal:

Program received signal SIGHUP, Hangup.
0x00007ffff41b00a3 in __epoll_wait_nocancel ()
    at ../sysdeps/unix/syscall-template.S:84
84	../sysdeps/unix/syscall-template.S: No such file or directory.
#0  0x00007ffff41b00a3 in __epoll_wait_nocancel ()
    at ../sysdeps/unix/syscall-template.S:84
#1  0x00005555559783f4 in Comm::DoSelect (msec=<optimized out>)
    at ModEpoll.cc:227
#2  0x000055555592197e in CommSelectEngine::checkEvents (
    this=<optimized out>, timeout=<optimized out>) at comm.cc:1847
#3  0x0000555555764345 in EventLoop::checkEngine (
    this=this at entry=0x7fffffffe940, engine=engine at entry=0x7fffffffe6a8, 
    primary=primary at entry=true) at EventLoop.cc:36
#4  0x000055555576452d in EventLoop::runOnce (this=this at entry=0x7fffffffe940)
    at EventLoop.cc:115
#5  0x0000555555764648 in EventLoop::run (this=0x7fffffffe940)
    at EventLoop.cc:83
#6  0x00005555557cf933 in SquidMain (argc=<optimized out>, 
    argv=<optimized out>) at main.cc:1707
#7  0x00005555556b53a4 in SquidMainSafe (argv=<optimized out>, argc=2)
    at main.cc:1415
#8  main (argc=2, argv=0x7fffffffed68) at main.cc:1403
A debugging session is active.

The crash (assertion failed)

2018/10/24 09:44:29| assertion failed: http.cc:1530: "!Comm::MonitorsRead(serverConnection->fd)"

Program received signal SIGABRT, Aborted.
__GI_raise (sig=sig at entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
51	../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
#0  __GI_raise (sig=sig at entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x00007ffff40fb42a in __GI_abort () at abort.c:89
#2  0x000055555573c57c in xassert (
    msg=msg at entry=0x555555aec788 "!Comm::MonitorsRead(serverConnection->fd)", 
    file=file at entry=0x555555aeb5c5 "http.cc", line=line at entry=1530)
    at debug.cc:618
#3  0x00005555557a1118 in HttpStateData::maybeReadVirginBody (
    this=0x5555574c6888) at http.cc:1530
#4  0x000055555579d3db in HttpStateData::sendRequest (
    this=this at entry=0x5555574c6888) at http.cc:2198
#5  0x000055555579df50 in HttpStateData::start (this=0x5555574c6888)
    at http.cc:2311
#6  0x00005555559156b2 in JobDialer<AsyncJob>::dial (this=0x55555db33390, 
    call=...) at ../../src/base/AsyncJobCalls.h:174
#7  0x00005555559110e1 in AsyncCall::make (this=this at entry=0x55555db33360)
    at AsyncCall.cc:40
#8  0x00005555559124b1 in AsyncCallQueue::fireNext (
    this=this at entry=0x555556105e00) at AsyncCallQueue.cc:56
#9  0x0000555555912819 in AsyncCallQueue::fire (this=0x555556105e00)
    at AsyncCallQueue.cc:42
#10 0x0000555555764549 in EventLoop::dispatchCalls (this=0x7fffffffe940)
    at EventLoop.cc:144
#11 EventLoop::runOnce (this=this at entry=0x7fffffffe940) at EventLoop.cc:121
#12 0x0000555555764648 in EventLoop::run (this=0x7fffffffe940)
    at EventLoop.cc:83
#13 0x00005555557cf933 in SquidMain (argc=<optimized out>, 
    argv=<optimized out>) at main.cc:1707
#14 0x00005555556b53a4 in SquidMainSafe (argv=<optimized out>, argc=2)
    at main.cc:1415
#15 main (argc=2, argv=0x7fffffffed68) at main.cc:1403
A debugging session is active.


The assertion seems to be similiar to bug https://bugs.squid-cache.org/show_bug.cgi?id=4896  as Alex said; but I am not sure.
Should I post there anyway?

Additional info: I can (now) reproduce the crash, it starts to happen when a client tried to connect to a server which (Chrome) returns: ERR_CONNECTION_RESET/EMPTY RESPONSE

I can confirm that site is not working (I tested without squid and from another internet access)

By other hand, what can I do with those squid's crashes?

Squid info (recompiled yesterday):

Squid Cache: Version 4.3-20181021-r17614d5
Service Name: squid

This binary uses OpenSSL 1.1.0f  25 May 2017. For legal restrictions on distribution see https://www.openssl.org/source/license.html

configure options:  '--prefix=/usr' '--build=x86_64-linux-gnu' '--libexecdir=/usr/lib/squid' '--srcdir=.' '--datadir=/usr/share/squid' '--with-cppunit-basedir=/usr' '--enable-inline' '--enable-delay-pools' '--sysconfdir=/etc/squid' '--with-logdir=/var/log/squid' '--with-pidfile=/var/run/squid.pid' '--with-openssl' '--enable-ssl-crtd' '--mandir=/usr/share/man' '--enable-arp-acl' '--enable-wccpv2' '--with-large-files' '--with-default-user=proxy' '--enable-linux-netfilter' '--enable-storeio=ufs,aufs,diskd,rock' '--enable-removal-policies=lru,heap' '--enable-icap' '--enable-icap-client' '--enable-cache-digests' '--enable-heap-replacement' 'build_alias=x86_64-linux-gnu'

If You want I post the config file.

Hope its helps,

Thank You