[squid-users] Squid SMP workers crash

[Deniz Eren]

On Sun, Oct 16, 2016 at 2:57 AM, Eliezer Croitoru <eliezer at ngtech.co.il> wrote:
> Hey,
>
> I can try to replicate the same configuration removing couple settings just to make it simpler to verify if the issue since it's similar to the next testing lab I have planned.
> Can you give more detail about the OS? CentOS, Ubuntu, Other?
CentOS 5

> If it's a self compiled versions then "squid -v" output.
Squid Cache: Version 3.5.20
Service Name: squid
configure options:  '--build=i686-redhat-linux-gnu'
'--host=i686-redhat-linux-gnu' '--target=i386-redhat-linux-gnu'
'--program-prefix=' '--exec-prefix=/opt/squid'
'--datadir=/opt/squid/share' '--libdir=/opt/squid/lib'
'--libexecdir=/opt/squid/libexec' '--localstatedir=/var'
'--sharedstatedir=/opt/squid/com' '--infodir=/usr/share/info'
'--prefix=/opt/squid' '--exec_prefix=/opt/squid'
'--bindir=/opt/squid/bin' '--sbindir=/opt/squid/sbin'
'--sysconfdir=/opt/squid/etc' '--datadir=/opt/squid/share/squid'
'--includedir=/opt/squid/include' '--libdir=/opt/squid/lib/squid'
'--libexecdir=/opt/squid/lib/squid' '--localstatedir=/opt/squid/var'
'--mandir=/opt/squid/share/man' '--infodir=/opt/squid/share/info'
'--enable-epoll' '--disable-dependency-tracking' '--enable-arp-acl'
'--enable-auth' '--enable-auth-negotiate' '--enable-auth-digest'
'--enable-auth-basic' '--enable-auth-ntlm' '--enable-cache-digests'
'--enable-cachemgr-hostname=localhost' '--enable-delay-pools'
'--enable-external-acl-helpers' '--enable-icap-client'
'--with-large-files' '--enable-linux-netfilter' '--enable-referer-log'
'--enable-removal-policies=heap,lru' '--enable-snmp' '--enable-ssl'
'--enable-storeio=aufs,diskd,ufs' '--enable-useragent-log'
'--enable-wccpv2' '--with-aio' '--with-default-user=squid'
'--with-filedescriptors=32768' '--with-dl' '--enable-ssl-crtd'
'--with-openssl=/opt/openssl101' '--with-pthreads'
'--enable-http-violations' '--enable-follow-x-forwarded-for'
'--disable-ipv6' 'build_alias=i686-redhat-linux-gnu'
'host_alias=i686-redhat-linux-gnu'
'target_alias=i386-redhat-linux-gnu' 'CFLAGS=-fPIE -Os -g -pipe
-fsigned-char -I /usr/kerberos/include -I/opt/openssl101/include -O2
-g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector
--param=ssp-buffer-size=4 -m32 -march=i386 -mtune=generic
-fasynchronous-unwind-tables' 'LDFLAGS=-pie -L/opt/openssl101/lib'
'CXXFLAGS=-fPIE -I/opt/openssl101/include -O2 -g -pipe -Wall
-Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector
--param=ssp-buffer-size=4 -m32 -march=i386 -mtune=generic
-fasynchronous-unwind-tables'
'PKG_CONFIG_PATH=/opt/squid/lib/pkgconfig:/opt/squid/share/pkgconfig'
--enable-ltdl-convenience


> I have also seen that you are intercepting both http and https traffic, have you tried looking at the logs?
You are right I'm intercepting both http and https traffic. Yes I have
looked at logs and only suspicious thing is this line:
2016/10/17 11:22:37 kid1| assertion failed:
../../src/ipc/AtomicWord.h:71: "Enabled()"

>
> If you don't hear me from me fast enough just bump me with an email.
>
> Eliezer
>
> ----
> Eliezer Croitoru
> Linux System Administrator
> Mobile+WhatsApp: +972-5-28704261
> Email: eliezer at ngtech.co.il
>
>
> -----Original Message-----
> From: squid-users [mailto:squid-users-bounces at lists.squid-cache.org] On Behalf Of Deniz Eren
> Sent: Thursday, October 13, 2016 10:53 AM
> To: squid-users at lists.squid-cache.org
> Subject: [squid-users] Squid SMP workers crash
>
> Hi,
>
> I'm using squid's SMP functionality to distribute requests to many
> squid instances and distribute workload to multiple processors.
> However while running squid's workers after a while worker processes
> crash with the error below and coordinator does not start them again:
> ...
> FATAL: Ipc::Mem::Segment::open failed to
> shm_open(/squid-cf__metadata.shm): (2) No such file or directory
> Squid Cache (Version 3.5.20): Terminated abnormally.
> ...
>
> Does a solution exists for this problem? (permissions are OK in /dev/shm)
>
>
> When everything is OK coordinator listens to http_ports/https_port and
> distributes connections to workers(at least that's the conclusion I
> got from looking access.logs).
> [root at squidbox ~]# netstat -nlp|grep squid
> tcp        0      0 0.0.0.0:8080                0.0.0.0:*
>      LISTEN      7887/(squid-coord-1
> tcp        0      0 0.0.0.0:3127                0.0.0.0:*
>      LISTEN      7887/(squid-coord-1
> tcp        0      0 0.0.0.0:3128                0.0.0.0:*
>      LISTEN      7887/(squid-coord-1
> tcp        0      0 0.0.0.0:3130                0.0.0.0:*
>      LISTEN      7887/(squid-coord-1
> tcp        0      0 0.0.0.0:8443                0.0.0.0:*
>      LISTEN      7887/(squid-coord-1
> udp        0      0 0.0.0.0:57850               0.0.0.0:*
>                  7897/(squid-1)
> udp        0      0 0.0.0.0:33643               0.0.0.0:*
>                  7894/(squid-4)
> udp        0      0 0.0.0.0:50485               0.0.0.0:*
>                  7896/(squid-2)
> udp        0      0 0.0.0.0:46427               0.0.0.0:*
>                  7887/(squid-coord-1
> udp        0      0 0.0.0.0:58938               0.0.0.0:*
>                  7895/(squid-3)
>
>
> Also is my way of using SMP functionality correct, since I want to
> distribute all connections between workers and to listen only specific
> ports?
>
> I have attached the squid.conf.
>
> Regards,
>