[squid-users] squid crash: ERROR: system call failure while accepting a TLS connection

[Alex Rousskov]

On 2024-11-18 09:59, Martin A. Brooks wrote:

> I am running 3 squid instances behind a load balancer.  It was running 
> fine for a couple of weeks but but there were suddenly tens of thousands 
> of this sort of message in the log:

> squid[507015]: ERROR: system call failure while accepting a TLS 
> connection on conn705596 local=[redacted]:3128 remote=[redacted]:34830 
> FD 56 flags=1: 0xaaaae5711490*1

If you upgrade to a supported Squid version (v6+), the trailing hex 
number will be replaced with potentially important error details. I do 
not recall whether Squid handles these particular TLS errors correctly, 
but modern Squids (v6+) have a higher chance of doing so.

To avoid a misunderstanding, while the subject of this email thread says 
"crash", the email describes a controlled/orderly/expected emergency 
exit after a fatal error beyond Squid control.


> squid[507015]: logfileHandleWrite: daemon:/var/log/squid/access.log: 
> error writing ((32) Broken pipe)

I suspect that something has killed (or prevented a restart of) helper 
processes that are responsible for writing access.log entries. Failure 
to log requests is currently fatal ("by design") for Squid, but the true 
problem/trigger lies elsewhere -- why did the helper process die? I 
recommend studying system logs for any unusual preceding 
events/activity/reports. For _example_, it is possible that your system 
ran out of some OS resources and/or is under DoS attack.

To work around logging limitations, you can use stdio module instead of 
"daemon" module. Such a workaround may have performance implications so 
proceed with care.


HTH,

Alex.


>                                                         connection: 
> conn17735104 local=[redacted]:3128 remote=[redacted]:44550 FD 75 flags=1
> squid[507015]: Closing HTTP(S) port [::]:3128
>                                                         connection: 
> conn17735104 local=[redacted]:3128 remote=[redacted]:44550 FD 75 flags=1
> squid[507015]: storeDirWriteCleanLogs: Starting...
>                                                         connection: 
> conn17735104 local=[redacted]:3128 remote=[redacted]:44550 FD 75 flags=1
> squid[507015]:   Finished.  Wrote 0 entries.
>                                                         connection: 
> conn17735104 local=[redacted]:3128 remote=[redacted]:44550 FD 75 flags=1
> squid[507015]:   Took 0.00 seconds (  0.00 entries/sec).
>                                                         connection: 
> conn17735104 local=[redacted]:3128 remote=[redacted]:44550 FD 75 flags=1
> squid[507015]: FATAL: I don't handle this error well!
>                                                         connection: 
> conn17735104 local=[redacted]:3128 remote=[redacted]:44550 FD 75 flags=1
> squid[507015]: Squid Cache (Version 5.7): Terminated abnormally.
>                                                         connection: 
> conn17735104 local=[redacted]:3128 remote=[redacted]:44550 FD 75 flags=1
> 
> 
> 
> systemd tried to restart squid a few times, but repeated failures meant 
> it eventually gave up. This is squid-openssl as packaged by Debian on 
> ARM64:
> 
> # squid --version
> Squid Cache: Version 5.7
> Service Name: squid
> Debian linux
> 
> This binary uses OpenSSL 3.0.15 3 Sep 2024. configure options: 
> '--build=aarch64-linux-gnu' '--prefix=/usr' 
> '--includedir=${prefix}/include' '--mandir=${prefix}/share/man' 
> '--infodir=${prefix}/share/info' '--sysconfdir=/etc' 
> '--localstatedir=/var' '--disable-option-checking' 
> '--disable-silent-rules' '--libdir=${prefix}/lib/aarch64-linux-gnu' 
> '--runstatedir=/run' '--disable-maintainer-mode' 
> '--disable-dependency-tracking' 'BUILDCXXFLAGS=-g -O2 
> -ffile-prefix-map=/build/reproducible-path/squid-5.7=. 
> -fstack-protector-strong -Wformat -Werror=format-security 
> -Wno-error=deprecated-declarations -Wdate-time -D_FORTIFY_SOURCE=2 
> -Wl,-z,relro -Wl,-z,now ' 'BUILDCXX=g++' 
> '--with-build-environment=default' '--enable-build-info=Debian linux' 
> '--datadir=/usr/share/squid' '--sysconfdir=/etc/squid' 
> '--libexecdir=/usr/lib/squid' '--mandir=/usr/share/man' 
> '--enable-inline' '--disable-arch-native' '--enable-async-io=8' 
> '--enable-storeio=ufs,aufs,diskd,rock' 
> '--enable-removal-policies=lru,heap' '--enable-delay-pools' 
> '--enable-cache-digests' '--enable-icap-client' 
> '--enable-follow-x-forwarded-for' 
> '--enable-auth-basic=DB,fake,getpwnam,LDAP,NCSA,PAM,POP3,RADIUS,SASL,SMB' '--enable-auth-digest=file,LDAP' '--enable-auth-negotiate=kerberos,wrapper' '--enable-auth-ntlm=fake,SMB_LM' '--enable-external-acl-helpers=file_userip,kerberos_ldap_group,LDAP_group,session,SQL_session,time_quota,unix_group,wbinfo_group' '--enable-security-cert-validators=fake' '--enable-storeid-rewrite-helpers=file' '--enable-url-rewrite-helpers=fake' '--enable-eui' '--enable-esi' '--enable-icmp' '--enable-zph-qos' '--enable-ecap' '--disable-translation' '--with-swapdir=/var/spool/squid' '--with-logdir=/var/log/squid' '--with-pidfile=/run/squid.pid' '--with-filedescriptors=65536' '--with-large-files' '--with-default-user=proxy' '--enable-linux-netfilter' '--with-systemd' '--with-openssl' '--enable-ssl-crtd' 'build_alias=aarch64-linux-gnu' 'CFLAGS=-g -O2 -ffile-prefix-map=/build/reproducible-path/squid-5.7=. -fstack-protector-strong -Wformat -Werror=format-security -Wno-error=deprecated-declarations' 'LDFLAGS=-Wl,-z,relro -Wl,-z,now ' 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2' 'CXXFLAGS=-g -O2 -ffile-prefix-map=/build/reproducible-path/squid-5.7=. -fstack-protector-strong -Wformat -Werror=format-security -Wno-error=deprecated-declarations'
> 
> 
> It looks like all three of the squid instances blew up at nearly the 
> same time.
> 
> Any ideas what could be going on here?
> 
> 
> Regards
> 
> 
> Martin.
> _______________________________________________
> squid-users mailing list
> squid-users at lists.squid-cache.org
> https://lists.squid-cache.org/listinfo/squid-users