[squid-users] squid-3.5.0.2-20141031-r13657 crashes

[Amos Jeffries]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 1/12/2014 9:44 a.m., James Harper wrote:
> I've been getting squid crashes with squid-3.5.0.2-20141031-r13657.
> Basically I think my cache got corrupt - started seeing
> TCP_SWAPFAIL_MISS and md5 mismatches. Config is cache_dir ufs
> /usr/local/squid/var/cache/squid 102400 16 256
> 
> It's possible that at one point I might have started 2 instances of
> squid running at once... could that cause corruption?

Yes, very likely. More so the longer they were both running.

I see you mention segfaults below, that can also cause it for any
objects in use at the time of the crash.

> 
> And if it happens again, what sort of things should I collect to
> better diagnose the problem?

Firstly, Squid is expected to cope with cache corruption by cleaning
up entries as it identifies them. The SWAPFAIL and MD5 details you
mention above are signs that the detection at least is occuring.

After any type of corruption these messages can be expected for a
while with an exponential drop-off in frequency as the cache gets
fixed. Only if it starts occuring with unknown cause or does not
decrease in frequency is there a serious problem to attend to (usually
a disk dying, Squid crashing a lot, or second Squid process started
wrongly).


The basic things required for bug reports
(http://wiki.squid-cache.org/SquidFaq/BugReporting). That also
includes investigation of the segfauls mentioned below.

Plus if you can the URL, HTTP request headers in full, any access.log
entry you can match up with the issue.

> As I see it there are two problems: 1. that the cache got corrupt
> in the first place 2. that a corrupt cache can crash squid

These may in fact be the reverse with cause (2) and effect (1). When a
segfault happens the details are not logged at all because the process
doing the logging is the Squid which has died.

Only in an assertion failure or exception error is Squid running well
enough and able to log why before exiting.


> 
> Unfortunately I did the stupid thing and deleted the cache without
> taking a copy for post-mortem... the best I can do is:
> 
> [31072.428922] squid[6317]: segfault at 58 ip 000000000061a6f9 sp
> 00007fff8b9e2d40 error 4 in squid[400000+4e9000] [31654.707792]
> squid[6329]: segfault at 58 ip 000000000061a6f9 sp 00007fff54358fe0
> error 4 in squid[400000+4e9000] [31783.399832] squid[6465]:
> segfault at 58 ip 000000000061a6f9 sp 00007fff82af0aa0 error 4 in
> squid[400000+4e9000] [31984.470507] squid[6509]: segfault at 58 ip
> 000000000061a6f9 sp 00007fff028a6640 error 4 in
> squid[400000+4e9000] [32178.270298] squid[6576]: segfault at 58 ip
> 000000000061a6f9 sp 00007fffe64a07e0 error 4 in
> squid[400000+4e9000] [32789.635935] squid[6626]: segfault at 58 ip
> 000000000061a6f9 sp 00007ffff6932960 error 4 in
> squid[400000+4e9000]
> 
> addr2line -e /usr/local/squid/sbin/squid 000000000061a6f9 
> /usr/local/src/squid-3.5.0.2-20141031-r13657/src/store.cc:962
> 

This looks like http://bugs.squid-cache.org/show_bug.cgi?id=4131. The
two patched posted there seem to be getting good results.

Amos
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (MingW32)

iQEcBAEBAgAGBQJUfEVFAAoJELJo5wb/XPRjgVkH/1gj4utqZanu9jtLFEaBuV9Q
c4K+pw7grL8s1CH8vKvyNZnhSiDi7FsPPaG1RQTkJplhsqswZ2rUcLkVwAEHb2Ug
cS4uH9y5nN/M+O7yqmx/29JS1ITaXnR2ooy8PctKZoYqizEIz6UhDDd2vFuKiPFJ
rlhP+gvd8fDACtZgWLnojl6OmrFXmD0RyxZE0r8Y5wQyzIkbqveJfHzRcl7hkZJh
xJLPfiakK0RBHQSEDRJg/Jui8hv2UeaqGd/YJcF+XJZW6USY6tB8sVnwd8zir6Aw
q2VVbofu2YRn7RJmUrwwppvbmQ+j9ykRS5VMFkJrDVGrf0RohIqn1d7OO3pNJu0=
=DOp+
-----END PGP SIGNATURE-----