<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:"Courier New";}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link="#0563C1" vlink="#954F72"><div class=WordSection1><p class=MsoNormal>Hello,<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I am having period crashes of my squid server, and I am not getting core dump files. I have set “workers 6” in my squid.conf, and I know that threads can cause trouble from reading the debugging wiki page. I have confirmed permissions on the directory I’m dumping to, so I don’t *<b>think</b>* that is the issue.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>FWIW (core dump to follow, I’ll retry without workers and see what happens) I am having squid crashes. Details I have so far as are as follows:<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I am running Squid-3.5.23-R14129 on a stock Ubuntu 16.04 configured with:<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>./configure --prefix=/usr --localstatedir=/var --libexecdir=/usr/lib/squid --srcdir=. --datadir=/usr/share/squid --sysconfdir=/etc/squid --with-default-user=proxy --with-logdir=/var/log --with-pidfile=/var/run/squid.pid --enable-linux-netfilter --enable-cache-digests --enable-storeio=ufs,aufs,diskd,rock --enable-async-io=30 --enable-http-violations --enable-zph-qos --with-netfilter-conntrack --with-filedescriptors=65536 --with-large-files<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>About once a day it is crashing with the following line as about my only lead in the cache.log:<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>assertion failed: MemBuf.cc:216: "0 <= tailSize && tailSize <= cSize"<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>From the possibly interesting-but-who-knows-maybe-unrelated-files, there is one additional detail. I had this version running on a Ubuntu 14.04 machine until last week, which I had installed GCC-4.9 on (so I could test squid 4.0), and that had ran stable from December 20th to January 5<sup>th</sup> without a any crashes. Then something totally outside of squid went horribly off the rails. Ubuntu dropped support for the 3.x series kernels, so I updated to 4.4 (from the Ubuntu repositories) and that caused /proc/sys/net/bridge to go away. While testing an unrelated issue, I ran a script that I adapted from <a href="http://wiki.squid-cache.org/ConfigExamples/Intercept/LinuxBridge">http://wiki.squid-cache.org/ConfigExamples/Intercept/LinuxBridge</a> which contains a dangerous couple lines I had not before contemplated:<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>cd /proc/sys/net/bridge/<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'> for i in *<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'> do<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'> echo 0 > $i<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'> done<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'> unset i<o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>When /proc/sys/net/bridge went away, the change directory failed, then the script proceeded to turn everything in that directory into 0’s. OOPS! I tell this bit so that my fellow admins get a laugh at my expense, and as a cautionary tale. CHECK the status of that command before you let it do other things! As it turns out, tproxy works fine without echoing ‘0’ at all those files, but if you want to leave it on the page, may I suggest the following revision to the wiki page:<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>#!/bin/bash<o:p></o:p></p><p class=MsoNormal>cd /proc/sys/net/bridge<o:p></o:p></p><p class=MsoNormal>if [ $? -eq 0 ]<o:p></o:p></p><p class=MsoNormal>then<o:p></o:p></p><p class=MsoNormal> for i in *<o:p></o:p></p><p class=MsoNormal> do<o:p></o:p></p><p class=MsoNormal> echo 0 > $i<o:p></o:p></p><p class=MsoNormal> done<o:p></o:p></p><p class=MsoNormal> unset i<o:p></o:p></p><p class=MsoNormal>else<o:p></o:p></p><p class=MsoNormal> echo "WARNING! /proc/sys/net/bridge does not exist, you can 'sudo modprobe br_netfilter' to get it, but you may not need it"<o:p></o:p></p><p class=MsoNormal>fi<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>That just checks whether the changedir worked, and if it didn’t it issues a warning instead of cooking all the files in your current directory, which is nice!<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Anyway, after that happened, for reasons completely unknown, but I suspect related to bridging, the machine that had been my squid server completely seized, so I installed Ubuntu 16, and have since run into this crash. Once I have a core dump, I’ll post it and my configuration, which is pretty stock with the exception of a couple ACLs and “workers 6”. I am wondering if the move from gcc 4.9 to gcc 5.4 (the stock gcc in Ubuntu 16) may be a culprit, might recompile with the downgrade and see if the issue resolves. <o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>In any event, is there a way to get a core with worker threads? My system benefits from them, so I’d rather not turn them off but I want a core dump. Also let me know if there are other details that would be useful. Adding much in the way of debugging is going to be a challenge because it takes a day or so to get to a crash, and I don’t necessarily have the disk space to hold the volume of cache log generated over that period of time (I have a 60-gb log partition). If there is some clever way of limiting cache.log and causing it to round-robin or something, I’m happy to try things. Thank you!<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Jester Purtteman<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p></div></body></html>