[squid-users] pinger crash - Bad opcode: 112

[Tomas Mozes]

After enabling IPv6 in the kernel, building squid with IPv6 and firewalling
IPv6 no crash was observed any more.

Thanks for the tip Amos.

On Sat, Jun 11, 2016 at 7:14 AM, Amos Jeffries <squid3 at treenet.co.nz> wrote:

> On 3/06/2016 3:47 a.m., Tomas Mozes wrote:
> > On Wed, Jun 1, 2016 at 1:53 PM, Amos Jeffries <squid3 at treenet.co.nz>
> wrote:
> >
> >> On 31/05/2016 9:56 p.m., Tomas Mozes wrote:
> >>> On Thu, May 26, 2016 at 8:04 AM, Amos Jeffries wrote:
> >>>
> >>>> On 24/05/2016 7:52 p.m., Tomas Mozes wrote:
> >>>>> Hello,
> >>>>> on two different squid servers I've observed a crash of pinger. First
> >> it
> >>>>> appeared on version 3.5.15 and later on version 3.5.17.
> >>>>>
> >>>>> Cache.log contains these lines:
> >>>>>
> >>>>> (pinger): Address.cc:671: void Ip::Address::getAddrInfo(addrinfo*&,
> >> int)
> >>>>> const: Assertion `false' failed.
> >>>>> 2016/05/14 21:55:25 kid1| Bad opcode: 112 from
> >>>>> [6661:6c73:6522:2061:7420:6c69:6e65:2036]
> >>>>> 2016/05/14 21:59:13 kid1| recv: (111) Connection refused
> >>>>> 2016/05/14 21:59:13 kid1| Closing Pinger socket on FD 17
> >>>>>
> >>>>> On both servers, that IPv6 address was the same -
> >>>>> 6661:6c73:6522:2061:7420:6c69:6e65:2036
> >>>>>
> >>>>
> >>>> That is the hexadecimal representation of the error:
> >>>>  false" at line 6
> >>>>
> >>>> Which means that your kernel is producing garbage when asked to
> resolve
> >>>> an IPv6 address or respond to an ICMPv6 packet.
> >>>>
> >>>
> >>> Cannot we prevent Squid from crashing in these cases?
> >>>
> >>
> >> Squid is not crashing. The pinger is. Squid continues with degraded
> >> service latency.
> >>
> >
> > Yes, sorry, I wasn't specific. I meant like some part of Squid, not the
> > Squid caching process itself.
> >
> >
> >> What kind of continued operations would you expect a program to do when
> >> it discovers at least some portion of its RAM has been filled with
> >> garbage by the system kernel?
> >>
> >
> >
> > In the worst case - wouldn't it be possible for the master process to
> > restart it?
> >
>
> For most helpers that is exacty what happens, but with frequent enough
> failures killing Squid. That latter bit has been part of the problem
> preventing this one being the same so far.
>
> This paticular helper needs root permission to use ICMP sockets. It has
> a long history of being installed with wrong ownership and access
> permissions, which cause it to exit early. So the most common form of it
> dying is not a situation in which Squid can reasonably restart it - or
> risks almost certainly dying itself.
>
> Thesituation has been improving in recent years though. So I'm hopeful
> we will get there eventually. If you wish to sponsor some work to speed
> that up it could be made a configurable action.
>
>
> >
> >>
> >> Now cross your fingers and pray that no other programs on your whole
> >> system (network, if you did the same "disable" on other machines) are
> >> behaving badly in secret when given the garbage by the kernel like
> >> Squid's pinger was.
> >>
> >
> > You know, the strange thing is I only found those strings on google
> > attached with Squid. No other place. If it's a general issue in the Linux
> > kernel, then it's been there for years, unnoticed. And as I mentioned
> > before, it happened on two machines, separate from each other, in
> > completely different data-centers with the same error message.
> >
> > What would you suggest?
> >
>
> I suggest enabling IPv6 within your network. :-) It can be firewalled if
> you want it not to be used, same as any other digital service.
>
> That will also give you a way to measure whether you have reached
> tipping point of it being useful. The global traffic passed 12% last
> weekend and growth rate is somewhere between exponential and hyperbolic
> despite a lot of networks still clinging to IPv4.
>
> Amos
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20160930/50fcca44/attachment-0001.html>