[squid-users] pinger crash - Bad opcode: 112

[Amos Jeffries]

On 3/06/2016 3:47 a.m., Tomas Mozes wrote:
> On Wed, Jun 1, 2016 at 1:53 PM, Amos Jeffries <squid3 at treenet.co.nz> wrote:
> 
>> On 31/05/2016 9:56 p.m., Tomas Mozes wrote:
>>> On Thu, May 26, 2016 at 8:04 AM, Amos Jeffries wrote:
>>>
>>>> On 24/05/2016 7:52 p.m., Tomas Mozes wrote:
>>>>> Hello,
>>>>> on two different squid servers I've observed a crash of pinger. First
>> it
>>>>> appeared on version 3.5.15 and later on version 3.5.17.
>>>>>
>>>>> Cache.log contains these lines:
>>>>>
>>>>> (pinger): Address.cc:671: void Ip::Address::getAddrInfo(addrinfo*&,
>> int)
>>>>> const: Assertion `false' failed.
>>>>> 2016/05/14 21:55:25 kid1| Bad opcode: 112 from
>>>>> [6661:6c73:6522:2061:7420:6c69:6e65:2036]
>>>>> 2016/05/14 21:59:13 kid1| recv: (111) Connection refused
>>>>> 2016/05/14 21:59:13 kid1| Closing Pinger socket on FD 17
>>>>>
>>>>> On both servers, that IPv6 address was the same -
>>>>> 6661:6c73:6522:2061:7420:6c69:6e65:2036
>>>>>
>>>>
>>>> That is the hexadecimal representation of the error:
>>>>  false" at line 6
>>>>
>>>> Which means that your kernel is producing garbage when asked to resolve
>>>> an IPv6 address or respond to an ICMPv6 packet.
>>>>
>>>
>>> Cannot we prevent Squid from crashing in these cases?
>>>
>>
>> Squid is not crashing. The pinger is. Squid continues with degraded
>> service latency.
>>
> 
> Yes, sorry, I wasn't specific. I meant like some part of Squid, not the
> Squid caching process itself.
> 
> 
>> What kind of continued operations would you expect a program to do when
>> it discovers at least some portion of its RAM has been filled with
>> garbage by the system kernel?
>>
> 
> 
> In the worst case - wouldn't it be possible for the master process to
> restart it?
> 

For most helpers that is exacty what happens, but with frequent enough
failures killing Squid. That latter bit has been part of the problem
preventing this one being the same so far.

This paticular helper needs root permission to use ICMP sockets. It has
a long history of being installed with wrong ownership and access
permissions, which cause it to exit early. So the most common form of it
dying is not a situation in which Squid can reasonably restart it - or
risks almost certainly dying itself.

Thesituation has been improving in recent years though. So I'm hopeful
we will get there eventually. If you wish to sponsor some work to speed
that up it could be made a configurable action.


> 
>>
>> Now cross your fingers and pray that no other programs on your whole
>> system (network, if you did the same "disable" on other machines) are
>> behaving badly in secret when given the garbage by the kernel like
>> Squid's pinger was.
>>
> 
> You know, the strange thing is I only found those strings on google
> attached with Squid. No other place. If it's a general issue in the Linux
> kernel, then it's been there for years, unnoticed. And as I mentioned
> before, it happened on two machines, separate from each other, in
> completely different data-centers with the same error message.
> 
> What would you suggest?
> 

I suggest enabling IPv6 within your network. :-) It can be firewalled if
you want it not to be used, same as any other digital service.

That will also give you a way to measure whether you have reached
tipping point of it being useful. The global traffic passed 12% last
weekend and growth rate is somewhere between exponential and hyperbolic
despite a lot of networks still clinging to IPv4.

Amos