[squid-users] squid hangs and dies and can not be killed - needs system reboot

[Amish]

Hi Amos,

On 19/12/23 20:25, Amos Jeffries wrote:
>
> On 19/12/23 16:29, Amish wrote:
>> Hi Alex,
>>
>> Thank you for replying.
>>
>> On 19/12/23 01:14, Alex Rousskov wrote:
>>> On 2023-12-18 09:35, Amish wrote:
>>>
>>>> I use Arch Linux and today I updated squid from squid 5.7 to squid 
>>>> 6.6.
>>>
>>> > Dec 18 13:01:24 mumbai squid[604]: kick abandoning conn199
>>>
>>> I do not know whether the above problem is the primary problem in 
>>> your setup, but it is a red flag. Transactions on the same 
>>> connection may get stuck after that message; it is essentially a 
>>> Squid bug.
>>>
>>> I am not sure at all, but this bug might be related to Bug 5187 
>>> workaround that went into Squid v6.2 (commit c44cfe7): 
>>> https://bugs.squid-cache.org/show_bug.cgi?id=5187
>>>
>>> Does Squid accept new TCP connections after it enters what you 
>>> describe as a dead state? For example, does "telnet 127.0.0.1 8080" 
>>> establishes a connection if executed on the same machine as Squid?
>>>
>> Yes it establishes connection. But I do not know what to do next. 
>> Browser showed "Connection timed out" message. But I believe 
>> browser's also connected but nothing happened afterwards.
>>
>
> Ah ... that port is an interception port. It should *not* connect.
>
> Please ensure your firewall contains the "-t mangle" rules for each 
> interception port you use. As shown at 
> <https://wiki.squid-cache.org/ConfigExamples/Intercept/LinuxDnat>
>
No, port 8080 is not an interception port. And firewall is fine. 
Everything worked before upgrade to 6.6.

>>> > kill -9 does nothing
>>>
>>> Is it possible that you are trying to kill the wrong process? You 
>>> should be killing this process AFAICT:
>>>
>>> > root         601  0.0  0.2  73816 22528 ?        Ss 12:59 0:02
>>> > /usr/bin/squid -f /etc/squid/btnet/squid.btnet.conf --foreground -sYC
>>
>> I did not clarify but all processes needed SIGKILL and vanished 
>> except the Dead squid process which still remained.
>>
>> # systemctl stop squid
>>
>> Dec 19 08:46:38 mumbai systemd[1]: squid.service: State 
>> 'stop-sigterm' timed out. Killing.
>
> FWIW, Squid default shutdown grace period for clients to disconnect is 
> longer that systemd typically is willing to wait for a service shutdown.
>
> Please set "shutdown_lifetime 10 seconds" in your squid.conf.
>
Default shutdown_lifetime is 30 seconds.

I think systemd waits 90 seconds.

>
>> Dec 19 08:46:38 mumbai systemd[1]: squid.service: Killing process 601 
>> (squid) with signal SIGKILL.
>> Dec 19 08:46:38 mumbai systemd[1]: squid.service: Killing process 604 
>> (squid) with signal SIGKILL.
>
> This is systemd running the command " kill -9 604 ".
>
> Per the Squid code: "XXX: In SMP mode, uncatchable SIGKILL only kills 
> the master process".
>
> You can try SIGTERM instead, and repeat up to 3 times if the first 
> does not close the process.

kill -9 is supposed to kill the process immediately. But it didnt. 601 
(master process) got killed but 604 did not.

Even when powering off, systemd "finished" shutdown target, unmounted 
partitions and then killed all squid processes but could not kill 
process 604.

After waiting for 2-3 minutes, I forced power off the system and restarted.

Regards,

Amish

>
> HTH
> Amos