<div dir="ltr">Hi Amish,<div> the message you posted really looks like a kernel bug, possibly due to faulty code, or resulting from a hardware problem.</div><div>Nothing squid can do can cause that kind of stack traces in kernel-space.</div><div><br></div><div>A quick search online results in <a href="https://lkml.kernel.org/netdev/20230526163458.2880232-1-edumazet@google.com/T/">https://lkml.kernel.org/netdev/20230526163458.2880232-1-edumazet@google.com/T/</a> , which looks very much like your message</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Dec 21, 2023 at 1:10 PM Amish <<a href="mailto:anon.amish@gmail.com">anon.amish@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi Alex,<br>
<br>
Sorry for late reply. But this is a production system and hence <br>
debugging is tough. May take few days.<br>
<br>
However I noticed that everytime squid hangs (goes dead), the dmesg has <br>
these errors.<br>
<br>
[63567.802188] divide error: 0000 [#1] PREEMPT SMP PTI<br>
[63567.802215] CPU: 3 PID: 617 Comm: squid Not tainted 6.6.7-arch1-1 #1 <br>
4505c4baa0b3d7c4037b0e8f5402626fa360717f<br>
[63567.802238] Hardware name: Gigabyte Technology Co., Ltd. <br>
H81M-S/H81M-S, BIOS F2 08/19/2015<br>
[63567.802255] RIP: 0010:tcp_rcv_space_adjust+0xbe/0x160<br>
[63567.802267] Code: f8 41 89 d0 29 d0 31 d2 49 0f af c2 49 f7 f0 45 8b <br>
81 bc 04 00 00 44 0f b6 8b 10 06 00 00 31 d2 49 8d 04 42 48 98 48 c1 e0 <br>
08 <49> f7 f1 49 63 d0<br>
48 98 48 39 d0 48 0f 47 c2 39 83 18 01 00 00 7c<br>
[63567.802287] RSP: 0018:ffffba9f00da3c18 EFLAGS: 00010206<br>
[63567.802296] RAX: 0000000004fb2800 RBX: ffff9e124acc1c80 RCX: <br>
0000000eccd9dace<br>
[63567.802304] RDX: 0000000000000000 RSI: 0000000037fa4ae8 RDI: <br>
0000000000008349<br>
[63567.802314] RBP: ffff9e124acc1c80 R08: 0000000000600000 R09: <br>
0000000000000000<br>
[63567.802322] R10: 00000000000161d2 R11: ffff9e1346621700 R12: <br>
0000000000000000<br>
[63567.802331] R13: ffff9e124acc1d58 R14: 0000000000000000 R15: <br>
ffff9e124acc2214<br>
[63567.802340] FS: 00007fc3d627c840(0000) GS:ffff9e1457380000(0000) <br>
knlGS:0000000000000000<br>
[63567.802350] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033<br>
[63567.802358] CR2: 0000560b7e86d690 CR3: 0000000102f44001 CR4: <br>
00000000001706e0<br>
[63567.802367] Call Trace:<br>
[63567.802372] <TASK><br>
[63567.802378] ? die+0x36/0x90<br>
[63567.802386] ? do_trap+0xda/0x100<br>
[63567.802392] ? tcp_rcv_space_adjust+0xbe/0x160<br>
[63567.802401] ? do_error_trap+0x6a/0x90<br>
[63567.802408] ? tcp_rcv_space_adjust+0xbe/0x160<br>
[63567.802415] ? exc_divide_error+0x38/0x50<br>
[63567.802423] ? tcp_rcv_space_adjust+0xbe/0x160<br>
[63567.802431] ? asm_exc_divide_error+0x1a/0x20<br>
[63567.802441] ? tcp_rcv_space_adjust+0xbe/0x160<br>
[63567.802449] ? tcp_rcv_space_adjust+0x1a/0x160<br>
[63567.802456] tcp_recvmsg_locked+0x2d4/0x960<br>
[63567.802464] tcp_recvmsg+0x87/0x1f0<br>
[63567.802471] inet6_recvmsg+0x56/0x130<br>
[63567.802479] ? __pfx_bpf_lsm_socket_recvmsg+0x10/0x10<br>
[63567.802488] ? security_socket_recvmsg+0x44/0x70<br>
[63567.802497] sock_recvmsg+0x57/0xd0<br>
[63567.802505] sock_read_iter+0x96/0x100<br>
[63567.802513] vfs_read+0x303/0x350<br>
[63567.802796] ksys_read+0xbb/0xf0<br>
[63567.803074] do_syscall_64+0x60/0x90<br>
[63567.803346] ? syscall_exit_to_user_mode+0x2b/0x40<br>
[63567.803619] ? do_syscall_64+0x6c/0x90<br>
[63567.803890] ? syscall_exit_to_user_mode+0x2b/0x40<br>
[63567.804163] ? do_syscall_64+0x6c/0x90<br>
[63567.804438] ? do_syscall_64+0x6c/0x90<br>
[63567.804710] entry_SYSCALL_64_after_hwframe+0x6e/0xd8<br>
[63567.804984] RIP: 0033:0x7fc3d5f21531<br>
[63567.805271] Code: ff ff eb c3 67 e8 2f c9 01 00 66 2e 0f 1f 84 00 00 <br>
00 00 00 0f 1f 44 00 00 f3 0f 1e fa 80 3d 35 ce 0d 00 00 74 13 31 c0 0f <br>
05 <48> 3d 00 f0 ff ff<br>
77 57 c3 66 0f 1f 44 00 00 48 83 ec 28 48 89 54<br>
[63567.805909] RSP: 002b:00007fffa66f6748 EFLAGS: 00000246 ORIG_RAX: <br>
0000000000000000<br>
[63567.806225] RAX: ffffffffffffffda RBX: 00007fffa66f6830 RCX: <br>
00007fc3d5f21531<br>
[63567.806542] RDX: 000000000000191a RSI: 000055d106003d80 RDI: <br>
00000000000000bd<br>
[63567.806855] RBP: 000055d10620bc30 R08: 000055d10620bc30 R09: <br>
000055d1036a8e78<br>
[63567.807163] R10: 000055d1037ad630 R11: 0000000000000246 R12: <br>
000000000000191a<br>
[63567.807472] R13: 000055d10664f410 R14: 000055d106003d80 R15: <br>
00007fc3d627c6b8<br>
[63567.807768] </TASK><br>
[63567.808037] Modules linked in: nfnetlink_queue nf_conntrack_netlink <br>
xt_connmark xt_mark ip_set_hash_net xt_NFQUEUE xt_set ip_set_hash_ip <br>
ip_set cls_fw sch_sfq sch_h<br>
tb nf_nat_ftp nf_conntrack_ftp tls tun cfg80211 rfkill nft_chain_nat <br>
xt_MASQUERADE xt_nat xt_REDIRECT nf_nat nft_limit xt_limit xt_conntrack <br>
nf_conntrack nf_defrag_ipv<br>
6 nf_defrag_ipv4 xt_multiport xt_tcpudp nft_compat nf_tables libcrc32c <br>
intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp <br>
snd_hda_codec_realtek cor<br>
etemp snd_hda_codec_generic ledtrig_audio snd_hda_codec_hdmi kvm_intel <br>
snd_hda_intel kvm snd_intel_dspcfg snd_intel_sdw_acpi irqbypass <br>
snd_hda_codec snd_hda_core crct1<br>
0dif_pclmul r8169 crc32_pclmul snd_hwdep polyval_clmulni snd_pcm <br>
polyval_generic gf128mul snd_timer realtek ax88179_178a spi_nor <br>
mdio_devres usbnet snd libphy ghash_cl<br>
mulni_intel mtd sha512_ssse3 at24 soundcore mii mei_pxp mei_hdcp <br>
sha256_ssse3 sha1_ssse3 iTCO_wdt i2c_i801 mei_me spi_intel_platform <br>
mousedev aesni_intel intel_pmc_bxt<br>
spi_intel iTCO_vendor_support<br>
[63567.808076] i2c_smbus mei crypto_simd lpc_ich cryptd rapl <br>
intel_cstate intel_uncore psmouse mac_hid pcspkr pkcs8_key_parser fuse <br>
dm_mod loop nfnetlink ip_tables x_<br>
tables ext4 crc32c_generic crc16 mbcache jbd2 i915 serio_raw <br>
i2c_algo_bit atkbd drm_buddy libps2 ttm vivaldi_fmap intel_gtt xhci_pci <br>
crc32c_intel drm_display_helper xh<br>
ci_pci_renesas cec i8042 video serio wmi<br>
[63567.811333] ---[ end trace 0000000000000000 ]---<br>
[63567.811701] RIP: 0010:tcp_rcv_space_adjust+0xbe/0x160<br>
[63567.812069] Code: f8 41 89 d0 29 d0 31 d2 49 0f af c2 49 f7 f0 45 8b <br>
81 bc 04 00 00 44 0f b6 8b 10 06 00 00 31 d2 49 8d 04 42 48 98 48 c1 e0 <br>
08 <49> f7 f1 49 63 d0<br>
48 98 48 39 d0 48 0f 47 c2 39 83 18 01 00 00 7c<br>
[63567.812848] RSP: 0018:ffffba9f00da3c18 EFLAGS: 00010206<br>
[63567.813252] RAX: 0000000004fb2800 RBX: ffff9e124acc1c80 RCX: <br>
0000000eccd9dace<br>
[63567.813651] RDX: 0000000000000000 RSI: 0000000037fa4ae8 RDI: <br>
0000000000008349<br>
[63567.814064] RBP: ffff9e124acc1c80 R08: 0000000000600000 R09: <br>
0000000000000000<br>
[63567.814470] R10: 00000000000161d2 R11: ffff9e1346621700 R12: <br>
0000000000000000<br>
[63567.814879] R13: ffff9e124acc1d58 R14: 0000000000000000 R15: <br>
ffff9e124acc2214<br>
[63567.815290] FS: 00007fc3d627c840(0000) GS:ffff9e1457380000(0000) <br>
knlGS:0000000000000000<br>
[63567.815700] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033<br>
[63567.816121] CR2: 0000560b7e86d690 CR3: 0000000102f44001 CR4: <br>
00000000001706e0<br>
<br>
After this the "child" squid process can not be killed, even by systemd, <br>
even by kill -9.<br>
<br>
It appears that squid "main" process (owned by root user) does not hang <br>
and hence connections on port 8080 are accepted.<br>
<br>
But squid child process (owned by proxy user) hangs and never recovers <br>
after above dmesg. And hence nothing happens and connection times out.<br>
<br>
# ps aux |grep squid<br>
root 612 0.0 0.2 75500 23680 ? Ss Dec20 0:02 <br>
/usr/bin/squid -f /etc/squid/btnet/squid.btnet.conf --foreground -sYC<br>
proxy 617 0.0 0.0 0 0 ? D Dec20 1:00 [squid]<br>
proxy 620 0.0 0.0 12152 7552 ? S Dec20 0:00 <br>
(security_file_certgen) -s /var/cache/squid/ssl_db -M 4MB<br>
proxy 621 0.0 0.0 12152 7552 ? S Dec20 0:00 <br>
(security_file_certgen) -s /var/cache/squid/ssl_db -M 4MB<br>
proxy 622 0.0 0.0 11888 5504 ? S Dec20 0:00 <br>
(security_file_certgen) -s /var/cache/squid/ssl_db -M 4MB<br>
proxy 623 0.0 0.0 11888 5632 ? S Dec20 0:00 <br>
(security_file_certgen) -s /var/cache/squid/ssl_db -M 4MB<br>
proxy 624 0.0 0.0 11888 5632 ? S Dec20 0:00 <br>
(security_file_certgen) -s /var/cache/squid/ssl_db -M 4MB<br>
proxy 643 0.0 0.0 6116 3200 ? S Dec20 0:01 <br>
(logfile-daemon) /var/log/squid/access.log<br>
<br>
# telnet 127.0.0.1 8080<br>
Trying 127.0.0.1...<br>
Connected to 127.0.0.1.<br>
Escape character is '^]'.<br>
^]<br>
telnet> c<br>
<br>
# curl -x <a href="http://127.0.0.1:8080" rel="noreferrer" target="_blank">127.0.0.1:8080</a> --max-time 30 <a href="https://google.com" rel="noreferrer" target="_blank">https://google.com</a><br>
curl: (28) Connection timed out after 30002 milliseconds<br>
<br>
# kill 612 617<br>
<br>
Ran kill 5 times, nothing happened. All squid processes remained.<br>
<br>
# kill -9 612 617<br>
<br>
Ran once, and all squid processes except 617 (child squid process) got <br>
killed<br>
<br>
# ps aux |grep squid<br>
proxy 617 0.0 0.0 0 0 ? D Dec20 1:00 [squid]<br>
root 90764 0.0 0.0 6552 2560 pts/5 S+ 17:36 0:00 grep <br>
--color=auto squid<br>
<br>
Can above information be of any help?<br>
<br>
Thanks and regards,<br>
<br>
Amish<br>
<br>
On 19/12/23 20:46, Alex Rousskov wrote:<br>
> On 2023-12-18 22:29, Amish wrote:<br>
>> On 19/12/23 01:14, Alex Rousskov wrote:<br>
>>> On 2023-12-18 09:35, Amish wrote:<br>
>>><br>
>>>> I use Arch Linux and today I updated squid from squid 5.7 to squid <br>
>>>> 6.6.<br>
>>><br>
>>> > Dec 18 13:01:24 mumbai squid[604]: kick abandoning conn199<br>
>>><br>
>>> I do not know whether the above problem is the primary problem in <br>
>>> your setup, but it is a red flag. Transactions on the same <br>
>>> connection may get stuck after that message; it is essentially a <br>
>>> Squid bug.<br>
>>><br>
>>> I am not sure at all, but this bug might be related to Bug 5187 <br>
>>> workaround that went into Squid v6.2 (commit c44cfe7): <br>
>>> <a href="https://bugs.squid-cache.org/show_bug.cgi?id=5187" rel="noreferrer" target="_blank">https://bugs.squid-cache.org/show_bug.cgi?id=5187</a><br>
>>><br>
>>> Does Squid accept new TCP connections after it enters what you <br>
>>> describe as a dead state? For example, does "telnet 127.0.0.1 8080" <br>
>>> establishes a connection if executed on the same machine as Squid?<br>
>>><br>
>> Yes it establishes connection. But I do not know what to do next. <br>
><br>
> This tells us that your Squid is still listening for incoming <br>
> connections. Most likely, it is not "dead" but running and just unable <br>
> to make progress with those connections (for yet unknown reasons). <br>
> That information is helpful but not sufficient (for me) to solve the <br>
> problem you are describing.<br>
><br>
> The next step that I would recommend is to collect debugging <br>
> information from the running process and share a pointer to the <br>
> corresponding compressed cache.log file:<br>
><br>
> * Ideally, start collection when Squid starts and reproduce the <br>
> problem while collecting full debugging information:<br>
> <a href="http://wiki.squid-cache.org/SquidFaq/BugReporting#full-debug-output" rel="noreferrer" target="_blank">http://wiki.squid-cache.org/SquidFaq/BugReporting#full-debug-output</a><br>
><br>
> * If you have to, start collection after Squid is already in bad state <br>
> and just before you use telnet or browser to tickle Squid:<br>
> <a href="http://wiki.squid-cache.org/SquidFaq/BugReporting#debugging-a-single-transaction" rel="noreferrer" target="_blank">http://wiki.squid-cache.org/SquidFaq/BugReporting#debugging-a-single-transaction</a> <br>
><br>
><br>
> Do not use any secret information (e.g., production certificate keys) <br>
> for these tests (unless you are going to share the logs privately with <br>
> those you trust).<br>
><br>
> Do not downgrade to v5 for these tests.<br>
><br>
><br>
> HTH,<br>
><br>
> Alex.<br>
><br>
><br>
>> Browser showed "Connection timed out" message. But I believe <br>
>> browser's also connected but nothing happened afterwards.<br>
>><br>
>>><br>
>>> > kill -9 does nothing<br>
>>><br>
>>> Is it possible that you are trying to kill the wrong process? You <br>
>>> should be killing this process AFAICT:<br>
>>><br>
>>> > root 601 0.0 0.2 73816 22528 ? Ss 12:59 0:02<br>
>>> > /usr/bin/squid -f /etc/squid/btnet/squid.btnet.conf --foreground -sYC<br>
>><br>
>> I did not clarify but all processes needed SIGKILL and vanished <br>
>> except the Dead squid process which still remained.<br>
>><br>
>> # systemctl stop squid<br>
>><br>
>> Dec 19 08:46:38 mumbai systemd[1]: squid.service: State <br>
>> 'stop-sigterm' timed out. Killing.<br>
>> Dec 19 08:46:38 mumbai systemd[1]: squid.service: Killing process 601 <br>
>> (squid) with signal SIGKILL.<br>
>> Dec 19 08:46:38 mumbai systemd[1]: squid.service: Killing process 604 <br>
>> (squid) with signal SIGKILL.<br>
>> Dec 19 08:46:38 mumbai systemd[1]: squid.service: Killing process 607 <br>
>> (security_file_c) with signal SIGKILL.<br>
>> Dec 19 08:46:38 mumbai systemd[1]: squid.service: Killing process 608 <br>
>> (security_file_c) with signal SIGKILL.<br>
>> Dec 19 08:46:38 mumbai systemd[1]: squid.service: Killing process 609 <br>
>> (security_file_c) with signal SIGKILL.<br>
>> Dec 19 08:46:38 mumbai systemd[1]: squid.service: Killing process 610 <br>
>> (security_file_c) with signal SIGKILL.<br>
>> Dec 19 08:46:38 mumbai systemd[1]: squid.service: Killing process 611 <br>
>> (security_file_c) with signal SIGKILL.<br>
>> Dec 19 08:46:38 mumbai systemd[1]: squid.service: Killing process 622 <br>
>> (log_file_daemon) with signal SIGKILL.<br>
>> Dec 19 08:46:38 mumbai systemd[1]: squid.service: Main process <br>
>> exited, code=killed, status=9/KILL<br>
>> Dec 19 08:46:38 mumbai systemd[1]: squid.service: Killing process 604 <br>
>> (squid) with signal SIGKILL.<br>
>><br>
>> Waited for 2 minutes for squid to stop then pressed ctrl-c to <br>
>> systemctl stop squid command.<br>
>><br>
>> As you can see in last line shows that attempt was made to kill DEAD <br>
>> process with PID 604.<br>
>><br>
>> # ps aux |grep squid<br>
>> proxy 604 0.0 0.0 0 0 ? D Dec18 0:03 [squid]<br>
>><br>
>> Now only DEAD squid process remains.<br>
>><br>
>> What next? Should I downgrade to 5.9 and check?<br>
>><br>
>> Regards<br>
>><br>
>> Amish<br>
>><br>
>>> Alex.<br>
>>><br>
>>><br>
>>>> After the update from 5.7 to 6.6, squid starts but then reaches <br>
>>>> Dead state in a minute or two.<br>
>>>><br>
>>>> # ps aux | grep squid<br>
>>>> root 601 0.0 0.2 73816 22528 ? Ss 12:59 0:02 <br>
>>>> /usr/bin/squid -f /etc/squid/btnet/squid.btnet.conf --foreground -sYC<br>
>>>> proxy 604 0.0 0.0 0 0 ? D 12:59 0:03 <br>
>>>> [squid]<br>
>>>> proxy 607 0.0 0.0 11976 7424 ? S 12:59 0:00 <br>
>>>> (security_file_certgen) -s /var/cache/squid/ssl_db -M 4MB<br>
>>>> proxy 608 0.0 0.0 11976 7168 ? S 12:59 0:00 <br>
>>>> (security_file_certgen) -s /var/cache/squid/ssl_db -M 4MB<br>
>>>> proxy 609 0.0 0.0 11712 5632 ? S 12:59 0:00 <br>
>>>> (security_file_certgen) -s /var/cache/squid/ssl_db -M 4MB<br>
>>>> proxy 610 0.0 0.0 11712 5376 ? S 12:59 0:00 <br>
>>>> (security_file_certgen) -s /var/cache/squid/ssl_db -M 4MB<br>
>>>> proxy 611 0.0 0.0 11712 5504 ? S 12:59 0:00 <br>
>>>> (security_file_certgen) -s /var/cache/squid/ssl_db -M 4MB<br>
>>>> proxy 622 0.0 0.0 6116 3200 ? S 12:59 0:00 <br>
>>>> (logfile-daemon) /var/log/squid/access.log<br>
>>>><br>
>>>> And then all requests get stuck. Notice the D (dead) state of squid.<br>
>>>><br>
>>>> I use multiple ports for multiple purposes. (It all worked fine in <br>
>>>> squid 5.7)<br>
>>>><br>
>>>> Dec 18 12:59:10 mumbai squid[601]: Starting Authentication on port <br>
>>>> [::]:3128<br>
>>>> Dec 18 12:59:10 mumbai squid[601]: Disabling Authentication on port <br>
>>>> [::]:3128 (interception enabled)<br>
>>>> Dec 18 12:59:10 mumbai squid[601]: Starting Authentication on port <br>
>>>> [::]:8081<br>
>>>> Dec 18 12:59:10 mumbai squid[601]: Disabling Authentication on port <br>
>>>> [::]:8081 (interception enabled)<br>
>>>> Dec 18 12:59:12 mumbai squid[601]: Starting Authentication on port <br>
>>>> [::]:8082<br>
>>>> Dec 18 12:59:12 mumbai squid[601]: Disabling Authentication on port <br>
>>>> [::]:8082 (interception enabled)<br>
>>>> Dec 18 12:59:12 mumbai squid[601]: Starting Authentication on port <br>
>>>> [::]:8083<br>
>>>> Dec 18 12:59:12 mumbai squid[601]: Disabling Authentication on port <br>
>>>> [::]:8083 (interception enabled)<br>
>>>> Dec 18 12:59:13 mumbai squid[601]: Starting Authentication on port <br>
>>>> [::]:8084<br>
>>>> Dec 18 12:59:13 mumbai squid[601]: Disabling Authentication on port <br>
>>>> [::]:8084 (interception enabled)<br>
>>>> Dec 18 12:59:13 mumbai squid[601]: Starting Authentication on port <br>
>>>> [::]:3136<br>
>>>> Dec 18 12:59:13 mumbai squid[601]: Disabling Authentication on port <br>
>>>> [::]:3136 (interception enabled)<br>
>>>> Dec 18 12:59:13 mumbai squid[601]: Starting Authentication on port <br>
>>>> [::]:3137<br>
>>>> Dec 18 12:59:13 mumbai squid[601]: Disabling Authentication on port <br>
>>>> [::]:3137 (interception enabled)<br>
>>>> ...<br>
>>>> Dec 18 12:59:29 mumbai squid[604]: Adaptation support is on<br>
>>>> Dec 18 12:59:29 mumbai squid[604]: Accepting NAT intercepted HTTP <br>
>>>> Socket connections at conn19 local=[::]:3128 remote=[::] FD 27 <br>
>>>> flags=41<br>
>>>> listening port: 3128<br>
>>>> Dec 18 12:59:29 mumbai squid[604]: Accepting SSL bumped HTTP Socket <br>
>>>> connections at conn21 local=[::]:8080 remote=[::] FD 28 flags=9<br>
>>>> listening port: 8080<br>
>>>> Dec 18 12:59:29 mumbai squid[604]: Accepting NAT intercepted SSL <br>
>>>> bumped HTTPS Socket connections at conn23 local=[::]:8081 <br>
>>>> remote=[::] FD 29 flags=41<br>
>>>> listening port: 8081<br>
>>>> Dec 18 12:59:29 mumbai squid[604]: Accepting SSL bumped HTTP Socket <br>
>>>> connections at conn25 local=[::]:8092 remote=[::] FD 30 flags=9<br>
>>>> listening port: 8092<br>
>>>> Dec 18 12:59:29 mumbai systemd[1]: Started Squid Web Proxy Server.<br>
>>>> Dec 18 12:59:29 mumbai squid[604]: Accepting SSL bumped HTTP Socket <br>
>>>> connections at conn27 local=[::]:8093 remote=[::] FD 31 flags=9<br>
>>>> listening port: 8093<br>
>>>> Dec 18 12:59:29 mumbai squid[604]: Accepting SSL bumped HTTP Socket <br>
>>>> connections at conn29 local=[::]:8094 remote=[::] FD 32 flags=9<br>
>>>> listening port: 8094<br>
>>>> Dec 18 12:59:29 mumbai squid[604]: Accepting NAT intercepted SSL <br>
>>>> bumped HTTPS Socket connections at conn31 local=[::]:8082 <br>
>>>> remote=[::] FD 33 flags=41<br>
>>>> listening port: 8082<br>
>>>> Dec 18 12:59:29 mumbai squid[604]: Accepting NAT intercepted SSL <br>
>>>> bumped HTTPS Socket connections at conn33 local=[::]:8083 <br>
>>>> remote=[::] FD 34 flags=41<br>
>>>> listening port: 8083<br>
>>>> Dec 18 12:59:29 mumbai squid[604]: Accepting NAT intercepted SSL <br>
>>>> bumped HTTPS Socket connections at conn35 local=[::]:8084 <br>
>>>> remote=[::] FD 35 flags=41<br>
>>>> listening port: 8084<br>
>>>> Dec 18 12:59:29 mumbai squid[604]: Accepting NAT intercepted HTTP <br>
>>>> Socket connections at conn37 local=[::]:3136 remote=[::] FD 36 <br>
>>>> flags=41<br>
>>>> listening port: 3136<br>
>>>> Dec 18 12:59:29 mumbai squid[604]: Accepting NAT intercepted HTTP <br>
>>>> Socket connections at conn39 local=[::]:3137 remote=[::] FD 37 <br>
>>>> flags=41<br>
>>>> listening port: 3137<br>
>>>><br>
>>>> And then following errors came:<br>
>>>><br>
>>>><br>
>>>> Dec 18 12:59:45 mumbai squid[604]: ERROR: failure while accepting a <br>
>>>> TLS connection on conn41 local=<a href="http://192.168.0.1:8080" rel="noreferrer" target="_blank">192.168.0.1:8080</a> <br>
>>>> remote=<a href="http://192.168.0.111:53867" rel="noreferrer" target="_blank">192.168.0.111:53867</a> FD 12 flags=1: SQUID_TLS<br>
>>>> _ERR_ACCEPT+TLS_LIB_ERR=A000416+TLS_IO_ERR=1<br>
>>>> current master transaction: <br>
>>>> master53<br>
>>>> Dec 18 12:59:45 mumbai squid[604]: ERROR: failure while accepting a <br>
>>>> TLS connection on conn42 local=<a href="http://192.168.0.1:8080" rel="noreferrer" target="_blank">192.168.0.1:8080</a> <br>
>>>> remote=<a href="http://192.168.0.111:53868" rel="noreferrer" target="_blank">192.168.0.111:53868</a> FD 14 flags=1: SQUID_TLS<br>
>>>> _ERR_ACCEPT+TLS_LIB_ERR=A000416+TLS_IO_ERR=1<br>
>>>> current master transaction: <br>
>>>> master53<br>
>>>> Dec 18 12:59:45 mumbai squid[604]: ERROR: failure while accepting a <br>
>>>> TLS connection on conn43 local=<a href="http://192.168.0.1:8080" rel="noreferrer" target="_blank">192.168.0.1:8080</a> <br>
>>>> remote=<a href="http://192.168.0.111:53869" rel="noreferrer" target="_blank">192.168.0.111:53869</a> FD 16 flags=1: SQUID_TLS<br>
>>>> _ERR_ACCEPT+TLS_LIB_ERR=A000416+TLS_IO_ERR=1<br>
>>>> current master transaction: <br>
>>>> master57<br>
>>>> Dec 18 12:59:45 mumbai squid[604]: ERROR: failure while accepting a <br>
>>>> TLS connection on conn44 local=<a href="http://192.168.0.1:8080" rel="noreferrer" target="_blank">192.168.0.1:8080</a> <br>
>>>> remote=<a href="http://192.168.0.111:53870" rel="noreferrer" target="_blank">192.168.0.111:53870</a> FD 12 flags=1: SQUID_TLS<br>
>>>> _ERR_ACCEPT+TLS_LIB_ERR=A000416+TLS_IO_ERR=1<br>
>>>> current master transaction: <br>
>>>> master57<br>
>>>> Dec 18 12:59:56 mumbai squid[604]: ERROR: failure while accepting a <br>
>>>> TLS connection on conn62 local=<a href="http://192.168.0.1:8080" rel="noreferrer" target="_blank">192.168.0.1:8080</a> <br>
>>>> remote=<a href="http://192.168.0.111:53887" rel="noreferrer" target="_blank">192.168.0.111:53887</a> FD 12 flags=1: SQUID_TLS<br>
>>>> _ERR_ACCEPT+TLS_LIB_ERR=A000416+TLS_IO_ERR=1<br>
>>>> current master transaction: <br>
>>>> master95<br>
>>>> Dec 18 12:59:59 mumbai squid[604]: ERROR: failure while accepting a <br>
>>>> TLS connection on conn64 local=<a href="http://192.168.0.1:8080" rel="noreferrer" target="_blank">192.168.0.1:8080</a> <br>
>>>> remote=<a href="http://192.168.0.111:53888" rel="noreferrer" target="_blank">192.168.0.111:53888</a> FD 12 flags=1: SQUID_TLS<br>
>>>> _ERR_ACCEPT+TLS_LIB_ERR=A000416+TLS_IO_ERR=1<br>
>>>> current master transaction: <br>
>>>> master99<br>
>>>> Dec 18 13:00:02 mumbai squid[604]: ERROR: failure while accepting a <br>
>>>> TLS connection on conn65 local=<a href="http://192.168.0.1:8080" rel="noreferrer" target="_blank">192.168.0.1:8080</a> <br>
>>>> remote=<a href="http://192.168.0.178:56115" rel="noreferrer" target="_blank">192.168.0.178:56115</a> FD 12 flags=1: SQUID_TLS<br>
>>>> _ERR_ACCEPT+TLS_LIB_ERR=A000418+TLS_IO_ERR=1<br>
>>>> current master transaction: <br>
>>>> master53<br>
>>>> Dec 18 13:01:24 mumbai squid[604]: kick abandoning conn199 <br>
>>>> local=<a href="http://192.168.0.1:8093" rel="noreferrer" target="_blank">192.168.0.1:8093</a> remote=<a href="http://192.168.0.101:52211" rel="noreferrer" target="_blank">192.168.0.101:52211</a> FD 52 flags=1<br>
>>>> connection: conn199 <br>
>>>> local=<a href="http://192.168.0.1:8093" rel="noreferrer" target="_blank">192.168.0.1:8093</a> remote=<a href="http://192.168.0.101:52211" rel="noreferrer" target="_blank">192.168.0.101:52211</a> FD 52 flags=1<br>
>>>> Dec 18 13:01:45 mumbai squid[604]: ERROR: failure while accepting a <br>
>>>> TLS connection on conn240 local=<a href="http://192.168.0.1:8093" rel="noreferrer" target="_blank">192.168.0.1:8093</a> <br>
>>>> remote=<a href="http://192.168.0.111:53931" rel="noreferrer" target="_blank">192.168.0.111:53931</a> FD 48 flags=1: SQUID_TL<br>
>>>> S_ERR_ACCEPT+TLS_LIB_ERR=A000416+TLS_IO_ERR=1<br>
>>>> current master transaction: <br>
>>>> master314<br>
>>>><br>
>>>><br>
>>>> After this point there is nothing in systemd journal (via: <br>
>>>> journalctl -f -u squid) and same lines are in cache.log.<br>
>>>><br>
>>>> Squid got stuck (DEAD state) at 13:01 and right now it 19:26 (6 <br>
>>>> hours passed) and squid is still in dead state.<br>
>>>><br>
>>>> kill -9 or kill -ALRM or -HUP also does nothing.<br>
>>>><br>
>>>> So to restart squid - I will need to restart whole system.<br>
>>>><br>
>>>> I have sslbump directives but it is not really applied.<br>
>>>><br>
>>>> #NOTE: nosslbump_ips below contains <a href="http://192.168.0.0/24" rel="noreferrer" target="_blank">192.168.0.0/24</a> (whole LAN) so <br>
>>>> effectively there is no SSL bump after step1.<br>
>>>><br>
>>>> acl nosslbump_ips src <a href="http://192.168.0.0/24" rel="noreferrer" target="_blank">192.168.0.0/24</a><br>
>>>> ssl_bump splice ssl_step1 nosslbump_ips<br>
>>>> ssl_bump peek ssl_step1<br>
>>>> ssl_bump splice nosslbump_domains<br>
>>>> ssl_bump stare sslbump_domains<br>
>>>> ssl_bump splice ssl_step2<br>
>>>> ssl_bump bump all<br>
>>>><br>
>>>><br>
>>>> Any idea? If anything changed from 5.7 to 6.6 that may cause this <br>
>>>> behaviour?<br>
>>>><br>
>>>> Looking at changelog:<br>
>>>><br>
>>>> Bug 5256: Intercepting port fails to accept<br>
>>>> <a href="https://bugs.squid-cache.org/show_bug.cgi?id=5256" rel="noreferrer" target="_blank">https://bugs.squid-cache.org/show_bug.cgi?id=5256</a><br>
>>>><br>
>>>> Bug 5154: Do not open IPv6 sockets when IPv6 is disabled<br>
>>>> <a href="https://bugs.squid-cache.org/show_bug.cgi?id=5154" rel="noreferrer" target="_blank">https://bugs.squid-cache.org/show_bug.cgi?id=5154</a><br>
>>>><br>
>>>> Not sure if above two bug FIXES (in between v5.7 to v6.6) are <br>
>>>> related to my issue.<br>
>>>><br>
>>>> I ran netstat:<br>
>>>><br>
>>>> # netstat -ntlp<br>
>>>> Active Internet connections (only servers)<br>
>>>> Proto Recv-Q Send-Q Local Address Foreign Address <br>
>>>> State PID/Program name<br>
>>>> ...<br>
>>>> tcp6 33 0 :::3137 :::* LISTEN -<br>
>>>> tcp6 0 0 :::3136 :::* LISTEN -<br>
>>>> tcp6 4 0 :::3128 :::* LISTEN -<br>
>>>> tcp6 0 0 :::8081 :::* LISTEN -<br>
>>>> tcp6 0 0 :::8080 :::* LISTEN -<br>
>>>> tcp6 0 0 :::8083 :::* LISTEN -<br>
>>>> tcp6 0 0 :::8082 :::* LISTEN -<br>
>>>> tcp6 0 0 :::8084 :::* LISTEN -<br>
>>>> tcp6 4097 0 :::8093 :::* LISTEN -<br>
>>>> tcp6 0 0 :::8092 :::* LISTEN -<br>
>>>> tcp6 0 0 :::8094 :::* LISTEN -<br>
>>>> ...<br>
>>>><br>
>>>> I do not have IPv6 enabled, yet there are 33 and 4097 numbers in <br>
>>>> Recv-Q and also no process/PID owns these ports.<br>
>>>><br>
>>>> Same IPv4 ports are not shown in use by netstat, so only IPv6 ports <br>
>>>> remain open, that too orphaned!<br>
>>>><br>
>>>> So what is happening?<br>
>>>><br>
>>>> Any idea to solve or any workaround?<br>
>>>><br>
>>>> Thank you,<br>
>>>><br>
>>>> Amish.<br>
>> _______________________________________________<br>
>> squid-users mailing list<br>
>> <a href="mailto:squid-users@lists.squid-cache.org" target="_blank">squid-users@lists.squid-cache.org</a><br>
>> <a href="https://lists.squid-cache.org/listinfo/squid-users" rel="noreferrer" target="_blank">https://lists.squid-cache.org/listinfo/squid-users</a><br>
><br>
_______________________________________________<br>
squid-users mailing list<br>
<a href="mailto:squid-users@lists.squid-cache.org" target="_blank">squid-users@lists.squid-cache.org</a><br>
<a href="https://lists.squid-cache.org/listinfo/squid-users" rel="noreferrer" target="_blank">https://lists.squid-cache.org/listinfo/squid-users</a><br>
</blockquote></div><br clear="all"><div><br></div><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"> Francesco</div>