[squid-users] squid hangs and dies and can not be killed - needs system reboot
Francesco Chemolli
gkinkie at gmail.com
Thu Dec 21 16:30:51 UTC 2023
I can't correlate the two changes, it might be some other external factor.
On Thu, Dec 21, 2023 at 2:00 PM Amish <anon.amish at gmail.com> wrote:
>
> On 21/12/23 17:55, Francesco Chemolli wrote:
>
> Hi Amish,
> the message you posted really looks like a kernel bug, possibly due to
> faulty code, or resulting from a hardware problem.
> Nothing squid can do can cause that kind of stack traces in kernel-space.
>
> A quick search online results in
> https://lkml.kernel.org/netdev/20230526163458.2880232-1-edumazet@google.com/T/
> , which looks very much like your message
>
> Yes even I believe so but I wonder, why only after updating to squid 6.6?
>
> Regards
>
> Amish.
>
> On Thu, Dec 21, 2023 at 1:10 PM Amish <anon.amish at gmail.com> wrote:
>
>> Hi Alex,
>>
>> Sorry for late reply. But this is a production system and hence
>> debugging is tough. May take few days.
>>
>> However I noticed that everytime squid hangs (goes dead), the dmesg has
>> these errors.
>>
>> [63567.802188] divide error: 0000 [#1] PREEMPT SMP PTI
>> [63567.802215] CPU: 3 PID: 617 Comm: squid Not tainted 6.6.7-arch1-1 #1
>> 4505c4baa0b3d7c4037b0e8f5402626fa360717f
>> [63567.802238] Hardware name: Gigabyte Technology Co., Ltd.
>> H81M-S/H81M-S, BIOS F2 08/19/2015
>> [63567.802255] RIP: 0010:tcp_rcv_space_adjust+0xbe/0x160
>> [63567.802267] Code: f8 41 89 d0 29 d0 31 d2 49 0f af c2 49 f7 f0 45 8b
>> 81 bc 04 00 00 44 0f b6 8b 10 06 00 00 31 d2 49 8d 04 42 48 98 48 c1 e0
>> 08 <49> f7 f1 49 63 d0
>> 48 98 48 39 d0 48 0f 47 c2 39 83 18 01 00 00 7c
>> [63567.802287] RSP: 0018:ffffba9f00da3c18 EFLAGS: 00010206
>> [63567.802296] RAX: 0000000004fb2800 RBX: ffff9e124acc1c80 RCX:
>> 0000000eccd9dace
>> [63567.802304] RDX: 0000000000000000 RSI: 0000000037fa4ae8 RDI:
>> 0000000000008349
>> [63567.802314] RBP: ffff9e124acc1c80 R08: 0000000000600000 R09:
>> 0000000000000000
>> [63567.802322] R10: 00000000000161d2 R11: ffff9e1346621700 R12:
>> 0000000000000000
>> [63567.802331] R13: ffff9e124acc1d58 R14: 0000000000000000 R15:
>> ffff9e124acc2214
>> [63567.802340] FS: 00007fc3d627c840(0000) GS:ffff9e1457380000(0000)
>> knlGS:0000000000000000
>> [63567.802350] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [63567.802358] CR2: 0000560b7e86d690 CR3: 0000000102f44001 CR4:
>> 00000000001706e0
>> [63567.802367] Call Trace:
>> [63567.802372] <TASK>
>> [63567.802378] ? die+0x36/0x90
>> [63567.802386] ? do_trap+0xda/0x100
>> [63567.802392] ? tcp_rcv_space_adjust+0xbe/0x160
>> [63567.802401] ? do_error_trap+0x6a/0x90
>> [63567.802408] ? tcp_rcv_space_adjust+0xbe/0x160
>> [63567.802415] ? exc_divide_error+0x38/0x50
>> [63567.802423] ? tcp_rcv_space_adjust+0xbe/0x160
>> [63567.802431] ? asm_exc_divide_error+0x1a/0x20
>> [63567.802441] ? tcp_rcv_space_adjust+0xbe/0x160
>> [63567.802449] ? tcp_rcv_space_adjust+0x1a/0x160
>> [63567.802456] tcp_recvmsg_locked+0x2d4/0x960
>> [63567.802464] tcp_recvmsg+0x87/0x1f0
>> [63567.802471] inet6_recvmsg+0x56/0x130
>> [63567.802479] ? __pfx_bpf_lsm_socket_recvmsg+0x10/0x10
>> [63567.802488] ? security_socket_recvmsg+0x44/0x70
>> [63567.802497] sock_recvmsg+0x57/0xd0
>> [63567.802505] sock_read_iter+0x96/0x100
>> [63567.802513] vfs_read+0x303/0x350
>> [63567.802796] ksys_read+0xbb/0xf0
>> [63567.803074] do_syscall_64+0x60/0x90
>> [63567.803346] ? syscall_exit_to_user_mode+0x2b/0x40
>> [63567.803619] ? do_syscall_64+0x6c/0x90
>> [63567.803890] ? syscall_exit_to_user_mode+0x2b/0x40
>> [63567.804163] ? do_syscall_64+0x6c/0x90
>> [63567.804438] ? do_syscall_64+0x6c/0x90
>> [63567.804710] entry_SYSCALL_64_after_hwframe+0x6e/0xd8
>> [63567.804984] RIP: 0033:0x7fc3d5f21531
>> [63567.805271] Code: ff ff eb c3 67 e8 2f c9 01 00 66 2e 0f 1f 84 00 00
>> 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 80 3d 35 ce 0d 00 00 74 13 31 c0 0f
>> 05 <48> 3d 00 f0 ff ff
>> 77 57 c3 66 0f 1f 44 00 00 48 83 ec 28 48 89 54
>> [63567.805909] RSP: 002b:00007fffa66f6748 EFLAGS: 00000246 ORIG_RAX:
>> 0000000000000000
>> [63567.806225] RAX: ffffffffffffffda RBX: 00007fffa66f6830 RCX:
>> 00007fc3d5f21531
>> [63567.806542] RDX: 000000000000191a RSI: 000055d106003d80 RDI:
>> 00000000000000bd
>> [63567.806855] RBP: 000055d10620bc30 R08: 000055d10620bc30 R09:
>> 000055d1036a8e78
>> [63567.807163] R10: 000055d1037ad630 R11: 0000000000000246 R12:
>> 000000000000191a
>> [63567.807472] R13: 000055d10664f410 R14: 000055d106003d80 R15:
>> 00007fc3d627c6b8
>> [63567.807768] </TASK>
>> [63567.808037] Modules linked in: nfnetlink_queue nf_conntrack_netlink
>> xt_connmark xt_mark ip_set_hash_net xt_NFQUEUE xt_set ip_set_hash_ip
>> ip_set cls_fw sch_sfq sch_h
>> tb nf_nat_ftp nf_conntrack_ftp tls tun cfg80211 rfkill nft_chain_nat
>> xt_MASQUERADE xt_nat xt_REDIRECT nf_nat nft_limit xt_limit xt_conntrack
>> nf_conntrack nf_defrag_ipv
>> 6 nf_defrag_ipv4 xt_multiport xt_tcpudp nft_compat nf_tables libcrc32c
>> intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp
>> snd_hda_codec_realtek cor
>> etemp snd_hda_codec_generic ledtrig_audio snd_hda_codec_hdmi kvm_intel
>> snd_hda_intel kvm snd_intel_dspcfg snd_intel_sdw_acpi irqbypass
>> snd_hda_codec snd_hda_core crct1
>> 0dif_pclmul r8169 crc32_pclmul snd_hwdep polyval_clmulni snd_pcm
>> polyval_generic gf128mul snd_timer realtek ax88179_178a spi_nor
>> mdio_devres usbnet snd libphy ghash_cl
>> mulni_intel mtd sha512_ssse3 at24 soundcore mii mei_pxp mei_hdcp
>> sha256_ssse3 sha1_ssse3 iTCO_wdt i2c_i801 mei_me spi_intel_platform
>> mousedev aesni_intel intel_pmc_bxt
>> spi_intel iTCO_vendor_support
>> [63567.808076] i2c_smbus mei crypto_simd lpc_ich cryptd rapl
>> intel_cstate intel_uncore psmouse mac_hid pcspkr pkcs8_key_parser fuse
>> dm_mod loop nfnetlink ip_tables x_
>> tables ext4 crc32c_generic crc16 mbcache jbd2 i915 serio_raw
>> i2c_algo_bit atkbd drm_buddy libps2 ttm vivaldi_fmap intel_gtt xhci_pci
>> crc32c_intel drm_display_helper xh
>> ci_pci_renesas cec i8042 video serio wmi
>> [63567.811333] ---[ end trace 0000000000000000 ]---
>> [63567.811701] RIP: 0010:tcp_rcv_space_adjust+0xbe/0x160
>> [63567.812069] Code: f8 41 89 d0 29 d0 31 d2 49 0f af c2 49 f7 f0 45 8b
>> 81 bc 04 00 00 44 0f b6 8b 10 06 00 00 31 d2 49 8d 04 42 48 98 48 c1 e0
>> 08 <49> f7 f1 49 63 d0
>> 48 98 48 39 d0 48 0f 47 c2 39 83 18 01 00 00 7c
>> [63567.812848] RSP: 0018:ffffba9f00da3c18 EFLAGS: 00010206
>> [63567.813252] RAX: 0000000004fb2800 RBX: ffff9e124acc1c80 RCX:
>> 0000000eccd9dace
>> [63567.813651] RDX: 0000000000000000 RSI: 0000000037fa4ae8 RDI:
>> 0000000000008349
>> [63567.814064] RBP: ffff9e124acc1c80 R08: 0000000000600000 R09:
>> 0000000000000000
>> [63567.814470] R10: 00000000000161d2 R11: ffff9e1346621700 R12:
>> 0000000000000000
>> [63567.814879] R13: ffff9e124acc1d58 R14: 0000000000000000 R15:
>> ffff9e124acc2214
>> [63567.815290] FS: 00007fc3d627c840(0000) GS:ffff9e1457380000(0000)
>> knlGS:0000000000000000
>> [63567.815700] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [63567.816121] CR2: 0000560b7e86d690 CR3: 0000000102f44001 CR4:
>> 00000000001706e0
>>
>> After this the "child" squid process can not be killed, even by systemd,
>> even by kill -9.
>>
>> It appears that squid "main" process (owned by root user) does not hang
>> and hence connections on port 8080 are accepted.
>>
>> But squid child process (owned by proxy user) hangs and never recovers
>> after above dmesg. And hence nothing happens and connection times out.
>>
>> # ps aux |grep squid
>> root 612 0.0 0.2 75500 23680 ? Ss Dec20 0:02
>> /usr/bin/squid -f /etc/squid/btnet/squid.btnet.conf --foreground -sYC
>> proxy 617 0.0 0.0 0 0 ? D Dec20 1:00 [squid]
>> proxy 620 0.0 0.0 12152 7552 ? S Dec20 0:00
>> (security_file_certgen) -s /var/cache/squid/ssl_db -M 4MB
>> proxy 621 0.0 0.0 12152 7552 ? S Dec20 0:00
>> (security_file_certgen) -s /var/cache/squid/ssl_db -M 4MB
>> proxy 622 0.0 0.0 11888 5504 ? S Dec20 0:00
>> (security_file_certgen) -s /var/cache/squid/ssl_db -M 4MB
>> proxy 623 0.0 0.0 11888 5632 ? S Dec20 0:00
>> (security_file_certgen) -s /var/cache/squid/ssl_db -M 4MB
>> proxy 624 0.0 0.0 11888 5632 ? S Dec20 0:00
>> (security_file_certgen) -s /var/cache/squid/ssl_db -M 4MB
>> proxy 643 0.0 0.0 6116 3200 ? S Dec20 0:01
>> (logfile-daemon) /var/log/squid/access.log
>>
>> # telnet 127.0.0.1 8080
>> Trying 127.0.0.1...
>> Connected to 127.0.0.1.
>> Escape character is '^]'.
>> ^]
>> telnet> c
>>
>> # curl -x 127.0.0.1:8080 --max-time 30 https://google.com
>> curl: (28) Connection timed out after 30002 milliseconds
>>
>> # kill 612 617
>>
>> Ran kill 5 times, nothing happened. All squid processes remained.
>>
>> # kill -9 612 617
>>
>> Ran once, and all squid processes except 617 (child squid process) got
>> killed
>>
>> # ps aux |grep squid
>> proxy 617 0.0 0.0 0 0 ? D Dec20 1:00 [squid]
>> root 90764 0.0 0.0 6552 2560 pts/5 S+ 17:36 0:00 grep
>> --color=auto squid
>>
>> Can above information be of any help?
>>
>> Thanks and regards,
>>
>> Amish
>>
>> On 19/12/23 20:46, Alex Rousskov wrote:
>> > On 2023-12-18 22:29, Amish wrote:
>> >> On 19/12/23 01:14, Alex Rousskov wrote:
>> >>> On 2023-12-18 09:35, Amish wrote:
>> >>>
>> >>>> I use Arch Linux and today I updated squid from squid 5.7 to squid
>> >>>> 6.6.
>> >>>
>> >>> > Dec 18 13:01:24 mumbai squid[604]: kick abandoning conn199
>> >>>
>> >>> I do not know whether the above problem is the primary problem in
>> >>> your setup, but it is a red flag. Transactions on the same
>> >>> connection may get stuck after that message; it is essentially a
>> >>> Squid bug.
>> >>>
>> >>> I am not sure at all, but this bug might be related to Bug 5187
>> >>> workaround that went into Squid v6.2 (commit c44cfe7):
>> >>> https://bugs.squid-cache.org/show_bug.cgi?id=5187
>> >>>
>> >>> Does Squid accept new TCP connections after it enters what you
>> >>> describe as a dead state? For example, does "telnet 127.0.0.1 8080"
>> >>> establishes a connection if executed on the same machine as Squid?
>> >>>
>> >> Yes it establishes connection. But I do not know what to do next.
>> >
>> > This tells us that your Squid is still listening for incoming
>> > connections. Most likely, it is not "dead" but running and just unable
>> > to make progress with those connections (for yet unknown reasons).
>> > That information is helpful but not sufficient (for me) to solve the
>> > problem you are describing.
>> >
>> > The next step that I would recommend is to collect debugging
>> > information from the running process and share a pointer to the
>> > corresponding compressed cache.log file:
>> >
>> > * Ideally, start collection when Squid starts and reproduce the
>> > problem while collecting full debugging information:
>> > http://wiki.squid-cache.org/SquidFaq/BugReporting#full-debug-output
>> >
>> > * If you have to, start collection after Squid is already in bad state
>> > and just before you use telnet or browser to tickle Squid:
>> >
>> http://wiki.squid-cache.org/SquidFaq/BugReporting#debugging-a-single-transaction
>> >
>> >
>> > Do not use any secret information (e.g., production certificate keys)
>> > for these tests (unless you are going to share the logs privately with
>> > those you trust).
>> >
>> > Do not downgrade to v5 for these tests.
>> >
>> >
>> > HTH,
>> >
>> > Alex.
>> >
>> >
>> >> Browser showed "Connection timed out" message. But I believe
>> >> browser's also connected but nothing happened afterwards.
>> >>
>> >>>
>> >>> > kill -9 does nothing
>> >>>
>> >>> Is it possible that you are trying to kill the wrong process? You
>> >>> should be killing this process AFAICT:
>> >>>
>> >>> > root 601 0.0 0.2 73816 22528 ? Ss 12:59 0:02
>> >>> > /usr/bin/squid -f /etc/squid/btnet/squid.btnet.conf --foreground
>> -sYC
>> >>
>> >> I did not clarify but all processes needed SIGKILL and vanished
>> >> except the Dead squid process which still remained.
>> >>
>> >> # systemctl stop squid
>> >>
>> >> Dec 19 08:46:38 mumbai systemd[1]: squid.service: State
>> >> 'stop-sigterm' timed out. Killing.
>> >> Dec 19 08:46:38 mumbai systemd[1]: squid.service: Killing process 601
>> >> (squid) with signal SIGKILL.
>> >> Dec 19 08:46:38 mumbai systemd[1]: squid.service: Killing process 604
>> >> (squid) with signal SIGKILL.
>> >> Dec 19 08:46:38 mumbai systemd[1]: squid.service: Killing process 607
>> >> (security_file_c) with signal SIGKILL.
>> >> Dec 19 08:46:38 mumbai systemd[1]: squid.service: Killing process 608
>> >> (security_file_c) with signal SIGKILL.
>> >> Dec 19 08:46:38 mumbai systemd[1]: squid.service: Killing process 609
>> >> (security_file_c) with signal SIGKILL.
>> >> Dec 19 08:46:38 mumbai systemd[1]: squid.service: Killing process 610
>> >> (security_file_c) with signal SIGKILL.
>> >> Dec 19 08:46:38 mumbai systemd[1]: squid.service: Killing process 611
>> >> (security_file_c) with signal SIGKILL.
>> >> Dec 19 08:46:38 mumbai systemd[1]: squid.service: Killing process 622
>> >> (log_file_daemon) with signal SIGKILL.
>> >> Dec 19 08:46:38 mumbai systemd[1]: squid.service: Main process
>> >> exited, code=killed, status=9/KILL
>> >> Dec 19 08:46:38 mumbai systemd[1]: squid.service: Killing process 604
>> >> (squid) with signal SIGKILL.
>> >>
>> >> Waited for 2 minutes for squid to stop then pressed ctrl-c to
>> >> systemctl stop squid command.
>> >>
>> >> As you can see in last line shows that attempt was made to kill DEAD
>> >> process with PID 604.
>> >>
>> >> # ps aux |grep squid
>> >> proxy 604 0.0 0.0 0 0 ? D Dec18 0:03
>> [squid]
>> >>
>> >> Now only DEAD squid process remains.
>> >>
>> >> What next? Should I downgrade to 5.9 and check?
>> >>
>> >> Regards
>> >>
>> >> Amish
>> >>
>> >>> Alex.
>> >>>
>> >>>
>> >>>> After the update from 5.7 to 6.6, squid starts but then reaches
>> >>>> Dead state in a minute or two.
>> >>>>
>> >>>> # ps aux | grep squid
>> >>>> root 601 0.0 0.2 73816 22528 ? Ss 12:59 0:02
>> >>>> /usr/bin/squid -f /etc/squid/btnet/squid.btnet.conf --foreground -sYC
>> >>>> proxy 604 0.0 0.0 0 0 ? D 12:59 0:03
>> >>>> [squid]
>> >>>> proxy 607 0.0 0.0 11976 7424 ? S 12:59 0:00
>> >>>> (security_file_certgen) -s /var/cache/squid/ssl_db -M 4MB
>> >>>> proxy 608 0.0 0.0 11976 7168 ? S 12:59 0:00
>> >>>> (security_file_certgen) -s /var/cache/squid/ssl_db -M 4MB
>> >>>> proxy 609 0.0 0.0 11712 5632 ? S 12:59 0:00
>> >>>> (security_file_certgen) -s /var/cache/squid/ssl_db -M 4MB
>> >>>> proxy 610 0.0 0.0 11712 5376 ? S 12:59 0:00
>> >>>> (security_file_certgen) -s /var/cache/squid/ssl_db -M 4MB
>> >>>> proxy 611 0.0 0.0 11712 5504 ? S 12:59 0:00
>> >>>> (security_file_certgen) -s /var/cache/squid/ssl_db -M 4MB
>> >>>> proxy 622 0.0 0.0 6116 3200 ? S 12:59 0:00
>> >>>> (logfile-daemon) /var/log/squid/access.log
>> >>>>
>> >>>> And then all requests get stuck. Notice the D (dead) state of squid.
>> >>>>
>> >>>> I use multiple ports for multiple purposes. (It all worked fine in
>> >>>> squid 5.7)
>> >>>>
>> >>>> Dec 18 12:59:10 mumbai squid[601]: Starting Authentication on port
>> >>>> [::]:3128
>> >>>> Dec 18 12:59:10 mumbai squid[601]: Disabling Authentication on port
>> >>>> [::]:3128 (interception enabled)
>> >>>> Dec 18 12:59:10 mumbai squid[601]: Starting Authentication on port
>> >>>> [::]:8081
>> >>>> Dec 18 12:59:10 mumbai squid[601]: Disabling Authentication on port
>> >>>> [::]:8081 (interception enabled)
>> >>>> Dec 18 12:59:12 mumbai squid[601]: Starting Authentication on port
>> >>>> [::]:8082
>> >>>> Dec 18 12:59:12 mumbai squid[601]: Disabling Authentication on port
>> >>>> [::]:8082 (interception enabled)
>> >>>> Dec 18 12:59:12 mumbai squid[601]: Starting Authentication on port
>> >>>> [::]:8083
>> >>>> Dec 18 12:59:12 mumbai squid[601]: Disabling Authentication on port
>> >>>> [::]:8083 (interception enabled)
>> >>>> Dec 18 12:59:13 mumbai squid[601]: Starting Authentication on port
>> >>>> [::]:8084
>> >>>> Dec 18 12:59:13 mumbai squid[601]: Disabling Authentication on port
>> >>>> [::]:8084 (interception enabled)
>> >>>> Dec 18 12:59:13 mumbai squid[601]: Starting Authentication on port
>> >>>> [::]:3136
>> >>>> Dec 18 12:59:13 mumbai squid[601]: Disabling Authentication on port
>> >>>> [::]:3136 (interception enabled)
>> >>>> Dec 18 12:59:13 mumbai squid[601]: Starting Authentication on port
>> >>>> [::]:3137
>> >>>> Dec 18 12:59:13 mumbai squid[601]: Disabling Authentication on port
>> >>>> [::]:3137 (interception enabled)
>> >>>> ...
>> >>>> Dec 18 12:59:29 mumbai squid[604]: Adaptation support is on
>> >>>> Dec 18 12:59:29 mumbai squid[604]: Accepting NAT intercepted HTTP
>> >>>> Socket connections at conn19 local=[::]:3128 remote=[::] FD 27
>> >>>> flags=41
>> >>>> listening port: 3128
>> >>>> Dec 18 12:59:29 mumbai squid[604]: Accepting SSL bumped HTTP Socket
>> >>>> connections at conn21 local=[::]:8080 remote=[::] FD 28 flags=9
>> >>>> listening port: 8080
>> >>>> Dec 18 12:59:29 mumbai squid[604]: Accepting NAT intercepted SSL
>> >>>> bumped HTTPS Socket connections at conn23 local=[::]:8081
>> >>>> remote=[::] FD 29 flags=41
>> >>>> listening port: 8081
>> >>>> Dec 18 12:59:29 mumbai squid[604]: Accepting SSL bumped HTTP Socket
>> >>>> connections at conn25 local=[::]:8092 remote=[::] FD 30 flags=9
>> >>>> listening port: 8092
>> >>>> Dec 18 12:59:29 mumbai systemd[1]: Started Squid Web Proxy Server.
>> >>>> Dec 18 12:59:29 mumbai squid[604]: Accepting SSL bumped HTTP Socket
>> >>>> connections at conn27 local=[::]:8093 remote=[::] FD 31 flags=9
>> >>>> listening port: 8093
>> >>>> Dec 18 12:59:29 mumbai squid[604]: Accepting SSL bumped HTTP Socket
>> >>>> connections at conn29 local=[::]:8094 remote=[::] FD 32 flags=9
>> >>>> listening port: 8094
>> >>>> Dec 18 12:59:29 mumbai squid[604]: Accepting NAT intercepted SSL
>> >>>> bumped HTTPS Socket connections at conn31 local=[::]:8082
>> >>>> remote=[::] FD 33 flags=41
>> >>>> listening port: 8082
>> >>>> Dec 18 12:59:29 mumbai squid[604]: Accepting NAT intercepted SSL
>> >>>> bumped HTTPS Socket connections at conn33 local=[::]:8083
>> >>>> remote=[::] FD 34 flags=41
>> >>>> listening port: 8083
>> >>>> Dec 18 12:59:29 mumbai squid[604]: Accepting NAT intercepted SSL
>> >>>> bumped HTTPS Socket connections at conn35 local=[::]:8084
>> >>>> remote=[::] FD 35 flags=41
>> >>>> listening port: 8084
>> >>>> Dec 18 12:59:29 mumbai squid[604]: Accepting NAT intercepted HTTP
>> >>>> Socket connections at conn37 local=[::]:3136 remote=[::] FD 36
>> >>>> flags=41
>> >>>> listening port: 3136
>> >>>> Dec 18 12:59:29 mumbai squid[604]: Accepting NAT intercepted HTTP
>> >>>> Socket connections at conn39 local=[::]:3137 remote=[::] FD 37
>> >>>> flags=41
>> >>>> listening port: 3137
>> >>>>
>> >>>> And then following errors came:
>> >>>>
>> >>>>
>> >>>> Dec 18 12:59:45 mumbai squid[604]: ERROR: failure while accepting a
>> >>>> TLS connection on conn41 local=192.168.0.1:8080
>> >>>> remote=192.168.0.111:53867 FD 12 flags=1: SQUID_TLS
>> >>>> _ERR_ACCEPT+TLS_LIB_ERR=A000416+TLS_IO_ERR=1
>> >>>> current master transaction:
>> >>>> master53
>> >>>> Dec 18 12:59:45 mumbai squid[604]: ERROR: failure while accepting a
>> >>>> TLS connection on conn42 local=192.168.0.1:8080
>> >>>> remote=192.168.0.111:53868 FD 14 flags=1: SQUID_TLS
>> >>>> _ERR_ACCEPT+TLS_LIB_ERR=A000416+TLS_IO_ERR=1
>> >>>> current master transaction:
>> >>>> master53
>> >>>> Dec 18 12:59:45 mumbai squid[604]: ERROR: failure while accepting a
>> >>>> TLS connection on conn43 local=192.168.0.1:8080
>> >>>> remote=192.168.0.111:53869 FD 16 flags=1: SQUID_TLS
>> >>>> _ERR_ACCEPT+TLS_LIB_ERR=A000416+TLS_IO_ERR=1
>> >>>> current master transaction:
>> >>>> master57
>> >>>> Dec 18 12:59:45 mumbai squid[604]: ERROR: failure while accepting a
>> >>>> TLS connection on conn44 local=192.168.0.1:8080
>> >>>> remote=192.168.0.111:53870 FD 12 flags=1: SQUID_TLS
>> >>>> _ERR_ACCEPT+TLS_LIB_ERR=A000416+TLS_IO_ERR=1
>> >>>> current master transaction:
>> >>>> master57
>> >>>> Dec 18 12:59:56 mumbai squid[604]: ERROR: failure while accepting a
>> >>>> TLS connection on conn62 local=192.168.0.1:8080
>> >>>> remote=192.168.0.111:53887 FD 12 flags=1: SQUID_TLS
>> >>>> _ERR_ACCEPT+TLS_LIB_ERR=A000416+TLS_IO_ERR=1
>> >>>> current master transaction:
>> >>>> master95
>> >>>> Dec 18 12:59:59 mumbai squid[604]: ERROR: failure while accepting a
>> >>>> TLS connection on conn64 local=192.168.0.1:8080
>> >>>> remote=192.168.0.111:53888 FD 12 flags=1: SQUID_TLS
>> >>>> _ERR_ACCEPT+TLS_LIB_ERR=A000416+TLS_IO_ERR=1
>> >>>> current master transaction:
>> >>>> master99
>> >>>> Dec 18 13:00:02 mumbai squid[604]: ERROR: failure while accepting a
>> >>>> TLS connection on conn65 local=192.168.0.1:8080
>> >>>> remote=192.168.0.178:56115 FD 12 flags=1: SQUID_TLS
>> >>>> _ERR_ACCEPT+TLS_LIB_ERR=A000418+TLS_IO_ERR=1
>> >>>> current master transaction:
>> >>>> master53
>> >>>> Dec 18 13:01:24 mumbai squid[604]: kick abandoning conn199
>> >>>> local=192.168.0.1:8093 remote=192.168.0.101:52211 FD 52 flags=1
>> >>>> connection: conn199
>> >>>> local=192.168.0.1:8093 remote=192.168.0.101:52211 FD 52 flags=1
>> >>>> Dec 18 13:01:45 mumbai squid[604]: ERROR: failure while accepting a
>> >>>> TLS connection on conn240 local=192.168.0.1:8093
>> >>>> remote=192.168.0.111:53931 FD 48 flags=1: SQUID_TL
>> >>>> S_ERR_ACCEPT+TLS_LIB_ERR=A000416+TLS_IO_ERR=1
>> >>>> current master transaction:
>> >>>> master314
>> >>>>
>> >>>>
>> >>>> After this point there is nothing in systemd journal (via:
>> >>>> journalctl -f -u squid) and same lines are in cache.log.
>> >>>>
>> >>>> Squid got stuck (DEAD state) at 13:01 and right now it 19:26 (6
>> >>>> hours passed) and squid is still in dead state.
>> >>>>
>> >>>> kill -9 or kill -ALRM or -HUP also does nothing.
>> >>>>
>> >>>> So to restart squid - I will need to restart whole system.
>> >>>>
>> >>>> I have sslbump directives but it is not really applied.
>> >>>>
>> >>>> #NOTE: nosslbump_ips below contains 192.168.0.0/24 (whole LAN) so
>> >>>> effectively there is no SSL bump after step1.
>> >>>>
>> >>>> acl nosslbump_ips src 192.168.0.0/24
>> >>>> ssl_bump splice ssl_step1 nosslbump_ips
>> >>>> ssl_bump peek ssl_step1
>> >>>> ssl_bump splice nosslbump_domains
>> >>>> ssl_bump stare sslbump_domains
>> >>>> ssl_bump splice ssl_step2
>> >>>> ssl_bump bump all
>> >>>>
>> >>>>
>> >>>> Any idea? If anything changed from 5.7 to 6.6 that may cause this
>> >>>> behaviour?
>> >>>>
>> >>>> Looking at changelog:
>> >>>>
>> >>>> Bug 5256: Intercepting port fails to accept
>> >>>> https://bugs.squid-cache.org/show_bug.cgi?id=5256
>> >>>>
>> >>>> Bug 5154: Do not open IPv6 sockets when IPv6 is disabled
>> >>>> https://bugs.squid-cache.org/show_bug.cgi?id=5154
>> >>>>
>> >>>> Not sure if above two bug FIXES (in between v5.7 to v6.6) are
>> >>>> related to my issue.
>> >>>>
>> >>>> I ran netstat:
>> >>>>
>> >>>> # netstat -ntlp
>> >>>> Active Internet connections (only servers)
>> >>>> Proto Recv-Q Send-Q Local Address Foreign Address
>> >>>> State PID/Program name
>> >>>> ...
>> >>>> tcp6 33 0 :::3137 :::* LISTEN -
>> >>>> tcp6 0 0 :::3136 :::* LISTEN -
>> >>>> tcp6 4 0 :::3128 :::* LISTEN -
>> >>>> tcp6 0 0 :::8081 :::* LISTEN -
>> >>>> tcp6 0 0 :::8080 :::* LISTEN -
>> >>>> tcp6 0 0 :::8083 :::* LISTEN -
>> >>>> tcp6 0 0 :::8082 :::* LISTEN -
>> >>>> tcp6 0 0 :::8084 :::* LISTEN -
>> >>>> tcp6 4097 0 :::8093 :::* LISTEN -
>> >>>> tcp6 0 0 :::8092 :::* LISTEN -
>> >>>> tcp6 0 0 :::8094 :::* LISTEN -
>> >>>> ...
>> >>>>
>> >>>> I do not have IPv6 enabled, yet there are 33 and 4097 numbers in
>> >>>> Recv-Q and also no process/PID owns these ports.
>> >>>>
>> >>>> Same IPv4 ports are not shown in use by netstat, so only IPv6 ports
>> >>>> remain open, that too orphaned!
>> >>>>
>> >>>> So what is happening?
>> >>>>
>> >>>> Any idea to solve or any workaround?
>> >>>>
>> >>>> Thank you,
>> >>>>
>> >>>> Amish.
>> >> _______________________________________________
>> >> squid-users mailing list
>> >> squid-users at lists.squid-cache.org
>> >> https://lists.squid-cache.org/listinfo/squid-users
>> >
>> _______________________________________________
>> squid-users mailing list
>> squid-users at lists.squid-cache.org
>> https://lists.squid-cache.org/listinfo/squid-users
>>
>
>
> --
> Francesco
>
>
--
Francesco
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20231221/9692a87d/attachment-0001.htm>
More information about the squid-users
mailing list