[squid-users] squid hangs and dies and can not be killed - needs system reboot
Amish
anon.amish at gmail.com
Thu Dec 21 12:59:59 UTC 2023
On 21/12/23 17:55, Francesco Chemolli wrote:
> Hi Amish,
> the message you posted really looks like a kernel bug, possibly due
> to faulty code, or resulting from a hardware problem.
> Nothing squid can do can cause that kind of stack traces in kernel-space.
>
> A quick search online results in
> https://lkml.kernel.org/netdev/20230526163458.2880232-1-edumazet@google.com/T/
> , which looks very much like your message
>
Yes even I believe so but I wonder, why only after updating to squid 6.6?
Regards
Amish.
> On Thu, Dec 21, 2023 at 1:10 PM Amish <anon.amish at gmail.com> wrote:
>
> Hi Alex,
>
> Sorry for late reply. But this is a production system and hence
> debugging is tough. May take few days.
>
> However I noticed that everytime squid hangs (goes dead), the
> dmesg has
> these errors.
>
> [63567.802188] divide error: 0000 [#1] PREEMPT SMP PTI
> [63567.802215] CPU: 3 PID: 617 Comm: squid Not tainted
> 6.6.7-arch1-1 #1
> 4505c4baa0b3d7c4037b0e8f5402626fa360717f
> [63567.802238] Hardware name: Gigabyte Technology Co., Ltd.
> H81M-S/H81M-S, BIOS F2 08/19/2015
> [63567.802255] RIP: 0010:tcp_rcv_space_adjust+0xbe/0x160
> [63567.802267] Code: f8 41 89 d0 29 d0 31 d2 49 0f af c2 49 f7 f0
> 45 8b
> 81 bc 04 00 00 44 0f b6 8b 10 06 00 00 31 d2 49 8d 04 42 48 98 48
> c1 e0
> 08 <49> f7 f1 49 63 d0
> 48 98 48 39 d0 48 0f 47 c2 39 83 18 01 00 00 7c
> [63567.802287] RSP: 0018:ffffba9f00da3c18 EFLAGS: 00010206
> [63567.802296] RAX: 0000000004fb2800 RBX: ffff9e124acc1c80 RCX:
> 0000000eccd9dace
> [63567.802304] RDX: 0000000000000000 RSI: 0000000037fa4ae8 RDI:
> 0000000000008349
> [63567.802314] RBP: ffff9e124acc1c80 R08: 0000000000600000 R09:
> 0000000000000000
> [63567.802322] R10: 00000000000161d2 R11: ffff9e1346621700 R12:
> 0000000000000000
> [63567.802331] R13: ffff9e124acc1d58 R14: 0000000000000000 R15:
> ffff9e124acc2214
> [63567.802340] FS: 00007fc3d627c840(0000) GS:ffff9e1457380000(0000)
> knlGS:0000000000000000
> [63567.802350] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [63567.802358] CR2: 0000560b7e86d690 CR3: 0000000102f44001 CR4:
> 00000000001706e0
> [63567.802367] Call Trace:
> [63567.802372] <TASK>
> [63567.802378] ? die+0x36/0x90
> [63567.802386] ? do_trap+0xda/0x100
> [63567.802392] ? tcp_rcv_space_adjust+0xbe/0x160
> [63567.802401] ? do_error_trap+0x6a/0x90
> [63567.802408] ? tcp_rcv_space_adjust+0xbe/0x160
> [63567.802415] ? exc_divide_error+0x38/0x50
> [63567.802423] ? tcp_rcv_space_adjust+0xbe/0x160
> [63567.802431] ? asm_exc_divide_error+0x1a/0x20
> [63567.802441] ? tcp_rcv_space_adjust+0xbe/0x160
> [63567.802449] ? tcp_rcv_space_adjust+0x1a/0x160
> [63567.802456] tcp_recvmsg_locked+0x2d4/0x960
> [63567.802464] tcp_recvmsg+0x87/0x1f0
> [63567.802471] inet6_recvmsg+0x56/0x130
> [63567.802479] ? __pfx_bpf_lsm_socket_recvmsg+0x10/0x10
> [63567.802488] ? security_socket_recvmsg+0x44/0x70
> [63567.802497] sock_recvmsg+0x57/0xd0
> [63567.802505] sock_read_iter+0x96/0x100
> [63567.802513] vfs_read+0x303/0x350
> [63567.802796] ksys_read+0xbb/0xf0
> [63567.803074] do_syscall_64+0x60/0x90
> [63567.803346] ? syscall_exit_to_user_mode+0x2b/0x40
> [63567.803619] ? do_syscall_64+0x6c/0x90
> [63567.803890] ? syscall_exit_to_user_mode+0x2b/0x40
> [63567.804163] ? do_syscall_64+0x6c/0x90
> [63567.804438] ? do_syscall_64+0x6c/0x90
> [63567.804710] entry_SYSCALL_64_after_hwframe+0x6e/0xd8
> [63567.804984] RIP: 0033:0x7fc3d5f21531
> [63567.805271] Code: ff ff eb c3 67 e8 2f c9 01 00 66 2e 0f 1f 84
> 00 00
> 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 80 3d 35 ce 0d 00 00 74 13 31
> c0 0f
> 05 <48> 3d 00 f0 ff ff
> 77 57 c3 66 0f 1f 44 00 00 48 83 ec 28 48 89 54
> [63567.805909] RSP: 002b:00007fffa66f6748 EFLAGS: 00000246 ORIG_RAX:
> 0000000000000000
> [63567.806225] RAX: ffffffffffffffda RBX: 00007fffa66f6830 RCX:
> 00007fc3d5f21531
> [63567.806542] RDX: 000000000000191a RSI: 000055d106003d80 RDI:
> 00000000000000bd
> [63567.806855] RBP: 000055d10620bc30 R08: 000055d10620bc30 R09:
> 000055d1036a8e78
> [63567.807163] R10: 000055d1037ad630 R11: 0000000000000246 R12:
> 000000000000191a
> [63567.807472] R13: 000055d10664f410 R14: 000055d106003d80 R15:
> 00007fc3d627c6b8
> [63567.807768] </TASK>
> [63567.808037] Modules linked in: nfnetlink_queue
> nf_conntrack_netlink
> xt_connmark xt_mark ip_set_hash_net xt_NFQUEUE xt_set ip_set_hash_ip
> ip_set cls_fw sch_sfq sch_h
> tb nf_nat_ftp nf_conntrack_ftp tls tun cfg80211 rfkill nft_chain_nat
> xt_MASQUERADE xt_nat xt_REDIRECT nf_nat nft_limit xt_limit
> xt_conntrack
> nf_conntrack nf_defrag_ipv
> 6 nf_defrag_ipv4 xt_multiport xt_tcpudp nft_compat nf_tables
> libcrc32c
> intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal
> intel_powerclamp
> snd_hda_codec_realtek cor
> etemp snd_hda_codec_generic ledtrig_audio snd_hda_codec_hdmi
> kvm_intel
> snd_hda_intel kvm snd_intel_dspcfg snd_intel_sdw_acpi irqbypass
> snd_hda_codec snd_hda_core crct1
> 0dif_pclmul r8169 crc32_pclmul snd_hwdep polyval_clmulni snd_pcm
> polyval_generic gf128mul snd_timer realtek ax88179_178a spi_nor
> mdio_devres usbnet snd libphy ghash_cl
> mulni_intel mtd sha512_ssse3 at24 soundcore mii mei_pxp mei_hdcp
> sha256_ssse3 sha1_ssse3 iTCO_wdt i2c_i801 mei_me spi_intel_platform
> mousedev aesni_intel intel_pmc_bxt
> spi_intel iTCO_vendor_support
> [63567.808076] i2c_smbus mei crypto_simd lpc_ich cryptd rapl
> intel_cstate intel_uncore psmouse mac_hid pcspkr pkcs8_key_parser
> fuse
> dm_mod loop nfnetlink ip_tables x_
> tables ext4 crc32c_generic crc16 mbcache jbd2 i915 serio_raw
> i2c_algo_bit atkbd drm_buddy libps2 ttm vivaldi_fmap intel_gtt
> xhci_pci
> crc32c_intel drm_display_helper xh
> ci_pci_renesas cec i8042 video serio wmi
> [63567.811333] ---[ end trace 0000000000000000 ]---
> [63567.811701] RIP: 0010:tcp_rcv_space_adjust+0xbe/0x160
> [63567.812069] Code: f8 41 89 d0 29 d0 31 d2 49 0f af c2 49 f7 f0
> 45 8b
> 81 bc 04 00 00 44 0f b6 8b 10 06 00 00 31 d2 49 8d 04 42 48 98 48
> c1 e0
> 08 <49> f7 f1 49 63 d0
> 48 98 48 39 d0 48 0f 47 c2 39 83 18 01 00 00 7c
> [63567.812848] RSP: 0018:ffffba9f00da3c18 EFLAGS: 00010206
> [63567.813252] RAX: 0000000004fb2800 RBX: ffff9e124acc1c80 RCX:
> 0000000eccd9dace
> [63567.813651] RDX: 0000000000000000 RSI: 0000000037fa4ae8 RDI:
> 0000000000008349
> [63567.814064] RBP: ffff9e124acc1c80 R08: 0000000000600000 R09:
> 0000000000000000
> [63567.814470] R10: 00000000000161d2 R11: ffff9e1346621700 R12:
> 0000000000000000
> [63567.814879] R13: ffff9e124acc1d58 R14: 0000000000000000 R15:
> ffff9e124acc2214
> [63567.815290] FS: 00007fc3d627c840(0000) GS:ffff9e1457380000(0000)
> knlGS:0000000000000000
> [63567.815700] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [63567.816121] CR2: 0000560b7e86d690 CR3: 0000000102f44001 CR4:
> 00000000001706e0
>
> After this the "child" squid process can not be killed, even by
> systemd,
> even by kill -9.
>
> It appears that squid "main" process (owned by root user) does not
> hang
> and hence connections on port 8080 are accepted.
>
> But squid child process (owned by proxy user) hangs and never
> recovers
> after above dmesg. And hence nothing happens and connection times out.
>
> # ps aux |grep squid
> root 612 0.0 0.2 75500 23680 ? Ss Dec20 0:02
> /usr/bin/squid -f /etc/squid/btnet/squid.btnet.conf --foreground -sYC
> proxy 617 0.0 0.0 0 0 ? D Dec20 1:00
> [squid]
> proxy 620 0.0 0.0 12152 7552 ? S Dec20 0:00
> (security_file_certgen) -s /var/cache/squid/ssl_db -M 4MB
> proxy 621 0.0 0.0 12152 7552 ? S Dec20 0:00
> (security_file_certgen) -s /var/cache/squid/ssl_db -M 4MB
> proxy 622 0.0 0.0 11888 5504 ? S Dec20 0:00
> (security_file_certgen) -s /var/cache/squid/ssl_db -M 4MB
> proxy 623 0.0 0.0 11888 5632 ? S Dec20 0:00
> (security_file_certgen) -s /var/cache/squid/ssl_db -M 4MB
> proxy 624 0.0 0.0 11888 5632 ? S Dec20 0:00
> (security_file_certgen) -s /var/cache/squid/ssl_db -M 4MB
> proxy 643 0.0 0.0 6116 3200 ? S Dec20 0:01
> (logfile-daemon) /var/log/squid/access.log
>
> # telnet 127.0.0.1 8080
> Trying 127.0.0.1...
> Connected to 127.0.0.1.
> Escape character is '^]'.
> ^]
> telnet> c
>
> # curl -x 127.0.0.1:8080 <http://127.0.0.1:8080> --max-time 30
> https://google.com
> curl: (28) Connection timed out after 30002 milliseconds
>
> # kill 612 617
>
> Ran kill 5 times, nothing happened. All squid processes remained.
>
> # kill -9 612 617
>
> Ran once, and all squid processes except 617 (child squid process)
> got
> killed
>
> # ps aux |grep squid
> proxy 617 0.0 0.0 0 0 ? D Dec20 1:00
> [squid]
> root 90764 0.0 0.0 6552 2560 pts/5 S+ 17:36 0:00 grep
> --color=auto squid
>
> Can above information be of any help?
>
> Thanks and regards,
>
> Amish
>
> On 19/12/23 20:46, Alex Rousskov wrote:
> > On 2023-12-18 22:29, Amish wrote:
> >> On 19/12/23 01:14, Alex Rousskov wrote:
> >>> On 2023-12-18 09:35, Amish wrote:
> >>>
> >>>> I use Arch Linux and today I updated squid from squid 5.7 to
> squid
> >>>> 6.6.
> >>>
> >>> > Dec 18 13:01:24 mumbai squid[604]: kick abandoning conn199
> >>>
> >>> I do not know whether the above problem is the primary problem in
> >>> your setup, but it is a red flag. Transactions on the same
> >>> connection may get stuck after that message; it is essentially a
> >>> Squid bug.
> >>>
> >>> I am not sure at all, but this bug might be related to Bug 5187
> >>> workaround that went into Squid v6.2 (commit c44cfe7):
> >>> https://bugs.squid-cache.org/show_bug.cgi?id=5187
> >>>
> >>> Does Squid accept new TCP connections after it enters what you
> >>> describe as a dead state? For example, does "telnet 127.0.0.1
> 8080"
> >>> establishes a connection if executed on the same machine as Squid?
> >>>
> >> Yes it establishes connection. But I do not know what to do next.
> >
> > This tells us that your Squid is still listening for incoming
> > connections. Most likely, it is not "dead" but running and just
> unable
> > to make progress with those connections (for yet unknown reasons).
> > That information is helpful but not sufficient (for me) to solve
> the
> > problem you are describing.
> >
> > The next step that I would recommend is to collect debugging
> > information from the running process and share a pointer to the
> > corresponding compressed cache.log file:
> >
> > * Ideally, start collection when Squid starts and reproduce the
> > problem while collecting full debugging information:
> > http://wiki.squid-cache.org/SquidFaq/BugReporting#full-debug-output
> >
> > * If you have to, start collection after Squid is already in bad
> state
> > and just before you use telnet or browser to tickle Squid:
> >
> http://wiki.squid-cache.org/SquidFaq/BugReporting#debugging-a-single-transaction
>
> >
> >
> > Do not use any secret information (e.g., production certificate
> keys)
> > for these tests (unless you are going to share the logs
> privately with
> > those you trust).
> >
> > Do not downgrade to v5 for these tests.
> >
> >
> > HTH,
> >
> > Alex.
> >
> >
> >> Browser showed "Connection timed out" message. But I believe
> >> browser's also connected but nothing happened afterwards.
> >>
> >>>
> >>> > kill -9 does nothing
> >>>
> >>> Is it possible that you are trying to kill the wrong process? You
> >>> should be killing this process AFAICT:
> >>>
> >>> > root 601 0.0 0.2 73816 22528 ? Ss 12:59 0:02
> >>> > /usr/bin/squid -f /etc/squid/btnet/squid.btnet.conf
> --foreground -sYC
> >>
> >> I did not clarify but all processes needed SIGKILL and vanished
> >> except the Dead squid process which still remained.
> >>
> >> # systemctl stop squid
> >>
> >> Dec 19 08:46:38 mumbai systemd[1]: squid.service: State
> >> 'stop-sigterm' timed out. Killing.
> >> Dec 19 08:46:38 mumbai systemd[1]: squid.service: Killing
> process 601
> >> (squid) with signal SIGKILL.
> >> Dec 19 08:46:38 mumbai systemd[1]: squid.service: Killing
> process 604
> >> (squid) with signal SIGKILL.
> >> Dec 19 08:46:38 mumbai systemd[1]: squid.service: Killing
> process 607
> >> (security_file_c) with signal SIGKILL.
> >> Dec 19 08:46:38 mumbai systemd[1]: squid.service: Killing
> process 608
> >> (security_file_c) with signal SIGKILL.
> >> Dec 19 08:46:38 mumbai systemd[1]: squid.service: Killing
> process 609
> >> (security_file_c) with signal SIGKILL.
> >> Dec 19 08:46:38 mumbai systemd[1]: squid.service: Killing
> process 610
> >> (security_file_c) with signal SIGKILL.
> >> Dec 19 08:46:38 mumbai systemd[1]: squid.service: Killing
> process 611
> >> (security_file_c) with signal SIGKILL.
> >> Dec 19 08:46:38 mumbai systemd[1]: squid.service: Killing
> process 622
> >> (log_file_daemon) with signal SIGKILL.
> >> Dec 19 08:46:38 mumbai systemd[1]: squid.service: Main process
> >> exited, code=killed, status=9/KILL
> >> Dec 19 08:46:38 mumbai systemd[1]: squid.service: Killing
> process 604
> >> (squid) with signal SIGKILL.
> >>
> >> Waited for 2 minutes for squid to stop then pressed ctrl-c to
> >> systemctl stop squid command.
> >>
> >> As you can see in last line shows that attempt was made to kill
> DEAD
> >> process with PID 604.
> >>
> >> # ps aux |grep squid
> >> proxy 604 0.0 0.0 0 0 ? D Dec18 0:03
> [squid]
> >>
> >> Now only DEAD squid process remains.
> >>
> >> What next? Should I downgrade to 5.9 and check?
> >>
> >> Regards
> >>
> >> Amish
> >>
> >>> Alex.
> >>>
> >>>
> >>>> After the update from 5.7 to 6.6, squid starts but then reaches
> >>>> Dead state in a minute or two.
> >>>>
> >>>> # ps aux | grep squid
> >>>> root 601 0.0 0.2 73816 22528 ? Ss 12:59 0:02
> >>>> /usr/bin/squid -f /etc/squid/btnet/squid.btnet.conf
> --foreground -sYC
> >>>> proxy 604 0.0 0.0 0 0 ? D 12:59 0:03
> >>>> [squid]
> >>>> proxy 607 0.0 0.0 11976 7424 ? S 12:59 0:00
> >>>> (security_file_certgen) -s /var/cache/squid/ssl_db -M 4MB
> >>>> proxy 608 0.0 0.0 11976 7168 ? S 12:59 0:00
> >>>> (security_file_certgen) -s /var/cache/squid/ssl_db -M 4MB
> >>>> proxy 609 0.0 0.0 11712 5632 ? S 12:59 0:00
> >>>> (security_file_certgen) -s /var/cache/squid/ssl_db -M 4MB
> >>>> proxy 610 0.0 0.0 11712 5376 ? S 12:59 0:00
> >>>> (security_file_certgen) -s /var/cache/squid/ssl_db -M 4MB
> >>>> proxy 611 0.0 0.0 11712 5504 ? S 12:59 0:00
> >>>> (security_file_certgen) -s /var/cache/squid/ssl_db -M 4MB
> >>>> proxy 622 0.0 0.0 6116 3200 ? S 12:59 0:00
> >>>> (logfile-daemon) /var/log/squid/access.log
> >>>>
> >>>> And then all requests get stuck. Notice the D (dead) state of
> squid.
> >>>>
> >>>> I use multiple ports for multiple purposes. (It all worked
> fine in
> >>>> squid 5.7)
> >>>>
> >>>> Dec 18 12:59:10 mumbai squid[601]: Starting Authentication on
> port
> >>>> [::]:3128
> >>>> Dec 18 12:59:10 mumbai squid[601]: Disabling Authentication
> on port
> >>>> [::]:3128 (interception enabled)
> >>>> Dec 18 12:59:10 mumbai squid[601]: Starting Authentication on
> port
> >>>> [::]:8081
> >>>> Dec 18 12:59:10 mumbai squid[601]: Disabling Authentication
> on port
> >>>> [::]:8081 (interception enabled)
> >>>> Dec 18 12:59:12 mumbai squid[601]: Starting Authentication on
> port
> >>>> [::]:8082
> >>>> Dec 18 12:59:12 mumbai squid[601]: Disabling Authentication
> on port
> >>>> [::]:8082 (interception enabled)
> >>>> Dec 18 12:59:12 mumbai squid[601]: Starting Authentication on
> port
> >>>> [::]:8083
> >>>> Dec 18 12:59:12 mumbai squid[601]: Disabling Authentication
> on port
> >>>> [::]:8083 (interception enabled)
> >>>> Dec 18 12:59:13 mumbai squid[601]: Starting Authentication on
> port
> >>>> [::]:8084
> >>>> Dec 18 12:59:13 mumbai squid[601]: Disabling Authentication
> on port
> >>>> [::]:8084 (interception enabled)
> >>>> Dec 18 12:59:13 mumbai squid[601]: Starting Authentication on
> port
> >>>> [::]:3136
> >>>> Dec 18 12:59:13 mumbai squid[601]: Disabling Authentication
> on port
> >>>> [::]:3136 (interception enabled)
> >>>> Dec 18 12:59:13 mumbai squid[601]: Starting Authentication on
> port
> >>>> [::]:3137
> >>>> Dec 18 12:59:13 mumbai squid[601]: Disabling Authentication
> on port
> >>>> [::]:3137 (interception enabled)
> >>>> ...
> >>>> Dec 18 12:59:29 mumbai squid[604]: Adaptation support is on
> >>>> Dec 18 12:59:29 mumbai squid[604]: Accepting NAT intercepted
> HTTP
> >>>> Socket connections at conn19 local=[::]:3128 remote=[::] FD 27
> >>>> flags=41
> >>>> listening port: 3128
> >>>> Dec 18 12:59:29 mumbai squid[604]: Accepting SSL bumped HTTP
> Socket
> >>>> connections at conn21 local=[::]:8080 remote=[::] FD 28 flags=9
> >>>> listening port: 8080
> >>>> Dec 18 12:59:29 mumbai squid[604]: Accepting NAT intercepted SSL
> >>>> bumped HTTPS Socket connections at conn23 local=[::]:8081
> >>>> remote=[::] FD 29 flags=41
> >>>> listening port: 8081
> >>>> Dec 18 12:59:29 mumbai squid[604]: Accepting SSL bumped HTTP
> Socket
> >>>> connections at conn25 local=[::]:8092 remote=[::] FD 30 flags=9
> >>>> listening port: 8092
> >>>> Dec 18 12:59:29 mumbai systemd[1]: Started Squid Web Proxy
> Server.
> >>>> Dec 18 12:59:29 mumbai squid[604]: Accepting SSL bumped HTTP
> Socket
> >>>> connections at conn27 local=[::]:8093 remote=[::] FD 31 flags=9
> >>>> listening port: 8093
> >>>> Dec 18 12:59:29 mumbai squid[604]: Accepting SSL bumped HTTP
> Socket
> >>>> connections at conn29 local=[::]:8094 remote=[::] FD 32 flags=9
> >>>> listening port: 8094
> >>>> Dec 18 12:59:29 mumbai squid[604]: Accepting NAT intercepted SSL
> >>>> bumped HTTPS Socket connections at conn31 local=[::]:8082
> >>>> remote=[::] FD 33 flags=41
> >>>> listening port: 8082
> >>>> Dec 18 12:59:29 mumbai squid[604]: Accepting NAT intercepted SSL
> >>>> bumped HTTPS Socket connections at conn33 local=[::]:8083
> >>>> remote=[::] FD 34 flags=41
> >>>> listening port: 8083
> >>>> Dec 18 12:59:29 mumbai squid[604]: Accepting NAT intercepted SSL
> >>>> bumped HTTPS Socket connections at conn35 local=[::]:8084
> >>>> remote=[::] FD 35 flags=41
> >>>> listening port: 8084
> >>>> Dec 18 12:59:29 mumbai squid[604]: Accepting NAT intercepted
> HTTP
> >>>> Socket connections at conn37 local=[::]:3136 remote=[::] FD 36
> >>>> flags=41
> >>>> listening port: 3136
> >>>> Dec 18 12:59:29 mumbai squid[604]: Accepting NAT intercepted
> HTTP
> >>>> Socket connections at conn39 local=[::]:3137 remote=[::] FD 37
> >>>> flags=41
> >>>> listening port: 3137
> >>>>
> >>>> And then following errors came:
> >>>>
> >>>>
> >>>> Dec 18 12:59:45 mumbai squid[604]: ERROR: failure while
> accepting a
> >>>> TLS connection on conn41 local=192.168.0.1:8080
> <http://192.168.0.1:8080>
> >>>> remote=192.168.0.111:53867 <http://192.168.0.111:53867> FD 12
> flags=1: SQUID_TLS
> >>>> _ERR_ACCEPT+TLS_LIB_ERR=A000416+TLS_IO_ERR=1
> >>>> current master transaction:
> >>>> master53
> >>>> Dec 18 12:59:45 mumbai squid[604]: ERROR: failure while
> accepting a
> >>>> TLS connection on conn42 local=192.168.0.1:8080
> <http://192.168.0.1:8080>
> >>>> remote=192.168.0.111:53868 <http://192.168.0.111:53868> FD 14
> flags=1: SQUID_TLS
> >>>> _ERR_ACCEPT+TLS_LIB_ERR=A000416+TLS_IO_ERR=1
> >>>> current master transaction:
> >>>> master53
> >>>> Dec 18 12:59:45 mumbai squid[604]: ERROR: failure while
> accepting a
> >>>> TLS connection on conn43 local=192.168.0.1:8080
> <http://192.168.0.1:8080>
> >>>> remote=192.168.0.111:53869 <http://192.168.0.111:53869> FD 16
> flags=1: SQUID_TLS
> >>>> _ERR_ACCEPT+TLS_LIB_ERR=A000416+TLS_IO_ERR=1
> >>>> current master transaction:
> >>>> master57
> >>>> Dec 18 12:59:45 mumbai squid[604]: ERROR: failure while
> accepting a
> >>>> TLS connection on conn44 local=192.168.0.1:8080
> <http://192.168.0.1:8080>
> >>>> remote=192.168.0.111:53870 <http://192.168.0.111:53870> FD 12
> flags=1: SQUID_TLS
> >>>> _ERR_ACCEPT+TLS_LIB_ERR=A000416+TLS_IO_ERR=1
> >>>> current master transaction:
> >>>> master57
> >>>> Dec 18 12:59:56 mumbai squid[604]: ERROR: failure while
> accepting a
> >>>> TLS connection on conn62 local=192.168.0.1:8080
> <http://192.168.0.1:8080>
> >>>> remote=192.168.0.111:53887 <http://192.168.0.111:53887> FD 12
> flags=1: SQUID_TLS
> >>>> _ERR_ACCEPT+TLS_LIB_ERR=A000416+TLS_IO_ERR=1
> >>>> current master transaction:
> >>>> master95
> >>>> Dec 18 12:59:59 mumbai squid[604]: ERROR: failure while
> accepting a
> >>>> TLS connection on conn64 local=192.168.0.1:8080
> <http://192.168.0.1:8080>
> >>>> remote=192.168.0.111:53888 <http://192.168.0.111:53888> FD 12
> flags=1: SQUID_TLS
> >>>> _ERR_ACCEPT+TLS_LIB_ERR=A000416+TLS_IO_ERR=1
> >>>> current master transaction:
> >>>> master99
> >>>> Dec 18 13:00:02 mumbai squid[604]: ERROR: failure while
> accepting a
> >>>> TLS connection on conn65 local=192.168.0.1:8080
> <http://192.168.0.1:8080>
> >>>> remote=192.168.0.178:56115 <http://192.168.0.178:56115> FD 12
> flags=1: SQUID_TLS
> >>>> _ERR_ACCEPT+TLS_LIB_ERR=A000418+TLS_IO_ERR=1
> >>>> current master transaction:
> >>>> master53
> >>>> Dec 18 13:01:24 mumbai squid[604]: kick abandoning conn199
> >>>> local=192.168.0.1:8093 <http://192.168.0.1:8093>
> remote=192.168.0.101:52211 <http://192.168.0.101:52211> FD 52 flags=1
> >>>> connection: conn199
> >>>> local=192.168.0.1:8093 <http://192.168.0.1:8093>
> remote=192.168.0.101:52211 <http://192.168.0.101:52211> FD 52 flags=1
> >>>> Dec 18 13:01:45 mumbai squid[604]: ERROR: failure while
> accepting a
> >>>> TLS connection on conn240 local=192.168.0.1:8093
> <http://192.168.0.1:8093>
> >>>> remote=192.168.0.111:53931 <http://192.168.0.111:53931> FD 48
> flags=1: SQUID_TL
> >>>> S_ERR_ACCEPT+TLS_LIB_ERR=A000416+TLS_IO_ERR=1
> >>>> current master transaction:
> >>>> master314
> >>>>
> >>>>
> >>>> After this point there is nothing in systemd journal (via:
> >>>> journalctl -f -u squid) and same lines are in cache.log.
> >>>>
> >>>> Squid got stuck (DEAD state) at 13:01 and right now it 19:26 (6
> >>>> hours passed) and squid is still in dead state.
> >>>>
> >>>> kill -9 or kill -ALRM or -HUP also does nothing.
> >>>>
> >>>> So to restart squid - I will need to restart whole system.
> >>>>
> >>>> I have sslbump directives but it is not really applied.
> >>>>
> >>>> #NOTE: nosslbump_ips below contains 192.168.0.0/24
> <http://192.168.0.0/24> (whole LAN) so
> >>>> effectively there is no SSL bump after step1.
> >>>>
> >>>> acl nosslbump_ips src 192.168.0.0/24 <http://192.168.0.0/24>
> >>>> ssl_bump splice ssl_step1 nosslbump_ips
> >>>> ssl_bump peek ssl_step1
> >>>> ssl_bump splice nosslbump_domains
> >>>> ssl_bump stare sslbump_domains
> >>>> ssl_bump splice ssl_step2
> >>>> ssl_bump bump all
> >>>>
> >>>>
> >>>> Any idea? If anything changed from 5.7 to 6.6 that may cause
> this
> >>>> behaviour?
> >>>>
> >>>> Looking at changelog:
> >>>>
> >>>> Bug 5256: Intercepting port fails to accept
> >>>> https://bugs.squid-cache.org/show_bug.cgi?id=5256
> >>>>
> >>>> Bug 5154: Do not open IPv6 sockets when IPv6 is disabled
> >>>> https://bugs.squid-cache.org/show_bug.cgi?id=5154
> >>>>
> >>>> Not sure if above two bug FIXES (in between v5.7 to v6.6) are
> >>>> related to my issue.
> >>>>
> >>>> I ran netstat:
> >>>>
> >>>> # netstat -ntlp
> >>>> Active Internet connections (only servers)
> >>>> Proto Recv-Q Send-Q Local Address Foreign Address
> >>>> State PID/Program name
> >>>> ...
> >>>> tcp6 33 0 :::3137 :::* LISTEN -
> >>>> tcp6 0 0 :::3136 :::* LISTEN -
> >>>> tcp6 4 0 :::3128 :::* LISTEN -
> >>>> tcp6 0 0 :::8081 :::* LISTEN -
> >>>> tcp6 0 0 :::8080 :::* LISTEN -
> >>>> tcp6 0 0 :::8083 :::* LISTEN -
> >>>> tcp6 0 0 :::8082 :::* LISTEN -
> >>>> tcp6 0 0 :::8084 :::* LISTEN -
> >>>> tcp6 4097 0 :::8093 :::* LISTEN -
> >>>> tcp6 0 0 :::8092 :::* LISTEN -
> >>>> tcp6 0 0 :::8094 :::* LISTEN -
> >>>> ...
> >>>>
> >>>> I do not have IPv6 enabled, yet there are 33 and 4097 numbers in
> >>>> Recv-Q and also no process/PID owns these ports.
> >>>>
> >>>> Same IPv4 ports are not shown in use by netstat, so only IPv6
> ports
> >>>> remain open, that too orphaned!
> >>>>
> >>>> So what is happening?
> >>>>
> >>>> Any idea to solve or any workaround?
> >>>>
> >>>> Thank you,
> >>>>
> >>>> Amish.
> >> _______________________________________________
> >> squid-users mailing list
> >> squid-users at lists.squid-cache.org
> >> https://lists.squid-cache.org/listinfo/squid-users
> >
> _______________________________________________
> squid-users mailing list
> squid-users at lists.squid-cache.org
> https://lists.squid-cache.org/listinfo/squid-users
>
>
>
> --
> Francesco
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20231221/a1b1a966/attachment-0001.htm>
More information about the squid-users
mailing list