[squid-users] squid hangs and dies and can not be killed - needs system reboot
NgTech LTD
ngtech1ltd at gmail.com
Tue Dec 19 04:21:30 UTC 2023
Hey Amish,
I want to replicate this issue on a local vm.
Can you give us some details on the version of arch and the relevant
settings for recreating the issue?
How did you installed arch and also squid?
Thanks,
Eliezer
בתאריך יום ב׳, 18 בדצמ׳ 2023, 16:36, מאת Amish <anon.amish at gmail.com>:
> Hello,
>
> I use Arch Linux and today I updated squid from squid 5.7 to squid 6.6.
>
> After the update from 5.7 to 6.6, squid starts but then reaches Dead
> state in a minute or two.
>
> # ps aux | grep squid
> root 601 0.0 0.2 73816 22528 ? Ss 12:59 0:02
> /usr/bin/squid -f /etc/squid/btnet/squid.btnet.conf --foreground -sYC
> proxy 604 0.0 0.0 0 0 ? D 12:59 0:03 [squid]
> proxy 607 0.0 0.0 11976 7424 ? S 12:59 0:00
> (security_file_certgen) -s /var/cache/squid/ssl_db -M 4MB
> proxy 608 0.0 0.0 11976 7168 ? S 12:59 0:00
> (security_file_certgen) -s /var/cache/squid/ssl_db -M 4MB
> proxy 609 0.0 0.0 11712 5632 ? S 12:59 0:00
> (security_file_certgen) -s /var/cache/squid/ssl_db -M 4MB
> proxy 610 0.0 0.0 11712 5376 ? S 12:59 0:00
> (security_file_certgen) -s /var/cache/squid/ssl_db -M 4MB
> proxy 611 0.0 0.0 11712 5504 ? S 12:59 0:00
> (security_file_certgen) -s /var/cache/squid/ssl_db -M 4MB
> proxy 622 0.0 0.0 6116 3200 ? S 12:59 0:00
> (logfile-daemon) /var/log/squid/access.log
>
> And then all requests get stuck. Notice the D (dead) state of squid.
>
> I use multiple ports for multiple purposes. (It all worked fine in squid
> 5.7)
>
> Dec 18 12:59:10 mumbai squid[601]: Starting Authentication on port
> [::]:3128
> Dec 18 12:59:10 mumbai squid[601]: Disabling Authentication on port
> [::]:3128 (interception enabled)
> Dec 18 12:59:10 mumbai squid[601]: Starting Authentication on port
> [::]:8081
> Dec 18 12:59:10 mumbai squid[601]: Disabling Authentication on port
> [::]:8081 (interception enabled)
> Dec 18 12:59:12 mumbai squid[601]: Starting Authentication on port
> [::]:8082
> Dec 18 12:59:12 mumbai squid[601]: Disabling Authentication on port
> [::]:8082 (interception enabled)
> Dec 18 12:59:12 mumbai squid[601]: Starting Authentication on port
> [::]:8083
> Dec 18 12:59:12 mumbai squid[601]: Disabling Authentication on port
> [::]:8083 (interception enabled)
> Dec 18 12:59:13 mumbai squid[601]: Starting Authentication on port
> [::]:8084
> Dec 18 12:59:13 mumbai squid[601]: Disabling Authentication on port
> [::]:8084 (interception enabled)
> Dec 18 12:59:13 mumbai squid[601]: Starting Authentication on port
> [::]:3136
> Dec 18 12:59:13 mumbai squid[601]: Disabling Authentication on port
> [::]:3136 (interception enabled)
> Dec 18 12:59:13 mumbai squid[601]: Starting Authentication on port
> [::]:3137
> Dec 18 12:59:13 mumbai squid[601]: Disabling Authentication on port
> [::]:3137 (interception enabled)
> ...
> Dec 18 12:59:29 mumbai squid[604]: Adaptation support is on
> Dec 18 12:59:29 mumbai squid[604]: Accepting NAT intercepted HTTP Socket
> connections at conn19 local=[::]:3128 remote=[::] FD 27 flags=41
> listening port: 3128
> Dec 18 12:59:29 mumbai squid[604]: Accepting SSL bumped HTTP Socket
> connections at conn21 local=[::]:8080 remote=[::] FD 28 flags=9
> listening port: 8080
> Dec 18 12:59:29 mumbai squid[604]: Accepting NAT intercepted SSL bumped
> HTTPS Socket connections at conn23 local=[::]:8081 remote=[::] FD 29
> flags=41
> listening port: 8081
> Dec 18 12:59:29 mumbai squid[604]: Accepting SSL bumped HTTP Socket
> connections at conn25 local=[::]:8092 remote=[::] FD 30 flags=9
> listening port: 8092
> Dec 18 12:59:29 mumbai systemd[1]: Started Squid Web Proxy Server.
> Dec 18 12:59:29 mumbai squid[604]: Accepting SSL bumped HTTP Socket
> connections at conn27 local=[::]:8093 remote=[::] FD 31 flags=9
> listening port: 8093
> Dec 18 12:59:29 mumbai squid[604]: Accepting SSL bumped HTTP Socket
> connections at conn29 local=[::]:8094 remote=[::] FD 32 flags=9
> listening port: 8094
> Dec 18 12:59:29 mumbai squid[604]: Accepting NAT intercepted SSL bumped
> HTTPS Socket connections at conn31 local=[::]:8082 remote=[::] FD 33
> flags=41
> listening port: 8082
> Dec 18 12:59:29 mumbai squid[604]: Accepting NAT intercepted SSL bumped
> HTTPS Socket connections at conn33 local=[::]:8083 remote=[::] FD 34
> flags=41
> listening port: 8083
> Dec 18 12:59:29 mumbai squid[604]: Accepting NAT intercepted SSL bumped
> HTTPS Socket connections at conn35 local=[::]:8084 remote=[::] FD 35
> flags=41
> listening port: 8084
> Dec 18 12:59:29 mumbai squid[604]: Accepting NAT intercepted HTTP Socket
> connections at conn37 local=[::]:3136 remote=[::] FD 36 flags=41
> listening port: 3136
> Dec 18 12:59:29 mumbai squid[604]: Accepting NAT intercepted HTTP Socket
> connections at conn39 local=[::]:3137 remote=[::] FD 37 flags=41
> listening port: 3137
>
> And then following errors came:
>
>
> Dec 18 12:59:45 mumbai squid[604]: ERROR: failure while accepting a TLS
> connection on conn41 local=192.168.0.1:8080 remote=192.168.0.111:53867
> FD 12 flags=1: SQUID_TLS
> _ERR_ACCEPT+TLS_LIB_ERR=A000416+TLS_IO_ERR=1
> current master transaction:
> master53
> Dec 18 12:59:45 mumbai squid[604]: ERROR: failure while accepting a TLS
> connection on conn42 local=192.168.0.1:8080 remote=192.168.0.111:53868
> FD 14 flags=1: SQUID_TLS
> _ERR_ACCEPT+TLS_LIB_ERR=A000416+TLS_IO_ERR=1
> current master transaction:
> master53
> Dec 18 12:59:45 mumbai squid[604]: ERROR: failure while accepting a TLS
> connection on conn43 local=192.168.0.1:8080 remote=192.168.0.111:53869
> FD 16 flags=1: SQUID_TLS
> _ERR_ACCEPT+TLS_LIB_ERR=A000416+TLS_IO_ERR=1
> current master transaction:
> master57
> Dec 18 12:59:45 mumbai squid[604]: ERROR: failure while accepting a TLS
> connection on conn44 local=192.168.0.1:8080 remote=192.168.0.111:53870
> FD 12 flags=1: SQUID_TLS
> _ERR_ACCEPT+TLS_LIB_ERR=A000416+TLS_IO_ERR=1
> current master transaction:
> master57
> Dec 18 12:59:56 mumbai squid[604]: ERROR: failure while accepting a TLS
> connection on conn62 local=192.168.0.1:8080 remote=192.168.0.111:53887
> FD 12 flags=1: SQUID_TLS
> _ERR_ACCEPT+TLS_LIB_ERR=A000416+TLS_IO_ERR=1
> current master transaction:
> master95
> Dec 18 12:59:59 mumbai squid[604]: ERROR: failure while accepting a TLS
> connection on conn64 local=192.168.0.1:8080 remote=192.168.0.111:53888
> FD 12 flags=1: SQUID_TLS
> _ERR_ACCEPT+TLS_LIB_ERR=A000416+TLS_IO_ERR=1
> current master transaction:
> master99
> Dec 18 13:00:02 mumbai squid[604]: ERROR: failure while accepting a TLS
> connection on conn65 local=192.168.0.1:8080 remote=192.168.0.178:56115
> FD 12 flags=1: SQUID_TLS
> _ERR_ACCEPT+TLS_LIB_ERR=A000418+TLS_IO_ERR=1
> current master transaction:
> master53
> Dec 18 13:01:24 mumbai squid[604]: kick abandoning conn199
> local=192.168.0.1:8093 remote=192.168.0.101:52211 FD 52 flags=1
> connection: conn199
> local=192.168.0.1:8093 remote=192.168.0.101:52211 FD 52 flags=1
> Dec 18 13:01:45 mumbai squid[604]: ERROR: failure while accepting a TLS
> connection on conn240 local=192.168.0.1:8093 remote=192.168.0.111:53931
> FD 48 flags=1: SQUID_TL
> S_ERR_ACCEPT+TLS_LIB_ERR=A000416+TLS_IO_ERR=1
> current master transaction:
> master314
>
>
> After this point there is nothing in systemd journal (via: journalctl -f
> -u squid) and same lines are in cache.log.
>
> Squid got stuck (DEAD state) at 13:01 and right now it 19:26 (6 hours
> passed) and squid is still in dead state.
>
> kill -9 or kill -ALRM or -HUP also does nothing.
>
> So to restart squid - I will need to restart whole system.
>
> I have sslbump directives but it is not really applied.
>
> #NOTE: nosslbump_ips below contains 192.168.0.0/24 (whole LAN) so
> effectively there is no SSL bump after step1.
>
> acl nosslbump_ips src 192.168.0.0/24
> ssl_bump splice ssl_step1 nosslbump_ips
> ssl_bump peek ssl_step1
> ssl_bump splice nosslbump_domains
> ssl_bump stare sslbump_domains
> ssl_bump splice ssl_step2
> ssl_bump bump all
>
>
> Any idea? If anything changed from 5.7 to 6.6 that may cause this
> behaviour?
>
> Looking at changelog:
>
> Bug 5256: Intercepting port fails to accept
> https://bugs.squid-cache.org/show_bug.cgi?id=5256
>
> Bug 5154: Do not open IPv6 sockets when IPv6 is disabled
> https://bugs.squid-cache.org/show_bug.cgi?id=5154
>
> Not sure if above two bug FIXES (in between v5.7 to v6.6) are related to
> my issue.
>
> I ran netstat:
>
> # netstat -ntlp
> Active Internet connections (only servers)
> Proto Recv-Q Send-Q Local Address Foreign Address
> State PID/Program name
> ...
> tcp6 33 0 :::3137 :::* LISTEN -
> tcp6 0 0 :::3136 :::* LISTEN -
> tcp6 4 0 :::3128 :::* LISTEN -
> tcp6 0 0 :::8081 :::* LISTEN -
> tcp6 0 0 :::8080 :::* LISTEN -
> tcp6 0 0 :::8083 :::* LISTEN -
> tcp6 0 0 :::8082 :::* LISTEN -
> tcp6 0 0 :::8084 :::* LISTEN -
> tcp6 4097 0 :::8093 :::* LISTEN -
> tcp6 0 0 :::8092 :::* LISTEN -
> tcp6 0 0 :::8094 :::* LISTEN -
> ...
>
> I do not have IPv6 enabled, yet there are 33 and 4097 numbers in Recv-Q
> and also no process/PID owns these ports.
>
> Same IPv4 ports are not shown in use by netstat, so only IPv6 ports
> remain open, that too orphaned!
>
> So what is happening?
>
> Any idea to solve or any workaround?
>
> Thank you,
>
> Amish.
>
> _______________________________________________
> squid-users mailing list
> squid-users at lists.squid-cache.org
> https://lists.squid-cache.org/listinfo/squid-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20231219/1e3b14d2/attachment-0001.htm>
More information about the squid-users
mailing list