[squid-users] squid 3.5.2==> HTTPS FATAL: The ssl_crtd helpers are crashing too rapidly, need help!

Fri Mar 3 13:17:59 UTC 2017

Lets hope for the best.
When you will be with 3.5.24 we would be able to handle things in a simpler manner.
There are couple options to handle a situation like this and one of the options is to fence the ssl_crtd with some kind of lock mechanism for the DB rebuild time.
For example a bash script that will run the ssl_crtd and will try to trace if there is an exit code else then 0.
If such a case exists it will check for some lock file.
If it doesn't exit it will create it and will send a healing request to another daemon or will start the self healing by itself.
There are pros and cons but I believe that 3.5.24 will help more then many other options.

All The Bests,
Eliezer

----
http://ngtech.co.il/lmgtfy/
Linux System Administrator
Mobile: +972-5-28704261
Email: eliezer at ngtech.co.il

From: --Ahmad-- [mailto:ahmed.zaeem at netstream.ps] 
Sent: Friday, March 3, 2017 2:20 PM
To: Yuri Voinov <yvoinov at gmail.com>
Cc: Eliezer Croitoru <eliezer at ngtech.co.il>; squid-users at lists.squid-cache.org
Subject: Re: [squid-users] squid 3.5.2==> HTTPS FATAL: The ssl_crtd helpers are crashing too rapidly, need help!

@ eliezer 
i was using children as 10
ans faced the problem 

so i trued to increase children to 1000 to see if this was the reason 
and unfortunately the same problem .

ys I’m using debian 6 os .

i appreciable the helping from all the replies below but so far i havent got any clear solution .

now i updated to 3.5.24 last one .
and will see it if comes back …i will update the list with result .

if it failed … I’m forced to create cron job to remove the certs like every 24 hours .

thank you  guys all of you .
thanks amos , thanks eliezer , thanks yuri

kind regards
On Mar 3, 2017, at 1:37 PM, Yuri Voinov <mailto:yvoinov at gmail.com> wrote:

03.03.2017 6:32, Eliezer Croitoru пишет:

Hey Yuri,

This issue is not 100% squid but I think it's related to the way ssl_crtd works.
I am not sure if it has some locking or other things to prevent such issues.
The first solution is to somehow defend the DB from corruption, like in a case that more then a dozen identical requests are being done towards a single site and two ssl_crtd helpers are trying to do the same things.
I believe that something to fence this should already be inside squid and ssl_crtd but I am pretty sure this is the main issue.
I suggests this can be external reason to occurs this issue. Somehow,
for example, BlueCoat on ISP upstream, tcp packets corruption, etc. I
dont know, just guessing.

Alex and his team should know the answer for this subject and if I'm not wrong theoretically there are couple ways to prevent the mentioned issues.
I had a plan to try and understand the ssl_crtd code and interface but yet to do so.

I hope this issue will be resolved in a way that it can be backported to 3.5 in the worst case.
I hope too, but if it external..... fewwwwwwwwwww.

Anyway, watchdog is good backup to preventing manual interventions by SA.

Eliezer

----
http://ngtech.co.il/lmgtfy/
Linux System Administrator
Mobile: +972-5-28704261
Email: mailto:eliezer at ngtech.co.il

From: squid-users [mailto:squid-users-bounces at lists.squid-cache.org] On Behalf Of Yuri Voinov
Sent: Thursday, March 2, 2017 11:46 PM
To: mailto:squid-users at lists.squid-cache.org
Subject: Re: [squid-users] squid 3.5.2==> HTTPS FATAL: The ssl_crtd helpers are crashing too rapidly, need help!

This problem, in principle, is common to all versions of ssl-bumped Squid from version 3.4 and 5.0, inclusive, and occurs when the stored certificate is damaged for any reason. The only thing vorkeraund that I could find - a monitor kesh.log and initialize the certificate database again with squid restart automatically.
In some installations, this problem does not occur over the years. In other - almost daily. I have no desire to find out why this is happening exactly. For me it was easier to make the watchdog, which will follow up on this.
03.03.2017 3:40, Yuri Voinov пишет:
One hint finally:
'([^ ]*) helper database ([^ ]*) failed: The SSL certificate database ([^ ]*) is corrupted. Please rebuild' - - - 0    exec "/usr/local/bin/crtd_create.sh -r >/dev/null 2>&1"
'FATAL: ([^ ]*) helpers are crashing too rapidly, need help!' - - - 0    exec "/usr/local/bin/crtd_create.sh -r >/dev/null 2>&1"
'Cannot add certificate to db.' - - - 0        exec "/usr/local/bin/crtd_create.sh -r >/dev/null 2>&1"
PS. This is from logsurfer.conf.

03.03.2017 3:34, Yuri Voinov пишет:
This error is usually preceded by another error in cache.log associated with the certificates.
I will show you the direction. Then go himself.
This software will useful for you to solve:
http://www.crypt.gen.nz/logsurfer/
HTH, Yuri

03.03.2017 2:47, --Ahmad-- пишет:
hey folks . 
i have a problem with squid it get crashed after i enabled https !
cache log error => FATAL: The ssl_crtd helpers are crashing too rapidly, need help!

i googled many topics and relevant pages and couldnt find a clear solution .

the quick solution i made was i  removed the certs in file :
rm -rfv /var/lib/ssl_db/

then reinitiated the DB using cmd below :
/lib/squid/ssl_crtd -c -s /var/lib/ssl_db
chown -R squid.squid /var/lib/ssl_db
chown -R squid.squid /var/lib/ssl_db

the restarted squid .

but this is not a solution becuase squid get crashed again after certain time and i don’t know why !
my version is 3.5.2

here is squid.conf :
/etc/squid/squid.conf
visible_hostname pcloud
acl ip1 myip 10.1.0.1
acl ip2 myip 192.168.10.210
tcp_outgoing_address 192.168.10.210 ip1
tcp_outgoing_address 192.168.10.210 ip2
#
# Recommended minimum configuration:
#

# Example rule allowing access from your local networks.
# Adapt to list your (internal) IP networks from where browsing
# should be allowed
acl localnet src 10.0.0.0/8     # RFC1918 possible internal network
acl localnet src 172.16.0.0/12  # RFC1918 possible internal network
acl localnet src 192.168.0.0/16 # RFC1918 possible internal network
acl localnet src fc00::/7       # RFC 4193 local private network range
acl localnet src fe80::/10      # RFC 4291 link-local (directly plugged) machines

acl SSL_ports port 443
acl Safe_ports port 80          # http
acl Safe_ports port 21          # ftp
acl Safe_ports port 443         # https
acl Safe_ports port 70          # gopher
acl Safe_ports port 210         # wais
acl Safe_ports port 1025-65535  # unregistered ports
acl Safe_ports port 280         # http-mgmt
acl Safe_ports port 488         # gss-http
acl Safe_ports port 591         # filemaker
acl Safe_ports port 777         # multiling http
acl CONNECT method CONNECT

#
# Recommended minimum Access Permission configuration:
#
# Deny requests to certain unsafe ports
http_access deny !Safe_ports

# Deny CONNECT to other than secure SSL ports
http_access deny CONNECT !SSL_ports
http_access allow  CONNECT 
# Only allow cachemgr access from localhost
http_access allow localhost manager
http_access deny manager

# We strongly recommend the following be uncommented to protect innocent
# web applications running on the proxy server who think the only
# one who can access services on "localhost" is a local user
#http_access deny to_localhost

#
# INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS
#

# Example rule allowing access from your local networks.
# Adapt localnet in the ACL section to list your (internal) IP networks
# from where browsing should be allowed
http_access allow localnet
http_access allow localhost

# And finally deny all other access to this proxy
http_access deny all

# Squid normally listens to port 3128
http_port 3128

# Uncomment and adjust the following to add a disk cache directory.
#cache_dir ufs /var/cache/squid 100 16 256

# Leave coredumps in the first cache dir
#coredump_dir /var/cache/squid

#
# Add any of your own refresh_pattern entries above these.
#
#

http_port 3126
#http_port 3128
#######################################
#cache_swap_low 90
#cache_swap_high 95
############################
cache_effective_user squid
cache_effective_group squid
memory_replacement_policy lru
cache_replacement_policy heap LFUDA
########################
maximum_object_size 10000 MB
#cache_mem 5000 MB
maximum_object_size_in_memory 10 MB
#########################
logfile_rotate 2
max_filedescriptors 131072
###############################
############
cache_dir aufs /var/cache/squid 600000 64 128
#######################################
https_port 3129 intercept ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=4MB cert=/usr/local/squid/ssl_cert/myca.pem key=/usr/local/squid/ssl_cert/myca.pem
ssl_bump server-first all
sslcrtd_program /lib/squid/ssl_crtd -s /var/lib/ssl_db -M 4MB
sslcrtd_children 1000 startup=1 idle=1
###
minimum_object_size 0 bytes
#refresh patterns for caching static files
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern -i .(gif|png|jpg|jpeg|ico)$ 10080 90% 43200 override-expire ignore-no-cache ignore-no-store ignore-private
refresh_pattern -i .(iso|avi|wav|mp3|mp4|mpeg|swf|flv|x-flv)$ 43200 90% 432000 override-expire ignore-no-cache ignore-no-store ignore-private
refresh_pattern -i .(deb|rpm|exe|zip|tar|tgz|ram|rar|bin|ppt|doc|tiff)$ 10080 90% 43200 override-expire ignore-no-cache ignore-no-store ignore-private
refresh_pattern -i .index.(html|htm)$ 0 40% 10080
refresh_pattern -i .(html|htm|css|js)$ 1440 40% 40320
refresh_pattern . 0 40% 40320

any Joy Guys ?

should i update squid ? or downgrade squid ?

kind regards 

_______________________________________________
squid-users mailing list
mailto:squid-users at lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

-- 
Bugs to the Future
<0x613DEC46.asc>_______________________________________________
squid-users mailing list
mailto:squid-users at lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users