[squid-users] [SPAM] Squid stops serving requests after squid -k reconfigure

Wed Mar 31 17:36:14 UTC 2021

Hey Acid,

I created as an example scripts the next:

https://github.com/elico/squid-external-matchers

The bash script is the pseudo and the ruby are production grade services.

I have use these services in production for a very long time and it works great.

Auto reloads the files as you append the content/domains/urls/sni.

All The Bests,

Eliezer

----

Eliezer Croitoru

Tech Support

Mobile: +972-5-28704261

Email: ngtech1ltd at gmail.com <mailto:ngtech1ltd at gmail.com> 

Zoom: Coming soon

From: NgTech LTD <ngtech1ltd at gmail.com> 
Sent: Tuesday, March 30, 2021 11:08 PM
To: acidflash acidflash <acidflash_ at linuxmail.org>
Cc: Squid Users <squid-users at lists.squid-cache.org>
Subject: Re: [squid-users] [SPAM] Squid stops serving requests after squid -k reconfigure

Hey Acid,

First you should try 4.x latest.

a reload of 50 domains every 10 second doesn't make sense.

I don't understand the config and the setup.

for 50 sites you just need a basic script and even one in bash will work dor you with grep.

I wrote an example in ruby a while ago, I will try to find it in the next week.

Maybe the server is overloaded.

I will try to give you an example later on.

Eliezer

בתאריך יום ג׳, 30 במרץ 2021, 20:41, מאת acidflash acidflash ‏<acidflash_ at linuxmail.org <mailto:acidflash_ at linuxmail.org> >:

Hi Eliezer,

I hope your doing ok. Thanks for the reply. Yeah currently what I am doing is:

include /etc/squid/blockedsites.list
and adding the ACL's and the denies in the list file. What version do you recommend I upgrade to, and is this a known issue? The list is actually pretty small, probably no more than 50 sites or so, and thats split across 4 or 5 groups (different ACL's). I'll look into ufdbguard and the other projects as well, does this sound familiar though? If you think that the best path forward is to alleviate the burden off of squid to some external tool, I could probably think up a few hacks to for that, but would obviously prefer to keep it all within squid. Is this occurance common with squid -k reconfigure and dstdomain matching? Thanks for your time. Stay safe. 

Sent: Sunday, March 28, 2021 at 4:42 AM
From: "Eliezer Croitoru" <ngtech1ltd at gmail.com <mailto:ngtech1ltd at gmail.com> >
To: squid-users at lists.squid-cache.org <mailto:squid-users at lists.squid-cache.org> 
Subject: Re: [squid-users] [SPAM] Squid stops serving requests after squid -k reconfigure

Hey Acid,

Haven’t seen you here for a very long time.

The first thing is to upgrade squid if possible…

It’s better that you won’t use squid -kreconf for big blacklists.

Instead you should use some external software to match the blacklists.

The most recommended software these days is ufdbguard.

Depends on the size of your blacklist your might need to find the right solution.

The best solution would be to store the list in ram somehow.

Have you tried some kind of rbl server?

At the time I wrote some code to and some of it was merged into:

https://github.com/looterz/grimd

It has a reload url so you can update the files on disk and send a reload.

Another service I am using is:

https://github.com/andybalholm/redwood

Which has a “Classification Service” function.

It’s pretty easy to write a json http client that can run queries against this classification service.

Also you’d better use a file In the dstdomain ac ie:

acl Blacklist dstdomain “/var/blacklists/xyx.list”

http_access deny Blacklist

and inside the xyx.list file just add lines of domains like

.blacklisted-domain.com <http://blacklisted-domain.com> 

.example.com <http://example.com> 

Etc..

All The Bests,

Eliezer

----

Eliezer Croitoru

Tech Support

Mobile: +972-5-28704261

Email: ngtech1ltd at gmail.com <mailto:ngtech1ltd at gmail.com> 

Zoom: Coming soon

From: squid-users <squid-users-bounces at lists.squid-cache.org <mailto:squid-users-bounces at lists.squid-cache.org> > On Behalf Of acidflash acidflash
Sent: Saturday, March 27, 2021 10:55 AM
To: squid-users at lists.squid-cache.org <mailto:squid-users at lists.squid-cache.org> 
Subject: [SPAM] [squid-users] Squid stops serving requests after squid -k reconfigure

I have gone through the forums, and I haven't found an answer to the question, although it has been asked more than once.

I am running squid 3.5.X on Centos 7, the compile options are:
"configure options:  '--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu' '--program-prefix=' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64' '--libexecdir=/usr/libexec' '--sharedstatedir=/var/lib' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--disable-strict-error-checking' '--exec_prefix=/usr' '--libexecdir=/usr/lib64/squid' '--localstatedir=/var' '--datadir=/usr/share/squid' '--sysconfdir=/etc/squid' '--with-logdir=$(localstatedir)/log/squid' '--with-pidfile=$(localstatedir)/run/squid.pid' '--disable-dependency-tracking' '--enable-eui' '--enable-follow-x-forwarded-for' '--enable-auth' '--enable-auth-basic=DB,LDAP,MSNT-multi-domain,NCSA,NIS,PAM,POP3,RADIUS,SASL,SMB,SMB_LM,getpwnam' '--enable-auth-ntlm=smb_lm,fake' '--enable-auth-digest=file,LDAP,eDirectory' '--enable-auth-negotiate=kerberos' '--enable-external-acl-helpers=file_userip,LDAP_group,time_quota,session,unix_group,wbinfo_group,kerberos_ldap_group' '--enable-cache-digests' '--enable-cachemgr-hostname=localhost' '--enable-delay-pools' '--enable-epoll' '--enable-ident-lookups' '--enable-linux-netfilter' '--enable-removal-policies=heap,lru' '--enable-snmp' '--enable-ssl-crtd' '--enable-storeio=aufs,diskd,rock,ufs' '--enable-wccpv2' '--enable-esi' '--enable-ecap' '--with-aio' '--with-default-user=squid' '--with-dl' '--with-openssl' '--with-pthreads' '--disable-arch-native' 'build_alias=x86_64-redhat-linux-gnu' 'host_alias=x86_64-redhat-linux-gnu' 'CFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches   -m64 -mtune=generic -fpie' 'LDFLAGS=-Wl,-z,relro  -pie -Wl,-z,relro -Wl,-z,now' 'CXXFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches   -m64 -mtune=generic -fpie' 'PKG_CONFIG_PATH=:/usr/lib64/pkgconfig:/usr/share/pkgconfig'"

I have a service which adds domains to a blacklist file, and then calls squid -k reconfigure.
Instead of writing to the file, this service updates the file completely with new rules,
by deleting the old file, and creating a new one in its place, and then calling squid -k reconfigure.
After doing this, on odd occasions, squid will stop serving traffic completely,
until you do a squid stop, and squid start. After shutting down squid,
and starting squid up with the same rules, squid will continue to work normally.
Its probably worth mentioning that during the time that these events are taking place,
the server is under quite a bit of load, and clients don't stop sending requests via the server.
What these directives look like:

acl Porn dstdomain .xnxx.com <http://xnxx.com>  .sex.com <http://sex.com>  
acl Drugs dstdomain .drugs.com <http://drugs.com>  .silkroad.eu <http://silkroad.eu>  
http_access deny Porn
http_access deny Drugs

This also seems to be amplified when there are several squid workers (child processes) running.
In regards to order, these ACL's are above any other ACL's in the list. We have a very basic squid conf file that looks like this:
## START OF FILE
http_port 3128
cache deny all
#
access_log /var/log/squid/access.log 
cache_store_log none
cache_log /dev/null
logfile_rotate 4
#
auth_param basic program /usr/lib64/squid/basic_db_auth --dsn "DBI:mysql:host=XX.XX.XX.XX;port=XXXX;database=XXXXX" --user XXXXXX --password XXXXXXXX --plaintext --persist
# 
acl db-auth proxy_auth REQUIRED
auth_param basic credentialsttl 2 hours
auth_param basic casesensitive on
#
connect_timeout 55 minutes
#
request_header_access Allow allow all
request_header_access Authorization allow all
request_header_access WWW-Authenticate allow all
request_header_access Proxy-Authorization allow all
request_header_access Proxy-Authenticate allow all
request_header_access Cache-Control allow all
request_header_access Content-Encoding allow all
request_header_access Content-Length allow all
request_header_access Content-Type allow all
request_header_access Date allow all
request_header_access Expires allow all
request_header_access Host allow all
request_header_access If-Modified-Since allow all
request_header_access Last-Modified allow all
request_header_access Location allow all
request_header_access Pragma allow all
request_header_access Accept allow all
request_header_access Accept-Charset allow all
request_header_access Accept-Encoding allow all
request_header_access Accept-Language allow all
request_header_access Content-Language allow all
request_header_access Mime-Version allow all
request_header_access Retry-After allow all
request_header_access Title allow all
request_header_access Connection allow all
request_header_access Proxy-Connection allow all
request_header_access User-Agent allow all
request_header_access Cookie allow all
request_header_access All deny all
dns_v4_first on
via off
forwarded_for off
follow_x_forwarded_for deny all
dns_nameservers 8.8.8.8 8.8.4.4
## END OF FILE

Your help is greatly appreciated, maybe there has been some insight into this issue after 10+ years since the last time it was asked.

_______________________________________________ squid-users mailing list squid-users at lists.squid-cache.org <mailto:squid-users at lists.squid-cache.org>  http://lists.squid-cache.org/listinfo/squid-users

_______________________________________________
squid-users mailing list
squid-users at lists.squid-cache.org <mailto:squid-users at lists.squid-cache.org> 
http://lists.squid-cache.org/listinfo/squid-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20210331/c872f363/attachment-0001.htm>