[squid-users] [squid ] externalAclLookup: 'wbinfo_group_helper' queue overload.
Amos Jeffries
squid3 at treenet.co.nz
Tue Apr 21 03:45:57 UTC 2015
On 20/04/2015 7:31 p.m., Jagannath Naidu wrote:
> Hi,
>
> I am having this issue very frequently. Please help on this.
>
> I get these errors randomly, mostly when usage is at very peak. (800 users)
>
>
> /var/log/squid/cache.log
>
> 2015/04/20 12:37:40| externalAclLookup: 'wbinfo_group_helper' queue
> overload (ch=0x7fc99e2ce518)
What do you think "overload" means?
The helper is unable to cope with the traffic load being passed to it.
Here is the biggest hint:
>
> in /var/log/messages, I get the following errors
>
> pr 20 12:59:15 GGNPROXY01 winbindd[1910]: winbindd: Exceeding 200 client
> connections, no idle connection found
> Then squid stops working. For squid to start work again, I have to dlete
> the cache and restart the squid "squid -k reconfigure", and then squid
> restart.
What Squid version are you using?
>
> squid.conf
>
> max_filedesc 17192
> acl manager proto cache_object
> acl localhost src 172.16.50.61/24
You have an entire /24 (256 IPs) assigned to this machine?
I think you need to remove that "/24" part if the *.61 is the local
machines *public* IP.
> http_access allow manager localhost
> dns_nameservers 172.16.3.34 10.1.2.91
> acl allowips src 172.16.58.187 172.16.16.192 172.16.58.113 172.16.58.63
> 172.16.58.98 172.16.60.244 172.16.58.165 172.16.58.157
> http_access allow allowips
> auth_param basic realm Squid proxy-caching web server
> auth_param basic credentialsttl 2 hours external_acl_type nt_group ttl=0
> children=60 %LOGIN /usr/lib64/squid/wbinfo_group.pl
The above two very mangled config lines are useless. Remove them.
> acl localnet src 172.16.0.0/24
Its a bit strange that none of the localhost machine IPs
(172.16.50.0-172.16.50.255) are part of the LAN its plugged into
172.16.0.0-172.16.0.255.
> acl localnet src fc00::/7 # RFC 4193 local private network range
> acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged) machines
> auth_param ntlm program /usr/bin/ntlm_auth --diagnostics
> --helper-protocol=squid-2.5-ntlmssp --domain=HTMEDIA.NET
Okay you have configured NTLM...
> auth_param ntlm program /usr/bin/ntlm_auth
> --helper-protocol=squid-2.5-ntlmssp --domain=HTMEDIA.NET
... but twice. With different settings. Only these last ones will have
any effect.
> auth_param ntlm children 600
> auth_param ntlm keep_alive off
> auth_param negotiate children 150
> auth_param negotiate keep_alive off
> visible_hostname GGNPROXY01.HTMEDIA.NET
> external_acl_type wbinfo_group_helper ttl=0 children=40 %LOGIN
> /usr/lib64/squid/wbinfo_group.pl -d
> auth_param negotiate keep_alive off
You have several useless configuration lines for Negotiate auth which is
not being used in any way. Remove those.
> acl Safe_ports port 8080 #https
> acl SSL_ports port 443
> acl Safe_ports port 80 # http
> acl Safe_ports port 21 # ftp
> acl Safe_ports port 443 # https
> acl Safe_ports port 70 # gopher
> acl Safe_ports port 210 # wais
> acl Safe_ports port 1025-65535 # unregistered ports
> acl Safe_ports port 280 # http-mgmt
> acl Safe_ports port 488 # gss-http
> acl Safe_ports port 591 # filemaker
> acl Safe_ports port 777 # multiling http
> acl CONNECT method CONNECT
> acl auth proxy_auth REQUIRED
> acl google dstdomain -i "/etc/squid/google_site.com"
> http_access allow google
> acl sq1 external wbinfo_group_helper "/etc/squid/HT/sq1"
> acl sq2 external wbinfo_group_helper "/etc/squid/HT/sq2"
> acl sq3 external wbinfo_group_helper "/etc/squid/HT/sq3"
> acl sq4 external wbinfo_group_helper "/etc/squid/HT/sq4"
> acl sq5 external wbinfo_group_helper "/etc/squid/HT/sq5"
> acl pro1 external wbinfo_group_helper "/etc/squid/HT/pro1"
> acl pro2 external wbinfo_group_helper "/etc/squid/HT/pro2"
> acl pro3 external wbinfo_group_helper "/etc/squid/HT/pro3"
> acl pro4 external wbinfo_group_helper "/etc/squid/HT/pro4"
> acl pro5 external wbinfo_group_helper "/etc/squid/HT/pro5"
> acl pro6 external wbinfo_group_helper "/etc/squid/HT/pro6"
> acl webvip external wbinfo_group_helper "/etc/squid/HT/webvip"
> acl allgroup external wbinfo_group_helper "/etc/squid/HT/allgreop"
> acl restricted external wbinfo_group_helper "/etc/squid/HT/restricted"
> acl ad_auth proxy_auth REQUIRE
You already have an ACL named "auth" which performs authentication.
The above line is not useful. Remove it and replace all uses of
"ad_auth" ACL with "auth" ACL.
> acl allowwebsites dstdomain -i "/blacklists/allowedwebsite/domains"
> acl allowwebsites_url url_regex -i "/blacklists/allowedwebsite/url"
> http_access allow allowwebsites
> http_access allow allowwebsites_url
> acl shopping dstdomain -i "/etc/squid/shopping.txt"
> acl social_networking dstdomain -i "/blacklists/social/social.networking"
> acl youtube dstdomain -i .youtube.com
> http_access allow Safe_ports pro1 pro2 pro3 pro4 pro5 pro6 webvip
Incorrect use of "Safe_ports" security check. Correct usage is to deny
access to all *unsafe* ports. They are unsafe because HTTP can be
smuggled within the ports native protocol to attack your proxy.
Once the correct security protections for Safe_port and CONNECT tunnels
have been moved up the top remove the "Safe_ports" check from this line.
This line is also very odd in another way. ACL tests in a single line
are AND'ed together - so this means the request must be from a user who is:
authenticated AND a member of group pro1 AND pro2 AND pro3 AND pro4
AND pro5 AND pro6 AND webvip
This hints at what your main helper problem is. The above line requires
7 group helper lookups *per request*. The winbind helper has a maximum
of 200 simultaneous connections. This line alone will limit your proxy
just under 30 new visitors per second (that becomes 60 lookups/sec
before queue overload).
The helper result caching will help a lot, but you also have a LOT of
other group checks being made and 800 users.
> http_access allow youtube pro5
> http_access allow youtube pro6
> http_access allow youtube webvip
> http_access deny youtube
> http_access allow shopping pro5
> http_access allow shopping pro6
> http_access allow shopping webvip
> http_access deny shopping
Optimization hint:
"youtube" and "shopping" have the same allow/deny criteria. It would be
worth combining them into one ACL.
> http_access allow social_networking pro2
> http_access allow social_networking pro4
> http_access allow social_networking pro6
> http_access allow social_networking webvip
> http_access deny social_networking
> acl porn_site1 dstdomain "/etc/squid/blacklists/porn/domains.txt"
> acl porn_site2 dstdom_regex -i "/etc/squid/blacklists/porn/expressions"
> acl porn_site3 dstdom_regex -i "/etc/squid/blacklists/porn/urls.txt"
> acl audio_video1 dstdomain "/etc/squid/blacklists/audio-video/urls.txt"
> ###################### THERE ARE TOO MANY acls and http_access , so not
> bothering with vast linux
I will bet a lot of those ACLs are also calling the group helper too yes?
> http_access allow liquorinfo webvip
> http_access deny liquorinfo
> http_access allow ad_auth
> http_access allow auth
Once you have removed ad_auth ACL, this becomes:
http_access allow auth
http_access allow auth
I hope you can see how redundant that is.
Also, its very likely that the "allow auth" is a useless operation after
a great many group checks have also performed authentication. That "TOO
MANY acls and https_access" list you omitted will be needed to determine
that.
> http_access allow sq1 sq2
> acl NTLMUsers proxy_auth REQUIRED
You already have an ACL named "auth" which performs authentication.
The above line is not being used in any way. Remove it.
> http_access deny !Safe_ports
> http_access deny CONNECT !SSL_ports
These are basic security protection against Denial of Service and other
types of protocol smuggling attacks. They only work when they are used
*above* your custom "allow" rules.
Move these two lines above your "http_access allow google" line.
> http_port 8080
> hierarchy_stoplist cgi-bin ?
The above line is not useful these days. Remove it.
> cache_effective_user squid
> cache_dir aufs /var/spool/squid 20384 32 512
> cache_mem 50 MB
> cache_replacement_policy heap LFUDA
> cache_swap_low 85
> cache_swap_high 95
> maximum_object_size 5 MB
> maximum_object_size_in_memory 50 KB
> ipcache_size 5240
> ipcache_low 90
> ipcache_high 95
> cache_mgr amit
> cachemgr_passwd ****
I hope that was not your real cachemgr password you just published on a
public mailing list.
> acl SSL_ports port 443
The above is a duplicate config line. Remove it.
> http_access allow CONNECT SSL_ports
> coredump_dir /var/spool/squid
> refresh_pattern ^ftp: 1440 20% 10080
> refresh_pattern ^gopher: 1440 0% 1440
> refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
> refresh_pattern . 0 20% 4320
> url_rewrite_program /usr/local/bin/squidGuard -c
> /usr/local/squidGuard/squidGuard.conf
>
Now, as to solving your problem:
1) Clean up your config. Reduce the amount of redundant or unused
things. I've mentioned a few above.
2) Run "squid -k parse" and fix any other problems it highlights.
3) optimize your ACls and http_access rules. I've mentioned a few, such
as moving the main security checks to the top so DoS traffic does not
put load on the helpers and other ACLs.
I believe though that you will probably find Squid works much better
having the following access controls pattern:
"
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
# if they are not authenticated, they will not be in a group
http_access deny !auth
# assuming that webvip are the group with full access?
http_access allow webvip
# your long list of per-site group check ACLs go here
...
# this is where defining the LAN ranges correctly comes in.
# note that users have authenticated simply to get near here
http_access allow localnet
http_access deny all
"
4) consider an upgrade to Squid 3.4+. The "notes" ACL type offers much
more efficient ACL testing with a custom group lookup helper. The all-of
and any-of ACL types can also much reduce your http_access lines.
HTH
Amos
More information about the squid-users
mailing list