[squid-users] [squid ] externalAclLookup: 'wbinfo_group_helper' queue overload.

Amos Jeffries squid3 at treenet.co.nz
Tue Apr 21 03:45:57 UTC 2015


On 20/04/2015 7:31 p.m., Jagannath Naidu wrote:
> Hi,
> 
> I am having this issue very frequently. Please help on this.
> 
> I get these errors randomly, mostly when usage is at very peak. (800 users)
> 
> 
> /var/log/squid/cache.log
> 
> 2015/04/20 12:37:40| externalAclLookup: 'wbinfo_group_helper' queue
> overload (ch=0x7fc99e2ce518)

What do you think "overload" means?
 The helper is unable to cope with the traffic load being passed to it.

Here is the biggest hint:
>
> in /var/log/messages,  I get the following errors
>
> pr 20 12:59:15 GGNPROXY01 winbindd[1910]:   winbindd: Exceeding 200 client
> connections, no idle connection found




> Then squid stops working. For squid to start work again, I have to dlete
> the cache and restart the squid "squid -k reconfigure", and then squid
> restart.

What Squid version are you using?

> 
> squid.conf
> 
> max_filedesc 17192
> acl manager proto cache_object
> acl localhost src 172.16.50.61/24

You have an entire /24 (256 IPs) assigned to this machine?

I think you need to remove that "/24" part if the *.61 is the local
machines *public* IP.


> http_access allow manager localhost
> dns_nameservers 172.16.3.34 10.1.2.91
> acl allowips src 172.16.58.187 172.16.16.192 172.16.58.113 172.16.58.63
> 172.16.58.98 172.16.60.244 172.16.58.165 172.16.58.157
> http_access allow allowips

> auth_param basic realm Squid proxy-caching web server
> auth_param basic credentialsttl 2 hours external_acl_type nt_group ttl=0
> children=60 %LOGIN /usr/lib64/squid/wbinfo_group.pl

The above two very mangled config lines are useless. Remove them.

> acl localnet src 172.16.0.0/24

Its a bit strange that none of the localhost machine IPs
(172.16.50.0-172.16.50.255) are part of the LAN its plugged into
172.16.0.0-172.16.0.255.


> acl localnet src fc00::/7 # RFC 4193 local private network range
> acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged) machines
> auth_param ntlm program /usr/bin/ntlm_auth --diagnostics
> --helper-protocol=squid-2.5-ntlmssp --domain=HTMEDIA.NET

Okay you have configured NTLM...

> auth_param ntlm program /usr/bin/ntlm_auth
> --helper-protocol=squid-2.5-ntlmssp --domain=HTMEDIA.NET

... but twice. With different settings. Only these last ones will have
any effect.


> auth_param ntlm children 600
> auth_param ntlm keep_alive off

> auth_param negotiate children 150
> auth_param negotiate keep_alive off
> visible_hostname GGNPROXY01.HTMEDIA.NET
> external_acl_type wbinfo_group_helper ttl=0 children=40 %LOGIN
> /usr/lib64/squid/wbinfo_group.pl -d
> auth_param negotiate keep_alive off

You have several useless configuration lines for Negotiate auth which is
not being used in any way. Remove those.


> acl Safe_ports port 8080 #https
> acl SSL_ports port 443
> acl Safe_ports port 80          # http
> acl Safe_ports port 21          # ftp
> acl Safe_ports port 443 # https
> acl Safe_ports port 70          # gopher
> acl Safe_ports port 210         # wais
> acl Safe_ports port 1025-65535  # unregistered ports
> acl Safe_ports port 280         # http-mgmt
> acl Safe_ports port 488         # gss-http
> acl Safe_ports port 591         # filemaker
> acl Safe_ports port 777         # multiling http
> acl CONNECT method CONNECT
> acl auth proxy_auth REQUIRED
> acl google dstdomain -i "/etc/squid/google_site.com"
> http_access allow google
> acl sq1 external wbinfo_group_helper "/etc/squid/HT/sq1"
> acl sq2 external wbinfo_group_helper "/etc/squid/HT/sq2"
> acl sq3 external wbinfo_group_helper "/etc/squid/HT/sq3"
> acl sq4 external wbinfo_group_helper "/etc/squid/HT/sq4"
> acl sq5 external wbinfo_group_helper "/etc/squid/HT/sq5"
> acl pro1 external wbinfo_group_helper "/etc/squid/HT/pro1"
> acl pro2 external wbinfo_group_helper "/etc/squid/HT/pro2"
> acl pro3 external wbinfo_group_helper "/etc/squid/HT/pro3"
> acl pro4 external wbinfo_group_helper "/etc/squid/HT/pro4"
> acl pro5 external wbinfo_group_helper "/etc/squid/HT/pro5"
> acl pro6 external wbinfo_group_helper "/etc/squid/HT/pro6"
> acl webvip external wbinfo_group_helper "/etc/squid/HT/webvip"
> acl allgroup external wbinfo_group_helper "/etc/squid/HT/allgreop"
> acl restricted external wbinfo_group_helper "/etc/squid/HT/restricted"
> acl ad_auth proxy_auth REQUIRE

You already have an ACL named "auth" which performs authentication.
The above line is not useful. Remove it and replace all uses of
"ad_auth" ACL with "auth" ACL.

> acl allowwebsites dstdomain -i "/blacklists/allowedwebsite/domains"
> acl allowwebsites_url url_regex -i "/blacklists/allowedwebsite/url"
> http_access allow allowwebsites
> http_access allow allowwebsites_url
> acl shopping dstdomain -i "/etc/squid/shopping.txt"
> acl social_networking dstdomain -i "/blacklists/social/social.networking"
> acl youtube dstdomain -i .youtube.com
> http_access allow Safe_ports pro1 pro2 pro3 pro4 pro5 pro6 webvip

Incorrect use of "Safe_ports" security check. Correct usage is to deny
access to all *unsafe* ports. They are unsafe because HTTP can be
smuggled within the ports native protocol to attack your proxy.

Once the correct security protections for Safe_port and CONNECT tunnels
have been moved up the top remove the "Safe_ports" check from this line.

This line is also very odd in another way. ACL tests in a single line
are AND'ed together - so this means the request must be from a user who is:
  authenticated AND a member of group pro1 AND pro2 AND pro3 AND pro4
AND pro5 AND pro6 AND webvip

This hints at what your main helper problem is. The above line requires
7 group helper lookups *per request*. The winbind helper has a maximum
of 200 simultaneous connections. This line alone will limit your proxy
just under 30 new visitors per second (that becomes 60 lookups/sec
before queue overload).
 The helper result caching will help a lot, but you also have a LOT of
other group checks being made and 800 users.


> http_access allow youtube pro5
> http_access allow youtube pro6
> http_access allow youtube webvip
> http_access deny youtube
> http_access allow shopping pro5
> http_access allow shopping pro6
> http_access allow shopping webvip
> http_access deny shopping

Optimization hint:
 "youtube" and "shopping" have the same allow/deny criteria. It would be
worth combining them into one ACL.

> http_access allow social_networking pro2
> http_access allow social_networking pro4
> http_access allow social_networking pro6
> http_access allow social_networking webvip
> http_access deny social_networking
> acl porn_site1   dstdomain "/etc/squid/blacklists/porn/domains.txt"
> acl porn_site2   dstdom_regex -i "/etc/squid/blacklists/porn/expressions"
> acl porn_site3   dstdom_regex -i "/etc/squid/blacklists/porn/urls.txt"
> acl audio_video1   dstdomain "/etc/squid/blacklists/audio-video/urls.txt"
> ###################### THERE ARE TOO MANY acls and http_access , so not
> bothering with vast linux

I will bet a lot of those ACLs are also calling the group helper too yes?

> http_access allow liquorinfo webvip
> http_access deny liquorinfo
> http_access allow ad_auth
> http_access allow auth

Once you have removed ad_auth ACL, this becomes:
 http_access allow auth
 http_access allow auth

I hope you can see how redundant that is.

Also, its very likely that the "allow auth" is a useless operation after
a great many group checks have also performed authentication. That "TOO
MANY acls and https_access" list you omitted will be needed to determine
that.


> http_access allow sq1 sq2
> acl NTLMUsers proxy_auth REQUIRED

You already have an ACL named "auth" which performs authentication.
The above line is not being used in any way. Remove it.

> http_access deny !Safe_ports
> http_access deny CONNECT !SSL_ports

These are basic security protection against Denial of Service and other
types of protocol smuggling attacks. They only work when they are used
*above* your custom "allow" rules.

Move these two lines above your "http_access allow google" line.



> http_port 8080
> hierarchy_stoplist cgi-bin ?

The above line is not useful these days. Remove it.

> cache_effective_user squid
> cache_dir aufs /var/spool/squid 20384 32 512
> cache_mem 50 MB
> cache_replacement_policy heap LFUDA
> cache_swap_low 85
> cache_swap_high 95
> maximum_object_size 5 MB
> maximum_object_size_in_memory 50 KB
> ipcache_size 5240
> ipcache_low 90
> ipcache_high 95
> cache_mgr amit
> cachemgr_passwd ****

I hope that was not your real cachemgr password you just published on a
public mailing list.


> acl SSL_ports port 443

The above is a duplicate config line. Remove it.

> http_access allow CONNECT SSL_ports
> coredump_dir /var/spool/squid
> refresh_pattern ^ftp:           1440    20%     10080
> refresh_pattern ^gopher:        1440    0%      1440
> refresh_pattern -i (/cgi-bin/|\?) 0     0%      0
> refresh_pattern .               0       20%     4320
> url_rewrite_program /usr/local/bin/squidGuard -c
> /usr/local/squidGuard/squidGuard.conf
> 


Now, as to solving your problem:

1) Clean up your config. Reduce the amount of redundant or unused
things. I've mentioned a few above.

2) Run "squid -k parse" and fix any other problems it highlights.

3) optimize your ACls and http_access rules. I've mentioned a few, such
as moving the main security checks to the top so DoS traffic does not
put load on the helpers and other ACLs.

I believe though that you will probably find Squid works much better
having the following access controls pattern:
"
 http_access deny !Safe_ports
 http_access deny CONNECT !SSL_ports

 # if they are not authenticated, they will not be in a group
 http_access deny !auth

 # assuming that webvip are the group with full access?
 http_access allow webvip

 # your long list of per-site group check ACLs go here
 ...

 # this is where defining the LAN ranges correctly comes in.
 # note that users have authenticated simply to get near here
 http_access allow localnet
 http_access deny all
"


4) consider an upgrade to Squid 3.4+. The "notes" ACL type offers much
more efficient ACL testing with a custom group lookup helper. The all-of
and any-of ACL types can also much reduce your http_access lines.

HTH
Amos


More information about the squid-users mailing list