[squid-users] [squid ] externalAclLookup: 'wbinfo_group_helper' queue overload.
Jagannath Naidu
jagannath.naidu at fosteringlinux.com
Thu Apr 23 09:41:09 UTC 2015
Hi Amos,
regrets, I am late.
On 21 April 2015 at 09:15, Amos Jeffries <squid3 at treenet.co.nz> wrote:
> On 20/04/2015 7:31 p.m., Jagannath Naidu wrote:
> > Hi,
> >
> > I am having this issue very frequently. Please help on this.
> >
> > I get these errors randomly, mostly when usage is at very peak. (800
> users)
> >
> >
> > /var/log/squid/cache.log
> >
> > 2015/04/20 12:37:40| externalAclLookup: 'wbinfo_group_helper' queue
> > overload (ch=0x7fc99e2ce518)
>
> What do you think "overload" means?
> The helper is unable to cope with the traffic load being passed to it.
>
> Here is the biggest hint:
> >
> > in /var/log/messages, I get the following errors
> >
> > pr 20 12:59:15 GGNPROXY01 winbindd[1910]: winbindd: Exceeding 200
> client
> > connections, no idle connection found
>
>
>
>
> > Then squid stops working. For squid to start work again, I have to dlete
> > the cache and restart the squid "squid -k reconfigure", and then squid
> > restart.
>
> What Squid version are you using?
>
> my squid version squid-3.1.10-19.el6_4.x86_64
> >
> > squid.conf
> >
> > max_filedesc 17192
> > acl manager proto cache_object
> > acl localhost src 172.16.50.61/24
>
> changed to "acl localhost src 172.16.50.6*1*" already
> You have an entire /24 (256 IPs) assigned to this machine?
>
> I think you need to remove that "/24" part if the *.61 is the local
> machines *public* IP.
>
>
> > http_access allow manager localhost
> > dns_nameservers 172.16.3.34 10.1.2.91
> > acl allowips src 172.16.58.187 172.16.16.192 172.16.58.113 172.16.58.63
> > 172.16.58.98 172.16.60.244 172.16.58.165 172.16.58.157
> > http_access allow allowips
>
> > auth_param basic realm Squid proxy-caching web server
> > auth_param basic credentialsttl 2 hours external_acl_type nt_group ttl=0
> > children=60 %LOGIN /usr/lib64/squid/wbinfo_group.pl
>
> The above two very mangled config lines are useless. Remove them.
>
> > acl localnet src 172.16.0.0/24
>
changed
> Its a bit strange that none of the localhost machine IPs
> (172.16.50.0-172.16.50.255) are part of the LAN its plugged into
> 172.16.0.0-172.16.0.255.
>
>
> > acl localnet src fc00::/7 # RFC 4193 local private network range
> > acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged)
> machines
> > auth_param ntlm program /usr/bin/ntlm_auth --diagnostics
> > --helper-protocol=squid-2.5-ntlmssp --domain=HTMEDIA.NET
>
> Okay you have configured NTLM...
>
> > auth_param ntlm program /usr/bin/ntlm_auth
> > --helper-protocol=squid-2.5-ntlmssp --domain=HTMEDIA.NET
>
> ... but twice. With different settings. Only these last ones will have
> any effect.
>
>
> > auth_param ntlm children 600
> > auth_param ntlm keep_alive off
>
> > auth_param negotiate children 150
> > auth_param negotiate keep_alive off
> > visible_hostname GGNPROXY01.HTMEDIA.NET
> > external_acl_type wbinfo_group_helper ttl=0 children=40 %LOGIN
> > /usr/lib64/squid/wbinfo_group.pl -d
> > auth_param negotiate keep_alive off
>
> You have several useless configuration lines for Negotiate auth which is
> not being used in any way. Remove those.
>
>
> > acl Safe_ports port 8080 #https
> > acl SSL_ports port 443
> > acl Safe_ports port 80 # http
> > acl Safe_ports port 21 # ftp
> > acl Safe_ports port 443 # https
> > acl Safe_ports port 70 # gopher
> > acl Safe_ports port 210 # wais
> > acl Safe_ports port 1025-65535 # unregistered ports
> > acl Safe_ports port 280 # http-mgmt
> > acl Safe_ports port 488 # gss-http
> > acl Safe_ports port 591 # filemaker
> > acl Safe_ports port 777 # multiling http
> > acl CONNECT method CONNECT
> > acl auth proxy_auth REQUIRED
> > acl google dstdomain -i "/etc/squid/google_site.com"
> > http_access allow google
> > acl sq1 external wbinfo_group_helper "/etc/squid/HT/sq1"
> > acl sq2 external wbinfo_group_helper "/etc/squid/HT/sq2"
> > acl sq3 external wbinfo_group_helper "/etc/squid/HT/sq3"
> > acl sq4 external wbinfo_group_helper "/etc/squid/HT/sq4"
> > acl sq5 external wbinfo_group_helper "/etc/squid/HT/sq5"
> > acl pro1 external wbinfo_group_helper "/etc/squid/HT/pro1"
> > acl pro2 external wbinfo_group_helper "/etc/squid/HT/pro2"
> > acl pro3 external wbinfo_group_helper "/etc/squid/HT/pro3"
> > acl pro4 external wbinfo_group_helper "/etc/squid/HT/pro4"
> > acl pro5 external wbinfo_group_helper "/etc/squid/HT/pro5"
> > acl pro6 external wbinfo_group_helper "/etc/squid/HT/pro6"
> > acl webvip external wbinfo_group_helper "/etc/squid/HT/webvip"
> > acl allgroup external wbinfo_group_helper "/etc/squid/HT/allgreop"
> > acl restricted external wbinfo_group_helper "/etc/squid/HT/restricted"
> > acl ad_auth proxy_auth REQUIRE
>
> You already have an ACL named "auth" which performs authentication.
> The above line is not useful. Remove it and replace all uses of
> "ad_auth" ACL with "auth" ACL.
>
> > acl allowwebsites dstdomain -i "/blacklists/allowedwebsite/domains"
> > acl allowwebsites_url url_regex -i "/blacklists/allowedwebsite/url"
> > http_access allow allowwebsites
> > http_access allow allowwebsites_url
> > acl shopping dstdomain -i "/etc/squid/shopping.txt"
> > acl social_networking dstdomain -i "/blacklists/social/social.networking"
> > acl youtube dstdomain -i .youtube.com
> > http_access allow Safe_ports pro1 pro2 pro3 pro4 pro5 pro6 webvip
>
> Incorrect use of "Safe_ports" security check. Correct usage is to deny
> access to all *unsafe* ports. They are unsafe because HTTP can be
> smuggled within the ports native protocol to attack your proxy.
>
> Once the correct security protections for Safe_port and CONNECT tunnels
> have been moved up the top remove the "Safe_ports" check from this line.
>
> This line is also very odd in another way. ACL tests in a single line
> are AND'ed together - so this means the request must be from a user who is:
> authenticated AND a member of group pro1 AND pro2 AND pro3 AND pro4
> AND pro5 AND pro6 AND webvip
>
> This hints at what your main helper problem is. The above line requires
> 7 group helper lookups *per request*. The winbind helper has a maximum
> of 200 simultaneous connections. This line alone will limit your proxy
> just under 30 new visitors per second (that becomes 60 lookups/sec
> before queue overload).
> The helper result caching will help a lot, but you also have a LOT of
> other group checks being made and 800 users.
>
>
> > http_access allow youtube pro5
> > http_access allow youtube pro6
> > http_access allow youtube webvip
> > http_access deny youtube
> > http_access allow shopping pro5
> > http_access allow shopping pro6
> > http_access allow shopping webvip
> > http_access deny shopping
>
> Optimization hint:
> "youtube" and "shopping" have the same allow/deny criteria. It would be
> worth combining them into one ACL.
>
> > http_access allow social_networking pro2
> > http_access allow social_networking pro4
> > http_access allow social_networking pro6
> > http_access allow social_networking webvip
> > http_access deny social_networking
> > acl porn_site1 dstdomain "/etc/squid/blacklists/porn/domains.txt"
> > acl porn_site2 dstdom_regex -i "/etc/squid/blacklists/porn/expressions"
> > acl porn_site3 dstdom_regex -i "/etc/squid/blacklists/porn/urls.txt"
> > acl audio_video1 dstdomain "/etc/squid/blacklists/audio-video/urls.txt"
> > ###################### THERE ARE TOO MANY acls and http_access , so not
> > bothering with vast linux
>
> I will bet a lot of those ACLs are also calling the group helper too yes?
>
> > http_access allow liquorinfo webvip
> > http_access deny liquorinfo
> > http_access allow ad_auth
> > http_access allow auth
>
> Once you have removed ad_auth ACL, this becomes:
> http_access allow auth
> http_access allow auth
>
> I hope you can see how redundant that is.
>
> Also, its very likely that the "allow auth" is a useless operation after
> a great many group checks have also performed authentication. That "TOO
> MANY acls and https_access" list you omitted will be needed to determine
> that.
>
>
> > http_access allow sq1 sq2
> > acl NTLMUsers proxy_auth REQUIRED
>
> You already have an ACL named "auth" which performs authentication.
> The above line is not being used in any way. Remove it.
>
> > http_access deny !Safe_ports
> > http_access deny CONNECT !SSL_ports
>
> These are basic security protection against Denial of Service and other
> types of protocol smuggling attacks. They only work when they are used
> *above* your custom "allow" rules.
>
> Move these two lines above your "http_access allow google" line.
>
>
>
> > http_port 8080
> > hierarchy_stoplist cgi-bin ?
>
> The above line is not useful these days. Remove it.
>
> > cache_effective_user squid
> > cache_dir aufs /var/spool/squid 20384 32 512
> > cache_mem 50 MB
> > cache_replacement_policy heap LFUDA
> > cache_swap_low 85
> > cache_swap_high 95
> > maximum_object_size 5 MB
> > maximum_object_size_in_memory 50 KB
> > ipcache_size 5240
> > ipcache_low 90
> > ipcache_high 95
> > cache_mgr amit
> > acl SSL_ports port 443
>
> The above is a duplicate config line. Remove it.
>
> > http_access allow CONNECT SSL_ports
> > coredump_dir /var/spool/squid
> > refresh_pattern ^ftp: 1440 20% 10080
> > refresh_pattern ^gopher: 1440 0% 1440
> > refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
> > refresh_pattern . 0 20% 4320
> > url_rewrite_program /usr/local/bin/squidGuard -c
> > /usr/local/squidGuard/squidGuard.conf
> >
>
>
> Now, as to solving your problem:
>
> 1) Clean up your config. Reduce the amount of redundant or unused
> things. I've mentioned a few above.
>
> 2) Run "squid -k parse" and fix any other problems it highlights.
>
> 3) optimize your ACls and http_access rules. I've mentioned a few, such
> as moving the main security checks to the top so DoS traffic does not
> put load on the helpers and other ACLs.
>
> I believe though that you will probably find Squid works much better
> having the following access controls pattern:
> "
> http_access deny !Safe_ports
> http_access deny CONNECT !SSL_ports
>
> # if they are not authenticated, they will not be in a group
> http_access deny !auth
>
> # assuming that webvip are the group with full access?
> http_access allow webvip
>
> # your long list of per-site group check ACLs go here
> ...
>
> # this is where defining the LAN ranges correctly comes in.
> # note that users have authenticated simply to get near here
> http_access allow localnet
> http_access deny all
> "
>
>
> 4) consider an upgrade to Squid 3.4+. The "notes" ACL type offers much
> more efficient ACL testing with a custom group lookup helper. The all-of
> and any-of ACL types can also much reduce your http_access lines.
>
> HTH
> Amos
> _______________________________________________
> squid-users mailing list
> squid-users at lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users
>
Thank you Amos, I will check and will update the list.
--
Thanks & Regards
B Jagannath
Keen & Able Computers Pvt. Ltd.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20150423/4e7744c9/attachment-0001.html>
More information about the squid-users
mailing list