[squid-users] Squid + kerberos, all childrens are busy

[Amos Jeffries]

On 14/07/2015 5:54 a.m., Дмитрий Рукавцов wrote:
> 
> 
> 
> 
> 
>     >>>>> Hello, i have a problem here :) System - freebsd 10.1, squid 3.5.5 + kerberos (MIT), 50 users total.
>     >>>>>
>     >>>>> Without any auth my squid works fine, system is not loaded. When i enable Kerberos auth internet slowly goes down and crushing after a while, at logs i see:
>     >>>>>
>     >>>>> 2015/07/09 11:47:14 kid1| WARNING: All 60/60 negotiateauthenticator processes are busy.
>     >>>>> 2015/07/09 11:47:14 kid1| WARNING: 72 pending requests queued
>     >>>>>
>     >>>>
>     >>>> So 50 users / 60 helpers ... how many requests per second? and how
>     >>>> fast/slow is the helper responding?
>     >> Could you clarify how I can get value of requests per second and respond?
>     >
>     >The cachemgr "info" report. From the cachemgr.cgi tool, or "squidclient
>     >mgr:info" command line, or
>     >http://$visible_hostname:3128/squid-internal-mgr/info
>     >
>     > Or calculated from a quick count of the access.log lines over a few mins.
> 
>     ~600 lines per minute,
> 
> 
> 
>     >> Debugs show like 3-4 message per second like:
>     >>
<snip>
>     >> negotiate_kerberos_auth.cc(783): pid=1456 :2015/07/09 13:26:48| negotiate_kerberos_auth: DEBUG: AF oYGyMIGvoAMKAQChCwYJKoZIgvcSAQICooGaBIGXYIGUBgkqhkiG9xIBAgICAG+BhDCBgaADAgEFoQMCAQ+idTBzoAMCAReibARqY4fSYtg+X4HhiH8dFmWxdn3wxtoKKZzEfUjLYibMoy0XAAWgkSYVXgC7gxO7cgCkOofEqZQhi/GKa4NZqn2dQqOJU/3y4zkPqBP9Ialh//BL5ov03L5BqjgthrbYbrcxJTo57EJIdO8O1g== avialex
>     >>
>     >> And errors like:
>     >> 2015/07/09 13:28:03 kid1| ERROR: Negotiate Authentication validating user. Result: {result=BH, notes={message: received type 1 NTLM token; }}
>     >> All my friends get the same error, but their squid is working fine.
>     >>


Okay, so the traffic arriving at Squid is ~10 req/sec, and the helpers
are processing only 4 req/sec successfully.

If we assume that also each client connection is attempting one NTLM
request before it gets to Kerberos (when it should be doing the
opposite). That allows for the helper rejecting 3-4 req/sec.

That makes a total of up to 8 req/sec being handled by the helpers.
Still leaving 2 req/sec building up in the queue.


I see two problems there.

Firstly, 3-4 req/sec seems to be a very slow response rate by the
helpers. If you can find some way to improve that enough to stop the
queue building up your problem should go away.
 - that might be done by increasing the startup= helpers count (and
maximum count)
 - that might be by improving the helper connectivity speed and access
to DNS and the backend AD system.


Secondly, that NTLM issue. The only fix for that is to get the client
devices configured to try the more secure Kerberos auth first (like they
should be doing anyway).
 - that may require disabling NTLM entirely for them.



>     >> Don't see anything else
>     >>
>     >
>     >Aha. So your users browsers are sending NTLM auth instead of Kerberos.
>     >That is at least one part of the problem. NTLM handshake can take whole
>     >seconds and places a lot of extra load on the helpers. To resolve these
>     >the users software needs fixing to use Kerberos properly when Negotiate
> 
>     >is offered.
>     >
>     >The other part is figuring out what amount of helpers is needed to meet
>     >the load requirements. With NTLM it is usually several hundred.
>     >
> 
> 
> When i'm using proxy alone, squid stars 33 childrens, don't recive any NTLM errors, but internet start to lag. So the problem not in the NTLM software. I tryed to start 300 children for my 60 users, but still have huge lags, even when half was free.
> 

I suggest For 60 users doing 10req/sec I suggest configuring Squid with:
 auth_param negotiate children 500 startup=120 idle=10


So what do you think the lag is coming from then?

And how are you defining "free" in terms of helpers?

Amos