[squid-users] Squid + kerberos, all childrens are busy

[Дмитрий Рукавцов]

>>     >>>>> Hello, i have a problem here :) System - freebsd 10.1, squid 3.5.5 + kerberos (MIT), 50 users total.
>>>     >>>>>
>>>     >>>>> Without any auth my squid works fine, system is not loaded. When i enable Kerberos auth internet slowly goes down and crushing after a while, at logs i see:
>>     >>>>>>
>>     >>>>>> 2015/07/09 11:47:14 kid1| WARNING: All 60/60 negotiateauthenticator processes are busy.
>>     >>>>>> 2015/07/09 11:47:14 kid1| WARNING: 72 pending requests queued
>>>     >>>>>>
>>>     >>>>
>>>     >>>> So 50 users / 60 helpers ... how many requests per second? and how
>>>     >>>> fast/slow is the helper responding?
>>>     >> Could you clarify how I can get value of requests per second and respond?
>>>     >
>>>     >The cachemgr "info" report. From the cachemgr.cgi tool, or "squidclient
>>>     >mgr:info" command line, or
>>>     >http://$visible_hostname:3128/squid-internal-mgr/info
>>>     >
>>>     > Or calculated from a quick count of the access.log lines over a few mins.
>>> 
>>>     ~600 lines per minute,
>>> 
>>> 
>>> 
>>>     >> Debugs show like 3-4 message per second like:
>>>     >>
>>>     >> negotiate_kerberos_auth.cc(783): pid=1456 :2015/07/09 13:26:48| negotiate_kerberos_auth: DEBUG: AF oYGyMIGvoAMKAQChCwYJKoZIgvcSAQICooGaBIGXYIGUBgkqhkiG9xIBAgICAG+BhDCBgaADAgEFoQMCAQ+idTBzoAMCAReibARqY4fSYtg+X4HhiH8dFmWxdn3wxtoKKZzEfUjLYibMoy0XAAWgkSYVXgC7gxO7cgCkOofEqZQhi/GKa4NZqn2dQqOJU/3y4zkPqBP9Ialh//BL5ov03L5BqjgthrbYbrcxJTo57EJIdO8O1g== avialex
>>>     >>
>>>     >> And errors like:
>>>     >> 2015/07/09 13:28:03 kid1| ERROR: Negotiate Authentication validating user. Result: {result=BH, notes={message: received type 1 NTLM token; }}
>>     >>> All my friends get the same error, but their squid is working fine.
>>     >>>
>>
>>
>>Okay, so the traffic arriving at Squid is ~10 req/sec, and the helpers
>>are processing only 4 req/sec successfully.
>>
>>If we assume that also each client connection is attempting one NTLM
>>request before it gets to Kerberos (when it should be doing the
>>opposite). That allows for the helper rejecting 3-4 req/sec.
>>
>>That makes a total of up to 8 req/sec being handled by the helpers.
>>Still leaving 2 req/sec building up in the queue.
>>
>>
>>I see two problems there.
>>
>>Firstly, 3-4 req/sec seems to be a very slow response rate by the
>>helpers. If you can find some way to improve that enough to stop the
>>queue building up your problem should go away.
>>- that might be done by increasing the startup= helpers count (and
>>maximum count)
>>- that might be by improving the helper connectivity speed and access
>>to DNS and the backend AD system.
>>
>>
>>Secondly, that NTLM issue. The only fix for that is to get the client
>>devices configured to try the more secure Kerberos auth first (like they
>>should be doing anyway).
> 
>- that may require disabling NTLM entirely for them.
>>
>>
>>
>>>     >> Don't see anything else
>>>     >>
>>>     >
>>>     >Aha. So your users browsers are sending NTLM auth instead of Kerberos.
>>>     >That is at least one part of the problem. NTLM handshake can take whole
>>     >>seconds and places a lot of extra load on the helpers. To resolve these
>>     >>the users software needs fixing to use Kerberos properly when Negotiate
>> >
>>     >>is offered.
>>     >>
>>     >>The other part is figuring out what amount of helpers is needed to meet
>>     >>the load requirements. With NTLM it is usually several hundred.
>>     >>
>>> 
>> >
>> >When i'm using proxy alone, squid stars 33 childrens, don't recive any NTLM errors, but internet start to lag. So the problem not in the NTLM software. I tryed to start 300 children for my 60 users, but still have huge lags, even when half was free.
>>> 
>>
>>I suggest For 60 users doing 10req/sec I suggest configuring Squid with:
> 
>auth_param negotiate children 500 startup=120 idle=10
>>
>>
>>So what do you think the lag is coming from then?
>>
>>And how are you defining "free" in terms of helpers?
>>
>>Amos
when i started 150/300 children, 
ps -ax | grep negotiate | wc -l 
shows me that only 151 launched but there was lags

So i decided that settings isn't my problem, so only sofware left. So what i did:

I upgraded squid from 3.5.5 to 3.5.6 and changed kerberos realisation from MIT to heimdal and it works perfectly! Only 18 childrens are launched right now, NTLM errors still presents but i already know what soft makes it and will fix it.
Don't know what was wrong with MIT realisation of kerberos on my freebsd server, but heimdal works just fine.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20150714/53900658/attachment-0001.html>