[squid-users] 3.3.x -> 3.4.x: huge performance regression

[Diego Woitasen]

On Sat, Oct 25, 2014 at 11:20 AM, Alan <lameventanas at gmail.com> wrote:
> I had the same problem when I tried to move from 3.3 to 3.4.  Because
> of this, I had to go back to 3.3.
>
> I don't remember the CPU being stuck at 100%, but it was certainly
> higher, and the Internet browsing experience was slower.
>
> I don't use the NTLM helper, but I do use the Kerberos one, and also a
> couple of custom external acls.
>
> Alan
>
>
> On Fri, Oct 24, 2014 at 9:49 PM, Eliezer Croitoru <eliezer at ngtech.co.il> wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Hey Eugene,
>>
>> I will try to clear things out.
>> Any helper is an external software but each and every one of them
>> answers to a specific logic.
>> external_acl url_rewrite and store_id helpers works in one way while
>> authentication works in another.
>>
>> To make the story short external_acl url_rewrite and store_id helpers
>> acquire a single line from squid via STDIN and responses to his own
>> STDOUT towards squid STDIN.
>> an example would be:
>> 1 http://www.yahoo.com "more things such as requester and others"
>>
>> The helper responses to this with a line such as:
>> 1 ERR
>> or
>> 1 OK "more stuff"
>>
>> The above example applies only to squid 3.4 while in squid 3.3 this is
>> being used only on the external_acl interface.
>>
>> In auth helpers there are couple "stages" that applies on each requests.
>> In external_acl and the similar interfaces the client do not need to
>> send any details for the helper to analyze else then these that exists
>> on each request.
>> Auth helpers has a logic that denies with request for more details.
>> NTLM specifically has overload on the whole process since it can have
>> 3 stages against squid.
>> Kerberous has the benefit of only 2 stages against squid:
>> 1 - request -> denial
>> 2 - request + credentials -> denial or approval
>>
>> Comparing NTLM to external_acl helpers it is very different.
>> But comparing Kerberous to external_acl it's almost similar.
>>
>> If for example NTML + Kerberous + external_acl + others suffers from
>> the same issue while using only one of them at a time it will prove
>> that there is a specific direction to the issue.
>> But if Kerberous and NTLM differ it will prove that there are other
>> things in hand.
>>
>> Another approach to test the issue is to write an ECAP or ICAP helper
>> that will implement the whole authentication logic.
>> This will also prove that the helpers code is broken in 3.4 compared
>> to 3.3.
>>
>> There is also the comparison of basic authentication compared to NTLM
>> + Keberous which is also important.
>>
>> The squid team needs a testing environment for the issue for quite
>> some time.
>>
>> Now that Victor Sudakov has used Kerberous instead of NTLM we might be
>> able to put some efforts and to pinpoint the issue.
>>
>> All The Bests,
>> Eliezer
>>
>> On 10/24/2014 03:18 PM, Eugene M. Zheganin wrote:
>>>> see http://bugs.squid-cache.org/show_bug.cgi?id=3997
>>> There seems to be a large mix of terms, like "external helper" and
>>> "external program". The ntlm_auth, which is external to squid and
>>> part of the samba package, doesnt' change between version
>>> uprgrades, so I don't see how it can be involved (and why replace
>>> it with a fake helper ?). In the same time it's an authentication
>>> helper. The squid_kerb_ldap, also known as
>>> ext_kerberos_ldap_group_acl, is, indeed, an "external acl" helper,
>>> but it has nothing to do with NTLM, as it doesn't use it. My
>>> opinion - this pr is flooded with error messages of almost any
>>> kind, and it's very hard to see the point.
>>>
>>> We need to understand what's happening to squid when it starts to
>>> hit the 100% CPU usage, last time I saw this during the night
>>> without any possible user traffic load.
>>>
>>> Eugene.
>>
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG v1
>>
>> iQEcBAEBAgAGBQJUSkrVAAoJENxnfXtQ8ZQUp8UIAI2MUcnASF98RWQECoKFczS9
>> E7SWOZMjQQLIy80NAId9MdiZ79XPNTF6a1VaGcdP4cAc78XgNqk6NdBF0eFIKSgy
>> 31W48GGt8p0spJ0o8+NVijV9O/2pGcIW8M3psbnYG/fcgAzWq4DLz9Q/tFTtHsIa
>> UzY56BMuSX1+0d2IeZSO2bXxEnOyiyAJKrQ96H/7tlg/9VV5mHOB7Py1b6G+O1YP
>> nU/+vj4GppHYOYwPdqX9MwKfXwHzFPLHGXdafIJT1/Ci1ApEWW+137PMrvG1uvg3
>> pzSF9amhAyBpAjIA1WRgoYb5gtUciK43K0EIXYkmZnmmuw3mEFOv4WWyDApFBgs=
>> =TLyW
>> -----END PGP SIGNATURE-----
>> _______________________________________________
>> squid-users mailing list
>> squid-users at lists.squid-cache.org
>> http://lists.squid-cache.org/listinfo/squid-users
> _______________________________________________
> squid-users mailing list
> squid-users at lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users

Same problem here. New users, only a few users from IT testing it and
CPU usage is really high from time to time.

Switched to basic auth for a few days. Looks like everybody is having
issues with NTLM/SPNEGO.

Keep in touch and we'll fix it :)

Regards,
   Diego

-- 
Diego Woitasen
Infrastructure Developer, DevOps Engineer, Linux and Open Source expert
http://www.woitasen.com.ar