[squid-users] 3.3.x -> 3.4.x: huge performance regression

[Eliezer Croitoru]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hey Eugene,

I will try to clear things out.
Any helper is an external software but each and every one of them
answers to a specific logic.
external_acl url_rewrite and store_id helpers works in one way while
authentication works in another.

To make the story short external_acl url_rewrite and store_id helpers
acquire a single line from squid via STDIN and responses to his own
STDOUT towards squid STDIN.
an example would be:
1 http://www.yahoo.com "more things such as requester and others"

The helper responses to this with a line such as:
1 ERR
or
1 OK "more stuff"

The above example applies only to squid 3.4 while in squid 3.3 this is
being used only on the external_acl interface.

In auth helpers there are couple "stages" that applies on each requests.
In external_acl and the similar interfaces the client do not need to
send any details for the helper to analyze else then these that exists
on each request.
Auth helpers has a logic that denies with request for more details.
NTLM specifically has overload on the whole process since it can have
3 stages against squid.
Kerberous has the benefit of only 2 stages against squid:
1 - request -> denial
2 - request + credentials -> denial or approval

Comparing NTLM to external_acl helpers it is very different.
But comparing Kerberous to external_acl it's almost similar.

If for example NTML + Kerberous + external_acl + others suffers from
the same issue while using only one of them at a time it will prove
that there is a specific direction to the issue.
But if Kerberous and NTLM differ it will prove that there are other
things in hand.

Another approach to test the issue is to write an ECAP or ICAP helper
that will implement the whole authentication logic.
This will also prove that the helpers code is broken in 3.4 compared
to 3.3.

There is also the comparison of basic authentication compared to NTLM
+ Keberous which is also important.

The squid team needs a testing environment for the issue for quite
some time.

Now that Victor Sudakov has used Kerberous instead of NTLM we might be
able to put some efforts and to pinpoint the issue.

All The Bests,
Eliezer

On 10/24/2014 03:18 PM, Eugene M. Zheganin wrote:
>> see http://bugs.squid-cache.org/show_bug.cgi?id=3997
> There seems to be a large mix of terms, like "external helper" and 
> "external program". The ntlm_auth, which is external to squid and
> part of the samba package, doesnt' change between version
> uprgrades, so I don't see how it can be involved (and why replace
> it with a fake helper ?). In the same time it's an authentication
> helper. The squid_kerb_ldap, also known as
> ext_kerberos_ldap_group_acl, is, indeed, an "external acl" helper,
> but it has nothing to do with NTLM, as it doesn't use it. My 
> opinion - this pr is flooded with error messages of almost any
> kind, and it's very hard to see the point.
> 
> We need to understand what's happening to squid when it starts to
> hit the 100% CPU usage, last time I saw this during the night
> without any possible user traffic load.
> 
> Eugene.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBAgAGBQJUSkrVAAoJENxnfXtQ8ZQUp8UIAI2MUcnASF98RWQECoKFczS9
E7SWOZMjQQLIy80NAId9MdiZ79XPNTF6a1VaGcdP4cAc78XgNqk6NdBF0eFIKSgy
31W48GGt8p0spJ0o8+NVijV9O/2pGcIW8M3psbnYG/fcgAzWq4DLz9Q/tFTtHsIa
UzY56BMuSX1+0d2IeZSO2bXxEnOyiyAJKrQ96H/7tlg/9VV5mHOB7Py1b6G+O1YP
nU/+vj4GppHYOYwPdqX9MwKfXwHzFPLHGXdafIJT1/Ci1ApEWW+137PMrvG1uvg3
pzSF9amhAyBpAjIA1WRgoYb5gtUciK43K0EIXYkmZnmmuw3mEFOv4WWyDApFBgs=
=TLyW
-----END PGP SIGNATURE-----