[squid-users] Debugging slow access

[Steve Hill]

On 10.12.14 17:09, Amos Jeffries wrote:

>> I'm looking for advice on figuring out what is causing intermittent
>> high CPU usage.

It appears that the connections gradually gain more and more notes with 
the key "token" (and values containing Kerberos tokens).  I haven't been 
able to reproduce the problem reliably enough to determine if this is 
the root of the high CPU usage problem, but it certainly doesn't look right:

When an ACL is executed that requires the login name (e.g. the 
proxy_auth ACL, or an external ACL using the %LOGIN format specifier), 
Acl.cc:AuthenticateAcl() is called.  This, in turn, calls
UserRequest.cc:tryToAuthenticateAndSetAuthUser(), which calls 
UserRequest.cc:authTryGetUser().  Here we get a call to 
Notes.cc:appendNewOnly() which appends all the notes from 
checklist->auth_user_request->user()->notes.

I can see the appendNewOnly() call sometimes ends up appending a large 
number of "token" notes (I've observed requests with a couple of hundred 
"token" notes attached to them) - the number of notes increases each 
time a Kerberos authentication is performed.  My suspicion is that this 
growth is unbounded and in some cases the number of notes could become 
large enough to be a significant performance hit.

A couple of questions spring to mind:

1. HelperReply.cc:parse() calls notes.add("token",authToken.content()) 
(i.e. it adds a token rather than replacing an existing one).  As far as 
I can tell, Squid only ever uses the first "token" note, so maybe we 
should be removing the old notes when we add a new one?

[Actually, on closer inspection, NotePairs::add() appends to the end of 
the list but NotePairs::findFirst() finds the note closest to the start 
of the list.  Unless I'm missing something, this means the newer "token" 
notes are added but never used?]

2. I'm not sure on how the ACL checklists and User objects are shared 
between connections/requests and how they are supposed to persist.  It 
seems to me that there is something wrong with the sharing/persistence 
if we're accumulating so many "token" notes.  As well as the performance 
problems, there could be some race conditions lurking here?

-- 
  - Steve Hill
    Technical Director
    Opendium Limited     http://www.opendium.com

Direct contacts:
    Instant messager: xmpp:steve at opendium.com
    Email:            steve at opendium.com
    Phone:            sip:steve at opendium.com

Sales / enquiries contacts:
    Email:            sales at opendium.com
    Phone:            +44-1792-824568 / sip:sales at opendium.com

Support contacts:
    Email:            support at opendium.com
    Phone:            +44-1792-825748 / sip:support at opendium.com