[squid-users] Debugging slow access

[Steve Hill]

I'm looking for advice on figuring out what is causing intermittent high 
CPU usage.

I'm seeing this on multiple servers - most of the time everything is 
fine and I see the Squid workers using maybe 20% CPU each, but every so 
often all the workers sit at the top of the process list in "top", using 
 > 97% CPU each and users report very sluggish web access.

Using squidclient during "sluggish" periods is also very slow, with 
Squid taking several seconds to respond to the http requests.  The 
number of requests being handled by squid during the slow periods isn't 
especially high (maybe ~20 / second) and is certainly lower than the 
number of requests at other times - probably because it is taking so 
long to answer requests, but this seems to indicate that it isn't simply 
overloaded and having to deal with too many requests at once.

The during the "slow" periods, squid's servicing of requests seems very 
bursty in nature - I see a whole bunch of requests over a few hundred 
milliseconds and then nothing for maybe half a second.  There are no log 
entries that seem to coincide with these problems.

If I firewall off the clients, the load drops back to zero, so it seems 
this is something a client is doing that is causing Squid to expend a 
huge amount of CPU handling the request, rather than Squid getting stuck 
in a loop or similar.

Restarting squid seems to temporarily fix the problem, but it invariably 
comes back again at some point.

Notably the median service time go up:
	HTTP Requests (All):   0.30178  0.40454
	Cache Misses:          0.70906  0.65348
	Cache Hits:            0.00000  0.00000
	Near Hits:             0.00000  0.00000
	Not-Modified Replies:  0.00000  0.00000
	DNS Lookups:           0.02893  0.03092
	ICP Queries:           0.00000  0.00000

	UP Time:	11657.399 seconds
	CPU Time:	8843.268 seconds
	CPU Usage:	111.23%
	CPU Usage, 5 minute avg:	144.81%
	CPU Usage, 60 minute avg:	153.58%
	Maximum Resident Size: 2937536 KB
	Page faults with physical i/o: 3


Compared to (recently restarted):
	HTTP Requests (All):   0.09477  0.09477
	Cache Misses:          0.11465  0.11465
	Cache Hits:            0.00000  0.00000
	Near Hits:             0.00000  0.00000
	Not-Modified Replies:  0.00000  0.00000
	DNS Lookups:           0.00953  0.00953
	ICP Queries:           0.00000  0.00000

	UP Time:	293.336 seconds
	CPU Time:	127.775 seconds
	CPU Usage:	43.56%
	CPU Usage, 5 minute avg:	47.40%
	CPU Usage, 60 minute avg:	47.40%
	Maximum Resident Size: 799808 KB
	Page faults with physical i/o: 0


Is there any advice on how to track down what the problem is?

This Squid is doing:
  - No caching
  - ICAP
  - External ACLs
  - Auth (Negotiate and Basic)
  - SSL bump
  - Both TPROXY and non-transparent (majority of the traffic is 
non-transparent)
  - Uses an upstream proxy for most HTTP (not HTTPS)

-- 
  - Steve Hill
    Technical Director
    Opendium Limited     http://www.opendium.com

Direct contacts:
    Instant messager: xmpp:steve at opendium.com
    Email:            steve at opendium.com
    Phone:            sip:steve at opendium.com

Sales / enquiries contacts:
    Email:            sales at opendium.com
    Phone:            +44-1792-824568 / sip:sales at opendium.com

Support contacts:
    Email:            support at opendium.com
    Phone:            +44-1792-825748 / sip:support at opendium.com