[squid-users] Squid and CPU 100%

[Alex Rousskov]

On 11/07/2017 10:16 AM, Vieri wrote:

> A quick grep at access.log before the issue I reported shows that
> there were 1350 lines during a full minute. So I understand that
> would mean there were 1350 requests during that minute even though
> some of them were denied by squid.conf's policies. So I should
> estimate less than 2 * 1350. I would use that value anyway, and add
> 30% to that. I'd end up with 3510.

You are talking about request rate. To get the number of connections,
you need to account for response time:

    number of concurrent connections = request rate * response time

And then add 30% overhead for idle persistent connections. That 30%
number is just a rough guess of mine. You can possibly do (much) better
if you study your actual Squid stats.


> So, correct me if I'm wrong, but raising max_filedescriptors
> (currently at 32768) won't solve the root cause of the problem.

Even though you have not computed the right kind of value, I do not
expect you to reach a different conclusion when you fix your formula
(unless your response times are huge). With 1350 requests/minute, your
response times would have to be in the order of minutes to get to 32K
open connections!

Please note that mgr:info and other cheap cache manager pages report
request rates (or equivalent). You do not need to count access log lines
to access and monitor that data.


> You mention that maybe another unknown process may suddenly consume
> almost all CPU cycles,

I did not. I was talking about Squid problems, not another process.


> Wouldn't I need to monitor CPU usage by all processes at all times?

This is not my area of expertise, but, ideally, one should monitor
server usage (using appropriate system administration/monitoring tools).


> Also, wouldn't it be useful to check "squidclient
> mgr:filedescriptors" every 10 minutes or so? I have the feeling it's
> steadily growing over time, even when overall CPU usage is low. So
> the second less likely theory may also be a candidate.

To monitor the number of file descriptors, use mgr:info. It is a much
cheaper action than mgr:filedescriptors, which has to list every
descriptor in every worker. Using mgr:filedescriptors is a good idea in
many cases, but using it every minute may have a noticeable negative
performance impact on concurrent user transactions.


> Under which circumstances would "squidclient mgr:filedescriptors"
> show ever growing numbers even on very low CPU and network usage
> (squid seems to be very responsive)?

A lot of factors, including idle connection pool warmup, bots, and bugs,
may lead to a steady increase in the number of idle and/or stuck
connections. Each idle or stuck connection does not consume much CPU or
network resources...


Alex.