[squid-users] I want to know the concerns of load testing

[Amos Jeffries]

On 2/10/20 3:15 pm, m k wrote:
> Hello,
> 
> I'm planning a proxy renewal for a company with 45k clients.
> I'm looking at the performance of a single Squid to determine the number
> of Squids.
> 
> Environment: Virtual (OpenStack)
> OS: CentOS8.1
> CPU: 4 cores
> MEM: 8GB
> DISK: SATA30GB / 100GB

See our notes on relative disk JBOD / RAID performances.
<https://wiki.squid-cache.org/SquidFaq/RAID>


> Squid 4.4

I know it can be hard to get hold of newer packages on CentOS. Please do
try hard to upgrade to the 4.13 release for production. There have been
more than a few critical security issues fixed this past year.


>  SSL Bump
>  Blacklist: 1,700k
>  auth: NTLM

NTLM is a major performance issue. With every request needing to be sent
twice it will essentially halve the traffic your proxy can serve to clients.

I do know that Squid used to be able to handle way more RPS than Windows
DC would like to handle. So the DC may be a bottleneck there.

Negotiate/Kerberos auth is the solution to all those problems. If you
are really interested in good performance avoid NTLM.


>  cache: 4GB
> 
> In an environment with authentication disabled and SSL decoding enabled
> A load test was performed with Jmeter.
> 
> Result: CPU high load (100rps-1000rps: CPU Usage 80-90%)
> (Confirm with top command)
> 

If the proxy is not using 100% of the core(s) it is supposed to be
using. Then you have not reached the capacity limits of the proxy.

What you do about that depends on whether you are trying to find
theoretical limits, or performance for a specific traffic profile.


For a specific traffic profile the measurement is likely hitting disk
I/O or network I/O limits. Double-check which it was - that is what to
change to improve performance.


For theoretical limits the same detail about I/O applies. But also to
max the proxy out fully you may need to tune the test traffic load for
either higher TCP connection concurrency, or to utilize less resource
consuming features. eg requests that will HIT on memory cached (small)
objects and only need simple fast-type ACL checks. Memory-only traffic
is approximately 100x faster than any involving disk I/O.

 To be clear this is to find the theoretical maximum performance. You
cannot tune clients real traffic like this.



> Added multi-core support settings to squid.conf
> "workers 4"
> 
> A load test with Jmeter was performed again.
> 
> Result: CPU load is distributed to 4 cores (CPU Usage 20-40%)
> (Confirm with top command)

See above. That 20% implies the same 80% is spread over 4 cores.


> 
> Question
> 1. 1. How much will CPU Usage increase if NTLM authentication is enabled?

NTLM requires 2 HTTP messages to authenticate every new TCP connection.
So there will be one extra HTTP message on every set of pipelined requests.

It depends on how many requests are pipelined on each TCP connection as
to how much impact that auth overhead is.


After disk I/O capacity the CPU cycles are what limit Squid most. The
RPS achievable is capped out when all CPU cores assigned for Squid reach
100%.


> 2. 2. Are there any concerns other than CPU Usage in Squid?

The usual bottlenecks:

 * disk I/O limits
 * Network latency (DNS in particular. In general, TCP to _everywhere_)
 * features used (CPU drains)
 * memory

The order is my own experience of service impact, YMMV


> 3. 3. When I enabled the cache in this test, the CPU Usage decreased,
> but in general, does the Squid cache increase the CPU Usage?


In general cache should have little effect on CPU. Processing HTTP
headers is by far the major use of CPU cycles in Squid. SSL-Bump is
expected to be a close second, especially if decrypting.

In some cases it can. A large number of small cache objects can consume
many cycles CPU searching for an object. Or Range requests on very large
objects can spend a lot of cycles to generate the Range response HIT
payload.



HTH
Amos