[squid-users] Linearly increasing delays in HTTPS proxy CONNECTS / 3.5.20

[Alex Rousskov]

On 8/20/19 9:12 AM, Ilari Laitinen wrote:

> I am experiencing a slowdown between CONNECT requests and corresponding "200 Connection established" responses.

I assume you are not using any SslBump features but please correct me if
I am wrong.


> source —> squid [SYN]
> source <— squid [SYN, ACK]
> source —> squid [ACK]
> source —> squid [CONNECT target:443 HTTP/1.1]
> source <— squid [ACK]
> 
> [Unexpected delay here]
> 
> squid —> target [SYN]
> squid <— target [SYN, ACK]
> squid —> target [ACK]
> source <— squid [HTTP/1.1 200 Connection established]
> source —> squid [ACK]
> 
> [Rest of the connection here without such delays]


> I noticed small but consistent spikes in squid's disk cache usage
> coinciding with the issue at hand. This seemed strange, given there
> was no other traffic during the tests and proxied HTTPS means there's
> nothing to cache (right?).

Correct. To avoid suspecting disks, configure Squid to log to a
RAM-based partition and remove cache_dirs [until you solve the problem].


> The servers are located in a IPv4-only local network. Every outgoing
> request is supposed to be IPv4. The servers do have IPv6 interfaces
> but there is no traffic at all. Squid periodically queries AAAA
> records. Is it possible that new connections get queued while squid
> is busy trying to use IPv6 after receiving the new AAAAs?

If "no traffic at all" means "zero IPv6 packets", then it is not
possible. Otherwise, it is possible (only the latest Squid (i.e. future
v5) does not have this kind of problem).


> I have very little control over the environment. Is dns_v4_first
> worth a try in my scenario?

It is not a reliable solution, but it would not hurt as far as
performance is concerned.


> What should I look into next?

1. Check system logs.

2. Check atop output while the problem is present. If this is a resource
bottleneck, atop may expose it.

3. If there is IPv6 traffic, to eliminate IPv6 as a suspect, you can
disable IPv6 on the box, use Squid built without IPv6 support, or even
use a DNS forwarder that, for example, rejects all AAAA queries. All
these solutions can and should be validated by examining actual IPv6
traffic. And none of them are needed if there is no IPv6 traffic at all.

4. With delays ranging into _seconds_ it should be fairly easy for a
capable Squid developer to figure out what your Squid is doing by
looking at Squid debugging logs. You can post a link to compressed
cache.log here for analysis, but you should first simplify your workload
so that it has CONNECT tunnels and nothing else (if you have not
already) and enable debugging when the problem is present (e.g., use
"squid -k debug" although it is currently better to send the right
signal manually).


> Could setting up "workers N” help, for example?

The answer depends on your definition of "help": Large number of workers
may mask the problem to the point where it no longer bothers you, but I
would not make the setup a lot more complex until you know where the
current bottleneck is. In fact, I would go into the opposite direction
of making the setup as simple as possible!



HTH,

Alex.