[squid-dev] [RFC] Happy Eyeballs: Parallel TCP connection attempts

[Alex Rousskov]

Hello,

    Since r15240, Squid forwarding code gets a concurrently growing list
of destinations to forward the request to, but it only tries to
establish a connection to one destination at a time, going to the next
one only if the previous connection attempt has failed. Many admins
disable IPv6 because that previous (IPv6) attempt may fail very "slowly"
while it is possible to "quickly" establish a matching (IPv4) connection
instead. Users complain, and IPv6 gets turned off.

To combat this IPv6 adoption barrier, Happy Eyeballs (RFC 6555) suggests
to start establishing a second TCP connection if the first one appears
to be stuck while trying to connect(2). For example, if an IPv6
connect(2) does not succeed in 300ms, Chrome starts an IPv4 connect(2).
A "dumb" implementation of that algorithm is problematic for a proxy
because it can lead to excessive number of connections (in various
states) and might even resemble traffic amplification attacks in extreme
cases.

I propose to implement the following algorithm that follows the Happy
Eyeballs principles while limiting "environmental impact" of opening and
discarding lots of proxy-to-server connections:


* Bootstrapping:

When a new destination is received by FwdState::noteDestination():

0. If there are no in-use destinations, then proceed to use the new
destination as usual. No changes here.

1. If there is one in-use destination, then run the "Spare Connect"
algorithm described below. This is new.

2. If there are two in-use destinations, then add the new destination to
the list as usual. No changes here.

There is no item #3 because there cannot be more than two in-use
destinations at any given time.


* Spare Connect:

If/when allowed (see below), start a second or "spare" TCP connect()
attempt to race against the already in-progress attempt. The first
connect() to succeed wins. The loser is immediately discarded (its
socket is closed) and treated as connect() failure. For example, its
destination is marked as "bad" in the IP cache to prevent subsequent
slow connect() attempts.

Here are the preconditions for the "If/when allowed" part:

0. Both the IP address used by the in-progress connect() and the new
destination IP originate from the same domain name[0] but their address
families differ.

1. The other in-use destination is still in the "TCP connecting" state.
This algorithm works around IP connectivity problems, not slow HTTP(S)
servers.

2. Squid can open new connections and the total number of in-progress
spare connect() attempts (across all same-worker FwdStates) is below the
happy_eyeballs_connect_limit[1]. Please note that this precondition
changes as other spare connect() attempts end. However, I believe we can
ignore (i.e., do not monitor/wait for) such changes for the initial
implementation, for simplicity sake.

3. The last spare connect() attempt (across all same-worker FwdStates)
started at least happy_eyeballs_connect_gap[2] milliseconds ago. Please
note that this precondition changes with time. However, I believe we can
ignore (i.e., do not monitor/wait for) such changes for the initial
implementation, for simplicity sake.

4. The in-progress connect() attempt (for this FwdState object) started
at least happy_eyeballs_connect_timeout[3] milliseconds ago. Please note
that this precondition changes with time. FwdState may need to schedule
a timeout to comply (and re-evaluate preconditions #1-3 after that
timeout). The destination of the spare connect() is considered to be "in
use" while we wait for this timeout to expire.

Here are all the configurable checks and their intended purpose:

  happy_eyeballs_connect_limit: Do not consume too much over time.
  happy_eyeballs_connect_gap: Do not amplify flash crowds (rate limit).
  happy_eyeballs_connect_timeout: Do not try to fix fast-enough cases.


Did I miss any preconditions? Any better ideas?


Footnotes:

[0] The "same domain name" restriction ensures that the "preferred"
destination choices (e.g., cache_peer foo) are still always tried before
the "secondary" ones (e.g., DIRECT).

[1,2,3] The exact names, structure, and the defaults of the new
squid.conf directive(s) are to be determined. Suggestions welcomed. And
if we can hard-code something reasonable for some of them, that is even
better for the initial implementation!


Thank you,

Alex.