[squid-dev] RFC: tls_key_log: report TLS pre-master secrets, other key material

[Alex Rousskov]

Hello,

    I propose to add a new tls_key_log directive to record TLS
pre-master secret (and related encryption details) for to- and
from-Squid TLS connections. This very useful triage feature is common
for browsers and some networking tools. Wireshark supports it[1]. You
might know it as SSLKEYLOGFILE. It has been requested by several Squid
admins. A draft documentation of the proposed directive is at the end of
this email.

[1] https://wiki.wireshark.org/TLS#Using_the_.28Pre.29-Master-Secret

If you have any feature scope adjustments, implementation wishes, or
objections to this feature going in, please let me know!


FTR, we have considered providing similar support by adding new
logformat %code(s) to the existing access_log. While doing so would
reduce and simplify code changes, we ultimately rejected that design
because the combination of the following factors renders it inferior and
insufficient on several levels:

1. Some TLS connections are reflected in access logs a long time
   after they are established (e.g., when the connection serves
   a long CONNECT tunnel). During triage, admins may need the
   ability to decipher a live connection much sooner.

2. Some TLS connections are never reflected in access logs at all
   (e.g., when Squid opens but does not use an outgoing TLS connection
   or crashes when parsing the first request on an incoming one). Info
   gaps often create triage suspicions: Did we drop something else?

3. A single access_log record may correspond to many from-Squid
   connections, especially when Squid retries peer failures. Logging
   keys for all these connections would require accumulating the keys in
   the master transaction and then dumping them as a part of a new
   %code. Adding (and dumping) repeated ALE fields is awkward.

4. Manually creating ACLs to limit access log records to the first
   transaction on a connection would be a must for most deployments
   using this feature. Doing so is far from trivial and related
   configuration errors are difficult to triage. We could add a new
   ACL type for this purpose, but even that is tricky because a
   single master transaction may have many associated connections.
   And logging secrets for every transaction creates too much noise.

5. Configuration flexibility offered by logformat is likely to
   remain largely unused by the new feature because tools like
   Wireshark _automatically_ find relevant records when deciphering
   captured traffic. Augmenting these logs with other transaction
   details (typical for access log uses) would be mostly useless.

6. New %codes would be awkward to use in a regular access log because
   they may expand into a variable number of lines, going against the
   traditional line-oriented, "fixed" format access log use.

While some of the above items have workarounds, a few do not, and the
whole combination looks rather grim/unfriendly. We should attack this
problem from the other end -- a new simple configuration dedicated to
this useful feature.

We propose to structure this new directive so that it is easy to add
advanced access_log-like features later if needed (while reusing the
corresponding access_log code). For example, if users find that they
want to maintain multiple TLS key logs or augment log records with
connection details, we can add that support by borrowing access_log
options and code without backward compatibility concerns. The new
required "if" keyword in front of the ACL list allows for seamless
addition of new directive options in the future.


Cheers,

Alex.
---------- draft squid.conf directive documentation ------------

tls_key_log

Configures whether and where Squid records pre-master secret and
related encryption details for TLS connections accepted or established
by Squid. These connections include connections accepted at
https_port, TLS connections opened to origin servers/cache_peers/ICAP
services, and TLS tunnels bumped by Squid using the SslBump feature.
This log (a.k.a. SSLKEYLOGFILE) is meant for triage with traffic
inspection tools like Wireshark.

    tls_key_log <filename> if <acl>...

WARNING: This log allows anybody to decrypt the corresponding
encrypted TLS connections, both in-flight and postmortem.

At most one log file is supported at this time. Repeated tls_key_log
directives are treated as fatal configuration errors. By default, no
log is created or updated.

If the log file does not exist, Squid creates it. Otherwise, Squid
appends an existing log file.

The directive is consulted whenever a TLS connection is accepted or
established by Squid. TLS connections that fail the handshake may be
logged if Squid got enough information to form a log record. A record
is logged only if all of the configured ACLs match.

Squid does not buffer these log records -- the worker blocks until
each record is written. File system buffering may speed things up, but
consider placing this triage log in a memory-based partition.

This log is rotated based on the logfile_rotate settings.

Logging errors are reported to cache.log but are otherwise ignored.

While transport-related ACLs like src and dst should work, Squid may
not have access to higher-level information. For example, when logging
accepted https_port connections, Squid does not yet have access to the
expected HTTPS request. Similarly, an HTTPS response is not available
when logging most TLS connections established by Squid.

The log record format is meant to be compatible with TLS deciphering
features of Wireshark which include support for TLS v1.3 fields such
as CLIENT_EARLY_TRAFFIC_SECRET and SERVER_TRAFFIC_SECRET_0. A single
log record usually spans multiple lines. Technical documentation for
that format is maintained inside the Wireshark code (e.g., see
tls_keylog_process_lines() comments as of Wireshark commit
e3d44136f0f0026c5e893fa249f458073f3b7328).

This clause only supports fast acl types.
See http://wiki.squid-cache.org/SquidFaq/SquidAcl for details.