[squid-users] Wrong req_header result in cache_peer_access when using ssl_bump

[Amos Jeffries]

On 18/07/2016 10:23 p.m., Mihai Ene wrote:
> Hello,
> 
> I have created a gist with the relevant parts of `cache.log`
> (post-connection)
> https://gist.github.com/randunel/5c0d282c52e9135aa21b8c6e28925a52
> 
> The following logs are available:
> 
> 1. The initial HTTP CONNECT requests on port :8000 on line 51
> https://gist.github.com/randunel/5c0d282c52e9135aa21b8c6e28925a52#file-3-cache-log-L51
> 

Between lines 1 and line 462;

Your Squid is receiving an HTTP CONNECT tunnel request from curl on FD
12. Your custom header exists on that CONNECT request.

Your access controls determine that:
 a) this tunnel is *not* to be bumped (by http_port lacking ssl-bump
option).
 b) that a direct outgoing server TCP connection is required (by
always_direct allow),

So Squid is required to *decapsulate* the CONNECT request and send the
data inside it to upstream server.

At this point there is an opaque payload in the CONNECT tunnel arriving
from client and being delivered over an opaque TCP connection to the
server. There is no HTTP messaging over that server connection as far as
Squid is concerned, just opaque bytes of data.

==> Squid is not able to attach your custom HTTP header when there is no
outgoing HTTP message.


> 2. The mark 0x10 is set on line 435
> https://gist.github.com/randunel/5c0d282c52e9135aa21b8c6e28925a52#file-3-cache-log-L435
> 
> 3. The redirected HTTPS request on port :8443 comes in, line 467
> https://gist.github.com/randunel/5c0d282c52e9135aa21b8c6e28925a52#file-3-cache-log-L467
> 

By "redirected" you mean NAT'ed.

At line 466 your OS NAT configuration folds the outgoing opaque tunnel
connection back into Squid.

As far as Squid is concerned this is a *new* HTTPS connection on the
intercept port using FD 18. It is important to be aware that there is
*no* information about your custom header from the previous CONNECT. A
new inbound connection is a clean slate.


> 4. The forwarded CONNECT request is on line 526
> https://gist.github.com/randunel/5c0d282c52e9135aa21b8c6e28925a52#file-3-cache-log-L526
> 

No. Between line 467 and 628;

Squid is processing the fake CONNECT request representing the raw TCP
connection IP:port details. To see if the connection is to be rejected
immeditely, ssl-bump'ed or bypassed, etc.

==> Since the request is completely internal to Squid, on a fresh new
TCP connection, it does not contain any of the curl sent headers.

==> Since it is of course not an *outgoing* message. It does not have
any request_header_add custom alterations.

At line 611 your http_access rules fail to match (all of them). So the
implicit "allow all" happens to this pseudo-CONNECT.

at 628 ssl_bump indicates a need to peek at the TLS traffic.


between line 642 and 655; Squid is shoveling some of the opaque data
between its FD 12 and FD 16 connections.

Things then start interleaving between the inbound FD 18 connection and
the FD 12 <-> FD 16 opaque tunnel data transfer. I will ignore that
opaque shovelling from now in the between line X and Y statements.


between line 655 and 907 SSL-Bump is doing its peek thing with the TLS
now arriving on FD 18. At 907 it decides to bump using only the client
Hello details.


> 5. X-My-Header is *NOT* found on line 966
> https://gist.github.com/randunel/5c0d282c52e9135aa21b8c6e28925a52#file-3-cache-log-L966
> 

between lines 908 and 1043;

Squid is doing its outbound server destination selection to figure out
where this TLS client request is going to go to.

Note that at this point the only state Squid has associated with this
transaction is the TCP connections details, that fake-CONNECT request
(updated to contain the TLS SNI - if any), and the TLS clientHello details.
 Any rules that you have regarding HTTP state is using that fake-CONNECT
still.

At line 1044 Squid determines that there are no permitted ways to
contact any server and complete the TLS bumping.

Between lines 1045 and 1372; Squid is performing the SSL-Bump process on
the client connection so it can deliver the HTTP error message about
that server connectivity problem.

At line 1373 it begins receiving from FD 18 the HTTPS message that curl
sent inside the tunnel way back on FD 12.

Your custom header is visible on that curl request. But that is
irrelevant by now, Squid is delivering its message about no server being
permitted to be used for the SSL-Bump server connection. That response
gets delivered at line 1652.


> 5. The bumped contents are on line 1575
> https://gist.github.com/randunel/5c0d282c52e9135aa21b8c6e28925a52#file-3-cache-log-L1575
> , where X-My-Header is visible
> 
> 
> The following inconsistencies are seen:
> 
> a. The X-My-Header is *not* added to the CONNECT request according to
> config
> https://gist.github.com/randunel/5c0d282c52e9135aa21b8c6e28925a52#file-1-squid-conf-L19

There is never any outbound CONNECT request to send that header on.

> b. X-My-Header NOT found on line 966, although it is clearly visible on
> line 1579
> 

Your header is visible on every request where it can reasonably be
expected to exist.


> -----
> 
> The reason I was asking earlier whether this is the expected behaviour or
> not, was because I was wondering whether the ACLs apply to the CONNECT
> request contents, or to the ssl_bump contents.

The answer is yes. However they are not that problem, both of those you
refer to have the header and it is detected.

It is the side effects of NAT which is causing the headache. The
fake-CONNECT and lack of support for it in your squid.conf rules.


> 
> Is there a way to determine the cache_peer based on ssl bumped contents?

Yes, but that is not your problem.

> 
> If not, is there a way to add X-My-Header to the CONNECT request, as a
> workaround to ssl bumped contents not following any ACLs?

The way you are doing it is correct and will be applied on any CONNECT
request which leaves Squid to a cache_peer.

Your problem is that the CONNECT you receive from curl is being
tunneled, then redirected using NAT.

That process discards your header and any other state about the client
TCP connection that would have been useful during the http_access and
ssl_bump evaluation for bumping the TLS traffic.


> 
> If not, is there a way to set the cache_peer based on the headers of the
> bumped request?
> 
> If there's nothing I can do about this, is there a way to set the
> ssl_bumped cache_peer based on the earlier proxy_auth username?
> 

Start by adding the ssl-bump option on your http_port line which will
resolve this case of traffic event.

Properly NAT'ed client connections arriving on your intercept port will
still have the problem. So you will need to remove the never_direct rule
preventing server connections being used by the bumping process.

Amos