[squid-users] Wrong req_header result in cache_peer_access when using ssl_bump

Mon Jul 18 15:19:46 UTC 2016

Your details helped me understand a lot better.

It turns out squid correctly adds the header to the CONNECT request, when
that request is made to another proxy. It cannot be itself, unfortunately,
because then it complains about a loop.

Also unfortunately, your suggestion of doing `ssl-bump` on the http port
doesn't work because the squid process terminates with a failed assertion
when using cache_peer, it seems to be this bug
http://bugs.squid-cache.org/show_bug.cgi?id=3963 , which I get during with
my squid 3.5.20 `2016/07/18 15:07:50.566| assertion failed:
PeerConnector.cc:116: "peer->use_ssl"`.

Config used:

```
http_port 8000 ssl-bump generate-host-certificates=on
dynamic_cert_mem_cache_size=16MB cert=/etc/squid/ca.crt
key=/etc/squid/ca.key dhparams=/etc/squid/dh2048.pem options=NO_SSLv3

sslcrtd_program /usr/lib/squid/ssl_crtd -s /var/lib/squid_ssl_db -M 32MB
sslcrtd_children 32
acl step1 at_step SslBump1
ssl_bump peek step1
ssl_bump bump all

never_direct allow all

cache_peer 192.71.64.174 parent 6745 0 no-query no-digest default

http_access allow all
```

Considering the fact that I can't do `ssl-bump` on http port because of the
`peer-use_ssl` assertion (bug linked above), also considering the fact that
squid :8000 using itself as a proxy :8443 complains about a proxy loop, are
there any other options I might have to use ssl_bump *with* multiple
cache_peer, and cache_peer selection based on proxy_auth and/or req_header?

*Mihai Ene*
Software Developer

*UB | Your universal basket*

http://ub.io
me at ub.io
@shop_ub
+44 (0)7473 804972 <+447473804972>

On Mon, Jul 18, 2016 at 2:03 PM, Amos Jeffries <squid3 at treenet.co.nz> wrote:

> On 18/07/2016 10:23 p.m., Mihai Ene wrote:
> > Hello,
> >
> > I have created a gist with the relevant parts of `cache.log`
> > (post-connection)
> > https://gist.github.com/randunel/5c0d282c52e9135aa21b8c6e28925a52
> >
> > The following logs are available:
> >
> > 1. The initial HTTP CONNECT requests on port :8000 on line 51
> >
> https://gist.github.com/randunel/5c0d282c52e9135aa21b8c6e28925a52#file-3-cache-log-L51
> >
>
> Between lines 1 and line 462;
>
> Your Squid is receiving an HTTP CONNECT tunnel request from curl on FD
> 12. Your custom header exists on that CONNECT request.
>
> Your access controls determine that:
>  a) this tunnel is *not* to be bumped (by http_port lacking ssl-bump
> option).
>  b) that a direct outgoing server TCP connection is required (by
> always_direct allow),
>
> So Squid is required to *decapsulate* the CONNECT request and send the
> data inside it to upstream server.
>
> At this point there is an opaque payload in the CONNECT tunnel arriving
> from client and being delivered over an opaque TCP connection to the
> server. There is no HTTP messaging over that server connection as far as
> Squid is concerned, just opaque bytes of data.
>
> ==> Squid is not able to attach your custom HTTP header when there is no
> outgoing HTTP message.
>
>
> > 2. The mark 0x10 is set on line 435
> >
> https://gist.github.com/randunel/5c0d282c52e9135aa21b8c6e28925a52#file-3-cache-log-L435
> >
> > 3. The redirected HTTPS request on port :8443 comes in, line 467
> >
> https://gist.github.com/randunel/5c0d282c52e9135aa21b8c6e28925a52#file-3-cache-log-L467
> >
>
> By "redirected" you mean NAT'ed.
>
> At line 466 your OS NAT configuration folds the outgoing opaque tunnel
> connection back into Squid.
>
> As far as Squid is concerned this is a *new* HTTPS connection on the
> intercept port using FD 18. It is important to be aware that there is
> *no* information about your custom header from the previous CONNECT. A
> new inbound connection is a clean slate.
>
>
> > 4. The forwarded CONNECT request is on line 526
> >
> https://gist.github.com/randunel/5c0d282c52e9135aa21b8c6e28925a52#file-3-cache-log-L526
> >
>
> No. Between line 467 and 628;
>
> Squid is processing the fake CONNECT request representing the raw TCP
> connection IP:port details. To see if the connection is to be rejected
> immeditely, ssl-bump'ed or bypassed, etc.
>
> ==> Since the request is completely internal to Squid, on a fresh new
> TCP connection, it does not contain any of the curl sent headers.
>
> ==> Since it is of course not an *outgoing* message. It does not have
> any request_header_add custom alterations.
>
> At line 611 your http_access rules fail to match (all of them). So the
> implicit "allow all" happens to this pseudo-CONNECT.
>
> at 628 ssl_bump indicates a need to peek at the TLS traffic.
>
>
> between line 642 and 655; Squid is shoveling some of the opaque data
> between its FD 12 and FD 16 connections.
>
> Things then start interleaving between the inbound FD 18 connection and
> the FD 12 <-> FD 16 opaque tunnel data transfer. I will ignore that
> opaque shovelling from now in the between line X and Y statements.
>
>
> between line 655 and 907 SSL-Bump is doing its peek thing with the TLS
> now arriving on FD 18. At 907 it decides to bump using only the client
> Hello details.
>
>
> > 5. X-My-Header is *NOT* found on line 966
> >
> https://gist.github.com/randunel/5c0d282c52e9135aa21b8c6e28925a52#file-3-cache-log-L966
> >
>
> between lines 908 and 1043;
>
> Squid is doing its outbound server destination selection to figure out
> where this TLS client request is going to go to.
>
> Note that at this point the only state Squid has associated with this
> transaction is the TCP connections details, that fake-CONNECT request
> (updated to contain the TLS SNI - if any), and the TLS clientHello details.
>  Any rules that you have regarding HTTP state is using that fake-CONNECT
> still.
>
> At line 1044 Squid determines that there are no permitted ways to
> contact any server and complete the TLS bumping.
>
> Between lines 1045 and 1372; Squid is performing the SSL-Bump process on
> the client connection so it can deliver the HTTP error message about
> that server connectivity problem.
>
> At line 1373 it begins receiving from FD 18 the HTTPS message that curl
> sent inside the tunnel way back on FD 12.
>
> Your custom header is visible on that curl request. But that is
> irrelevant by now, Squid is delivering its message about no server being
> permitted to be used for the SSL-Bump server connection. That response
> gets delivered at line 1652.
>
>
> > 5. The bumped contents are on line 1575
> >
> https://gist.github.com/randunel/5c0d282c52e9135aa21b8c6e28925a52#file-3-cache-log-L1575
> > , where X-My-Header is visible
> >
> >
> > The following inconsistencies are seen:
> >
> > a. The X-My-Header is *not* added to the CONNECT request according to
> > config
> >
> https://gist.github.com/randunel/5c0d282c52e9135aa21b8c6e28925a52#file-1-squid-conf-L19
>
> There is never any outbound CONNECT request to send that header on.
>
> > b. X-My-Header NOT found on line 966, although it is clearly visible on
> > line 1579
> >
>
> Your header is visible on every request where it can reasonably be
> expected to exist.
>
>
> > -----
> >
> > The reason I was asking earlier whether this is the expected behaviour or
> > not, was because I was wondering whether the ACLs apply to the CONNECT
> > request contents, or to the ssl_bump contents.
>
> The answer is yes. However they are not that problem, both of those you
> refer to have the header and it is detected.
>
> It is the side effects of NAT which is causing the headache. The
> fake-CONNECT and lack of support for it in your squid.conf rules.
>
>
> >
> > Is there a way to determine the cache_peer based on ssl bumped contents?
>
> Yes, but that is not your problem.
>
> >
> > If not, is there a way to add X-My-Header to the CONNECT request, as a
> > workaround to ssl bumped contents not following any ACLs?
>
> The way you are doing it is correct and will be applied on any CONNECT
> request which leaves Squid to a cache_peer.
>
> Your problem is that the CONNECT you receive from curl is being
> tunneled, then redirected using NAT.
>
> That process discards your header and any other state about the client
> TCP connection that would have been useful during the http_access and
> ssl_bump evaluation for bumping the TLS traffic.
>
>
> >
> > If not, is there a way to set the cache_peer based on the headers of the
> > bumped request?
> >
> > If there's nothing I can do about this, is there a way to set the
> > ssl_bumped cache_peer based on the earlier proxy_auth username?
> >
>
> Start by adding the ssl-bump option on your http_port line which will
> resolve this case of traffic event.
>
> Properly NAT'ed client connections arriving on your intercept port will
> still have the problem. So you will need to remove the never_direct rule
> preventing server connections being used by the bumping process.
>
> Amos
>
> _______________________________________________
> squid-users mailing list
> squid-users at lists.squid-cache.org
> http://lists.squid-cache.org/listinfo/squid-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20160718/a299c7f6/attachment-0001.html>