<div dir="ltr">Your details helped me understand a lot better.<div><br></div><div>It turns out squid correctly adds the header to the CONNECT request, when that request is made to another proxy. It cannot be itself, unfortunately, because then it complains about a loop.</div><div><br></div><div>Also unfortunately, your suggestion of doing `ssl-bump` on the http port doesn't work because the squid process terminates with a failed assertion when using cache_peer, it seems to be this bug <a href="http://bugs.squid-cache.org/show_bug.cgi?id=3963">http://bugs.squid-cache.org/show_bug.cgi?id=3963</a> , which I get during with my squid 3.5.20 `2016/07/18 15:07:50.566| assertion failed: PeerConnector.cc:116: "peer->use_ssl"`.</div><div><br></div><div>Config used:</div><div><br></div><div>```</div><div><div>http_port 8000 ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=16MB cert=/etc/squid/ca.crt key=/etc/squid/ca.key dhparams=/etc/squid/dh2048.pem options=NO_SSLv3</div><div><br></div><div>sslcrtd_program /usr/lib/squid/ssl_crtd -s /var/lib/squid_ssl_db -M 32MB</div><div>sslcrtd_children 32</div><div>acl step1 at_step SslBump1</div><div>ssl_bump peek step1</div><div>ssl_bump bump all</div><div><br></div><div>never_direct allow all</div><div><br></div><div>cache_peer 192.71.64.174 parent 6745 0 no-query no-digest default</div><div><br></div><div>http_access allow all</div></div><div>```</div><div><br></div><div>Considering the fact that I can't do `ssl-bump` on http port because of the `peer-use_ssl` assertion (bug linked above), also considering the fact that squid :8000 using itself as a proxy :8443 complains about a proxy loop, are there any other options I might have to use ssl_bump *with* multiple cache_peer, and cache_peer selection based on proxy_auth and/or req_header?</div><div><br></div><div><br></div></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div></div><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><br><br><b>Mihai Ene</b><div><font color="#999999">Software Developer</font></div><div><font color="#999999"><br></font></div><div><b style="color:rgb(153,153,153);font-size:12.8000001907349px">UB | Your universal basket</b></div><div><font color="#999999"><br></font></div><div><font color="#999999"><a href="http://ub.io" target="_blank">http://ub.io</a></font></div><div><font color="#999999"><a href="mailto:me@ub.io" target="_blank">me@ub.io</a></font></div><div><span style="font-size:small">@shop_ub</span><br></div><div><font color="#999999"><a href="tel:+447473804972" value="+447881904476" target="_blank">+44 (0)7473 804972</a></font></div></div></div></div></div></div></div></div></div></div></div></div>
<br><div class="gmail_quote">On Mon, Jul 18, 2016 at 2:03 PM, Amos Jeffries <span dir="ltr"><<a href="mailto:squid3@treenet.co.nz" target="_blank">squid3@treenet.co.nz</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On 18/07/2016 10:23 p.m., Mihai Ene wrote:<br>
> Hello,<br>
><br>
> I have created a gist with the relevant parts of `cache.log`<br>
> (post-connection)<br>
> <a href="https://gist.github.com/randunel/5c0d282c52e9135aa21b8c6e28925a52" rel="noreferrer" target="_blank">https://gist.github.com/randunel/5c0d282c52e9135aa21b8c6e28925a52</a><br>
><br>
> The following logs are available:<br>
><br>
> 1. The initial HTTP CONNECT requests on port :8000 on line 51<br>
> <a href="https://gist.github.com/randunel/5c0d282c52e9135aa21b8c6e28925a52#file-3-cache-log-L51" rel="noreferrer" target="_blank">https://gist.github.com/randunel/5c0d282c52e9135aa21b8c6e28925a52#file-3-cache-log-L51</a><br>
><br>
<br>
</span>Between lines 1 and line 462;<br>
<br>
Your Squid is receiving an HTTP CONNECT tunnel request from curl on FD<br>
12. Your custom header exists on that CONNECT request.<br>
<br>
Your access controls determine that:<br>
a) this tunnel is *not* to be bumped (by http_port lacking ssl-bump<br>
option).<br>
b) that a direct outgoing server TCP connection is required (by<br>
always_direct allow),<br>
<br>
So Squid is required to *decapsulate* the CONNECT request and send the<br>
data inside it to upstream server.<br>
<br>
At this point there is an opaque payload in the CONNECT tunnel arriving<br>
from client and being delivered over an opaque TCP connection to the<br>
server. There is no HTTP messaging over that server connection as far as<br>
Squid is concerned, just opaque bytes of data.<br>
<br>
==> Squid is not able to attach your custom HTTP header when there is no<br>
outgoing HTTP message.<br>
<span class=""><br>
<br>
> 2. The mark 0x10 is set on line 435<br>
> <a href="https://gist.github.com/randunel/5c0d282c52e9135aa21b8c6e28925a52#file-3-cache-log-L435" rel="noreferrer" target="_blank">https://gist.github.com/randunel/5c0d282c52e9135aa21b8c6e28925a52#file-3-cache-log-L435</a><br>
><br>
> 3. The redirected HTTPS request on port :8443 comes in, line 467<br>
> <a href="https://gist.github.com/randunel/5c0d282c52e9135aa21b8c6e28925a52#file-3-cache-log-L467" rel="noreferrer" target="_blank">https://gist.github.com/randunel/5c0d282c52e9135aa21b8c6e28925a52#file-3-cache-log-L467</a><br>
><br>
<br>
</span>By "redirected" you mean NAT'ed.<br>
<br>
At line 466 your OS NAT configuration folds the outgoing opaque tunnel<br>
connection back into Squid.<br>
<br>
As far as Squid is concerned this is a *new* HTTPS connection on the<br>
intercept port using FD 18. It is important to be aware that there is<br>
*no* information about your custom header from the previous CONNECT. A<br>
new inbound connection is a clean slate.<br>
<span class=""><br>
<br>
> 4. The forwarded CONNECT request is on line 526<br>
> <a href="https://gist.github.com/randunel/5c0d282c52e9135aa21b8c6e28925a52#file-3-cache-log-L526" rel="noreferrer" target="_blank">https://gist.github.com/randunel/5c0d282c52e9135aa21b8c6e28925a52#file-3-cache-log-L526</a><br>
><br>
<br>
</span>No. Between line 467 and 628;<br>
<br>
Squid is processing the fake CONNECT request representing the raw TCP<br>
connection IP:port details. To see if the connection is to be rejected<br>
immeditely, ssl-bump'ed or bypassed, etc.<br>
<br>
==> Since the request is completely internal to Squid, on a fresh new<br>
TCP connection, it does not contain any of the curl sent headers.<br>
<br>
==> Since it is of course not an *outgoing* message. It does not have<br>
any request_header_add custom alterations.<br>
<br>
At line 611 your http_access rules fail to match (all of them). So the<br>
implicit "allow all" happens to this pseudo-CONNECT.<br>
<br>
at 628 ssl_bump indicates a need to peek at the TLS traffic.<br>
<br>
<br>
between line 642 and 655; Squid is shoveling some of the opaque data<br>
between its FD 12 and FD 16 connections.<br>
<br>
Things then start interleaving between the inbound FD 18 connection and<br>
the FD 12 <-> FD 16 opaque tunnel data transfer. I will ignore that<br>
opaque shovelling from now in the between line X and Y statements.<br>
<br>
<br>
between line 655 and 907 SSL-Bump is doing its peek thing with the TLS<br>
now arriving on FD 18. At 907 it decides to bump using only the client<br>
Hello details.<br>
<span class=""><br>
<br>
> 5. X-My-Header is *NOT* found on line 966<br>
> <a href="https://gist.github.com/randunel/5c0d282c52e9135aa21b8c6e28925a52#file-3-cache-log-L966" rel="noreferrer" target="_blank">https://gist.github.com/randunel/5c0d282c52e9135aa21b8c6e28925a52#file-3-cache-log-L966</a><br>
><br>
<br>
</span>between lines 908 and 1043;<br>
<br>
Squid is doing its outbound server destination selection to figure out<br>
where this TLS client request is going to go to.<br>
<br>
Note that at this point the only state Squid has associated with this<br>
transaction is the TCP connections details, that fake-CONNECT request<br>
(updated to contain the TLS SNI - if any), and the TLS clientHello details.<br>
Any rules that you have regarding HTTP state is using that fake-CONNECT<br>
still.<br>
<br>
At line 1044 Squid determines that there are no permitted ways to<br>
contact any server and complete the TLS bumping.<br>
<br>
Between lines 1045 and 1372; Squid is performing the SSL-Bump process on<br>
the client connection so it can deliver the HTTP error message about<br>
that server connectivity problem.<br>
<br>
At line 1373 it begins receiving from FD 18 the HTTPS message that curl<br>
sent inside the tunnel way back on FD 12.<br>
<br>
Your custom header is visible on that curl request. But that is<br>
irrelevant by now, Squid is delivering its message about no server being<br>
permitted to be used for the SSL-Bump server connection. That response<br>
gets delivered at line 1652.<br>
<span class=""><br>
<br>
> 5. The bumped contents are on line 1575<br>
> <a href="https://gist.github.com/randunel/5c0d282c52e9135aa21b8c6e28925a52#file-3-cache-log-L1575" rel="noreferrer" target="_blank">https://gist.github.com/randunel/5c0d282c52e9135aa21b8c6e28925a52#file-3-cache-log-L1575</a><br>
> , where X-My-Header is visible<br>
><br>
><br>
> The following inconsistencies are seen:<br>
><br>
> a. The X-My-Header is *not* added to the CONNECT request according to<br>
> config<br>
> <a href="https://gist.github.com/randunel/5c0d282c52e9135aa21b8c6e28925a52#file-1-squid-conf-L19" rel="noreferrer" target="_blank">https://gist.github.com/randunel/5c0d282c52e9135aa21b8c6e28925a52#file-1-squid-conf-L19</a><br>
<br>
</span>There is never any outbound CONNECT request to send that header on.<br>
<span class=""><br>
> b. X-My-Header NOT found on line 966, although it is clearly visible on<br>
> line 1579<br>
><br>
<br>
</span>Your header is visible on every request where it can reasonably be<br>
expected to exist.<br>
<span class=""><br>
<br>
> -----<br>
><br>
> The reason I was asking earlier whether this is the expected behaviour or<br>
> not, was because I was wondering whether the ACLs apply to the CONNECT<br>
> request contents, or to the ssl_bump contents.<br>
<br>
</span>The answer is yes. However they are not that problem, both of those you<br>
refer to have the header and it is detected.<br>
<br>
It is the side effects of NAT which is causing the headache. The<br>
fake-CONNECT and lack of support for it in your squid.conf rules.<br>
<span class=""><br>
<br>
><br>
> Is there a way to determine the cache_peer based on ssl bumped contents?<br>
<br>
</span>Yes, but that is not your problem.<br>
<span class=""><br>
><br>
> If not, is there a way to add X-My-Header to the CONNECT request, as a<br>
> workaround to ssl bumped contents not following any ACLs?<br>
<br>
</span>The way you are doing it is correct and will be applied on any CONNECT<br>
request which leaves Squid to a cache_peer.<br>
<br>
Your problem is that the CONNECT you receive from curl is being<br>
tunneled, then redirected using NAT.<br>
<br>
That process discards your header and any other state about the client<br>
TCP connection that would have been useful during the http_access and<br>
ssl_bump evaluation for bumping the TLS traffic.<br>
<span class=""><br>
<br>
><br>
> If not, is there a way to set the cache_peer based on the headers of the<br>
> bumped request?<br>
><br>
> If there's nothing I can do about this, is there a way to set the<br>
> ssl_bumped cache_peer based on the earlier proxy_auth username?<br>
><br>
<br>
</span>Start by adding the ssl-bump option on your http_port line which will<br>
resolve this case of traffic event.<br>
<br>
Properly NAT'ed client connections arriving on your intercept port will<br>
still have the problem. So you will need to remove the never_direct rule<br>
preventing server connections being used by the bumping process.<br>
<br>
Amos<br>
<br>
_______________________________________________<br>
squid-users mailing list<br>
<a href="mailto:squid-users@lists.squid-cache.org">squid-users@lists.squid-cache.org</a><br>
<a href="http://lists.squid-cache.org/listinfo/squid-users" rel="noreferrer" target="_blank">http://lists.squid-cache.org/listinfo/squid-users</a><br>
</blockquote></div><br></div>