[squid-users] Reverse proxying Exchange OWA wembail with SSL offloading

[Amos Jeffries]

On 31/10/20 3:20 am, Scott wrote:
> On Sat, Oct 31, 2020 at 12:49:16AM +1300, Amos Jeffries wrote:
>> On 30/10/20 3:27 pm, Scott wrote:
>>> On Thu, Oct 29, 2020 at 10:08:42PM +1300, Amos Jeffries wrote:
>>>> On 29/10/20 12:06 pm, Scott wrote:
>>>>> On Wed, Oct 28, 2020 at 12:00:01PM +0000, squid-users-reques wrote:
>>>>>> Date: Thu, 29 Oct 2020 00:08:34 +1300
>>>>>> From: Amos Jeffries
>>>>>>
>>>>>> On 28/10/20 5:25 pm, Scott wrote:
>>>>>>>
>>>>>>> Here are the logs (first not working, followed by working).
>>>>>>>
>>>>>>> Note this is the login attempt, not the loading of the initial page.  You'll
>>>>>>> see in the NOT WORKING section that the browser does NOT return a cookie to
>>>>>>> the server, which is where the problem may be.  Again, I'm not sure why - I'm
>>>>>>> thinking perhaps the browser/javascript is rejecting the cookie as it's
>>>>>>> missing the "secure" attribute (because the back-end is talking plain HTTP).
>>>>>>>
>>>>>>
>>>>>> The complete absence of a cookie may be expected to break something.
>>>>>>
>>>>>> The absence of a "secure" flag should only make the cookie vulnerable to
>>>>>> leaking. It should not affect anything depending on that cookies value.
>>>>>>
>>>>>>
>>>>>> Amos
>>>>>>
>>>>>
>>>>> After some more research and experimentation I've confirmed that my
>>>>> suspicions are correct.
>>>>>
>>>>> Recent browsers are no longer accepting cookies with the SameSite flag set
>>>>> without the Secure flag set.
>>>>>
>>>>> It's not an issue with Squid (although one Squid could solve - but I'm unsure
>>>>> of the implications).
>>>>
>>>> Implications are that the server may have intentionally used the
>>>> combination it did, no mistakes.
>>>>
>>>> The server is given "Front-End-Https: On" so that it is aware the client
>>>> is using HTTPS and can set (or not) the secure flag appropriately to
>>>> what it needs. Squid is not aware of whether the cookie is safe to use
>>>> on HTTP or restrict to just HTTPS.
>>>>
>>>>
>>>>>
>>>>> Here is a useful link:
>>>>> https://docs.microsoft.com/en-us/office365/troubleshoot/miscellaneous/chrome-behavior-affects-applications
>>>>>
>>>>> I tested Chrome 85 on Windows - the default settings DO NOT allow for these
>>>>> cookies.  However after setting
>>>>> 	Cookies without SameSite must be secure
>>>>> to Disabled, these cookies are permitted and OWA works.
>>>>>
>>>>> There are obvious implications for sites doing SSL offloading here.  Are
>>>>> sites no longer doing SSL offload?  Or are reverse proxies adding the Secure
>>>>> flag?
>>>>
>>>>
>>>> Neither. When a site frontend is entirely https:// with no http://
>>>> resources mixed in the Secure flag can be used by the server regardless
>>>> of what the internal connections are.
>>>>
>>>>
>>>> Amos
>>>
>>> My point is that, assuming browsers are now enforcing SameSite cookies must
>>> be secure, then doing SSL offload (whereby the origin server does NOT flag
>>> the cookie as secure) will break.
>>
>> But offloading does not mean the server omits the secure flag.
>>
>> Servers which choose to send it when offloading work fine. Servers that
>> choose to omit it have problems with the Google paranoid interpretation
>> of security.
> 
> Aren't origin servers oblivious to SSL offloading?  I thought they just
> happily accepted HTTP connections with no knowledge of any secure channel
> between client and reverse proxy.


It varies.  Mozilla have a brief writeup about the headers used:
<https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Forwarded-Proto>


> 
> If that's correct then, although not proscribed by the RFC, it's unlikely
> that a Set-Cookie header would ever contain a Secure flag because the server
> would be saying "I insist this Cookie be transmitted securely, but here it is
> over an insecure channel".
> 

That assumes the only type of security is HTTPS. The channel between 
proxy and server may be secured via other means than TLS, even as it 
transmits "plain text" HTTP.

There should be server settings to administrate that. I do not use OWA 
or IIS though, so now sure if they are available for your case.


>>>
>>> In this particular case, I use squid to do reverse proxy with SSL offload to
>>> a MS Exchange server.
>>
>> You also have a TLS connection to the exchange:443 peer.
> 
> I have both for testing, but only one is active at any one time (as
> referenced in a cache_peer_access).  Yes I now use the 443 peer because
> Chrome/IE/Edge + SSL Offloading no longer works.
> 

Please understand that with Squid acting as a reverse-proxy "offloading" 
is still happening. It is the cert you put in the https_port in 
Squid.conf that the clients are dealing with, and Squid which is doing 
any client cert validation.


For working security on the cache_peer you should identify the CA cert 
which signed the peer server's TLS certificate. Load that into Squid as 
cache_peer tls-cafile=  and you can drop the DONT_VERIFY_PEER flag.


>>
>> Notice that in the logs you showed transactions sent to that peer get
>> the secure flag set, despite the SSL offload being done by Squid at the
>> same time.
>>
>>
> 
> Not true AFAICT.  Any cookies you see in those logs with Secure set are shown
> under the "WORKING" heading whose entries are:
> 2020/10/28 12:01:23.549 kid1| 11,2| http.cc(2263) sendRequest: HTTP Server local=squid-internal:62597 remote=exchange:443 FD 30 flags=1
> 
> No SSL offloading there.  Otherwise it would say "remote=exchange:80"
> 

Squid is receiving the HTTP(S) message. That means Squid is terminating 
the TLS from client, decrypting the traffic. Which is what I understand 
to be offloading.

The connection to the server is independent. Squid will actually be 
multiplexing traffic for multiple clients on that connection. No client 
has participation or knowledge of that TLS existence.


Non-offloading would be a CONNECT tunnel through Squid to the server so 
TLS can be negotiated directly between client and server.


>>
>>> Because the requests are HTTP IIS does not set the secure flag on cookies,
>>> and browsers now reject them.  This breaks things.
>>>
>>> I've fixed it by switching back to HTTPS on the backend (no SSL offload), and
>>> so the secure flag is now being set by IIS.  Problem solved.
>>>
>>> In the long term it seems doing SSL offload to MS Exchange (and Sharepoint I
>>> think) will not be an option.  I've seen other proxy manufacturers provide
>>> cookie manipulation; I assume for this kind of issue.
>>>
>>> Do you think Squid should have such a feature?
>>
>> We have ICAP, eCAP, and *_header_replace features already. So any
>> proposed new feature would have to be better than what they already provide.
>>
> 
> Unless I'm reading the online command reference incorrectly, *_header_replace
> does not allow for the insertion of text so can't help here.  Please tell me
> I'm wrong - I'd love for Squid to have this capability like other proxies.


Aye. There are limits and issues with all those features. I was speaking 
there about what a new feature proposal would need to work with for 
consideration.


The options right now are (in order of preference):

* create an eCAP module that just edits the headers as-needed.

* create an ICAP service to do header edits as-needed.

* create an external_acl_type helper that takes the reply headers as 
input and delivers Squid a set of annotations that can be injected as 
Set-Cookie headers by http_reply_header_add. With *header_access to 
remove broken values first.

* new feature in Squid.


Amos