[squid-users] LEGACY_SERVER_CONNECT, ALLOW_UNSAFE_LEGACY_RENEGOTIATION does not work - SSL bump, OpenSSL 3

Wed Dec 28 16:49:44 UTC 2022

Hi Alex,

On 28/12/22 21:31, Alex Rousskov wrote:
> Hi Amish,
>
>     Squid parsing code is tricky. tls_outgoing_options parsing code is 
> triply so. Even its authors misinterpret it!
>
> I assume you have removed multiple tls_outgoing_options directives 
> from your configuration before testing. If you have not, please merge 
> those directives into one and retest. You should still see multiple 
> parsing paths, in part, due to (unfortunate) Security::PeerOptions 
> implementation and, in part, due to Squid parsing default options 
> before Squid parses your actual configuration files.
>
Yes I had combined all tls_outgoing_options in to a single directive 
(but in multiple lines ending with a backslash \). It can be seen in the 
grep command in my previous email.

But wait, I have found something.

And I have a doubt that parsedOptions work only for client to squid side 
of the connection and does not work for squid to server side of the 
connection.

What I did is changed my "http(s)_port" directives to include 
options=0x4. These directives control options for the client to squid 
side of TLS connection.

$ grep 'ssl-bump' /etc/squid/squid.conf
http_port 8080 ssl-bump generate-host-certificates=on 
tls-cert=/etc/squid/ssl_cert/squid.pem 
tls-dh=/etc/squid/ssl_cert/dhparam.pem options=0x4
https_port 8081 intercept ssl-bump generate-host-certificates=on 
tls-cert=/etc/squid/ssl_cert/squid.pem 
tls-dh=/etc/squid/ssl_cert/dhparam.pem options=0x4

Now lets reproduce the issue:
$ curl --no-progress-meter -kx 127.0.0.1:8080 https://www.jio.com |grep 
'TLS\|SSL'
<pre>[No Error] (TLS code: 
SQUID_TLS_ERR_CONNECT+TLS_LIB_ERR=A000152+TLS_IO_ERR=1)</pre>
<p>Failed to establish a secure connection: error:0A000152:SSL 
routines::unsafe legacy renegotiation disabled</p>

Lets see what cache.log has to say:

$ tail -f /var/log/squid/cache.log  |grep -i 'openssl\|parsed'
2022/12/28 21:52:56.532 kid1| 83,5| PeerOptions.cc(447) parseOptions: 
INFO: TLS parsedOptions(1)=4
2022/12/28 21:52:56.532 kid1| 83,5| PeerOptions.cc(643) 
updateContextOptions: set OpenSSL options for context=0x559075043e30, 
parsedOptions=4
2022/12/28 21:52:56.532 kid1| 83,5| PeerOptions.cc(645) 
updateContextOptions: get OpenSSL options for context=0x559075043e30, 
getOptions=1179652

Bingo!! As we can see, parsedOptions is now set to 4!

And it is also confirmed by ssl_get_options() - 1179652  = 0x120004 (4 
means the option is accepted by OpenSSL too)

But this also means that updateContextOptions() is called ONLY FOR 
client to squid side (curl to squid) and it is not called for squid to 
server (jio.com in this case) side.

The reason being that we do not see updateContextOptions() being called 
twice for a request. But only once. And that is why request still fails 
with the negotiation error.

So where exactly is the call for squid to server side being made?

> If merging directives does not deliver custom options to 
> SSL_CTX_set_options(), then let's attack this from the other end: 
> Supply the right options to each SSL_CTX_set_options() call:
>
>     const Security::ParsedOptions forcedParsedOptions = 0x4 | 0x40000;
>     SSL_CTX_set_options(ctx.get(), forcedParsedOptions);
>
> Does the above temporary hack fix the problem in your test?

I will try this tomorrow (its late night here).

But I think I will have to set this somewhere else and NOT in 
PeerOptions.cc. Because above code appears to be for client to squid side.

And I need to add forcedParsedOptions to the code which connects squid 
to server. But where?

Thank you,

Amish.

> Thank you,
>
> Alex.
>
> On 12/28/22 02:32, Amish wrote:
>> Hi Alex,
>>
>> Thanks again for your reply.
>>
>> To find answers to your questions, I added few debugs() lines to 
>> PeerOptions.cc.
>>
>> The diff file (patch) is attached.
>>
>> It prints parsedOptions and options retrieved from SSL context and 
>> session objects at several stages.
>>
>> Here is tls_outgoing_options setting:
>>
>> $ grep tls_outgoing_options /etc/squid/squid.conf
>> tls_outgoing_options \
>>      cafile=/etc/ssl/cert.pem \
>> cipher=ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS 
>> \
>>      options=0x4
>>
>>
>> Here is what squid logs on reload i.e. on parsing the squid.conf
>>
>> $ systemctl reload squid
>> $ tail -f /var/log/squid/cache.log  |grep -i 'openssl\|parsed'
>> 2022/12/28 12:19:30.596 kid1| 83,5| PeerOptions.cc(547) parseOptions: 
>> INFO: TLS parsedOptions(3)=0
>> 2022/12/28 12:19:30.598 kid1| 83,5| PeerOptions.cc(547) parseOptions: 
>> INFO: TLS parsedOptions(3)=0
>> 2022/12/28 12:19:30.723 kid1| 83,5| PeerOptions.cc(547) parseOptions: 
>> INFO: TLS parsedOptions(3)=4
>> 2022/12/28 12:19:30.729 kid1| 83,5| PeerOptions.cc(547) parseOptions: 
>> INFO: TLS parsedOptions(3)=0
>> 2022/12/28 12:19:30.729 kid1| 83,5| PeerOptions.cc(547) parseOptions: 
>> INFO: TLS parsedOptions(3)=0
>> 2022/12/28 12:19:32.147 kid1| 83,5| PeerOptions.cc(447) parseOptions: 
>> INFO: TLS parsedOptions(1)=0
>> 2022/12/28 12:19:32.147 kid1| 83,5| PeerOptions.cc(547) parseOptions: 
>> INFO: TLS parsedOptions(3)=0
>> 2022/12/28 12:19:33.524 kid1| 83,5| PeerOptions.cc(447) parseOptions: 
>> INFO: TLS parsedOptions(1)=0
>> 2022/12/28 12:19:33.532 kid1| 83,5| PeerOptions.cc(547) parseOptions: 
>> INFO: TLS parsedOptions(3)=0
>> 2022/12/28 12:19:33.695 kid1| 83,5| PeerOptions.cc(447) parseOptions: 
>> INFO: TLS parsedOptions(1)=0
>> 2022/12/28 12:19:33.695 kid1| 83,5| PeerOptions.cc(643) 
>> updateContextOptions: set OpenSSL options for context=0x562a5387fe30, 
>> parsedOptions=0
>> 2022/12/28 12:19:33.695 kid1| 83,5| PeerOptions.cc(645) 
>> updateContextOptions: get OpenSSL options for context=0x562a5387fe30, 
>> getOptions=1179648
>
>> 2022/12/28 12:19:33.708 kid1| 83,5| PeerOptions.cc(447) parseOptions: 
>> INFO: TLS parsedOptions(1)=0
>> 2022/12/28 12:19:33.708 kid1| 83,5| PeerOptions.cc(643) 
>> updateContextOptions: set OpenSSL options for context=0x562a53e6e740, 
>> parsedOptions=0
>> 2022/12/28 12:19:33.708 kid1| 83,5| PeerOptions.cc(645) 
>> updateContextOptions: get OpenSSL options for context=0x562a53e6e740, 
>> getOptions=1179648
>>
>> It seems that squid parses the options multiple times and only once 
>> it gets the value as 4. Rest are parsed as 0.
>>
>> The value of 1179648 (0x120000) corresponds to SSL_OP_NO_COMPRESSION 
>> (0x20000) and SSL_OP_ENABLE_MIDDLEBOX_COMPAT. (0x100000)
>>
>>
>> Now lets reproduce the issue:
>>
>> $ curl --no-progress-meter -kx 127.0.0.1:8080 https://www.jio.com 
>> |grep 'TLS\|SSL'
>> <pre>[No Error] (TLS code: 
>> SQUID_TLS_ERR_CONNECT+TLS_LIB_ERR=A000152+TLS_IO_ERR=1)</pre>
>> <p>Failed to establish a secure connection: error:0A000152:SSL 
>> routines::unsafe legacy renegotiation disabled</p>
>>
>> So, as we can see, we are still not able to access the site.
>>
>> Lets see what cache.log has to say.
>>
>> $ tail -f /var/log/squid/cache.log  |grep -i 'openssl\|parsed'
>> 2022/12/28 12:31:09.971 kid1| 83,5| PeerOptions.cc(447) parseOptions: 
>> INFO: TLS parsedOptions(1)=0
>> 2022/12/28 12:31:09.971 kid1| 83,5| PeerOptions.cc(643) 
>> updateContextOptions: set OpenSSL options for context=0x562a53eb14e0, 
>> parsedOptions=0
>> 2022/12/28 12:31:09.971 kid1| 83,5| PeerOptions.cc(645) 
>> updateContextOptions: get OpenSSL options for context=0x562a53eb14e0, 
>> getOptions=1179648
>>
>> Strangely parsedOptions was zero and not 4!
>>
>> Now we can answer your questions as below.
>>
>>
>> On 27/12/22 21:52, Alex Rousskov wrote:
>>> On 12/27/22 10:42, Amish wrote:
>>>> On 26/12/22 21:31, Alex Rousskov wrote:
>>>>> tls_outgoing_options options=0x4,0x40000
>>>
>>>> With numeric hex values, I do not see the ERROR on stderr.
>>>
>>>> But it still does not seem to be working as expected. Squid still 
>>>> does not open the page and gives same legacy negotiation error.
>>>
>>> There are still many unknowns (from my point of view), including:
>>>
>>> 1. Does OpenSSL accept the above options? You ask that question below.
>>
>> Google search shows some projects using OpenSSL v3 where there is 
>> mention to use above option when a similar error occurred to them.
>>
>> But in our case, its clear that squid does not pass value 4 to SSL 
>> context, hence we do not know yet if OpenSSL accepts above options.
>>
>>
>>> 2. Does Squid indeed stare at the server, as expected?
>>> 3. Does Squid apply the accepted options when staring at the server?
>>
>> A comment for parseOptions() in PeerOptions.cc states this:
>>
>> /**
>>   * Pre-parse TLS options= parameter to be applied when the TLS 
>> objects created.
>>   * Options must not used in the case of peek or stare bump mode.
>>   */
>> void Security::PeerOptions::parseOptions()
>>
>> So it appears that TLS options is NOT used for peek as well as stare. 
>> But why? I am not sure.
>>
>> How do I make squid use it for 'stare' atleast?
>>
>>
>>> 4. Why does TLS negotiation fail despite those options applied,
>>>    especially since it succeeds with using openssl s_client
>>
>> It possibly fails because options are not applied by squid.
>>
>> So, where do I check next on why parsedOptions is still set to 0 and 
>> why its not used for 'stare'?
>>
>> Thanks and regards,
>>
>> Amish.
>>