[squid-users] Rewriting HTTP to HTTPS for generic package proxy

Sun Jul 14 16:59:16 UTC 2024

Hi Alex,

sorry, I have not seen your message, yet. Thank you very much for your helping support.

(A) I will try to find a way to test, how a new Squid build based on OpenSSL behaves under those circumstances. It will take some time.

(B) Yes, Squid does nothing wrong, it is a very specific use case. I would prefer to utilize a dedicated package caching solution, but there are only a few available. It's a pity that Apt-Cacher-NG is in such a bad state. Unfortunately, I lack the required C/C++ skills to do the code extensions in Squid by myself.

Have a nice weekend.

Regards,
Christoph

>-----Ursprüngliche Nachricht-----
>Von: squid-users <squid-users-bounces at lists.squid-cache.org> Im Auftrag von Alex Rousskov
>Gesendet: Donnerstag, 11. Juli 2024 20:27
>An: squid-users at lists.squid-cache.org
>Betreff: Re: [squid-users] Rewriting HTTP to HTTPS for generic package proxy
>
>On 2024-07-11 13:37, Fiehe, Christoph wrote:
>> My proxy (the child proxy) already uses the OpenSSL library:
>
>Good.
>
>
>> The parent proxy was compiled ... '--with-gnutls'
>
>> The GnuTLS exception is thrown at my parent proxy.
>
>Thank you for reminding me of that fact; I did not notice or have
>forgotten about it. I assume you cannot rebuild your parent proxy to use
>OpenSSL.
>
>I see the following choice:
>
>A) Continue with the current no-CONNECT setup: Find somebody who can
>help you get Squid+GnuTLS code path working on the parent proxy. It
>might be impossible to get this working without making build or
>configuration changes at the parent proxy. Moreover, please note that
>your current no-CONNECT setup lacks encryption on the child-parent
>segment. If that was not intentional, then fixing that will increase
>TLS-related work for the parent, potentially triggering more problems there.
>
>B) Switch to a CONNECT-based setup: Find somebody who can enhance Squid
>code to establish a CONNECT tunnel through parent proxy when dealing
>with a GET-https request. Today, Squid will not do that AFAICT[^1].
>
>https://wiki.squid-cache.org/SquidFaq/AboutSquid#how-to-add-a-new-squid-feature-enhance-
>of-fix-something
>
>
>[^1]: AFAICT: Today, there are two primary Squid code conditions for
>establishing a CONNECT tunnel on a caching code path: Request method is
>CONNECT or SslBump is in use. Neither matches your GET-https request
>scenario. Squid current behavior is not "wrong" (as detailed in my
>earlier email about CONNECT and no-CONNECT scenarios), so, to make these
>changes official, the author will need to add a configuration option to
>let admins enable this behavior. The corresponding code changes feel
>straightforward to me, but I have not studied any details.
>
>
>HTH,
>
>Alex.
>
>
>
>> Unfortunately, I cannot make any changes here. So yes, I trust my parent proxy, but not
>using a tunnel between child and parent does not seem to work and results in the TLS
>exception on the parent proxy.
>>
>> I have not find a way to tell my child proxy to always setup a tunnel through the parent
>proxy, when the target server talks HTTPS. Do you know, how to achieve that? It would be a
>promising approach.
>>
>> Thank you very much help and your patience, Alex.
>>
>> Regards,
>> Christoph
>>
>>
>>> -----Ursprüngliche Nachricht-----
>>> Von: squid-users <squid-users-bounces at lists.squid-cache.org> Im Auftrag von Alex
>Rousskov
>>> Gesendet: Donnerstag, 11. Juli 2024 18:15
>>> An: squid-users at lists.squid-cache.org
>>> Betreff: Re: [squid-users] Rewriting HTTP to HTTPS for generic package proxy
>>>
>>> On 2024-07-10 16:57, Fiehe, Christoph wrote:
>>>
>>>> I am just trying to find something that helps to narrow down the
>>>> problem. What I want to achieve is, that a client can use HTTP in the
>>>> LAN, so that Squid can cache distribution packages without making use
>>>> of SSL intercepting when repos are only accessible via HTTPS.
>>>
>>> OK.
>>>
>>>
>>>> In that case the secure connection must start at the proxy and end on
>>>> the target server with or without any upstream proxies in betweem.
>>>
>>> It depends on whether you trust the parent proxy:
>>>
>>> If you trust the parent proxy, then you can use two secure connections:
>>>
>>> 1.1. child - parent (TLS; no CONNECT)
>>> 1.2. parent - origin (TLS; no CONNECT)
>>>
>>> If you do not trust the parent proxy, then, yes, you will need a tunnel:
>>>
>>> 2.1. child - parent (CONNECT)
>>> 2.2. child - origin (TLS inside the CONNECT tunnel)
>>>
>>> N.B. CONNECT request in 2.1 may be plain text (common) or encrypted
>>> (rare); I am ignoring the difference between those two subcases for now.
>>>
>>>
>>>> We have the following setup:
>>>>
>>>> client -> downstream proxy -> upstream proxy -> https://download.docker.com
>>>>
>>>> Now let us assume the client wants to retrieve the following resource
>>> http://download.docker.com/linux/ubuntu/dists/jammy/InRelease from the upstream proxy.
>>>>
>>>> The client initiates a HTTP GET request and sends it to the downstream proxy. Now, the
>>> URL gets rewritten.
>>>
>>> OK.
>>>
>>>
>>>> It indicates to use a HTTPS connection instead in order to talk to the target server,
>in
>>> our case the result is https://download.docker.com/linux/ubuntu/dists/jammy/InRelease.
>>>
>>> Yes, but HTTPS scheme does not imply that the child Squid has to use
>>> CONNECT. There are two possible scenarios detailed above. I do not know
>>> which of them applies to your use case.
>>>
>>>
>>>> Now comes the critical point: From my understanding – it may be
>>>> wrongof course - the downstream server now has to send a CONNECT
>>>> request to the upstream server
>>>
>>> Yes, provided the child (downstream) proxy does not trust that parent
>>> (upstream) proxy. That is scenario 2. Scenario 1 is different.
>>>
>>>
>>>> to advise him to establish a secure connection to the target server.
>>>
>>> No, the CONNECT tunnel itself is just a pair of TCP connections. The
>>> parent proxy "secures" nothing but basic TCP connectivity. It is the
>>> child proxy that negotiates TLS (over/inside that tunnel) with the
>>> origin server.
>>>
>>>
>>>> After creation, the downstream proxy can retrieve the resource and
>>>> send it back to the client via plain HTTP.
>>>
>>> Yes.
>>>
>>>
>>>
>>>> I suppose, that the GnuTLS occurs because of a missing SSL handshake
>>>> between downstream proxy and download.docker.com.
>>>
>>> At this time, I can only say that a TLS negotiation error occurs (while
>>> child Squid is using the encryption library it probably should not be
>>> using for this). It is not yet clear to me whether child Squid is
>>> negotiating with the wrong hop or something goes wrong during
>>> negotiation with the right hop.
>>>
>>> As the next steps, I recommend switching to OpenSSL and, if that alone
>>> does not help, sharing new errors and determining whether you want to
>>> use scenario 1 (no CONNECT), scenario 2 (CONNECT), or either (whichever
>>> works): Do you trust the parent Squid?
>>>
>>>
>>> HTH,
>>>
>>> Alex.
>>>
>>>
>>>>> -----Ursprüngliche Nachricht-----
>>>>> Von: Alex Rousskov <rousskov at measurement-factory.com>
>>>>> Gesendet: Mittwoch, 10. Juli 2024 22:15
>>>>> An: squid-users at lists.squid-cache.org
>>>>> Cc: Fiehe, Christoph <c.fiehe at eurodata.de>
>>>>> Betreff: AW: [squid-users] Rewriting HTTP to HTTPS for generic package proxy
>>>>>
>>>>> On 2024-07-10 15:31, Fiehe, Christoph wrote:
>>>>>> The problem is that the proxy just forwards the client GET request to the upstream
>>> proxy
>>>>>
>>>>> Why does sending a GET request to the upstream proxy represent a problem
>>>>> in your use case? I cannot find anything in your prior messages on this
>>>>> thread that would preclude sending a GET request to the upstream proxy.
>>>>>
>>>>>
>>>>>> but in that case a CONNECT is required.
>>>>>
>>>>> Why?
>>>>>
>>>>> Please do not interpret my response as implying that this "must send
>>>>> CONNECT" requirement is wrong (or correct). At this point, I am just
>>>>> trying to understand what problem(s) you are trying to solve beyond the
>>>>> one you have originally described.
>>>>>
>>>>>
>>>>> Thank you,
>>>>>
>>>>> Alex.
>>>>
>>>> _______________________________________________
>>>> squid-users mailing list
>>>> squid-users at lists.squid-cache.org
>>>> https://lists.squid-cache.org/listinfo/squid-users
>>>
>>> _______________________________________________
>>> squid-users mailing list
>>> squid-users at lists.squid-cache.org
>>> https://lists.squid-cache.org/listinfo/squid-users
>> _______________________________________________
>> squid-users mailing list
>> squid-users at lists.squid-cache.org
>> https://lists.squid-cache.org/listinfo/squid-users
>
>_______________________________________________
>squid-users mailing list
>squid-users at lists.squid-cache.org
>https://lists.squid-cache.org/listinfo/squid-users