[squid-users] ssl_bump with parent cache
Alex Rousskov
rousskov at measurement-factory.com
Wed Mar 9 21:43:10 UTC 2022
On 3/9/22 16:16, Aaron Dewell wrote:
> the difference in what works and what doesn't is:
>
> ssl_bump peek step1
> ssl_bump splice all
>
> (working) and this not working:
>
> ssl_bump peek all
> ssl_bump splice all
... which, for comparison purposes, is equivalent to:
ssl_bump peek step1
ssl_bump peek step2
ssl_bump splice all
In summary, bugs notwithstanding, the difference is in peeking at the
server certificate (during step2). The working configuration does not
look at the server certificate at all. The broken one does. Thus,
something goes wrong when Squid tries to look at the server certificate
and/or when Squid tries to deliver the problem it found (while looking
at that server certificate) to the client.
> I'm probably not going to be able to share any packet dumps so that does
> make things difficult. This device is under development and the details
> of "what it's doing" is likely to be sensitive.
Please keep in mind that HTTP traffic details would be encrypted and
non-encrypted TCP/TLS details may not be worth protecting.
> I appreciate that makes
> troubleshooting things difficult. What I could do is put a VM behind it
> and do manual wgets or something from it but that will take some changes
> to the environment, or perhaps just creating a new one to test this
> where this device isn't present. If that's where it needs to go, I'll
> have to start discussions on exactly how sensitive it is and/or start
> creating the new/modified setup.
If you are feeling lucky, you can collect an ALL,9 cache.log from one
transaction and grep for "check failed", "exception:", "FwdState", and
"PeerConnector". Then see whether you can share (a pointer to a
compressed file with) just the matching lines. It may take a few
iterations...
You can also find a Squid developer you can trust who is willing to sign
an NDA and look at your complete logs for a fee, of course.
Alex.
> On Mar 8 2022, at 4:16 pm, Alex Rousskov wrote:
>
> On 3/8/22 17:56, Aaron Dewell wrote:
> > Ok, with a bit more messing with it... Changing bump to splice
> does work:
>
> > ssl_bump splice all
>
> Noted.
>
>
> > Adding:
> >
> > acl step2 at_step SslBump2
> > ssl_bump peek step1
> > ssl_bump peek step2
> > ssl_bump splice step2
>
> The above is a bad configuration because no rule matches during step3.
>
> Please try this configuration instead, to make sure you are not seeing a
> side effect of various Squid bugs related to such configurations (we are
> working on fixing them):
>
> ssl_bump peek all
> ssl_bump splice all
>
> If the above fails, please see my earlier suggestions about sharing test
> artifacts. Otherwise, please try this:
>
> ssl_bump stare all
> ssl_bump bump all
>
> If the above fails, please see my earlier suggestions about sharing test
> artifacts. Otherwise, please try this:
>
> ssl_bump peek step1
> ssl_bump bump all
>
> If the above fails, please see my earlier suggestions about sharing test
> artifacts. Otherwise, you got a working config (which is the same config
> you have started with, so I doubt you will get to this stage!).
>
>
> > This appears to be the relevant data from the log:
> >
> > 2022/03/08 14:35:47.644 kid1| 83,7| bio.cc(163) stateChanged: FD
> 10 now: 0x4004 TED (TLSv1.3 early data)
> > 2022/03/08 14:35:47.644 kid1| 83,7| bio.cc(163) stateChanged: FD
> 10 now: 0x2002 SSLERR (error)
> > 2022/03/08 14:35:47.644 kid1| Error negotiating SSL connection on
> FD 10: error:00000001:lib(0):func(0):reason(1) (1/-1)
>
> YMMV, but I will need to see a lot more lines surrounding this snippet
> to understand what exactly went wrong. That is why I am asking for the
> entire transaction log rather than selected "error" lines.
>
> Sharing a packet capture might be sufficient if this is a basic
> misconfiguration error -- we will see wrong protocol traffic on one of
> both Squid sides. You can start with that (often, but not always, safer
> as far as sensitive info disclosure) step if you prefer.
>
>
> HTH,
>
> Alex.
>
>
> > On Mar 8 2022, at 2:57 pm, Alex Rousskov wrote:
> >
> > On 3/8/22 16:38, Aaron Dewell wrote:
> >
> > > Hi Alex, thanks for your reply! I did get access to the parent
> proxy
> > > and my assumption was wrong, it's doing minimal bumping.
> >
> > TLS inspection at the parent proxy does not affect what I was
> trying to
> > double check. What matters is whether it is a forward HTTP proxy
> (e.g.,
> > a Squid instance listening on an http_port configured without
> intercept,
> > tproxy, or accel flags). It sounds like that is what it is.
> That's OK!
> >
> > I do not remember whether your Squid version (v4.13) supports SslBump
> > with parent forward proxies, but I believe modern Squids do, and
> we can
> > assume that your Squid version does as well. The debugging should
> show
> > whether that is indeed the case.
> >
> >
> > > The parent is doing peek and splice to an exact list of internal
> > > destinations. Specifically, peek step1 all, peek step2
> > allowed_sites,
> > > splice allowed_sites, terminate all. That shouldn't (to my
> imperfect
> > > knowledge) interfere with what I'm doing though.
> >
> > Yes, assuming the parent does not terminate your Squid
> connections, of
> > course (i.e. assuming connections from your Squid always match
> > allowed_sites after step1 at the parent proxy).
> >
> >
> > > That's a good idea to do splice only and see if that's successful.
> > > Trying to do too much at once! I'll see what that does, then try
> > debug
> > > again. The debug output wasn't very helpful before, but stepwise
> > may be
> > > more useful.
> >
> > Just to clarify: The debugging output was not meant for you to
> > interpret. A Squid developer should do that. It may contain sensitive
> > details; it is best to not use anything but test traffic and test
> > certificate keys when sharing ALL,9 output.
> >
> > Alex.
> >
> >
> > > On Mar 8 2022, at 1:43 pm, Alex Rousskov wrote:
> > >
> > > On 3/8/22 14:16, Aaron Dewell wrote:
> > >
> > > > I'm trying to use these two features at the same time. The use
> > > case is
> > > > pretty simple. I want to capture all traffic from a single
> > source (a
> > > > device of mine) to another squid proxy server and decrypt/log
> > > it. I'm
> > > > using the Ubuntu 20 package of squid-ssl version 4.13.
> > > >
> > > > Device -> ssl_bump proxy -> upstream proxy -> website
> > > >
> > > > It was all successfully working without ssl_bump, so the
> cache_peer
> > > > configuration works. One side note: the parent proxy is running
> > > on 443
> > > > without SSL (I believe - I don't run it but I've asked those that
> > > do for
> > > > confirmation, but I do know it's a pretty standard destination
> > proxy
> > > > configuration).
> > > >
> > > > The website itself is not directly accessible thus the upstream
> > > proxy is
> > > > required.
> > > >
> > > > Adding the ssl_bump configuration caused it to not work, with
> > errors
> > > > about SSL versions and "Error negotiating SSL connection on FD
> > > xx". My
> > > > best guess is that it is attempting to establish an SSL
> > connection to
> > > > the upstream and failing.
> > > >
> > > > acl step1 at_step SslBump1
> > > > ssl_bump peek step1
> > > > ssl_bump bump all
> > > > http_port 3128 ssl-bump cert=/var/lib/squid/ssl_cert/myCA.pem
> > > > generate-host-certificates=on dynamic_cert_mem_cache_size=4MB
> > > > tls_outgoing_options options=NO_SSLv3
> > > > flags=DON'T_VERIFY_PEER,DONT_VERIFY_DOMAIN
> > > > cert=/var/lib/squid/ssl_cert/device.pem
> > > > key=/var/lib/squid/ssl_cert/client.key
> > > >
> > > > (client.key and client.pem are from the device and are needed due
> > > to the
> > > > authentication of the session at the destination server. Also, I
> > > > haven't looked at the packet logging yet. I assume that will be
> > > an easy
> > > > addition once the setup works generally.)
> > > >
> > > > However, my understanding is that the cache_peer configuration
> > should
> > > > NOT do TLS by default unless that was specified in the options,
> > and I
> > > > see no way to explicitly disable it.
> > > >
> > > > So first question: is that assumption accurate? No TLS to the
> > parent
> > > > unless explicitly configured?
> > >
> > > Yes, that is correct: cache_peers are plain HTTP forward proxies
> > unless
> > > explicitly configured otherwise. Their listening port value does
> > carry
> > > any special meaning as far as Squid code is concerned.
> > >
> > > However, it is very unusual to run a plain HTTP forward proxy
> on port
> > > 443. That port may imply that your parent proxy is an HTTPS reverse
> > > proxy. If it is, you need to use originserver flag when
> > configuring the
> > > corresponding cache_peer line. You can check by sending plain
> CONNECT
> > > requests to that upstream proxy using wget, curl, or some such. A
> > > reverse HTTPS proxy would reject such requests.
> > >
> > >
> > > > if the ssl_bump configuration is causing it to attempt an
> upstream
> > > > TLS connection ...
> > >
> > > Bugs notwithstanding, it should not.
> > >
> > >
> > > > Anything here that I'm doing obviously wrong?
> > >
> > > I see no red flags relevant to your specific question.
> > >
> > > Does replacing "bump all" with "splice all" fix the problem? I
> > realize
> > > that you do want to bump/see the device traffic, but I wonder
> whether
> > > the errors are not between Squid and the upstream proxy but
> Squid and
> > > the web site. Splicing would remove those errors while still
> keeping
> > > some SslBump code active.
> > >
> > > Sharing a (pointer to compressed) libpcap packet capture and/or a
> > > (pointer to compressed) ALL,9 cache.log while reproducing the
> problem
> > > with a single transaction may help:
> > >
> >
> https://wiki.squid-cache.org/SquidFaq/BugReporting#Debugging_a_single_transaction
> > >
> > > Alex.
> > > _______________________________________________
> > > squid-users mailing list
> > > squid-users at lists.squid-cache.org
> > > http://lists.squid-cache.org/listinfo/squid-users
> > >
> > > Sent from Mailspring
> >
> > Sent from Mailspring
>
More information about the squid-users
mailing list