[squid-users] ssl_bump with parent cache

Wed Mar 9 21:16:14 UTC 2022

So after a bit more experimentation. the difference in what works and what doesn't is:
ssl_bump peek step1
ssl_bump splice all

(working) and this not working:
ssl_bump peek all
ssl_bump splice all

I'm not clear on why peeking at "all" vs. "step1" would cause it to fail. However, I also note that not peeking at anything makes it work also (commenting out the "peek" line), so I'm suspicious that "peek step1" is effectively having it do nothing.
"stare step1" also works, but "stare all" does not (i.e. same result as "peek").
"stare step1" followed by "bump all" fails as well (i.e. changing "splice" to "bump"), as does "stare all/bump all".
I'm probably not going to be able to share any packet dumps so that does make things difficult. This device is under development and the details of "what it's doing" is likely to be sensitive. I appreciate that makes troubleshooting things difficult. What I could do is put a VM behind it and do manual wgets or something from it but that will take some changes to the environment, or perhaps just creating a new one to test this where this device isn't present. If that's where it needs to go, I'll have to start discussions on exactly how sensitive it is and/or start creating the new/modified setup.
Anyway, I do appreciate the suggestions, and the lightbulb is starting to go on, albeit dimly. :)
On Mar 8 2022, at 4:16 pm, Alex Rousskov <rousskov at measurement-factory.com> wrote:
> On 3/8/22 17:56, Aaron Dewell wrote:
> > Ok, with a bit more messing with it... Changing bump to splice does work:
>
> > ssl_bump splice all
> Noted.
>
> > Adding:
> >
> > acl step2 at_step SslBump2
> > ssl_bump peek step1
> > ssl_bump peek step2
> > ssl_bump splice step2
>
> The above is a bad configuration because no rule matches during step3.
> Please try this configuration instead, to make sure you are not seeing a
> side effect of various Squid bugs related to such configurations (we are
> working on fixing them):
>
> ssl_bump peek all
> ssl_bump splice all
>
> If the above fails, please see my earlier suggestions about sharing test
> artifacts. Otherwise, please try this:
>
> ssl_bump stare all
> ssl_bump bump all
>
> If the above fails, please see my earlier suggestions about sharing test
> artifacts. Otherwise, please try this:
>
> ssl_bump peek step1
> ssl_bump bump all
>
> If the above fails, please see my earlier suggestions about sharing test
> artifacts. Otherwise, you got a working config (which is the same config
> you have started with, so I doubt you will get to this stage!).
>
>
> > This appears to be the relevant data from the log:
> >
> > 2022/03/08 14:35:47.644 kid1| 83,7| bio.cc(163) stateChanged: FD 10 now: 0x4004 TED (TLSv1.3 early data)
> > 2022/03/08 14:35:47.644 kid1| 83,7| bio.cc(163) stateChanged: FD 10 now: 0x2002 SSLERR (error)
> > 2022/03/08 14:35:47.644 kid1| Error negotiating SSL connection on FD 10: error:00000001:lib(0):func(0):reason(1) (1/-1)
>
> YMMV, but I will need to see a lot more lines surrounding this snippet
> to understand what exactly went wrong. That is why I am asking for the
> entire transaction log rather than selected "error" lines.
>
> Sharing a packet capture might be sufficient if this is a basic
> misconfiguration error -- we will see wrong protocol traffic on one of
> both Squid sides. You can start with that (often, but not always, safer
> as far as sensitive info disclosure) step if you prefer.
>
>
> HTH,
> Alex.
>
> > On Mar 8 2022, at 2:57 pm, Alex Rousskov wrote:
> >
> > On 3/8/22 16:38, Aaron Dewell wrote:
> >
> > > Hi Alex, thanks for your reply! I did get access to the parent proxy
> > > and my assumption was wrong, it's doing minimal bumping.
> >
> > TLS inspection at the parent proxy does not affect what I was trying to
> > double check. What matters is whether it is a forward HTTP proxy (e.g.,
> > a Squid instance listening on an http_port configured without intercept,
> > tproxy, or accel flags). It sounds like that is what it is. That's OK!
> >
> > I do not remember whether your Squid version (v4.13) supports SslBump
> > with parent forward proxies, but I believe modern Squids do, and we can
> > assume that your Squid version does as well. The debugging should show
> > whether that is indeed the case.
> >
> >
> > > The parent is doing peek and splice to an exact list of internal
> > > destinations. Specifically, peek step1 all, peek step2
> > allowed_sites,
> > > splice allowed_sites, terminate all. That shouldn't (to my imperfect
> > > knowledge) interfere with what I'm doing though.
> >
> > Yes, assuming the parent does not terminate your Squid connections, of
> > course (i.e. assuming connections from your Squid always match
> > allowed_sites after step1 at the parent proxy).
> >
> >
> > > That's a good idea to do splice only and see if that's successful.
> > > Trying to do too much at once! I'll see what that does, then try
> > debug
> > > again. The debug output wasn't very helpful before, but stepwise
> > may be
> > > more useful.
> >
> > Just to clarify: The debugging output was not meant for you to
> > interpret. A Squid developer should do that. It may contain sensitive
> > details; it is best to not use anything but test traffic and test
> > certificate keys when sharing ALL,9 output.
> >
> > Alex.
> >
> >
> > > On Mar 8 2022, at 1:43 pm, Alex Rousskov wrote:
> > >
> > > On 3/8/22 14:16, Aaron Dewell wrote:
> > >
> > > > I'm trying to use these two features at the same time. The use
> > > case is
> > > > pretty simple. I want to capture all traffic from a single
> > source (a
> > > > device of mine) to another squid proxy server and decrypt/log
> > > it. I'm
> > > > using the Ubuntu 20 package of squid-ssl version 4.13.
> > > >
> > > > Device -> ssl_bump proxy -> upstream proxy -> website
> > > >
> > > > It was all successfully working without ssl_bump, so the cache_peer
> > > > configuration works. One side note: the parent proxy is running
> > > on 443
> > > > without SSL (I believe - I don't run it but I've asked those that
> > > do for
> > > > confirmation, but I do know it's a pretty standard destination
> > proxy
> > > > configuration).
> > > >
> > > > The website itself is not directly accessible thus the upstream
> > > proxy is
> > > > required.
> > > >
> > > > Adding the ssl_bump configuration caused it to not work, with
> > errors
> > > > about SSL versions and "Error negotiating SSL connection on FD
> > > xx". My
> > > > best guess is that it is attempting to establish an SSL
> > connection to
> > > > the upstream and failing.
> > > >
> > > > acl step1 at_step SslBump1
> > > > ssl_bump peek step1
> > > > ssl_bump bump all
> > > > http_port 3128 ssl-bump cert=/var/lib/squid/ssl_cert/myCA.pem
> > > > generate-host-certificates=on dynamic_cert_mem_cache_size=4MB
> > > > tls_outgoing_options options=NO_SSLv3
> > > > flags=DON'T_VERIFY_PEER,DONT_VERIFY_DOMAIN
> > > > cert=/var/lib/squid/ssl_cert/device.pem
> > > > key=/var/lib/squid/ssl_cert/client.key
> > > >
> > > > (client.key and client.pem are from the device and are needed due
> > > to the
> > > > authentication of the session at the destination server. Also, I
> > > > haven't looked at the packet logging yet. I assume that will be
> > > an easy
> > > > addition once the setup works generally.)
> > > >
> > > > However, my understanding is that the cache_peer configuration
> > should
> > > > NOT do TLS by default unless that was specified in the options,
> > and I
> > > > see no way to explicitly disable it.
> > > >
> > > > So first question: is that assumption accurate? No TLS to the
> > parent
> > > > unless explicitly configured?
> > >
> > > Yes, that is correct: cache_peers are plain HTTP forward proxies
> > unless
> > > explicitly configured otherwise. Their listening port value does
> > carry
> > > any special meaning as far as Squid code is concerned.
> > >
> > > However, it is very unusual to run a plain HTTP forward proxy on port
> > > 443. That port may imply that your parent proxy is an HTTPS reverse
> > > proxy. If it is, you need to use originserver flag when
> > configuring the
> > > corresponding cache_peer line. You can check by sending plain CONNECT
> > > requests to that upstream proxy using wget, curl, or some such. A
> > > reverse HTTPS proxy would reject such requests.
> > >
> > >
> > > > if the ssl_bump configuration is causing it to attempt an upstream
> > > > TLS connection ...
> > >
> > > Bugs notwithstanding, it should not.
> > >
> > >
> > > > Anything here that I'm doing obviously wrong?
> > >
> > > I see no red flags relevant to your specific question.
> > >
> > > Does replacing "bump all" with "splice all" fix the problem? I
> > realize
> > > that you do want to bump/see the device traffic, but I wonder whether
> > > the errors are not between Squid and the upstream proxy but Squid and
> > > the web site. Splicing would remove those errors while still keeping
> > > some SslBump code active.
> > >
> > > Sharing a (pointer to compressed) libpcap packet capture and/or a
> > > (pointer to compressed) ALL,9 cache.log while reproducing the problem
> > > with a single transaction may help:
> > >
> > https://wiki.squid-cache.org/SquidFaq/BugReporting#Debugging_a_single_transaction
> > >
> > > Alex.
> > > _______________________________________________
> > > squid-users mailing list
> > > squid-users at lists.squid-cache.org
> > > http://lists.squid-cache.org/listinfo/squid-users
> > >
> > > Sent from Mailspring
> >
> > Sent from Mailspring
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-users/attachments/20220309/ea6f561b/attachment.htm>