[squid-dev] cppunit -> googletest / gmock?

[Francesco Chemolli]

On Sat, May 30, 2020 at 7:38 PM Amos Jeffries <squid3 at treenet.co.nz> wrote:

> On 31/05/20 5:27 am, Francesco Chemolli wrote:
> > Hi all,
> >    starting from a PR in a conversation with Alex about our current
> > approach to unit testing being painful, I've checked what alternatives
> > would we have and how practical would they be.
> >
> > An easy first option would be googletest/googlemock.
> >
> > On a lazy afternoon, I've tried seeing how useful/painful it would be to
> > try it, by porting one test over - it's quite trivial and doesn't
> > require mocking, so I'll try a more complicated one next - to start a
> > conversation about the topic.
> >
> > You can find the test branch at
> https://github.com/kinkie/squid/tree/gtest .
> > I've only touched two files, a newly-created src/tests/testMemGtest and
> > src/Makefile.am .
> >
> > The output from the test run is at
> https://paste.ubuntu.com/p/3sgTDN7rNm/
> >
> > What do you think?
> >
> > My initial thoughts:
> > - it is somewhat simpler and more powerful than cppunit
> > - setting the test environment up is easy but at this time it can only
> > be done from source. Adding it to the build farm images is
> straightforward
>
> That is a problem. The unit tests are run by pretty much everyone
> building Squid.
>

We should be able to bundle gtest if needed; we only distribute in source
form and it wouldn't be part of distfiles so it shouldn't be a problem
downstream either, I believe


> It is not a complete blocker, but having a process more complex than
> simple dependency install does pose a relatively major hurdle that any
> framework has to get over to be of much utility.
>
>
>
> > - the license is BSD 3-clause new
> > (https://github.com/google/googletest/blob/master/LICENSE)
> > - googlemock promises to be vastly superior to our current approach
>
> Where are you seeing evidence of this?
>

Also including answer on Alex' question:
- https://stackoverflow.com/questions/7922289/googletest-vs-cppun
it-the-facts
<https://stackoverflow.com/questions/7922289/googletest-vs-cppunit-the-facts>
- trivial but it builds up: not generally necessary to have .h and .cc for
simple cases
- comparison table:
https://socialcompare.com/en/comparison/c-unit-testing-framework
- gmock/gtest is used in broadly-used projects (e.g. chromium) - I'm not
sure


So far I'm basing on documentation, found at
https://github.com/google/googletest/blob/master/googlemock/docs/cook_book.md
 .
I see:
- a more structured approach to mocking, and more infrastructure to do it
- ON_CALL and EXPECT_CALL patterns to define actions and expectations on
class methods (method called once / called multiple times). This can also
allow to change the behaviour of a mocked object on different tests without
having to remock it all
- matchers on called methods
- moched methods can different return values depending on arguments, and
check for complex sequences of calls to methods (call a() with some
arguments, then either b() or c(), if it's c() must be followe by d()
- for object, the concept of "uninteresting calls" that get ignored, and
can be defaulted



> The main issue we have with cppunit itself is that when a test fails it
> is not clear from the output which assertion occurred, nor why. One is
> left having to try and trigger the unit failure again manually and gdb
> from there.
>
> This can be worked around by following best-practice in unit test
> implementation. But people contributing to Squid have not been doing so
> consistently, and it is just a workaround.
>
> I do see somewhat more verbose output in the logs, and slightly less
> code to implement (no .h class). Which is a nice gain, but not what I
> would call "vastly superior".
>

Right. A lot of value comes from the combination with gmock.


>
>
> > - porting memTest took me about one hour, mostly caused by us including
> > cppunit headers from squid.h (WUT? A PR is coming up to unentangle that)
> >
>
> Converting tests from one framework to another is not a problem. We just
> have nobody doing the legwork. Case in point being the old tests not
> even using cppunit.
>

>
> The main problem(s) we have with testing of Squid is dev participation:
>
>  a) people are not writing tests to cover new code, and
>  b) people are not writing/updating tests to cover bug fixes, and
>  c) tests written are generally not following best practice design.
>
> IIRC Alex an I have different ideas about the ideal focus of testing;
>  * I prefer the micro-test approach to demonstrate a high quality proof
> of code reliability.
>  * Alex has stated a preference for high level black-box testing of
> behaviour vs design requirements.
>

TBH I favour unit tests as well. But if we can have black box texting (and
there are compliance testing frameworks to do that for us) that'd really
increase the confidence in correctness over time. The two do not cancel
each other, luckily. The former can do fast-feedback return, the latter can
say "something is wrong" and then we can - if needed - bisect to point to a
specific commit


-- 
    Francesco
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.squid-cache.org/pipermail/squid-dev/attachments/20200531/abfbc88a/attachment.html>