[squid-dev] cppunit -> googletest / gmock?

[Amos Jeffries]

On 31/05/20 5:27 am, Francesco Chemolli wrote:
> Hi all,
>    starting from a PR in a conversation with Alex about our current
> approach to unit testing being painful, I've checked what alternatives
> would we have and how practical would they be.
> 
> An easy first option would be googletest/googlemock.
> 
> On a lazy afternoon, I've tried seeing how useful/painful it would be to
> try it, by porting one test over - it's quite trivial and doesn't
> require mocking, so I'll try a more complicated one next - to start a
> conversation about the topic.
> 
> You can find the test branch at https://github.com/kinkie/squid/tree/gtest .
> I've only touched two files, a newly-created src/tests/testMemGtest and
> src/Makefile.am .
> 
> The output from the test run is at https://paste.ubuntu.com/p/3sgTDN7rNm/
> 
> What do you think?
> 
> My initial thoughts:
> - it is somewhat simpler and more powerful than cppunit
> - setting the test environment up is easy but at this time it can only
> be done from source. Adding it to the build farm images is straightforward

That is a problem. The unit tests are run by pretty much everyone
building Squid.

It is not a complete blocker, but having a process more complex than
simple dependency install does pose a relatively major hurdle that any
framework has to get over to be of much utility.



> - the license is BSD 3-clause new
> (https://github.com/google/googletest/blob/master/LICENSE) 
> - googlemock promises to be vastly superior to our current approach

Where are you seeing evidence of this?

The main issue we have with cppunit itself is that when a test fails it
is not clear from the output which assertion occurred, nor why. One is
left having to try and trigger the unit failure again manually and gdb
from there.

This can be worked around by following best-practice in unit test
implementation. But people contributing to Squid have not been doing so
consistently, and it is just a workaround.

I do see somewhat more verbose output in the logs, and slightly less
code to implement (no .h class). Which is a nice gain, but not what I
would call "vastly superior".


> - porting memTest took me about one hour, mostly caused by us including
> cppunit headers from squid.h (WUT? A PR is coming up to unentangle that)
> 

Converting tests from one framework to another is not a problem. We just
have nobody doing the legwork. Case in point being the old tests not
even using cppunit.


The main problem(s) we have with testing of Squid is dev participation:

 a) people are not writing tests to cover new code, and
 b) people are not writing/updating tests to cover bug fixes, and
 c) tests written are generally not following best practice design.

IIRC Alex an I have different ideas about the ideal focus of testing;
 * I prefer the micro-test approach to demonstrate a high quality proof
of code reliability.
 * Alex has stated a preference for high level black-box testing of
behaviour vs design requirements.


Amos