[squid-dev] How to run the unit tests of Squid?

[Amos Jeffries]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 14/10/2014 8:40 p.m., Tianyin Xu wrote:
> Thanks, Amos!
> 
> One more question. I see test-builds.sh tries to run the tests in
> the "test-suites" at the top directory. What's the difference
> between the test under "src/tests" and "test-suites"?
> 
> I'm a little bit confused since you said the tests in "src/tests"
> contains all the unit tests.
> 
> Also, I can successfully ran make check. But when I checked out the
> trunk and ran ./test-builds.sh at the top directory, many tests
> fail. Is this normal? (sorry I actually do not understand what
> "test-suites" does)

That depends on how complete your build environment is. The script
begins with testing ./bootstrap.sh to ensure the toolchain is complete
and working. Recent autoconf produce a lot of warnings about
subdir-objects feature, but should still finish saying "success".
 If it does not then the error it exits with should explain what needs
to be resolved, usually some portion of the autotools build
environment is outdated or missing.

For the later "layers" being tested (eg the ESI layer) there are some
additional libraries needed to be installed which are optional for
normal users.

You *need* this script to run to completion with no errors on the
current 3.HEAD code for it to be worth going any further into the testing.

So, what OS are you building on?
 and what are the specific error(s) the script is failing with?

> 
> Oh, I forget to mention the web page I referred to. Actually I'm
> referring to the "tasks needing a volunteer" in the following wiki
> page, http://wiki.squid-cache.org/Features/CodeTestBed Are these
> tasks still alive? Since I'm looking at the tests, I think I can 
> give a hand.

Aha. Wonderful thank you. In that case I better explain in a bit more
detail than just how to use them. :-)

src/tests/ content is the main code for what the wiki page Details
section refer to as "ad-hoc CPP Unit-tests". Other unit-tests also
exist elsewhere in the sources (eg pre-compiler unit tests in compat/).

Each of those src/tests/test*.cc is a test of a single "unit" (aka.
C++ class). "make check" turns it into a binary that is supposed to
exhaustively test what happens when any permutation of input is
presented to the class API/methods, including edge cases and soemtimes
wrong inputs.

Unit-tests like that can only test one way of compiling the code at a
time. Which is fine for validating that some pre-built binary will
work as intended. But fails miserably to validate source code for
software like Squid with many possible features built (or not) by users.

That is where the ./test-builds.sh script comes in. (See under
"Component Test Controls".) It runs the unit tests with different
permutations of ./configure settings.

It also runs some large scale tests, such as squid.conf parsing for
various known squid.conf contents which have or might cause trouble,
or the bootstrap to check the build chain is operational.

Ideally both unit-test and the ./configure tests would be exhaustive
and cover every bit of code at least once. That is a very long way off
still, and may actually never happen. But that is the goal.


So, yes the items in both "Tasks needing a volunteer:" lists still
need doing.

 The work is being coordinated in wiki page
<http://wiki.squid-cache.org/Features/SourceLayout>. The main table
first column indicates whether the new polished convenience libraries
have unit-tests for the classes inside.
 That column content is outdated, in fact many of those classes do
have unit-tests but nobody has audited to check if they are complete
and correct for the current API. So part of the needed work is
correcting that column entries.

 There is another item related to the testing which is not documented
there. That is the compiler build options which are available to
detect potential issues. Squid is only able to build cleanly with a
small set at present, additional work is needed to try building with
others and fix the issues they highlight.


I hope that is clear enough to understand. I am assuming you at least
know the basics of what testing is all about.


PS. since 3.5 is now in beta I will this week be auditing the
./configure tests options are up to date. That may affect the test
coverage and results somewhat in the coming days.

Amos
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (MingW32)

iQEcBAEBAgAGBQJUPPpqAAoJELJo5wb/XPRjKhEH/1QjhBXv6n3MM+EhduJ5q5NP
vKYzNTrg71vkoCcZ/7VoTTp5NXCdS9D4mg5cfaZtjHYU5HhZJqaJ3sxp3iuRE8Sh
EWHEIUWPu3+95PAdnnB4FAzEcwrDZoTvbMc7P7Mo+/vKQp3ljfA2x+uYLUZjtucV
h/N5W96ZfAWqYFdHUeZFh0xJ6+bNtsV1ndjlnSddPfMFfCCZIzEk/+QQacuZm9d7
aS7lO8Ig7IlZ+v6FJglWbVt7PGEnvQclS/J9hH6lQqV5bwQPnGZiS4zsLm2MFfPK
vMIRngmKrAw6vDrdCpuEXHjFBxosaGdLHTvrwwMMTFTMbFo0fRN9lcFJiZ7qI+g=
=zK+F
-----END PGP SIGNATURE-----