Unit testing

A place to discuss everything about the Orxonox framework.

Moderator: PPS-Leaders

Post Reply
User avatar
x3n
Baron Vladimir Harkonnen
Posts: 810
Joined: Mon Oct 30, 2006 5:40 pm
Contact:

Unit testing

Post by x3n » Wed May 18, 2011 4:05 pm

This post is just a stub for the moment. I want to add unit testing to Orxonox soon (or maybe someone of you can do this) and stumbled upon some helpful links.

http://en.wikipedia.org/wiki/List_of_un ... ks#C.2B.2B < list of unit testing frameworks

http://www.ibm.com/developerworks/aix/l ... index.html < introduction to boost::test
http://www.ibm.com/developerworks/aix/l ... index.html < introduction to CppUnit
http://www.ibm.com/developerworks/aix/l ... index.html < introduction to CppTest
http://www.ibm.com/developerworks/aix/l ... ework.html < introduction to Google C++ test framework

http://en.wikipedia.org/wiki/List_of_mo ... ks#C.2B.2B < list of mocking frameworks
http://www.assembla.com/spaces/hippomoc ... Comparison < comparison of mocking frameworks
Fabian 'x3n' Landau, Orxonox developer

User avatar
x3n
Baron Vladimir Harkonnen
Posts: 810
Joined: Mon Oct 30, 2006 5:40 pm
Contact:

Re: Unit testing

Post by x3n » Thu May 19, 2011 8:18 pm

So I read some comparisons and opinions and the conclusion is that there are about N different unit testing frameworks and about M people writing about them, so we end up with roughly N*M different opinions (or more).

Now I don't know any C++ unit testing framework so far and I assume neither do you (if someone of you DOES, please write about it as soon as possible so we can consider your opinion (the N*M+1th) as well). I really don't feel like making this a religious topic, especially since I don't think Orxonox will ever be a test driven project.

I don't have a strong preference for any particular framework, but I definitely want a simple solution. So I prefer not to add yet another dependency to Orxonox and thus I think we may use boost::test. I think I read it got a major update in boost 1.36, so that should be our minimum required version for unit testing (has to be verified).

Additionally I found a mocking framework which is called Turtle which looks like it integrates seemlessly into boost::test. It's only a few files, so we can easily add it to our repository.

This is by far not a definite decision, just something that I think could work well without being too much pain. If anyone of you is willing to read more about the different frameworks and maybe even test and compare them, that would be very nice of course. Otherwise I'll probably go with the solution mentioned above (note that I'm not really motivated - but "someone has to do it", right?).
Fabian 'x3n' Landau, Orxonox developer

User avatar
1337
Baron Vladimir Harkonnen
Posts: 521
Joined: Wed Oct 10, 2007 7:59 am

Re: Unit testing

Post by 1337 » Sun May 22, 2011 6:28 pm

That's definitely a good idea. I myself once looked into this topic, but not very closely.
The major findings were basically, that we would have to be careful with boost::test. The testing facilities should definitely be in dedicated test files that are not built by default. Or else we might easily double the Orxonox compilation time because boost::test again includes close to a thousand headers.
Apart from these general Boost problems (which is why I always prefer not to use boost in a project), the framework seems to be very suitable, but also very general (maybe too general?).

As I've not really looked into it properly, I stop talking here ^^
http://www.xkcd.com/
A webcomic of romance, sarcasm, math, and language.

User avatar
x3n
Baron Vladimir Harkonnen
Posts: 810
Joined: Mon Oct 30, 2006 5:40 pm
Contact:

Re: Unit testing

Post by x3n » Sun Mar 04, 2012 3:36 pm

As you probably noticed, I finally found time and motivation to dig further into this topic.

Since we use CMake, it was an obvious choice to look at CTest first. CTest is something like a test runner - it runs test-executables and checks their exit code. It's possible to run it automated on a build-server and send the results to a dashboard. The general usecases are nightly builds or per-commit builds. It depends on the effort to build and run all tests, as well as the committing-frequency.

Using CTest is really simple. Just enable tests in CMake and add test executables. 'make test' will do the rest.

However, this is by far not all we need for unit testing. We also need a framework to write the tests in those test-executables. I looked at the different unit testing frameworks again and it basically boiled down to boost test or google c++ testing framework (aka google test or gtest).

gtest looks a bit more elaborate, but boost test is also quite good and we already use boost in Orxonox. That's why I preferred boost test in my last post in this topic.

However, I changed my mind and want to go with gtest now. Why?
  • It's a bit more elaborate
  • I found the documentation a lot clearer than the one of boost test
  • There are some helpful projects that can be used with gtest: google mock (mocking framework), gbar (test runner), gtest-runner-qt (test runner), plugins for different IDEs (test runner). A test runner is able to run single tests or groups of them and show the result in a gui (in comparison to running the whole executable, which may contain hundreds of tests, in the console).
  • Orxonox is an educational project and it seems like a better idea to teach students google test than boost test if they consider a job in the software industry
Of course this is very subjective, and it doesn't really matter anyway, but I guess you're fine with my choice anyway. ;)

We can include gtest in the source repository of Orxonox, it's quite small and builds quickly, so we don't need to add it as an external dependency. I'm currently trying to figure out the best way to write tests. Tests are always in separate source files in the test/ directory and compiled into separate executables. The question is how to link the real code. Some tests will probably link to the whole libraries of Orxonox, but for other it might be needed to link only to one .cc file (to reduce dependencies). This requires also some stubs (replacements for real classes if we don't want to link them into the test executable).

I'd say we just start to write some tests and see how it evolves.
Fabian 'x3n' Landau, Orxonox developer

User avatar
1337
Baron Vladimir Harkonnen
Posts: 521
Joined: Wed Oct 10, 2007 7:59 am

Re: Unit testing

Post by 1337 » Mon Mar 05, 2012 2:38 am

I really like the part where you don't use boost ;)

And I realized that I haven't done something for Orxonox for a very long time... It kind of got replaced by photography. Maybe that gets old some day ^^
http://www.xkcd.com/
A webcomic of romance, sarcasm, math, and language.

User avatar
x3n
Baron Vladimir Harkonnen
Posts: 810
Joined: Mon Oct 30, 2006 5:40 pm
Contact:

Re: Unit testing

Post by x3n » Mon Mar 05, 2012 6:50 pm

Yeah, I've been quite lazy lately too. It's about time to change that and get back to work - or drop Orxonox. But that would be a pitty. Writing some tests looks like a good oportunity to get back into the code and then move on to enhance the framework.

I'll need your advice concerning CMake for this project. In ran into some difficulties this weekend while writing some sample tests. In particular, src/ and test/ have a lot in common, e.g. include directories, the need for OrxonoxConfig, and more. Moving test/ into src/ would be one option. Moving CMake code from src/ to root is another. Writing everything twice looks like the worst idea, but is also feasible. I have to think about it a little and then I'll hit you with an email. ;)
Fabian 'x3n' Landau, Orxonox developer

User avatar
1337
Baron Vladimir Harkonnen
Posts: 521
Joined: Wed Oct 10, 2007 7:59 am

Re: Unit testing

Post by 1337 » Tue Mar 06, 2012 4:48 am

Sure, anytime, but it might take a day or two for me to answer because I'm currently working on a photo album, on site:
http://www.flickr.com/photos/rgrieder/s ... 423938691/
http://www.xkcd.com/
A webcomic of romance, sarcasm, math, and language.

User avatar
x3n
Baron Vladimir Harkonnen
Posts: 810
Joined: Mon Oct 30, 2006 5:40 pm
Contact:

Re: Unit testing

Post by x3n » Tue Mar 06, 2012 8:00 pm

You've got some really awesome photos there. I see you're using a Nikon D7000 and up to 300mm zoom. Good stuff. I have to look at your album in more detail.
Fabian 'x3n' Landau, Orxonox developer

User avatar
1337
Baron Vladimir Harkonnen
Posts: 521
Joined: Wed Oct 10, 2007 7:59 am

Re: Unit testing

Post by 1337 » Tue Mar 06, 2012 10:31 pm

Thanks. As you noticed I'm using a D7000, which is an awesome camera. It can do just about anything you'd want and doesn't weight and cost too much.
For normal use, I have Nikon's 16-85mm zoom and when things get far, the 70-300mm (also Nikon) has incredibly fast AF for the price and delivers really good images.
There's also a 35mm f/1.8 laying around, but I rarely use it because with VR, the travel zoom gives me the same low light performance but with a larger DOF. And producing background blur at normal length is rarely what you're looking for (esp. only at f/2.7 in film format equiv.). But it was a bargain, so I'm gonna keep it, just to do this ;)
http://www.flickr.com/photos/rgrieder/6 ... hotostream
http://www.xkcd.com/
A webcomic of romance, sarcasm, math, and language.

User avatar
x3n
Baron Vladimir Harkonnen
Posts: 810
Joined: Mon Oct 30, 2006 5:40 pm
Contact:

Re: Unit testing

Post by x3n » Fri Nov 30, 2012 10:17 pm

Now that we're about to support unit tests in Orxonox, new questions arise. Who should run those tests? When should they be run? What happens if they fail?

In software engineering, one considers continuous integration as "best practice". In short this means that developers commit frequently to a shared repository (in our case: SVN). After each commit (or sometimes only once per day) a server builds the project and runs automated tests. It provides a dashboard where developers can observe the progress of the development and the check if all tests were successful.
(Feel free to follow the link above to read more about this topic on wikipedia.)

There is a number of software tools for continuous integration, both open source and propertary. Among the open source tools I already heard about
  • Cruise Control
  • Hudson
  • Jenkins
At work I use Bamboo, a propertary tool.

They all offere more or less the same features: Defining build plans (which repository to check out, how to build it, how to run tests), project overview (does it build, were all tests successful) and often some nice graphical charts that show the progress over time. They mostly differ in terms of usability and esthetic appeal.

Today I gave Hudson a try, mostly because it's easy to set up and I had some evidence that it would work in combination with google-test. Now that I know how to set it up, I'm pretty sure google-test would also work with any other common tool because it has a commandline option to create an XML report after running the tests. This report is written in a more or less standardized format, so we're thus still open to try some other tools (e.g. Cruise Control). But for the moment Hudson will do the job.

I set it up to build the "testing" Branch (the only one which contains unit tests so far) and it worked almost instantly, I only had to tweak some of my tests to run on Linux (I only tested it on Windows until then). It runs on Port 8080 on our server, but it's not public yet (you could connect with an SSH tunnel). Below are some screenshots.

The build overview: You can see 4 failed (red) tests, the 5th worked (blue). I assume it's optimized for red-green-colorblind people, hence the blue color for success. ;)
hudson1.JPG
hudson1.JPG (73.92 KiB) Viewed 30564 times
The details of the 5th build:
You can see my commit 9479 that made it work plus the information that all tests were successful.
hudson2.JPG
hudson2.JPG (69.29 KiB) Viewed 30564 times
A simple graphical chart that shows the execution time (or the number) of the tests (there were only tests executed in builds 4 and 5, hence there are only two samples visible). Obviously this doesn't make too much sense now (with execution times < 0.1s), but it could be handy later.
hudson3.JPG
hudson3.JPG (67.89 KiB) Viewed 30564 times
That's it so far, it's not much but it's a beginning.

We can do more fancy stuff with continuous integration: Measuring unit test code coverage (how much of our code is tested), computing software quality metrics (do we adhere to common standards), or building packages (e.g. nightly builds).

By the way, I just realized that the screenshots are in german. Sorry for this, but I think it doesn't really matter at the moment. ;)
Fabian 'x3n' Landau, Orxonox developer

User avatar
x3n
Baron Vladimir Harkonnen
Posts: 810
Joined: Mon Oct 30, 2006 5:40 pm
Contact:

Re: Unit testing

Post by x3n » Sun Dec 16, 2012 9:10 pm

After following this tutorial and installing gcovr on our server, I managed to get test coverage reports in Hudson. They show how much of our code is actually run by the tests.
Attachments
hudson4.PNG
hudson4.PNG (38.57 KiB) Viewed 30513 times
hudson5.PNG
hudson5.PNG (44.12 KiB) Viewed 30513 times
Fabian 'x3n' Landau, Orxonox developer

User avatar
x3n
Baron Vladimir Harkonnen
Posts: 810
Joined: Mon Oct 30, 2006 5:40 pm
Contact:

Re: Unit testing

Post by x3n » Sun Mar 10, 2013 8:10 pm

Recently I added some tests for the output systems which allowed me to dig further into google-test and google-mock. They provide many great features (I didn't use all of them by far) and I believe those libraries will serve us well.
The tests I added can be used as examples for future unit tests, either implemented by me or by anybody else. Feel welcome. :)

Our continuous integration tool Hudson of course noticed my new tests and plottet some nice graphs in the meantime, showing both the increase of successful tests as well as the increased code coverage in the src/libraries/util/output package.
Attachments
hudson6.PNG
hudson6.PNG (10.51 KiB) Viewed 30305 times
hudson7.PNG
hudson7.PNG (18.75 KiB) Viewed 30305 times
Fabian 'x3n' Landau, Orxonox developer

User avatar
x3n
Baron Vladimir Harkonnen
Posts: 810
Joined: Mon Oct 30, 2006 5:40 pm
Contact:

Re: Unit testing

Post by x3n » Tue Mar 12, 2013 10:17 pm

I merged the branch back to trunk. You can run the tests with

Code: Select all

make test ARGS="--output-on-failure"
or simply

Code: Select all

make test
Fabian 'x3n' Landau, Orxonox developer

User avatar
beni
Baron Vladimir Harkonnen
Posts: 949
Joined: Tue Oct 03, 2006 9:15 am
Location: Zurich
Contact:

Re: Unit testing

Post by beni » Wed Mar 13, 2013 5:03 pm

Great, I will test the testing :wink:
"I'm Commander Shepard and this is my favorite forum on the internet."

Post Reply

Who is online

Users browsing this forum: No registered users and 4 guests