Unit Testing Frameworks: Deciding
As we start migrating towards a more agile environment at work, I want to bring forward a unit testing framework that we could standardize on. The way I see it there are four unit testing frameworks available for the .NET framework right now. These are NUnit, MbUnit, csUnit and Zanebug.
NUnit is the de facto standard as far as unit testing goes. NUnit started as a direct port of Kent Beck’s JUnit framework. Over the course of time, it has evolved to use several features of the .NET framework that make it much easier to use such as custom attributes and reflection related capabilities. Most books and tutorials that are written around unit testing in the .NET framework use NUnit as the framework of choice. This is a big deal.
csUnit has not been updated since November 9, 2003. No, thanks! I want something that will have continuous development and improvements.
I believe that Zanebug is the latest player in the unit testing framework arena. Zanebug looks very promising as an open source unit testing tool. It includes NUnit compatibility to the point of providing the ability to run existing NUnit tests without modification. It also provides additional features such as performance metrics, results graphing, and perfmon integration.
MbUnit was written by Jonathon de Halleux and has quickly progressed into a very robust framework. One of my favorite features of MbUnit is the console runner that you can use to generate HTML reports. It allows you to give your unit test assembly a Main method and distribute it as an exe. This is kinda cool.
A very compelling argument for MbUnit is its packaging with TestDriven.NET which is an addin that lets you execute tests on any piece of code directly in the VS.NET IDE.
Similarly to Zanebug, MbUnit is compatible with NUnit in that you only have to change the namespaces. MbUnit seems to have a lot of the same additional features that Zanebug does, including test decorators which handle repeated tests, “timed” tests, load tests, multi-threaded tests. MbUnit offers a very rich feature set, but there is also a nice extensibility model should there be something that you need that isnt present.
At work, we are using CruiseControl.NET for all of our projects and so I think it would be really nice if the unit testing framework could be integrated into NAnt and CruiseControl. At this point, the only solutions that do this are NUnit and MbUnit.
As such, I think the recommendation that Im going to bring to our team will be MbUnit, as it seems to have the most features as well as integration with the tools that we’re using (VS.NET and CruiseControl).
Am I missing anything? Which framework are you using, and why?
Nice overview. I’m keen to hear how you get on.
Stuff you might or might not know:
I’ve used MbUnit on several projects but I would hesitate to base my testing on that framework in the future!
MbUnit is a great framework but has been built by a single developer. He has lately joined MS and there have been no updates to MbUnit since its release with TestDriven.NET. The latest entry on his blog is from January 9th (http://blog.dotnetwiki.org/) :( Further, he does not respond to emails (at least not mine).
The source code is not publicly available anymore after it was moved from tigris.org last year.
The TestDriven.NET wiki describes random thoughts about going commercial closed-source which further reduces my interest in the product (http://www.testdriven.net/wiki/default.aspx/MyWiki.TestDrivenPro). I don’t know for sure that it would include MbUnit but as TestDriven.NET is the only official way to get MbUnit I could imagine so.
I would really like an update from the authors of MbUnit and TestDriven.NET but I haven’t seen anything from a while.
Thanks Mr. B. Those are all things I’d consider pretty relevant. Maybe I won’t go look at MBUnit after all.
T.
Something even more critical, imo, is that the integration with CruiseControl.Net has a MAJOR drawback. (I found this out the hard way). CruiseControl fails a build when NUnit tests fail, but not when MbUnit tests fail. Technically, this is more of an issue with CC.NET than with MbUnit, but if you’re looking for something to deploy today, it should be taken into consideration. Of course, it may be that we’re doing something wrong, but I don’t think so. Anyway, I’m going to search the CC.NET issues and possibly submit a bug (if I’m right about this and there isn’t already an open case). I’ll keep you in the loop if you wish.
I really like MbUnit for the ways that it has innovated in the unit testing space. For example, you can create a unit test that runs through a table of data specified as attributes to the unit test. That’s pretty neat considering how many tests simply run the same code through a lot of different sample data.
Having said that, I’ve stuck with NUnit because although MbUnit can run NUnit, if you use the interesting features of MbUnit, you won’t be able to backport to NUnit. Most shops that do unit testing at all use NUnit.
As you said, it’s the de-facto standard and that’s in its favor.
Matt,
Thanks for the comments! FYI, there will be a new Zanebug release within the next few days (beta) that includes VS.NET integration similar to TestDriven.NET…we’re just doing some final testing.
NAnt integration and HTML reports are next on the list, and should be available within the next few weeks (Zanebug follows a 4-6 week release cycle). Here’s the release plan (bottom of the page): http://www.adapdev.com/zanebug/index.aspx
If you have any features you’d like to see added (such as the console runner that you mentioned), please let me know.
Sean
Sean: Thanks for the heads up!
Christian: Id love it if you let me know what you find out.
These are issues that are making me re-evaluate my recommendation. Thanks for the feedback!
Of the four you mentioned, only csUnit is currently capable of detecting tests in partially trusted assemblies. (A subset of this problem is documented for NUnit at http://sourceforge.net/tracker/?group_id=10749&atid=110749&func=detail&aid=1104516). This makes csUnit the only reasonable choice for conveniently testing how one’s code behaves when called from partially trusted code. Of course, csUnit is inconvenient in other ways, but there’s nothing locking one into using a single testing framework for all tests.
BTW, on the security side of things, you may also want to consider that, as installed, the current release of Zanebug doesn’t run properly under LUA. This can be fixed by adjusting ACLs to grant write permissions on certain files (or the whole app folder, if one is lazy;)), but this could be a bit of a pain on a large dev team.
I just recently switched from NUnit to MbUnit because of the extra functionallity and direct integration with VS via TD.Net (NUnitAddIn). MbUnit is my current recommendation.
I would suggest that more important than NAnt and CruiseControl integration is the ability for good command line execution with good xml output. If those two things exist you can integrate it with any build process and any continuous integration system. I’ve found that tight coupling of tools is as detrimental in the build process as it is beneficial in the development process (i.e. within the IDE).
I’d stick with NUnit and create custom attributes for that – it’s thesafest route I can see these days. I might use Zanebug to simply run the unit tests though.
I use MbUnit, and I really like it.
The only problem is that it’s apperantly not open source.
I’ve email Jhonatan about it, but I didn’t get an answer.
MbUnit allows to do some things very easily (TypeFixtures, factories, etc) that you need to do manually in NUnit.
I’ve found that I use different versions of different testing frameworks for different projects. From that I suggest picking the “best framework” from time to time. Because the code is easily converted this should not be an issue.
Having said that.
1) I use MBUnit at present. The test driven integration is really great.
2) Jonathan has joined MS and is currently going through a quiet period. This is common. Charles Stirling (MS Australia) suggested to me yesterday that the quietness is about absorbing a lot of new stuff and getting up to speed. Say that takes 9 months, then we might be looking at September for a restart from the remarkable de Halleux.
3) Like another poster here I’m concerned about the removal of the open source and related reference I’ve seen in blog entries.
I guess this will be a common issue as people decide on their NSuite policy. (The Team System licensing model doesn’t fit me, at all, so I’m going to use non MS tools for Source Control, Coverage, Testing, Documentation Generation etc. I call this range of things NSuite.) Part of my policy is to be open to change. I prefer MBUnit but can switch easily if I change my mind, Log4Net might be good in some cases while Enterprise Library is good in others.
PS. Unit testing has a clear cut meaning. A lot of what I do is technically not Unit Testing but Integration Testing. It might be time for a nomenclature shift!!
I see that Jonathan (on April 15) has passed the MBUnit material to Jamie Cansdale (of TestDriven.NET).
http://blog.dotnetwiki.org/archive/2005/04/15/1568.aspx?Pending=true
It’ll be interesting to see what happens next.
Peli has made an update on his blog: http://blog.dotnetwiki.org/archive/2005/04/15/1568.aspx
I really hope that this will get MbUnit up and running again as a community driven testing framework…
Matt,
From what I’ve been able to find out, the issue is that MbUnitConsole is returning 0 even when tests fail (because there isn’t an error in mbUnit itself I suppose) but the Nant exec task sees this as everything being a-Ok. Had mbUnit returned an error code for failing tests, Nant would report build failed, and subsequently so would CC.NET.
Peli recently announced that Jamie (of TD.NET) is taking over as project shepherd, so maybe we’ll get an updated source release soon and development can continue. Having used MbUnit, I can’t bear the thought of going back to NUnit.
Jamie is working through some coding and then expects to release on Tigris or Sourceforge.
Matt Berther just likes mbUnit because he likes to take credit for writing it.
Which is currently the best????????
