By accident I earlier today checked in a failing unit-test to our TFS source control. While waiting for the dreaded red light from my CC tray I was about to fix it, when I suddenly discovered…

It didn’t fail the build!

Going through the build report revealed that it did in fact discover the failing test:

I couldn’t really live with that so I started investigating. I tried the following:

• Testing the NAnt-target local on my machine – build successful
• Manually running NUnit with all the same parameters as in the build-file, checking errorcode after run – errorcode 0 (success)
• At least now I know it’s NUnit that is failing
• For no good reason at all I’m just trying the failing test-assembly, leaving the other test-assemblies alone – errorcode 1 (1 failure)
• I’m on to something
• I add another test-assembly that doesn’t have failing tests in it – errorcode 0 (success)

It looks like NUnit forgets the result from the first test, and is just giving me the errorcode from the latest test-assembly provided!

Adding several test-assemblies to a NUnit project however does work correctly.

The solution

I didn’t want to make “static” NUnit project files, so I had to look for answers somewhere else and turned to the NUnit home pages. Earlier searches for errorcodes didn’t bring me to these pages at all, but I started looking around on the site and I found my way into the release notes.

I was using version 2.4.1 from May, and there was a 2.4.3 out from August. I downloaded it, updated my solutions tools folder, and….

It was fixed (the build failed… confused yet?!?)!

So, just to wrap it up: If you’ve been running NUnit 2.4.1 with more than one test-assembly, and your build server is happily saying the build is fine, don’t believe it!

Upgrade to a newer version and find out if your build is still fine .