NUnit swallowed errorcodes
Update: The header is now in past tense as I figured out a GREAT solution; upgrade from 2.4.1 to 2.4.3!
By accident I earlier today checked in a failing unit-test to our TFS source control. While waiting for the dreaded red light from my CC tray I was about to fix it, when I suddenly discovered…
It didn’t fail the build!
Going through the build report revealed that it did in fact discover the failing test:
I couldn’t really live with that so I started investigating. I tried the following:
- Testing the NAnt-target local on my machine - build successful
- Manually running NUnit with all the same parameters as in the build-file, checking errorcode after run - errorcode 0 (success)
- At least now I know it’s NUnit that is failing
- For no good reason at all I’m just trying the failing test-assembly, leaving the other test-assemblies alone - errorcode 1 (1 failure)
- I’m on to something
- I add another test-assembly that doesn’t have failing tests in it - errorcode 0 (success)
It looks like NUnit forgets the result from the first test, and is just giving me the errorcode from the latest test-assembly provided!
Adding several test-assemblies to a NUnit project however does work correctly.
The solution
I didn’t want to make “static” NUnit project files, so I had to look for answers somewhere else and turned to the NUnit home pages. Earlier searches for errorcodes didn’t bring me to these pages at all, but I started looking around on the site and I found my way into the release notes.
I was using version 2.4.1 from May, and there was a 2.4.3 out from August. I downloaded it, updated my solutions tools folder, and….
It was fixed (the build failed… confused yet?!?)!
So, just to wrap it up: If you’ve been running NUnit 2.4.1 with more than one test-assembly, and your build server is happily saying the build is fine, don’t believe it!
Upgrade to a newer version and find out if your build is still fine .