16

I'm working on a Geometry library. There are 200+ unit tests.

There's a particularly stubborn test that fails whenever I select "Run All", but the test passes when I run that test individually, or use the debugger on it. I do believe the issue showed up about when I shifted over from visual studio '13 to the '15 edition.

Now some notes about the geometry library:

The objects are immutable.

The tests have no shared objects between them.

So my Question: What are the possible causes for this odd behavior?

Edit:

[Test()]
public void Plane_IntersectionWithPlane_IdenticalPlane()
{
     Plane testPlane = new Plane(new Direction(Point.MakePointWithInches(2, -1, 1)), 
                                               Point.MakePointWithInches(2, 1, 2));
     Line found = (testPlane.Intersection(testPlane));

     Line expected = new Line(new Direction(Point.MakePointWithInches(0, -1, -1)), 
                                            Point.MakePointWithInches(2, 1, 2));
     Assert.IsTrue(found.Equals(expected));
}
Improve this question
17
  • What's your question? Commented Jul 28, 2015 at 17:31
  • Are there any well known quirks to using NUnit that might cause a test to fail when run as part of a group, vs. run by itself? Commented Jul 28, 2015 at 17:32
  • 8
    No, this is very likely on your side. Show a short, reproducable example that demonstrates the problem and we can look at the exact issue. Likely there's a shared resource, incorrect asynchronicity or a myriad of other possibilities. Commented Jul 28, 2015 at 17:41
  • 1
    You are using Point class statically to call the function MakePointWithInches. Is the only case where one test can depend from another test that also use this funcition... Do you check it? Can you share the implementation of the Point class? Commented Aug 29, 2015 at 22:49
  • 1
    IMHO you need to provide more details. With the given information there are too many possibilities: Threading, static members, caching, ... Commented Sep 2, 2015 at 19:43

5 Answers 5

Reset to default
7
+50

I've come across similar issues in the past, but it's always turned out to be some unexpected interaction between the underlying code elements, or the way that the tests have been written. Some issues to check are:

  • Static object construction / destruction
  • Synchronisation issues
  • Mock setup / cleanup issues

I've found the best approach to track down the issue is follow an approach similar to that suggested by @Matthew Strawbridge in the comments. I add Ignore attributes to Tests/TestFixtures to remove tests until the run-all starts working and then start adding them back in until it breaks again narrowing down the problem.

Sometimes you will also find that ignoring the test that is currently failing will cause another test to fail instead. This is a good indication that the problem is actually being caused by another one of your tests not cleaning up after itself correctly.

Looking at the code between the tests that fail/seem to cause the fail can then help you to narrow down the interaction. The error/failure reason for the tests can also help of course...

Selecting 'Run-all' runs all of the tests in a predictable order, so the tests will generally be run the same way everytime. If you select a batch of tests then it's possible that the runner may choose to run them in a different order, based around your selection order which may be why you've experience different behaviour when selecting tests, rather than using run-all.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

3

Try the following:

Open Nunit GUI And before you run the test change the above in the settings:

enter image description here

It helped me once detect my problems.

BTW: what version of NUNIT are you using ? If you don't know what is Nunit GUI then you probably didn't downloaded Nunit separately. You can get the install from here :

http://www.nunit.org/index.php?p=download

Improve this answer

Comments

0

From what I see in the testcase, a diverging behavior is strange, but the test failing is a perfectly valid result:

A plane intersected with itself would be the plane, so if the result is restricted to being a line, then any line with 2 distinct points on the plane would be a valid result, but testing for one particular line looks like reverse engineering of execution results into test expectations.

It would be a different story if the requirement / specification is very specific about the direction of the resulting lines from identity intersection, so maybe you want to provide some more insight in this area.

I'm sorry for posting this as answer rather than a comment - missing rep.

Improve this answer

1 Comment

This is just an example there's other tests that have this odd behavior. In the case where a plane intersects self, it just picks a particular line in that plane. Whether or not that's a good way to handle the degeneracy is open to debate, but the line tested against is the line the method should have chosen. (and is the one it chooses when the test runs by itself)
0

Please check if there are any static variables involved. Static Variables are present at AppDomain level, therefor its a possibility that a static variable set by one test case may cause side effects to other test cases.

You may run this testcase in a separate app domain first to confirm the behaviour before starting the search for static variable. I'm sorry I never tried to create a new appdomain in nunit. But one of the answers here does give a hint about how to create a new appdomain for a test case - Run unit tests in different appdomain with NUnit

Improve this answer

Comments

0

If this test passes in isolation but fails when running alongside other tests then it is extremely likely that you have some shared state in your classes.

If you say that all our objects are immutable then my first guess would be to look at the implementation of

Line Intersection (Plane plane);

You expect this method to return a Line which is considered the same as your "expected" line, so probably Intersection returns a line based on some shared state.

Could you show us the implementation of Intersection?

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.