12

I have a complex set of PHPUnit tests, some of which involve connecting to servers around the world that, for whatever reason, timeout sometimes.

Rather than having the test fail when the server times out, I'd like to simply retry that test one or more times before actually marking it as failed.

Now, I understand that this may not be the best way to handle my situation at hand. One better solution would be to fix the servers. But, this is out of my control right now.

So, what I'd really like, is a way to tell PHPUnit to re-test each failing testcase X times, and only mark it as failed if it failed every time.

Any ideas?

Edit: Many of you have responded with helpful suggestions that I not do this. I understand, thank you. However, specifically what I am trying to do is create a test suite that tests the operation of the full system, including the remote servers. I understand the concept of testing certain parts of my code with "mock" responses from the outside...but I also sleep better at night if part of my tests test the "full stack".

Improve this question
1
  • 2
    According your edit: 1. When you say "its not your host" you say "its not part of your system". Thus you shouldn't include it into your test, because you cannot influence it anyway and it will falsify the results. 2. You would never create a test $this->assertEquals(1, rand(0,1)), would you? That's what happens, when you include uncertain external systems. Commented Oct 9, 2011 at 18:04

6 Answers 6

Reset to default
13

Edit: As per a comment below, this no longer works for PHPUnit 10+.

As PHPUnit doesn't support this behavior out of the box, you'll need to code the loop yourself. Instead of doing it in every test that requires it, create a custom test case base class (if you haven't already) that extends PHPUnit_Framework_TestCase and provides the feature.

You can either get fancy and override testBare() to check for an annotation such as @retry 5, loop that number of times, calling parent::testBare(), and swallow all exceptions (or a subset) except the last.

public function runBare() {
    // I'll leave this part to you. PHPUnit supplies methods for parsing annotations.
    $retryCount = $this->getNumberOfRetries();
    for ($i = 0; $i < $retryCount; $i++) {
        try {
            parent::runBare();
            return;
        }
        catch (Exception $e) {
            // last one thrown below
        }
    }
    if ($e) {
        throw $e;
    }
}

Or you can create a similar helper method that takes the number of retries and a closure/callable as parameters and call it from each test that needs it.

public function retryTest($count, $test) {
    // just like above without checking the annotation
    ...
        $test();
    ...
}

public function testLogin() {
    $this->retryTest(5, function() {
        $service = new LoginService();
        ...
    });
}
Improve this answer
Sign up to request clarification or add additional context in comments.

7 Comments

Hi David, Can you please provide more details what to do i have to retry test case once if it got failure.
Thanks for reply David. I want to simply retry my test case if it got failure without use of any annotations. What I did .. I put code of run bare function[as defined in your answer] in my base class using $retryCount = 2, after this point i am not clear :(
Also note that sample code also retries skipped/incomplete tests. You may want to catch and re-throw those ones to avoid retrying those needlessly.
The complete solution based on this answer gist.github.com/makasim/989fcaa6da8ff579f7914d973e68280c
Note that this only works up to PHPUnit 9 - starting with PHPUnit 10 runBare is final an cannot be overwritten!
|
6

Not exactly an answer to your question, but I'll say it anyway: Your tests should never include remote resources (especially when they are completely out of your hand (unlike local mirrors)). You should encapsulate your connections into separate classes (e.g. Connection) and within your tests you mock those objects and work with static responses, that your remote hosts would return.

Improve this answer

1 Comment

This is true for unit tests, but for automated acceptance tests, there's still a lot of debate out there on whether or not to mock external services. We chose to not, and it can increase the brittleness of a test. In such cases, retries can be helpful.
2

Rather than connecting to live servers, shouldn't you be using mock objects and fixtures so that responses from elsewhere don't have an effect on your tests?

You could possibly use dependency injection to use a specific HTTP client that will return the data and response code you tell it to (depending on how your code is written). Ideally, your unit tests should be independed of outside influences; you should be in control of what you're testing, and forcing a 404 or 500 error, for example, should be a separate part of your tests.

Rather than trying to hack around non-deterministic tests, you would be better served by looking at whether you can alter your code to enable mocking and test fixtures.

Aside from that, which you may already know of course, I'm afraid I don't know of a way to tell PHPUnit to allow a test to fail. It seems completely contrary to what the tool should be doing.

Improve this answer

2 Comments

+1, and important last sentence: "It seems completely contrary to what the tool should be doing."
If you mock, for example, the browser in a frontend test, you're doing it wrong.
2

If anyone is wondering how to implement this in PHPUnit 10+. I found a solution

This way all failed tests will be marked as skipped. And the new try will be initiated

use PHPUnit\Framework\TestCase as BaseTestCase; // or Illuminate\Foundation\Testing\TestCase if you use Laravel
use PHPUnit\Metadata\Annotation\Parser\DocBlock;
use Throwable;

abstract class TestCase extends BaseTestCase
{
    private int $try = 0;
    private ?int $retryCount;

    /** @var array<int>  */
    private array $retryDelayRange;

    protected function setUp(): void
    {
        parent::setUp();
        $this->setTries();
    }

    private function setTries(): void
    {   
        // Get annotations from docblock
        $annotations = DocBlock::ofMethod((new \ReflectionMethod($this::class, $this->name())))->symbolAnnotations();

        $this->retryCount = (int)($annotations['retry'][0] ?? null);
        
        // Get delay range
        $retryDelayMs = $annotations['retry-delay'][0];
        $retryDelayMsBetween = array_filter(explode('-', $retryDelayMs));
        if (count($retryDelayMsBetween) === 1) {
            $delay = (int)$retryDelayMsBetween[0];
            $this->retryDelayRange = [$delay, $delay];
        } else {
            $this->retryDelayRange = [$retryDelayMsBetween[0] ?? 500, $retryDelayMsBetween[1] ?? 1000];
        }
    }

    private function getRetryDelay(): int
    {
        return mt_rand(...$this->retryDelayRange) * 1000;
    }

    // Retry test here because this callback is called at the end of TestRunner::runBare()
    // This allows us to go through the entire test cycle
    // including dumping mock, static variables, database transactions and other things
    protected function onNotSuccessfulTest(Throwable $t): never
    {
        if ($this->try < $this->retryCount) {
            $this->try++;
            usleep($this->getRetryDelay());
            $this->runBare();
        }
        throw $t;
    }

    protected function runTest(): mixed
    {
        // If there are no retries then run as usual
        if (!$this->retryCount) {
            return parent::runTest();
        }
        
        // If test errored then mark it as skipped
        try {
            return parent::runTest();
        } catch (\Throwable $e) {
            $this->markTestSkipped();
        }
    }
}

And if you wanna retry your test on failure just add @retry annotation

/**
 * @retry 3 - Retry count
 * @retry-delay 100-500 - Just to randomize the retry delay in milliseconds
 */
public function testSomething(): void
{
    // test here
}
Improve this answer

1 Comment

This only works for phpunit 10, starting from phpunit 11, runTest is also final
1

I don't think there is a support for that, someone can prove me wrong, but I'd be surprised in that case.

I think what you can do is instead of asserting right away in the test methods to be retrieved is looping for X times and break out on success, outside the loop the result will be asserted.

This simple way, that you'd already thought about, has the disadvantage to add some more code to each test method. If you've many of them it adds to maintenance burden.

Otherwise, if you want to automate more, you could implement a PHPUnit_Framework_TestListener, keep a count of failed tests in associative array and compare with the test runs. Not sure how feasible is this route, but you could give a try.

Improve this answer

Comments

0

You should be able to create a DB connection assertion and implement that test in your other tests "as" your DB connection. Inside that test you could try as many times as you need and return false after X tries.

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.