Overview:

If you are a Test Automation Engineer working on a project, you’d know that the most difficult part of automation is the analysis of test executions. At the end of the execution, everytime we need to analyze failed test cases and try to figure out if there’s any false positive/flaky situation which may have been caused by network glitch, time-out, or some other problem.

Normally in automation after executing scripts/tests, we will check for the results and if the test fails we will re-run them again. Instead of that we can ask testNG to execute the failed test cases again for ‘n’ number of times and check for the updated results.

testNG’s IRetryAnalyser allows you to rerun a failed test method a set amount of times before declaring it as failed.

Why do we have to Re-Run Test or Why should we use IRetryAnalyzer?

Assume we have 10 test cases and out of which 8 Test cases are getting Pass and 2 Test cases failed. The failed test cases may be due to X-path issues or Browser got crashed or Application got crashed or may be due to element not found or time out exception or stale element exception etc, and these 2 test cases may get Pass on 2nd or 3rd attempt of run. So in order to give one more chance or couple of chances to execute failed test cases we use IRetryAnalyzer.

IRetryAnalyzer: 

• IRetryAnalyzer is a special Listener and an Interface which belongs to the org.testng package.

(Listeners are TestNG annotations that literally “listen” to the events in a script and modify TestNG behavior accordingly. These listeners are used as interfaces in the code)

• IRetryAnalyzer consists of one method retry and the return type is boolean.

How to Implement IRetryAnalyzer in your project?

This IRetryAnalyzer can be implemented in two ways:

1. @Test Level
2. At the Run Time

Steps to Implement Retry Logic in testNG project:

1. @Test Level

With @Test level we can apply the retry logic only for the specific test cases which are required.

Step 1:

Create a sample test class which includes a failed test case.

package Analyzer;
import org.testng.annotations.Test;
import junit.framework.Assert;
public class MyTests {
	@Test
	public void test1() 	{
		Assert.assertEquals(true, false);
	}
	@Test
	public void test2()	{
		Assert.assertEquals(true, false);
	}
	@Test
	public void test3()
	{
		Assert.assertEquals(true, true);
	}
}

Step 2:

Create a class which implements IRetryAnalyzer.

package Analyzer;

import org.testng.IRetryAnalyzer;
import org.testng.ITestResult;

public class RetryAnalyzerDemo implements IRetryAnalyzer
{
	int initialCount=0;
	int macCount=3;

	public boolean retry(ITestResult result) {
		if(initialCount<macCount)
		{
			initialCount++;
			return true;
		}
		return false;
	}

}

Note: The above class will help to run the failed test cases for 3 attempts and then finally it fails. Even though the test case fails in the first 3 attempts it results in skip and then it fails the test case. If in case the test case gets passed in the 2nd attempt itself it ignores the next attempt of retry.

Step 3:

Add the retryAnalyzer parameter in @Test annotation which has to retry.

package Analyzer;
import org.testng.annotations.Test;
import junit.framework.Assert;

public class MyTests {

	@Test(retryAnalyzer = RetryAnalyzerDemo.class)
	public void test1() 	{
		Assert.assertEquals(true, false);
	}
	@Test(retryAnalyzer = RetryAnalyzerDemo.class)
	public void test2()	{
		Assert.assertEquals(true, false);
	}
	@Test
	public void test3()
	{
		Assert.assertEquals(true, true);
	}
}

The Output of above code is as shown below:

Note: The main drawback of implementing IRetryAnalyzer @Test level is “If there are ‘n’ number of test cases that failed then we need to specify the retryAnalyzer parameter ‘n’ number of times. To overcome this problem, we make use of IAnnotationTransformer.

1. At Run Time:

This can be achieved using the testNG listener called IAnnotationTransformer.

IAnnotationTransformer:

IAnnotationTransformer transforms TestNG annotations at runtime. A scenario may arise where the user tries to override the annotation content based on a condition. In this case, it is not necessary to make changes to the source code. Simply use the IAnnotationTransformer to override the annotation content. The IAnnotationTransformer has only one method called transform() that accepts four parameters:

• Annotation ITestAnnotation
• Class testClass
• Constructor Test Constructor
• Method Test Method

Steps to Implement Retry Logic using IAnnotationTransformer:

Step 1:

Create a sample test class which includes a failed test case.

package Analyzer;
import org.testng.annotations.Test;
import junit.framework.Assert;

public class MyTests {

	@Test
	public void test1() 	{
		Assert.assertEquals(true, false);
	}
	@Test
	public void test2()	{
		Assert.assertEquals(true, false);
	}
	@Test
	public void test3()
	{
		Assert.assertEquals(true, true);
	}
}

Step 2:

Create a class which implements IRetryAnalyzer.

package Analyzer;

import org.testng.IRetryAnalyzer;
import org.testng.ITestResult;

public class RetryAnalyzerDemo implements IRetryAnalyzer
{
	int initialCount=0;
	int macCount=3;

	public boolean retry(ITestResult result) {
		if(initialCount<macCount)
		{
			initialCount++;
			return true;
		}
		return false;
	}

}

Step 3:

Create a class which implements IAnnotationTransformer.

package Analyzer;
import java.lang.reflect.Constructor;
import java.lang.reflect.Method;
import org.testng.IAnnotationTransformer;
import org.testng.annotations.ITestAnnotation;

public class MyTransformer implements IAnnotationTransformer
{
	public void transform(ITestAnnotation annotation, Class testClass, Constructor testConstructor, Method testMethod)
	{
		annotation.setRetryAnalyzer(RetryAnalyzerDemo.class);
	}
}

Step 4:

Create a testng.xml file for all the test suite and add a tag called listeners in the testng.xml file.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">
<suite name="Suite">
	<listeners>
		<listener class-name="Analyzer.MyTransformer"></listener>
	</listeners>
  <test thread-count="5" name="Test">

    <classes>
      <class name="Analyzer.MyTests"/>
    </classes>
  </test> <!-- Test -->
</suite> <!-- Suite -->

After executing the testng.xml file the output will be as shown below.

Note: We can also execute the failed test cases manually, i.e, Once after the test suite execution completes on refreshing the project it creates a .xml file which consists only of the failed test cases. We can run the testng-failed.xml for rerun of failed test cases, this also includes the dependency test cases.

testng-failed.xml

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">
<suite name="Failed suite [Default suite]" guice-stage="DEVELOPMENT">
  <test thread-count="5" name="Default test(failed)">
    <classes>
      <class name="Analyzer.MyTests">
        <methods>
          <include name="test2"/>
          <include name="test1"/>
        </methods>
      </class> <!-- Analyzer.MyTests -->
    </classes>
  </test> <!-- Default test(failed) -->
</suite> <!-- Failed suite [Default suite] -->

Advantages:

• We can optimize the failed test case count on re-executing the failed test for a couple of attempts.
• Retry effort will be reduced.

Limitations:

• TIme taken to complete the suite is more, as the failed test cases are re-trying.

Conclusion: 

WIth IRetryAnalyzer Listener interface we can achieve retry Logic .i.e, during runtime if any test cases are getting failed such test case will rerun couple of times (specified iteration) and even after execution for couple of time if the test case gets failed then its finally results to fail. If the test case is getting passed for 2nd or 3rd iteration then it results to pass, no further iterations will get executed.

This is applicable only for the failed test cases.