Discussing the flexibility of TAP to cover information generated by TestNG

Mar 20, 2011

In today's post I am expanding the previous one regarding the use of TAP with TestNG. Let's discuss about the flexibility of TAP to cover information generated by TestNG, more specifically:

Expected Exceptions
Data Providers
Groups
Tests that get skipped
Dependencies

We will be addressing these bullet points using tap4j, a TAP implementation for Java. The integration between TAP and TestNG is done through the use of TestNG Listeners developed in tap4j project (a big thanks here to Cesar Fernandes de Almeida for all his help).

There are two listeners in tap4j, br.eti.kinoshita.tap4j.ext.testng.TestTAPReporter and br.eti.kinoshita.tap4j.ext.testng.SuiteTAPReporter. The first was created based on org.testng.reporters.TestHTMLReporter and the latter was created based on org.testng.reporters.SuiteHTMLReporter.

In this figure you have testing tools as TAP producers (left) and different types of applications as consumers (right)

Expected Exceptions

There is no special treatment for the Expected Exceptions in the tap4j Listeners. When a test passes, tap4j listeners logs only that the test succeeded. But when it fails and the Test Result contains a Throwable object, it is logged to the TAP Stream. I am still discussing it with Cesar whether it would be a good idea to parse the Throwable message body to retrieve the 'line', 'expected' and 'got' values for this error (these are entries in TAP Diagnostics - YAML).

Particularly, I would like to have this information in TestException. Maybe it could have a constructor public TestException(String message, Throwable t, String expected, String got, long lineNumber) and TestNG could fill this information for me :-). I know I'm being lazy, but I like the description of laziness in Larry Wall's WikiPedia entry. The following test is expecting an exception that is never thrown, thus it raises an exception.

package example1;

import org.testng.Assert;
import org.testng.annotations.Test;

public class TestFoo
{

 // expecing npe instead of arith.
 @Test(expectedExceptions={NullPointerException.class})
 public void test()
 {
   Assert.assertTrue(1/0 > 0); // programmatic causing an error in the test
 }

}

And the TAP Stream with the exception in the 'backtrace' entry.

1..1
not ok 1 - example1.TestFoo#test
 ---
 message: TestNG Test test
 severity: High
 source: example1.TestFoo#test
 datetime: '2011-03-20T03:45:30'
 file: example1.TestFoo
 line: ''
 name: test
 extensions: '~'
 got: '~'
 expected: '~'
 display: '~'
 dump: '~'
 error: &quot;\nExpected exception java.lang.NullPointerException but got org.testng.TestException:\
 \ \nExpected exception java.lang.NullPointerException but got java.lang.ArithmeticException:\
 \ / by zero&quot;
 backtrace: &quot;org.testng.TestException: \nExpected exception java.lang.NullPointerException\
 \ but got org.testng.TestException: \nExpected exception java.lang.NullPointerException\
 \ but got java.lang.ArithmeticException: / by zero\n\tat org.testng.internal.Invoker.handleInvocationResults(Invoker.java:1416)\n\
 \tat org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1184)\n\tat org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)\n\
 \tat org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)\n\tat org.testng.TestRunner.runWorkers(TestRunner.java:1125)\n\
 \tat org.testng.TestRunner.privateRun(TestRunner.java:749)\n\tat org.testng.TestRunner.run(TestRunner.java:600)\n\
 \tat org.testng.SuiteRunner.runTest(SuiteRunner.java:317)\n\tat org.testng.SuiteRunner.runSequentially(SuiteRunner.java:312)\n\
 \tat org.testng.SuiteRunner.privateRun(SuiteRunner.java:274)\n\tat org.testng.SuiteRunner.run(SuiteRunner.java:223)\n\
 \tat org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)\n\tat org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)\n\
 \tat org.testng.TestNG.runSuitesSequentially(TestNG.java:995)\n\tat org.testng.TestNG.runSuitesLocally(TestNG.java:920)\n\
 \tat org.testng.TestNG.run(TestNG.java:856)\n\tat org.testng.remote.RemoteTestNG.run(RemoteTestNG.java:110)\n\
 \tat org.testng.remote.RemoteTestNG.initAndRun(RemoteTestNG.java:205)\n\tat org.testng.remote.RemoteTestNG.main(RemoteTestNG.java:174)\n\
 Caused by: org.testng.TestException: \nExpected exception java.lang.NullPointerException\
 \ but got java.lang.ArithmeticException: / by zero\n\tat org.testng.internal.Invoker.handleInvocationResults(Invoker.java:1416)\n\
 \tat org.testng.internal.Invoker.invokeMethod(Invoker.java:722)\n\tat org.testng.internal.Invoker.invokeTestMethod(Invoker.java:846)\n\
 \tat org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1170)\n\t... 17\
 \ more\nCaused by: java.lang.ArithmeticException: / by zero\n\tat example1.TestFoo.test(TestFoo.java:17)\n\
 \tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)\n\
 \tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)\n\
 \tat java.lang.reflect.Method.invoke(Method.java:597)\n\tat org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:74)\n\
 \tat org.testng.internal.Invoker.invokeMethod(Invoker.java:673)\n\t... 19 more\n&quot;
 ...

As you can see, the Exception contains the information of what the test expected (java.lang.NullPointerException) and what the test actually got (java.lang.ArithmeticException).

Data Providers

In order to use a data provider with TestNG you have to add parameters to your method. Look at the code below.

package example1;

import org.testng.Assert;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Listeners;
import org.testng.annotations.Test;

import br.eti.kinoshita.tap4j.ext.testng.TestTAPReporter;

@Listeners(value={TestTAPReporter.class})
public class TestWithDataProvider
{

 @DataProvider(name=&quot;names&quot;)
 public Object[][] getNames()
 {
   return new Object[][]
   {
     {&quot;Haruki&quot;},
     {&quot;Murakami&quot;}
   };
 }

 @Test(dataProvider=&quot;names&quot;)
 public void testName(String name)
 {
   Assert.assertNotNull(name);
 }

}

Running the test above results in the following TAP Stream:

1..2
ok 1 - example1.TestWithDataProvider#testName
 ---
 message: TestNG Test testName
 severity: '~'
 source: example1.TestWithDataProvider#testName
 datetime: '2011-03-20T03:09:50'
 file: example1.TestWithDataProvider
 line: '~'
 name: testName
 extensions: '~'
 got: '~'
 expected: '~'
 display: '~'
 dump: '{param1=Haruki}'
 error: '~'
 backtrace: '~'
 ...
ok 2 - example1.TestWithDataProvider#testName
 ---
 message: TestNG Test testName
 severity: '~'
 source: example1.TestWithDataProvider#testName
 datetime: '2011-03-20T03:09:50'
 file: example1.TestWithDataProvider
 line: '~'
 name: testName
 extensions: '~'
 got: '~'
 expected: '~'
 display: '~'
 dump: '{param1=Murakami}'
 error: '~'
 backtrace: '~'
 ...

As you can see, tap4j listeners output in the 'dump' entry the values for parameters used in the test. This way you have both the data obtained through Data Providers and what was passed as parameters during the test execution. One issue could be that this information can't be distinguished, i.e. you can't tell if the params are coming from a Data Provider or if they are parameters.

Groups

The listener br.eti.kinoshita.tap4j.ext.testng.SuiteTAPReporter contains the logic to generate a TAP Stream for each TestNG group found in the test suite. There was a BUG in this listener that was fixed in tap4j version 1.4.5 (thanks again to Cesar). Let's use the following code taken from TestNG documentation site.

package example1;

import org.testng.annotations.BeforeClass;
import org.testng.annotations.Test;

public class SimpleTest {

@BeforeClass
public void setUp() {
  // code that will be invoked when this test is instantiated
}

@Test(groups = { &quot;fast&quot; })
public void aFastTest() {
  System.out.println(&quot;Fast test&quot;);
}

@Test(groups = { &quot;slow&quot; })
public void aSlowTest() {
  System.out.println(&quot;Slow test&quot;);
}

}

And let's create a XML suite for running this test with tap4j SuiteTAPReporter listener.

<!DOCTYPE suite SYSTEM &quot;http://testng.org/testng-1.0.dtd&quot; >
  
<suite name=&quot;ExampleSuite&quot; verbose=&quot;1&quot; >
  <listeners>
    <listener class-name=&quot;br.eti.kinoshita.tap4j.ext.testng.SuiteTAPReporter&quot; />
  </listener>
  <test name=&quot;Sample test&quot;>
    <classes>
      <class name=&quot;example1.SimpleTest&quot; />
    </classes>
  </test>
</suite>

Running this XML suite will produce three files in the test-output directory.

ExampleSuite.tap

1..2
ok 1 - example1.SimpleTest#aFastTest
 ---
 message: TestNG Test aFastTest
 severity: '~'
 source: example1.SimpleTest#aFastTest
 datetime: '2011-03-20T01:47:11'
 file: example1.SimpleTest
 line: '~'
 name: aFastTest
 extensions: '~'
 got: '~'
 expected: '~'
 display: '~'
 dump: '~'
 error: '~'
 backtrace: '~'
 ...
ok 2 - example1.SimpleTest#aSlowTest
 ---
 message: TestNG Test aSlowTest
 severity: '~'
 source: example1.SimpleTest#aSlowTest
 datetime: '2011-03-20T01:47:11'
 file: example1.SimpleTest
 line: '~'
 name: aSlowTest
 extensions: '~'
 got: '~'
 expected: '~'
 display: '~'
 dump: '~'
 error: '~'
 backtrace: '~'
 ...

fast.tap

1..1
ok 1 - example1.SimpleTest#aFastTest
 ---
 message: TestNG Test aFastTest
 severity: '~'
 source: example1.SimpleTest#aFastTest
 datetime: '2011-03-20T01:47:11'
 file: example1.SimpleTest
 line: '~'
 name: aFastTest
 extensions: '~'
 got: '~'
 expected: '~'
 display: '~'
 dump: '~'
 error: '~'
 backtrace: '~'
 ...

slow.tap

1..1
ok 1 - example1.SimpleTest#aSlowTest
 ---
 message: TestNG Test aSlowTest
 severity: '~'
 source: example1.SimpleTest#aSlowTest
 datetime: '2011-03-20T01:47:11'
 file: example1.SimpleTest
 line: '~'
 name: aSlowTest
 extensions: '~'
 got: '~'
 expected: '~'
 display: '~'
 dump: '~'
 error: '~'
 backtrace: '~'
 ...

tap4j Listeners can generate TAP Streams for TestNG tests for each method, for each class, for each group and for each suite. Nice uhn? :-)

Tests that get skipped

In TAP there are two directives: SKIP and TODO. SKIP is used to mark a test result as skipped, and TODO says that certain method is still not completely implemented. Tests that are skipped by TestNG are then marked as 'not ok' with the SKIP directive. Let's see some sample code.

package example1;

import org.testng.Assert;
import org.testng.annotations.Listeners;
import org.testng.annotations.Test;

import br.eti.kinoshita.tap4j.ext.testng.TestTAPReporter;

@Listeners(value={TestTAPReporter.class})
public class SkipTest
{
 @Test
 public void aTestThatFails()
 {
   Assert.assertTrue(1/0 > 0); // programmatic causing an error in the test
 }

 @Test(dependsOnMethods={&quot;aTestThatFails&quot;})
 public void aDepedentMethod() // This method is always skipped
 {
   Assert.assertTrue( System.currentTimeMillis() > 0 );
 }
}

In the code above we are intentionally making a mistake in the method aTestThatFails (we try to divide a number by zero). The method aDepedentMethod that depends on this method is then always skipped. Here is the output TAP Stream:

example1.SkipTest.tap

1..2
not ok 1 - example1.SkipTest#aDepedentMethod # SKIP TestNG test was skipped
 ---
 message: TestNG Test aDepedentMethod
 severity: '~'
 source: example1.SkipTest#aDepedentMethod
 datetime: '2011-03-20T02:55:34'
 file: example1.SkipTest
 line: '~'
 name: aDepedentMethod
 extensions: '~'
 got: '~'
 expected: '~'
 display: '~'
 dump: '~'
 error: '~'
 backtrace: '~'
 ...
not ok 2 - example1.SkipTest#aTestThatFails
 ---
 message: TestNG Test aTestThatFails
 severity: High
 source: example1.SkipTest#aTestThatFails
 datetime: '2011-03-20T02:55:34'
 file: example1.SkipTest
 line: ''
 name: aTestThatFails
 extensions: '~'
 got: '~'
 expected: '~'
 display: '~'
 dump: '~'
 error: / by zero
 backtrace: &quot;java.lang.ArithmeticException: / by zero\n\tat example1.SkipTest.aTestThatFails(SkipTest.java:15)\n\
 \tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)\n\
 \tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)\n\
 \tat java.lang.reflect.Method.invoke(Method.java:597)\n\tat org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:74)\n\
 \tat org.testng.internal.Invoker.invokeMethod(Invoker.java:673)\n\tat org.testng.internal.Invoker.invokeTestMethod(Invoker.java:846)\n\
 \tat org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1170)\n\tat org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)\n\
 \tat org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)\n\tat org.testng.TestRunner.runWorkers(TestRunner.java:1125)\n\
 \tat org.testng.TestRunner.privateRun(TestRunner.java:749)\n\tat org.testng.TestRunner.run(TestRunner.java:600)\n\
 \tat org.testng.SuiteRunner.runTest(SuiteRunner.java:317)\n\tat org.testng.SuiteRunner.runSequentially(SuiteRunner.java:312)\n\
 \tat org.testng.SuiteRunner.privateRun(SuiteRunner.java:274)\n\tat org.testng.SuiteRunner.run(SuiteRunner.java:223)\n\
 \tat org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)\n\tat org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)\n\
 \tat org.testng.TestNG.runSuitesSequentially(TestNG.java:995)\n\tat org.testng.TestNG.runSuitesLocally(TestNG.java:920)\n\
 \tat org.testng.TestNG.run(TestNG.java:856)\n\tat org.testng.remote.RemoteTestNG.run(RemoteTestNG.java:110)\n\
 \tat org.testng.remote.RemoteTestNG.initAndRun(RemoteTestNG.java:205)\n\tat org.testng.remote.RemoteTestNG.main(RemoteTestNG.java:174)\n&quot;
 ...

You can use br.eti.kinoshita.tap4j.ext.testng.TestTAPReporter or br.eti.kinoshita.tap4j.ext.testng.SuiteTAPReporter, as both listeners generate information on skipped tests.

Dependencies

At moment tap4j listeners don't give you an idea of how is the dependency among the tests in your project. It is an issue that has to be better detailed to be implemented in a new version of tap4j. I know there are methods in the TestNG API that can tell you on which method/group a test depends upon. However I would need to sit down and think where to put this information in the TAP Stream. I like checking first how is it being done in Perl and then trying to code it in a way that the information generated by Java is more similar to that generated in Perl.

I created the enhancement issue 3228068 in tap4j SF.net project to discuss about dependencies in tap4j TestNG integration. I have to work on a few things in TestLink and Jenkins prior to start developing this integration, but it is in my list of things to do before Easter (this year's Easter, of course ;-)).

That's all folks! I hope this post clarifies a little bit more about TestNG and TAP. Although there is a lot of work to do in this integration, it is already being used by Jenkins TestLink Plug-in in productive environments as a way to retrieve information about the test results and update TestLink. Just for what it is worth, the plug-in reads the 'extensions' entry, looking for a file entry that represents a test evidence. This way automated tests can have evidences uploaded into TestLink test executions.

Ah, just one last note, in the following days the classes and packages of the tap4j project may be updated. Probably I will release tap4j 2.0 with the base package org.tap4j instead of br.eti.kinoshita.tap4j. I think it makes the project more 'free' and 'community-oriented' rather than using my own domain in it. As the API is very new I believe it is better to take this decision now rather than when it is being used in more projects.

Cheers

Categories: Blog

Tags: Test Anything Protocol, Software Quality, Java