Posts tagged with ‘test anything protocol’

Adding coverage reports in Jenkins to GoogleTest with gcovr

kinow @ Oct 16, 2012 16:09:33

After the last post about GoogleTest and TestLink using Jenkins TestLink Plug-in, I received an e-mail asking about coverage with GoogleTest and Jenkins. I’ve just updated the Makefile in the samples directory, of the GoogleTest TAP listener project, to output coverage data.

Basically, you have to add the compiler flags -fprofile-arcs -ftest-coverage and link the executable with -lgcov. Take a look at the project’s Makefile and you’ll notice how simple it is. In order to have Jenkins interpreting your coverage report, you’ll have to convert it to cobertura XML. There is a Python utility that can be used for this: gcovr. Download it and copy it to somewhere where Jenkins can execute it (e.g.: /usr/local/bin).

Now, if you’ve followed the instructions from the previous post, you should have a job that reports your GoogleTest tests from Jenkins to TestLink using the plug-in, and is downloading the source code from GitHub. Add an extra build step (Shell) to execute gcovr.

( Read more … )

Testing shell code and producing TAP using Jenkins?

kinow @ Dec 30, 2011 22:41:40

Definitely reading Hacker News before going to bed is a bad idea :-) Same goes for Reddit. So, I found a link about Bats, a tool to execute tests in Shell and output TAP - Test Anything Protocol. Then I thought; why not ask the author to include Bats under the list of Producers in

But you know what? Why not execute execute locally first to check if that’s working… moreover, why not use Jenkins TAP Plug-in and see what happens?

Here’s what is necessary.

( Read more … )

A comparison of TAP (Test Anything Protocol) and SubUnit

kinow @ Jun 04, 2011 02:04:45

I have been playing with TAP for some time and even implemented a Java API to let TestNG, JUnit and other Test Frameworks to produce and consume TAP. TAP is a standard format for test output that first appeared with Perl 1 in 1987. It is human and machine readable, easy to be serialized, language independent and extensible1 through the use of YAML.

Some days ago while I was designing a plug-in to show TAP test results in Jenkins I stumbled across a message in Jenkins dev-list where Max Magee and Nick Wiesmueller were discussing about a way of showing more details about the test executions. I thought that the TAP Plug-in would fit perfectly, until one of the users, Robert Collins, mentioned SubUnit.

Shame on me, but I hadn’t heard of SubUnit until that message. Max Magee and I exchanged some messages after that, talking about a initial design and analysis for the TAP Plug-in 2. Here is the initial idea:

  • The plug-in will be able to parse one or more test formats (maybe SubUnit, TAP and the formats available in xUnit?).
  • The test results will be displayed the same way JUnit tests are displayed in Jenkins (I think Jenkins supports JUnit format by default, but you can use objects and create test results data, independently of the test framework that you are using).
  • There will be a table containing the Test Name, Description and Status and an expandable section.
  • Inside this expandable section will be available all the details about the test.
  • In case there are images within the test details, they should be displayed as a lightbox gallery.3

Although I have worked with TAP and spent some good time writing the tap4j port for Java, I am not convinced it is the best solution for this issue yet. Hence I am posting this initial comparison between TAP and SubUnit hoping that more people will contribute with the design of this solution. My goal is not only having a super cool plug-in for Jenkins, but ease integration of test results in different tools and collaborate with both TAP and SubUnit. Another objective that I have in mind is improving the way that test results are displayed in Jenkins and enabling it to be an alternative for tools like Smolder, TestRepository or Tribunal. Because I believe the tasks done by these tools could be all done in my favorite CI Server, and it would increase the productivity of Build & Release professionals :-))

( Read more … )

Faça seus Testes Falarem TAP (Make your Tests Speak TAP) article published in MundoJ, issue 47, March/April 2011

kinow @ May 23, 2011 19:19:04

This is my first article for a Brazilian magazine! :-D It was published in this month’s issue of MundoJ magazine (previously called MundoJava).

In this article Cesar Fernandes de Almeida, Anderson dos Santos and I discuss how to make your Java tests output TAP (Test Anything Protocol) Streams. This test protocol has been used by Perl developers since Perl’s first version (1983~) but hasn’t been used by many Java developers yet, unfortunately. We hope it incentives other Java developers to use TAP for their tests. There is an alternative to TAP too, SubUnit, which was pointed out for me by Robert Collins in the Jenkins dev-list few weeks ago. Now I am preparing a comparison between TAP and SubUnit as an analysis before developing a plug-in for Jenkins to show detailed test results (such as exception, images, raw test, error line, etc).

Throughout the article we explain how to use tap4j to generate TAP Streams with TestNG, however it is possible to use the same approach to generate TAP Streams with JUnit.

Now I will translate the article to send it to Patrick LeBoutillier, who contributed to this article answering patiently my questions by mail and writing metatap, a utility that lets you check if a TAP Stream is valid, according to the program parameters. This utility is being used in tap4j for integration tests between Perl and Java.

Unfortunately I’m not allowed to redistribute the article, not even an English version of it :-(

Cheers :)

Discussing the flexibility of TAP to cover information generated by TestNG

kinow @ Mar 20, 2011 03:34:05

In today’s post I am expanding the previous one regarding the use of TAP with TestNG. Let’s discuss about the flexibility of TAP to cover information generated by TestNG, more specifically:

  • Expected Exceptions
  • Data Providers
  • Groups
  • Tests that get skipped
  • Dependencies

We will be addressing these bullet points using tap4j, a TAP implementation for Java. The integration between TAP and TestNG is done through the use of TestNG Listeners developed in tap4j project (a big thanks here to Cesar Fernandes de Almeida for all his help).

There are two listeners in tap4j, br.eti.kinoshita.tap4j.ext.testng.TestTAPReporter and br.eti.kinoshita.tap4j.ext.testng.SuiteTAPReporter. The first was created based on org.testng.reporters.TestHTMLReporter and the latter was created based on org.testng.reporters.SuiteHTMLReporter.

In this figure you have testing tools as TAP producers (left) and different types of applications as consumers (right)

( Read more … )