A comparison of TAP (Test Anything Protocol) and SubUnit

I have been playing with TAP for some time and even implemented a Java API to let TestNG, JUnit and other Test Frameworks to produce and consume TAP. TAP is a standard format for test output that first appeared with Perl 1 in 1987. It is human and machine readable, easy to be serialized, language independent and extensible1 through the use of YAML.

Some days ago while I was designing a plug-in to show TAP test results in Jenkins I stumbled across a message in Jenkins dev-list where Max Magee and Nick Wiesmueller were discussing about a way of showing more details about the test executions. I thought that the TAP Plug-in would fit perfectly, until one of the users, Robert Collins, mentioned SubUnit.

Shame on me, but I hadn't heard of SubUnit until that message. Max Magee and I exchanged some messages after that, talking about a initial design and analysis for the TAP Plug-in2. Here is the initial idea:

Although I have worked with TAP and spent some good time writing the tap4j port for Java, I am not convinced it is the best solution for this issue yet. Hence I am posting this initial comparison between TAP and SubUnit hoping that more people will contribute with the design of this solution. My goal is not only having a super cool plug-in for Jenkins, but ease integration of test results in different tools and collaborate with both TAP and SubUnit. Another objective that I have in mind is improving the way that test results are displayed in Jenkins and enabling it to be an alternative for tools like Smolder, TestRepository or Tribunal. Because I believe the tasks done by these tools could be all done in my favorite CI Server, and it would increase the productivity of Build & Release professionals :-))

Comparison table4

I believe tables are a good way to compare different technologies. However if anybody has any recommendation on a different way of doing it, I would be glad to give it a try. In case there are missing items or other suggestion, please, do not hesitate in getting in touch.

TAPSubUnit
Human and Machine readable formatYesYes
Language independentYesYes
Programming languages supported5Perl, Python, PHP, Java, C, C++, C#, Lua, Shell, Ruby, Javascript, Pascal, PostgreSQL, Haskell, Lisp, Forth, LimboPython, C, C++ and Shell
Since19872006
Grouping tests in some category/tag styleProposal67Yes
ExtensibleYes, YAMLN/A?
DocumentationGood, but old.Few examples, blogs or Wikis for beginners.
Used in real-world?Yes, an enormous number of modules in CPAN use itYes (e.g.: Samba)
Format specificationDraft at IETFInformation on Python Package Index 8
Show time of testsYes, with YAMLYes, natively
Use custom test statusNoNo
Attach files to test resultYes, Base64 encoded in YAMLYes, Base64 encoded in test output

Examples

TAP:

1..1
not ok 1 Wrong length 
    ---
    wanted: 5
    found: 4
    time: 2011-02-01 00:09:01-07
    extensions: 
        files: 
            1.txt:
                name: 1.txt 
                file-type: text/plain 
                file-size: 43 
                content: c2FtcGxl ...

SubUnit (using Python + nose):

time: 2011-05-23 22:49:38.856075Z
test: my_test.SampleTestCase.runTest 
failure: my_test.SampleTestCase.runTest 
[ 
    Traceback (most recent call last): File "/media/windows/dev/java/qa_workspace/python_nose_tests/src/my_test.py", line
    11, in runTest self.assertEqual(len(s), 4, 'Wrong length') AssertionError: Wrong length 
] 
time: 2011-05-2322:49:38.858163Z

Final considerations

Although I posted this comparison in my blog my intention is turning it to the community somehow, probably putting it in Wikipedia. Perhaps my thoughts here were biased by my proximity with TAP, however I am open to suggestions, ideas or critics (as a proof that I am open to SubUnit, I included it to the list of 'Other test protocols' in TAP Wiki, as there was only TST :-))

Initially I wrote this post as a draft and sent it to Max, Nick, Cesar Fernandes and to Robert Collins for revisions (hadn't heard back from Robert, unfortunately). Later I plan sending it to the TAP development team and for the guys responsible for the Automake GSoC TAP/SubUnit project. Then decide which protocol stick with to develop the TAP Plug-in (or SubUnit :-)). When this analysis is finished I will write an alpha version of this plug-in to send to the Jenkins dev-list, let me know if you would like to give it a try too. I believe that the easiest way to spread TAP or SubUnit as the de facto standard is using it, and asking for maintainers of test frameworks such as TestNG and JUnit to add support for these formats in theirs tools or make it the default output format.

Edit

As pointed by Renormalist, some tools that generate TAP also use another kind of diagnostics for extending the test protocol. In this approach, in the next line after a test result the first character is a '#' followed by a message. A test result may have several comment lines with diagnostic information. The comments in this case, belong to the test result above it. Perl Test::More module produces diagnostics in this way by default. Below you find an example of these diagnostics.

1..1 
not ok 1 - There's a foo user 
# Failed test 'There's a foo user' 
# at /home/kinow/perl/workspace/tests_with_testmore/main.pl line 2. 
# Since there's no foo, check that /etc/bar is set up right 
# Looks like you failed 1 test of 1.

1 Available in TAP version 13, http://testanything.org/wiki/index.php/YAMLish.

2 During this article I use TAP Plug-in to refer to the plug-in to display detailed test result, though after talking with Max we agreed that perhaps it would be a good idea implement it in some generic manner, not specific to TAP. We also agreed it would be good check other plug-ins like xUnit to see if we can extend it or use some code as basis.

3 Still got think more about it. Probably the images will be enclosed in the TAP Stream (or another format) Base64 encoded. Perhaps we would have to decode each attachment in the test result and display it according to its mimetype (zips, pdf, etc). But a lightbox gallery for attachments would be awesome!

4 For the first version of this comparison table I added the items that I could think of, and other items retrieved from the comparison done in Automake GSoC discussion list about the choice between TAP and SubUnit.

5 Here we considered languages that have at least one producer for the the protocol.

6 Test Blocks proposal, http://testanything.org/wiki/index.php/Test_Blocks .

7 Test Groups proposal, http://testanything.org/wiki/index.php/Test_Groups .

8 Python Package Index for python-subunit 0.0.6, http://pypi.python.org/pypi/python-subunit/0.0.6. This was the only one that I could find, but there may have another specification somewhere else.

Referecens

Categories: Blog

Tags: Test Anything Protocol, Software Quality