A comparison of TAP (Test Anything Protocol) and SubUnit
I have been playing with TAP for some time and even implemented a Java API to let TestNG, JUnit and other Test Frameworks to produce and consume TAP. TAP is a standard format for test output that first appeared with Perl 1 in 1987. It is human and machine readable, easy to be serialized, language independent and extensible1 through the use of YAML.
Some days ago while I was designing a plug-in to show TAP test results in Jenkins I stumbled across a message in Jenkins dev-list where Max Magee and Nick Wiesmueller were discussing about a way of showing more details about the test executions. I thought that the TAP Plug-in would fit perfectly, until one of the users, Robert Collins, mentioned SubUnit.
Shame on me, but I hadn't heard of SubUnit until that message. Max Magee and I exchanged some messages after that, talking about a initial design and analysis for the TAP Plug-in2. Here is the initial idea:
- The plug-in will be able to parse one or more test formats (maybe SubUnit, TAP and the formats available in xUnit?).
- The test results will be displayed the same way JUnit tests are displayed in Jenkins (I think Jenkins supports JUnit format by default, but you can use objects and create test results data, independently of the test framework that you are using).
- There will be a table containing the Test Name, Description and Status and an expandable section.
- Inside this expandable section will be available all the details about the test.
- In case there are images within the test details, they should be displayed as a lightbox gallery.3
Although I have worked with TAP and spent some good time writing the tap4j port for Java, I am not convinced it is the best solution for this issue yet. Hence I am posting this initial comparison between TAP and SubUnit hoping that more people will contribute with the design of this solution. My goal is not only having a super cool plug-in for Jenkins, but ease integration of test results in different tools and collaborate with both TAP and SubUnit. Another objective that I have in mind is improving the way that test results are displayed in Jenkins and enabling it to be an alternative for tools like Smolder, TestRepository or Tribunal. Because I believe the tasks done by these tools could be all done in my favorite CI Server, and it would increase the productivity of Build & Release professionals :-))
I believe tables are a good way to compare different technologies. However if anybody has any recommendation on a different way of doing it, I would be glad to give it a try. In case there are missing items or other suggestion, please, do not hesitate in getting in touch.
|Human and Machine readable format||Yes||Yes|
|Grouping tests in some category/tag style||Proposal67||Yes|
|Documentation||Good, but old.||Few examples, blogs or Wikis for beginners.|
|Used in real-world?||Yes, an enormous number of modules in CPAN use it||Yes (e.g.: Samba)|
|Format specification||Draft at IETF||Information on Python Package Index 8|
|Show time of tests||Yes, with YAML||Yes, natively|
|Use custom test status||No||No|
|Attach files to test result||Yes, Base64 encoded in YAML||Yes, Base64 encoded in test output|
1..1 not ok 1 Wrong length --- wanted: 5 found: 4 time: 2011-02-01 00:09:01-07 extensions: files: 1.txt: name: 1.txt file-type: text/plain file-size: 43 content: c2FtcGxl ...
SubUnit (using Python + nose):
time: 2011-05-23 22:49:38.856075Z test: my_test.SampleTestCase.runTest failure: my_test.SampleTestCase.runTest [ Traceback (most recent call last): File "/media/windows/dev/java/qa_workspace/python_nose_tests/src/my_test.py", line 11, in runTest self.assertEqual(len(s), 4, 'Wrong length') AssertionError: Wrong length ] time: 2011-05-2322:49:38.858163Z
Although I posted this comparison in my blog my intention is turning it to the community somehow, probably putting it in Wikipedia. Perhaps my thoughts here were biased by my proximity with TAP, however I am open to suggestions, ideas or critics (as a proof that I am open to SubUnit, I included it to the list of 'Other test protocols' in TAP Wiki, as there was only TST :-))
Initially I wrote this post as a draft and sent it to Max, Nick, Cesar Fernandes and to Robert Collins for revisions (hadn't heard back from Robert, unfortunately). Later I plan sending it to the TAP development team and for the guys responsible for the Automake GSoC TAP/SubUnit project. Then decide which protocol stick with to develop the TAP Plug-in (or SubUnit :-)). When this analysis is finished I will write an alpha version of this plug-in to send to the Jenkins dev-list, let me know if you would like to give it a try too. I believe that the easiest way to spread TAP or SubUnit as the de facto standard is using it, and asking for maintainers of test frameworks such as TestNG and JUnit to add support for these formats in theirs tools or make it the default output format.
As pointed by Renormalist, some tools that generate TAP also use another kind of diagnostics for extending the test protocol. In this approach, in the next line after a test result the first character is a '#' followed by a message. A test result may have several comment lines with diagnostic information. The comments in this case, belong to the test result above it. Perl Test::More module produces diagnostics in this way by default. Below you find an example of these diagnostics.
1..1 not ok 1 - There's a foo user # Failed test 'There's a foo user' # at /home/kinow/perl/workspace/tests_with_testmore/main.pl line 2. # Since there's no foo, check that /etc/bar is set up right # Looks like you failed 1 test of 1.
1 Available in TAP version 13, http://testanything.org/wiki/index.php/YAMLish.
2 During this article I use TAP Plug-in to refer to the plug-in to display detailed test result, though after talking with Max we agreed that perhaps it would be a good idea implement it in some generic manner, not specific to TAP. We also agreed it would be good check other plug-ins like xUnit to see if we can extend it or use some code as basis.
3 Still got think more about it. Probably the images will be enclosed in the TAP Stream (or another format) Base64 encoded. Perhaps we would have to decode each attachment in the test result and display it according to its mimetype (zips, pdf, etc). But a lightbox gallery for attachments would be awesome!
4 For the first version of this comparison table I added the items that I could think of, and other items retrieved from the comparison done in Automake GSoC discussion list about the choice between TAP and SubUnit.
5 Here we considered languages that have at least one producer for the the protocol.
6 Test Blocks proposal, http://testanything.org/wiki/index.php/Test_Blocks .
7 Test Groups proposal, http://testanything.org/wiki/index.php/Test_Groups .
8 Python Package Index for python-subunit 0.0.6, http://pypi.python.org/pypi/python-subunit/0.0.6. This was the only one that I could find, but there may have another specification somewhere else.
- Jenkins dev-list discussion where the TAP Plug-in idea was sent to http://jenkins.361315.n4.nabble.com/Re-Additional-Test-Result-Display-Idea-tt3510669.html
- Another discussion in Jenkins dev-list about Test Result refactoring http://jenkins.361315.n4.nabble.com/Review-requested-Test-Result-Refactoring-tt978100.html
- Test Anything Protocol Wiki - http://www.testanything.org
- Perl Wikipedia Article, History section - http://en.wikipedia.org/wiki/Perl#History
- automake - Interfacing with a test protocol like TAP or subunit (GSoC) http://www.google-melange.com/gsoc/proposal/review/google/gsoc2011/slattarini/1