Menu

Home

Performance problems in Jenkins TAP Plug-in — part 1

kinow @ Sep 03, 2016 23:28:03

JENKINS-17887 reports performance problems in the Jenkins TAP Plug-in. It also lists a series of suggestions on how to improve the Jenkins TAP Plug-in performance. On this initial post, we will get a general idea of how the plug-in performs for large projects.

BioPerl has over 21K tests. That should be enough for giving an initial idea of CPU, memory and disk usage for the plug-in.

git clone https://github.com/bioperl/bioperl-live.git
cd bioperl-live
sudo cpanm  -vv --installdeps --notest .
sudo cpanm Set::Scalar Graph::Directed XML::LibXML XML::SAX \
    SVG XML::Parser::PerlSAX Convert::Binary::C XML::SAX::Writer \
    XML::DOM::XPath Spreadsheet::ParseExcel XML::SAX::Writer \
    XML::DOM HTML::TableExtract XML::Simple Test::Pod DBI
prove -r t/ -a tests.tar.gz

All tests successful.
Files=325, Tests=21095, 94 wallclock secs ( 2.47 usr  0.55 sys + 88.29 cusr  3.85 csys = 95.16 CPU)
Result: PASS

When the test results are parsed, the plug-in also copies TAP files over to the master, in a folder called tap-master-files.

The BioPerl tests are not really big, just 1.7M. It gets doubled as there will be the workspace copy, and the tap-master-files directory copy, so 3.4M.

But several objects get created in memory, and persisted into the build.xml job file. BioPerl generates a build.xml file with 11M. So less than 15M. But the build.xml contains objects that are read via XStream by Jenkins and into the memory.

The build page with the graph, and the other two test result pages are rendering in more than 10 seconds in my computer. But the CPU load is OK, so a closer look at the memory use would probably be more interesting.

JENKINS-17887 YourKit profiler

The image shows one of the screens in YourKit profiler, where it is possible to see that org.tap4j.plugin.model.TapTestResultResult has over 6 million objects.

One build.xml for the BioPerl project gets over 80K entries for the TestResult object.

grep "org.tap4j.model.TestResult" builds/1/build.xml -o | wc -l
84522

This happens because each TAP file may contain multiple test results (lines with test results). Each of these test results gets turned into a Java object and loaded by the plug-in. So when loading the test result pages, Jenkins needs to wait until all these objects have been parsed, deserialized and read into the memory.

The next post will continue on code improvements, and another benchmark.

Happy profiling!

Revamping Frege logo — part 2

kinow @ Aug 24, 2016 23:54:03

Last time I used Blender was around 2007 I think, in University. But the bad weather in Auckland gives me plenty of time to have fun checking out Blender again :-)

Followed the following tutorials:

Here are the work in progress, created only with the bézier curve.

Frege updated logo

Frege 3D #1

Frege 3D #2

Frege 3D #3

Frege 3D #4

Frege 3D #5

Frege 3D #6

Frege 3D #7

Here’s the result after the mesh was created, and some material applied.

Frege 3D #8

Then using a plane as background, replacing the lamp by a sun, and tweaking a few parameters.

Frege 3D #9

And finally playing with animation. Not sure if there was a time line and animation controls in Blender the last time I used it, but the controls are not really complex.

I had to combine both meshes into a single object, in order to add a bone and rotate it. That is why the logo got back to a single material. The angle of the camera could probably do with some tweaking as well.

Frege 3D #10

But it was my very first time animating in Blender. Some day if I get access to one of those 3D printers, I will check what are the requirements for printing this logo.

Blender models can be downloaded here.

Now back to programming :-)

Revamping Frege logo

kinow @ Aug 16, 2016 23:54:03

For a while I had been wanting to try adding a flat colour to the Frege logo. This weekend had some bit of spare time, and here is the result.

Submitted to the project as pull request #299.

Current logo.

Frege current logo

And the updated logo.

Frege updated logo

The colour used was Flamingo (#EF4836), found in Flat UI Color Picker.

Perhaps not exactly revamping, since all I did was just change a colour. There were adjustments to the bézier curves as well, to better align them, but I updated both logos, not just the new one :-)

Reading notes about DRMAA v2

kinow @ Jul 20, 2016 19:59:03

The DRMAA v2 specification draft is ready to be published, and is in public comment until 31st July this year. I used DRMAA v1 to integrate Jenkins and PBS some time ago, but it was not a very elegant solution.

And in the end integrating other grid computing implementations like SGE would not be very simple.

This post contains my reading notes for DRMAA v2, and a short analysis of how this new specification could be used in a new tentative to integrate Jenkins and several grid computing implementations in a single plug-in.

( Read more ... )

Reasons for having pt and pt-BR in a software

kinow @ Jul 17, 2016 14:14:03

Some time ago I found some spare time to work on a different Open Source project: SKOSMOS. SKOSMOS is a web based SKOS browser and publishing tool, used to create vocabularies using the SKOS ontology.

I decided to help with translation, but there was no Brazilian Portuguese option, only Portuguese. I used a few arguments to suggest that having Brazilian Portuguese would be a good thing.

Another Open Source project that I use in a side project is LanguageTool. LanguageTool is used for proof-reading, and uses rules to find spelling and grammar errors.

Today I saw a message in the LanguageTool mailing list discussing whether having a Brazilian Portuguese page would make sense, or if it would be better to have just Portuguese, and then add rules for special cases.

( Read more ... )