Blog Taxonomy


When you don't realize you need a Comparable

kinow @ May 15, 2017 23:07:39 ()

In 2012, I wrote about how you always learn something new by following the Apache dev mailing lists.

After about five years, I am still learning, and still getting impressed by the knowledge of other developers. Days ago I was massaging some code in a pull request and a developer suggested me to simplify my code.

The suggestion was to make a class a Comparable type to both simplify the code, and also have a better design. I immediately agreed, and looking back in hindsight, it was the most logical choice. Yet, I simply did not think about that.

// What the code was
    int cmp = 0;
    String c1 = nv1.getCollation();
    String c2 = nv2.getCollation();
    if (c1 != null && c2 != null && c1.equals(c2)) {
        // locales are parsed. Here we could think about caching if necessary
        Locale desiredLocale = Locale.forLanguageTag(c1);
        Collator collator = Collator.getInstance(desiredLocale);
        cmp =, nv2.getString());
    } else {
        cmp = XSDFuncOp.compareString(nv1, nv2) ;
    return cmp;
// What the code is now
    return ((NodeValueSortKey) nv1).compareTo((NodeValueSortKey) nv2);

This moved the logic to a method in the NodeValueSortKey class. This reduced the complexity of the class with the switch statement. And it also made it easier to write unit tests.

If you are not involved in Open Source projects yet, I keep my suggestion from five years ago. Find a project related to something you like, and start reading the code, lurk in the mailing list or watch GitHub repositories.

You can always learn more!

♥ Open Source

Drawing sketch: Page 036

kinow @ Apr 24, 2017 00:41:03 ()
Page 036
Page 036

Almost ANZAC day. And almost time to switch back to coding Jenkins plug-ins.

Normally sketch more in-between development cycles for Open Source projects, or contract works.

  • Krill: Krill are small crustaceans of the order Euphausiacea, and are found in all the world’s oceans. The name krill comes from the Norwegian word krill, meaning “small fry of fish”, which is also often attributed to species of fish.
  • Tui: Tui are boisterous, medium-sized, common and widespread bird of forest and suburbia – unless you live in Canterbury. They look black from a distance, but in good light tui have a blue, green and bronze iridescent sheen, and distinctive white throat tufts (poi). They are usually very vocal, with a complicated mix of tuneful notes interspersed with coughs, grunts and wheezes. In flight, their bodies slant with the head higher than the tail, and their noisy whirring flight is interspersed with short glides.
  • Fatigue or Craziness: Lots of crazy people on Queen Street during holidays.

Troubleshooting a Jenkins Plug-in compatibility issue

kinow @ Apr 17, 2017 21:01:03 ()

This post is probably different from others. I will give a TL;DR, but will then give you a copy of a comment of a Jenkins JIRA issue. Hope you have fun reading it, specially if you maintain Jenkins servers or plug-ins.

TL;DR: there was an issue in Jenkins Job DSL Plug-in, that caused jobs created to have an invalid script. The fix had not been released, but was already in the master branch in GitHub.

Well, that was fun bug reproduced, I believe I know why that’s happening. Not so complicated to fix… but no fast way to fix it. Here’s the issue analysis (grab a coffee to read it).

  • Downloaded jenkins.war (2.32.3.war)
  • mkdir /tmp/123
  • JENKINS_HOME=/tmp/123 java -jar jenkins.war
  • Entered secret into form and submitted
  • Installed suggested plugins (boy that takes a while)
  • Created temp user
  • Installed (without restart) active-choices-plugin 1.5.3
  • Installed (without restart) job-dsl-plugin
  • Manually stopped Jenkins, and started it again with same command #3
  • Log in with user, all looking good
  • Created Freestyle job JENKINS-42655 (see attached config.xml)
  • Executed job, and found new job JENKINS-42655-1
  • Never opened the job configuration, clicked on the “Generated Items link to JENKINS-42655-1” to open in a new tab
  • Clicked on Build with Parameters
  • Looked at logs, and noticed the security-script-plugin exceptions
  • Went to “Manage Jenkins” / “In-process Script Approval” and approved scripts
  • Went back to the JENKINS-42655-1 build with parameters screen, and everything worked as expected

*Hummm. Issue more or less reproduced. Let’s investigate more.

  • Restarted Jenkins again
  • Changed the JENKINS-42655 seed job configuration to use a different script
  • Copied the config.xml file to another location
  • Went to build with parameters, and now it was broken again
  • Saved the job manually
  • Copied the config.xml file to yet another location
  • Went to build with parameters, and now it worked as reported in this issue

Now comes the interesting part. Looking at the diff. Attaching a screen shot so that others can have fun looking at it too. I installed Kompare as it has some cool features such as disabling diff for white spaces, blank lines, etc. The whole file changes as you save it. But if you ignore the number of white spaces… Then you can see that the Job DSL Plug-in is creating a <script> tag, as we used to do before 1.5 I think.

Screen shot
Screen shot

Now we use the script-security-plugin. So we need to wrap that around the script-security-plugin’s tags. Will report an issue for job-dsl-plugin, and will probably submit a pull request in the next days too. There’s not much left we can change in the plug-in code for that Piotr Tempes, so I’m afraid you will have to:

  • keep saving the job
  • perhaps work on the fix for Job DSL if you feel like doing it (as you could probably be faster than me in submitting the PR)
  • use an older version of the active-choices-plugin that doesn’t use security-script-plugin, though you could be bitten by other old bugs
  • write some script to replace the <script> tag and wrap it by the secureScript (doing what the pull request will do automatically later)

Sorry for not being able to quickly provide any workaround, nor to cut a quick bugfix release.

Cheers Bruno

ps: When I started getting involved in Open Source, I always felt extremely happy when people would spend their time troubleshooting my issues, or just educating me on how to behave in an Open Source community, or even how I should have troubleshooted initially the issue myself. So whenever I have time, I try to write detailed and polite replies. The person posting the comment is my past. The person posting the comment is my future :-) Kindness begets kindness.

♥ Open Source

Writing a binary parser in Python: NumPy vs. Construct

kinow @ Apr 14, 2017 19:21:03 ()

Some time ago I worked with researchers to write a parser for an old data format. The data was generated by device (radiosonde) using the vendor (Vaisala) specific binary format.

One of the researchers told me someone had written a parser for his work, and shared it on GitHub. To be honest, that was my first time parsing data in binary with Python. Did that before with C, C++, Perl, and Java, but never with Python.

The code on GitHub used NumPy and looked similar to this one.

import numpy as np

parse_header = np.dtype( [ (('field_a', 'b1'), ('field_b', '17b1') ] )

with open('input.dat', 'rb') as f:
    header = np.fromfile(f, dtype=parse_header, count=1)
    # ...

And it indeed worked fine. But in the end I used the code - after contacting the author and letting him know what I was about to do - as reference together with an old specification document for the format, and created a parser with Construct.

From Construct’s website:

Construct is a powerful declarative parser (and builder) for binary data.

This is what the code with construct looked like.

from construct import *

parse_header = Struct("parse_header",
        READY = 1,
        NOT_READY = 0,
        _default_ = "UNKNOWN",
    Bytes("reserved", 17)

# ...

parse_contents = Struct("parse_contents",
    Range(mincount=1, maxcout=5, subcon=pre_data),

with open('input.dat', 'rb') as f:
    parse_results = parse_contents.parse_stream(fid)
    # ...

Writing the parser with NumPy or Construct would achieve the same result. However, in the end this came down to personal preference, and my point of view as Software Engineer. This is the description of NumPy.

NumPy is the fundamental package for scientific computing with Python.

NumPy is a project tailored for scientific computing, with a focus on linear algebra, N-dimensional arrays, and so it goes. While it contains code that can parse binary data, the footprint added to a project that includes it as dependency is quite big.

The parser written with NumPy wasn’t using 5% of the NumPy code base. Probably less than 1%. Updates to NumPy could break the application compatibility, even if the update came due to some new matrix operation added to NumPy through some external and missing dependency.

In Java something similar happens with Google Guava. While I use it some times, most of the times I find myself using one of the Apache Commons libraries, or another dependency with just what I need. To avoid including unnecessary code to my application.

If you prefer to use NumPy that’s fine too :-) I just had the time enough to rewrite it instead of using the NumPy (took a couple of hours). In other cases it may still make sense to use another tool or library, even if it was not made specifically for the job ¯\(ツ)

♥ Open Source

Spring Cloud encrypted values and Spring PropertySources

kinow @ Apr 14, 2017 11:21:03 ()

As I could not find any documentation for that, I decided to write it as a note to myself in case I use the encryption and decryption with Spring Cloud again.

In Spring and Spring Boot, you normally have multiple sources of properties, like multiple properties files, environment properties and variables, and so it goes. In the Spring API, these are represented as PropertySource‘s.

In a Spring Boot application, you would be used to overriding certain properties by defining environments and using an file, or overriding values with environment properties.

This is common in Spring Boot applications deployed to Amazon Elastic Beanstalk.

Some time ago another team at work found that overriding did not always work when you have encrypted values in your properties files. Even if you specified new values in the Amazon Elastic Beanstalk application configuration.

Yesterday, while debugging the issue and reading Spring Cloud source code, I found its EnvironmentDecryptApplicationInitializer.

It basically iterates through all loaded property sources, looking for values that start with {cipher}. Then it calls the Spring Security TextEncryptor defined in the application.

Finally, it creates a new property source, called decrypted, with the decrypted values. So when your application looks for a property called XPTO, and if it has been encrypted, it will find the value in the decrypted propery source, regardless of whether you tried to override it or not.

# Property sources listed in Eclipse IDE


# When using encrypted values

  decrypted, <-------- created by Spring Cloud, with decrypted values. Prepended to the list of property sources

So in case you have encrypted values in your Spring application (and you are using Spring Cloud, of course) remember that these values will have higher priority, and can only be overriden by other encrypted values.

♥ Open Source