Blog Taxonomy

Home

Learning afl and testing MapServer

kinow @ Feb 27, 2016 23:21:03 ()

afl is a fuzzer, an application that combines a series of algorithms in order to try invoking programs with several different input values. It then analyses the application execution flow given different test case scenarios. You can read more about fuzzing at this OWASP page, or in other blogs that I also used while learning about afl 1 2

At work we are using MapServer for serving WFS and WMS. And I am using it for the NZ OpenStreetMap maps too. MapServer is written in C++ and is normally exposed as a CGI script, so I thought it was worth learning about afl and trying it on MapServer, as in case it finds any interesting bug I can submit it to the MapServer project.

( Read more ... )

How does the Jenkins Credentials Plug-in store passwords?

kinow @ Sep 07, 2015 01:25:03 ()

Jenkins Credentials Plug-in manages credentials stored in Jenkins. These credentials can be used in many jobs and by plug-ins for executing SSH commands, authenticating to systems, or running other commands that need some sort of authentication or authorisation.

I recently used its API for the first time in the BioUno figshare Plug-in to store OAuth 1.0 credentials (consumer key, consumer secret, token key, token secret). This blog post has more details about how we used the plug-in, but this post is specifically on how the passwords are stored by Jenkins.

Secret and ciphers

Jenkins stores its configuration on disk as XML using the XStream library. Plug-in developers using the Credentials Plug-in API must use the Secret class to encrypt sensitive information.

The Secret.fromString method is responsible for creating a cipher from a given String. As in the Secret Javadoc, “this is not meant as a protection against code running in the same VM, nor against an attacker who has local file system access on Jenkins master”. But at least makes things more complicated :-)

public static Secret fromString(String data) {
    data = Util.fixNull(data);
    Secret s = decrypt(data);
    if(s==null) s=new Secret(data);
    return s;
}

The first line simply replaces a null string by an empty “”, or keeps the current value of not null.

After that, the decrypt method is called.

public static Secret decrypt(String data) {
    if(data==null)      return null;
    try {
        byte[] in = Base64.decode(data.toCharArray());
        Secret s = tryDecrypt(KEY.decrypt(), in);
        if (s!=null)    return s;

        // try our historical key for backward compatibility
        Cipher cipher = getCipher("AES");
        cipher.init(Cipher.DECRYPT_MODE, getLegacyKey());
        return tryDecrypt(cipher, in);
    } catch (GeneralSecurityException e) {
        return null;
    } catch (UnsupportedEncodingException e) {
        throw new Error(e); // impossible
    } catch (IOException e) {
        return null;
    }
}

The KEY.decrypt() call will return a javax.crypto.Cipher. The Cipher class is handled in CryptoConfidentialKey in Jenkins API, where it defines the algorithm used to create the cipher: AES.

Jenkins has also a ConfidentialStore, that is required to create the cipher. This class must be initialized before someone tries to create or read a cipher. This extra step also increases security, though access to the JVM is still a problem.

It is a bit late, so it is all for today. In summary: the credentials plug-in gives you a central place to manage credentials, but it is up to plug-in developers to use it. Sensitive values can be encrypted with AES on disk. So it is important that your file permissions, ACL and system auditing processes are in place and well maintained and monitored.

Happy hacking!

Modeling observation data in SOS (Sensor Observation Service)

kinow @ Jun 23, 2015 13:40:03 ()

This week NZ Herald published an article about a device created by an Irish farmer enterpreneur that sends a message to a farmer when the cow is about to give birth. The device monitors “heightened tail moviment”.

In this post I will try to apply what I am learning following the SOS Tutorial (the Open Geospatial Consortium standard for Sensor Observation Service). Feel free to drop me a message via @kinow if you find any mistakes or have any suggestions.

Modeling the tail moviment observation data in SOS

SOS is a standard designed to provide access to observation data. There are several server implementations, such as Kisters KiWIS, istSOS and 52North SOS.

The standard mentions and utilises several other standards, such as SensorML, WFS, XML, WMS, etc. The SOS Tutorial on how to model your observation data for SOS starts by defining procedure, observed property, feature of interest, phenomenon and result times, and the result value.

Let’s try to model the data from the tail moviment sensors in the following table.

Name Description Our example
Procedure The process that has generated the observation, such as a sensor, from the O&M specification. In our example that could be a sensor identification moocall_001
Observed Property A property which is observed (look at NASA SWEET ontology for existing values) Heightned tail moviment
Feature of Interest A feature that carries the property which is observed Pregnant cow
Phenomenon and Result Times The phenomenon time is when the data has been taken, and the result time when it has been created. If both are the same, the resultTime can point to the phenomenonTime 20150623142000
Result value This is the result of the observation. Can be a OM_Measurement if numeric, OM_TruthObservation, etc (O&M) 3 (supposing we have a scale from 1 to 5)

In the next post I will try to show how to load this model and some dummy data into a fresh installation of 52North SOS server.

Groovy Hooks in Jenkins for increasing logging level

kinow @ Apr 12, 2015 11:30:03 ()

Yesterday, while debugging a problem we had in the BioUno update center, I realized that after increasing the logging level in the WEB interface, the messages that I needed weren’t being displayed in the logs.

It happened because some of the logging happened during Jenkins initialization, and before I could adjust the log level.

The solution was to use a Groovy Hook Script. If you are familiar with Linux init scripts, the idea is quite similar.

A Groovy script in the $JENKINS_ROOT_DIR/init.groovy.d/ directory is executed during Jenkins initialization. This way you can increase the global logger level with a script as the following below.

import java.util.logging.ConsoleHandler
import java.util.logging.LogManager
import java.util.logging.Logger
import java.util.logging.Level

def logger = Logger.getLogger("")
logger.setLevel(Level.FINEST)
logger.addHandler (new ConsoleHandler())

Happy logging!

Contributing to Apache Jena

kinow @ Jan 01, 2015 18:49:03 ()

As I mentioned in my previous post, I am using Apache Jena for a project of a customer. I had never used any triple store, nor a SPARQL Endpoint server before. But for being involved with the Apache Software Foundation, and since the company itself is using several Apache components, it was only natural Jena to be our first choice.

It has served us very well so far. At the moment we have less than 100 queries per day, but the project is still under development and we expect 1000 queries per day by the first quarter of 2015 and 1000000 near the end of 2015. We also have few entries in TDB, but expect to grow this number to a few million before 2016.

When I work for companies and we use Open Source Software (OSS) in a project, I always prepare assessment reports to include in the deliveries. In this report I justify the choice of Open Source Software (as well as commercial software). Sometimes I am lucky to work for a company that asks me to include hours to work on OSS :-)

I use Trello to triage issues in OSS projects (and for several other things). I have a board with several cards for Open Source. About a month ago I set up one for Jena and listed the issues that I thought I could contribute to.

Jena Trello card
Jena Trello card

I annotate easy issues with a “lhf” suffix for Low Hanging Fruit issues, and delete issues from the card once I submit a patch or update it (and include it in another card for the TupiLabs reports).

Most of the issues I included in the card for Jena had been created over two years ago, and hadn’t been updated in a while. When you test these issues against the current code, usually you find that some of them have already been fixed. Other issues included documentation problems, and minor features. I didn’t find any blocker issue that would impede us to use Jena in production.

Jena JIRA activity summary
Jena JIRA activity summary

The picture above shows the past 30 days activity summary in JIRA for Jena. The red line shows issues created, and the green line issues resolved. Andy Seaborne was very active in the past days and fixed several issues that were too old and had already been fixed in the trunk, and kindly merged patches and pull requests.

Some issues like JENA-632 will take a longer time to fix, but I’m getting used to Jena’s source code, and at the same getting more confident to use it in production - especially with a supportive OSS community. We are using Jena for RDF with Hadoop, and I learned that I can replace some custom Writables by others in the Jena Hadoop submodule.

By the way, even though this project ends in April, I intend to continue contributing to Jena. There is a lot of parts of the code that I would love to be able to understand and contribute, in special the Graph database, optimization techniques for SPARQL queries, the grammars used in the project, Fuseki v2 and enhance its testing harness (as well as the test coverage).

If you are looking for a interesting project to get you started with semantics, linked data, RDF, and even graphs and database querying, try contributing to Jena. I bet you’ll have a lot of fun!

Happy hacking and happy 2015!