What happens when you upload a Turtle file in Apache Jena Fuseki

kinow @ May 27, 2018 22:42:55

I am working on an issue for Skosmos that involves preparing some Turtle files and uploading them using Apache Jena Fuseki’s web interface.

The issue is now pending feedback, which gives me a moment to have fun with something else. So I decided to dig down the rabbit hole and start learning more about certain parts of the Apache Jena code base.

This post will be useful to myself in the future, as a note-taking in a series, so that I remember how things work - you never know right? But hopefully this will be useful to other wanting to understand more about the code of Apache Jena.

Where upload happens - SPARQL_Upload and Upload (Fuseki Core)

Knowing a bit of the code base, I went straight to the SPARQL_Upload class, from the Fuseki Core module. Set up a couple of breakpoints, uploaded my file, but nothing. Then tried on its package-neighbour class, Upload.

Actually, it is easier to understand seeing the class hierarchy, and knowing that when I run the application in Eclipse, it is running with Jetty, serving servlets (there is no framework like Wicket, Struts, etc, involved).

Several filters are applied to the HTTP request too, like Cross Origin, Shiro, and the FusekiFilter. The latter looks at the requests to see if it includes a dataset. If a dataset is found - it is in our case - then it hands the request over to the right class to handle it.

REST_Quads_RW will take care of the upload action, using the Upload class where my breakpoint stopped.

Upload#incomingData() (Fuseki Core)

Upload#incomingData() starts by checking the Content Type from the request. In my case it is a multipart/form-data. Then it calls its other method #fileUploadWorker().

#fileUploadWorker() creates a ServletFilterUpload, from Apache Commons FileUpload. With that, it opens a stream for the file, retrieves its name and other information, such as the content type.

Ah, the content type is interesting too. It defaults to RDFXML, but what’s interesting is the comment.

if ( lang == null )
    // Desperate.
    lang = RDFLanguages.RDFXML ;

Well, in this case we are getting a Lang:Turtle. So it now knows that it has a Turtle file, but it still needs to parse it.

ActionLib#parse() (Fuseki Core)

Upload calls ActionLib#parse(), which uses RDFParserBuilder to build a parser. It applies a nice fluent API design when doing that.

RDFParser.create()
    .errorHandler(errorHandler)
    .source(input)
    .lang(lang)
    .base(base)
    .parse(dest);

Side note to self: the `RDFParser` has a `canUse` flag. It seems to indicate the parser can be used just once. Though it looks actually it works until the stream is closed…

So RDFParserBuilder will call RDFParser, which in turn will use the classes LangTurtle and LangTurtleBase.

LangTurtle (ARQ)

ARQ is a low level module in Jena, responsible for parsing queries, and also some of the interaction with graphs and datasets.

LangTurtle extends LangTurtleBase. Their task starts by populating the prefixMap, which contains all those prefixes used in queries like rdfs, void, skos, etc.

Then it will keep parsing triples until it finds an EOF. For every triple, after the Predicate-Object-List is found, it calls LangTurtle#emit().

The #emit()method creates a Triple object (Jena Core, graph package). And also a StreamRDFCountingBase to keep track of statistics to display back to the user.

StreamRDFCountingBase extends StreamRDFWrapper, and wraps - as per name - other StreamRDF‘s, such as ParserOutputDataset.

ParserOutputDataset holds a reference to the DatasetGraph and also to the prefixMap populated earlier in LangTurtle. For each Triple that we have it will call the DatasetGraph#add method, creating a new Quad with the default graph name.

Conclusion

Finally, readers and streams are closed. An UploadDetails object is created holding stats ollected in StreamRDFCountingBase, which are also used for logging.

Upload#incomingPath() will return the UploadDetails. If there are no errors then the transaction will be commited. It involves again classes from ARQ and TDB (for journaling), but that will be for another post.

The final method called in the Upload class will be detailsJson(), which returns the object as JSON. This JSON string is then finally returned to the user.

So that’s it. Probably the next step will be to learn how DatasetGraph works, or maybe more about transactions in Jena.

Happy hacking !

tags [ apache software foundation ]  [ jena ]  [ opensource ]  [ programming ] 

Exif Odd Offsets

kinow @ Dec 25, 2017 21:43:33

A file format like JPEG may contain metadata in JFIF, Exif, or a vendor proprietary format. The Exif format is based - or uses parts of - on the TIFF format.

Within an Exif metadata block, you should see directories, with several entries. The entries have fields like description, value, and also an offset. The offset indicates the offset to the next entry.

The Exif specification defines that implementers must make sure to keep the offset an even number, within 4 bytes.

I recently worked on IMAGING-205, a ticket about odd offsets in files with Exif metadata. This issue was exactly to address that when files were rewritten with Apache Commons Imaging, even though the image initially had no odd offsets, after the entries were rearranged, we could have odd offsets.

The fix was simply checking for odd offsets, adding +1, and later it would be put within the 4 bytes limit.

Locating the bug

One interesting point, however, is that this is in the standard, but not all software that read and write Exif follow the specification. So it is quite common to find images with odd offsets.

Which means you could take a picture with your phone, that contains some Exif metadata, and be surprised to analyze it with exiftool and get warnings about odd offsets. Most viewers handle odd and even offsets, so it should work for most cases, unless you have a strict reader/viewer.

Happy hacking!

&heart; Open Source

tags [ programming ]  [ java ]  [ apache software foundation ]  [ opensource ] 

Remember to synchronize when iterating streams from a synchronized Collection

kinow @ Dec 03, 2017 23:56:13

When iterating collections created via Collections.synchronizedList for instance, you are required to obtain a lock on the actual list before doing so. So you normally end up with code similar to:

List list = Collections.synchronizedList(new ArrayList());
synchronized (list) {
  Iterator i = list.iterator(); // Must be in synchronized block
  while (i.hasNext())
      foo(i.next());
}

This requirement is documented in the javadocs.

Since lambdas and streams are being more widely used, it is important to remind that when iterating via a stream we also need to obtain a lock on the synchronized collection created.

List list = Collections.synchronizedList(new ArrayList());
synchronized (list) {
  list.stream()
    .anyMatch(...)
}

Here’s an example from Zalando Nakadi Event Broker.

Happy hacking!

tags [ programming ]  [ java ]  [ opensource ] 

Watch out for Locales when using NumberFormat with currencies

kinow @ Dec 02, 2017 22:51:00

In Java you have the NumberFormatException to help you formatting and parsing numbers for any locale. Said that, here’s some code.

BigDecimal negative = new BigDecimal("-1234.56");

DecimalFormat nf = (DecimalFormat) NumberFormat.getCurrencyInstance(Locale.UK);
String formattedNegative = nf.format(negative);

System.out.println(formattedNegative);

The output for this code is -£1,234.56. That’s expected, as the locale is set to UK, so the currency symbol used is for British Pounds. And as the number is negative, you get that minus sign as a prefix. For Japanese locale you’d get -￥1,235, and for Brazilian locale you’d get -R$ 1.234,56.

So far so good.

What about the following code, with nothing different except for the locale set to US.

BigDecimal negative = new BigDecimal("-1234.56");

DecimalFormat nf = (DecimalFormat) NumberFormat.getCurrencyInstance(Locale.US); // <--- US now
String formattedNegative = nf.format(negative);

System.out.println(formattedNegative);

Some could intuitively expect -$1,234.56. However, the output is actually ($1,234.56).

There are different prefixes and suffixes. But in some locales the prefix can be empty, or, as in the case of the US locale, it can be quite different than what you could expect.

Learned about this peculiarity from NumberFormat while working on VALIDATOR-433 for Apache Commons Validator.

Happy hacking!

tags [ programming ]  [ java ]  [ apache software foundation ] 

What Are Identification Keys

kinow @ Dec 01, 2017 23:26:09

In biology, an identification key is a printed or computer-aided device that aids the identification of biological entities, such as plants, animals, fossils, microorganisms, and pollen grains. Identification keys are also used in many other scientific and technical fields to identify various kinds of entities, such as diseases, soil types, minerals, or archaeological and anthropological artifacts

When you work writing software, it is common that at one point or another of the product development you will work with a SME, or subject-matter expert. This person is someone experienced in a field such as telecom billing, insurance, taxes, or even nature and environment.

Questions too specific, that land away from the knowledge area of software engineers, are normally addressed by the SME. Sometimes the knowledge from a SME can be spread in a team after they have worked together for a while.

Most of my time in New Zealand I have been working with environmental sciences. In one project, I remember hearing other developers mention ID’s, Keys, and Identification Keys. I had not worked with environmental sciences before, and only learned about its Portuguese translation (“Chave”, or “Chave de Identificação”, or “Chave Dicotêmica”) recently.

They are mechanisms, digital or physical, that help you identify plants, animals, fossils, etc. And I found out later that there are many identification key software written for the Web, desktop, and also other medias like PDF’s, and even printed version.

And also in New Zealand I found how fun identifying species can be with the iNaturalist app and community. If you are like me before, and had not heard about it before, search around more about it. It is a very simple concept, common to biologists, people who work with taxonomy, marine life, birds.

If you are in New Zealand, or just interested about it, here are some links to get you started identifying the local species.

• DOC (Department of Conservation) Species descriptions and threat classifications
• NatureWatch Helpful websites for exploring NZ nature
• NIWA (National Institute of Water and Atmospheric Research) Freshwater fish ID guides/keys
• Clues to Bird ID (PDF)

Have fun!

tags [ programming ]