Tag

Posts tagged with ‘opensource’

R Shiny + Ansible = shiny.nzoss.org.nz

kinow @ Feb 03, 2018 17:23:03

R Shiny + Ansible = shiny.nzoss.org.nz

Event
R Shiny + Ansible = shiny.nzoss.org.nz
Where ?
Auckland, New Zealand
When ?
2018-02-08

Exif Odd Offsets

kinow @ Dec 25, 2017 21:43:33

A file format like JPEG may contain metadata in JFIF, Exif, or a vendor proprietary format. The Exif format is based - or uses parts of - on the TIFF format.

Within an Exif metadata block, you should see directories, with several entries. The entries have fields like description, value, and also an offset. The offset indicates the offset to the next entry.

The Exif specification defines that implementers must make sure to keep the offset an even number, within 4 bytes.

I recently worked on IMAGING-205, a ticket about odd offsets in files with Exif metadata. This issue was exactly to address that when files were rewritten with Apache Commons Imaging, even though the image initially had no odd offsets, after the entries were rearranged, we could have odd offsets.

The fix was simply checking for odd offsets, adding +1, and later it would be put within the 4 bytes limit.

A screen shot of Eclipse with source code
Locating the bug

One interesting point, however, is that this is in the standard, but not all software that read and write Exif follow the specification. So it is quite common to find images with odd offsets.

Which means you could take a picture with your phone, that contains some Exif metadata, and be surprised to analyze it with exiftool and get warnings about odd offsets. Most viewers handle odd and even offsets, so it should work for most cases, unless you have a strict reader/viewer.

Happy hacking!

&heart; Open Source

Remember to synchronize when iterating streams from a synchronized Collection

kinow @ Dec 03, 2017 23:56:13

When iterating collections created via Collections.synchronizedList for instance, you are required to obtain a lock on the actual list before doing so. So you normally end up with code similar to:

List list = Collections.synchronizedList(new ArrayList());
synchronized (list) {
  Iterator i = list.iterator(); // Must be in synchronized block
  while (i.hasNext())
      foo(i.next());
}

This requirement is documented in the javadocs.

Since lambdas and streams are being more widely used, it is important to remind that when iterating via a stream we also need to obtain a lock on the synchronized collection created.

List list = Collections.synchronizedList(new ArrayList());
synchronized (list) {
  list.stream()
    .anyMatch(...)
}

Here’s an example from Zalando Nakadi Event Broker.

Happy hacking!

Using formatter exclusions with Eclipse

kinow @ Nov 06, 2017 21:56:56

Sometimes when you are formatting your code in Eclipse, you may want to prevent some parts of the code from being formatted. Especially when using Java 8 lambdas and optionals.

Here’s some code before being formatted by Eclipse’s default formatter rules.

Code adapted from: blog post Java d’eau ‐ Java 8: Streams in Hibernate and Beyond

session.createQuery("SELECT h FROM Hare h", Hare.class)
    .stream()
    .filter(h -> h.getId() == 1)
    .map(Hare::getName)
    .forEach(System.out::println);

Then after formatting.

session.createQuery("SELECT h FROM Hare h", Hare.class).stream().filter(h -> h.getId() == 1).map(Hare::getName)
                .forEach(System.out::println);

Which doesn’t look very appealing, ay? You can change this behaviour at least in two ways. The first by telling the formatter to ignore this block, through a special formatter tag in your code.

First you need to enable this feature in Eclipse, as it is disabled by default. This setting is found in the preferences JavaCode StyleFormatterEditOff/On Tags.

A screen shot of Eclipse formatter settings
Enabling formatter tags in Eclipse

Then formatting the following code won’t change a thing in the block surrounded by the formatter tags.

/* @Formatter:off */
session.createQuery("SELECT h FROM Hare h", Hare.class)
    .stream()
    .filter(h -> h.getId() == 1)
    .map(Hare::getName)
    .forEach(System.out::println);
/* @Formatter:on */

But having to type these tags can become annoying, and cause more commits and pull requests to be unnecessarily created. So an alternative approach can be to change the formatter behaviour globally.

This can be done in Eclipse in another option under the formatter options, JavaCode StyleFormatterEditLine WrappingFunction CallsQualified invocations.

You will have to choose “Wrap all elements, except first element if not necessary” under Line wrapping policy. And also check “Force split, even if line shorter than maximum line width”.

A screen shot of Eclipse formatter settings
Enabling custom formatter behaviour globally

Once it is done, your code will look like the following no matter what.

session.createQuery("SELECT h FROM Hare h", Hare.class)
    .stream()
    .filter(h -> h.getId() == 1)
    .map(Hare::getName)
    .forEach(System.out::println);

Happy coding!

♥ Open Source

Quickly Verifying jar Signatures For ASF Releases

kinow @ Oct 14, 2017 00:24:54

The release process within the Apache Software Foundation includes a series of steps. Amongst these steps is the voting process. In Apache Commons, the release instructions includes a note on artefact signatures.

During the course of the VOTE, make sure that one or more of the reviewers have verified the signatures and hash files included with the release artifacts. If no one specifically mentions having done that during the VOTE, ask on the dev list and make sure someone does this before you proceed with the release.

Tired of always having to manually check several artefacts, or having to come up with the correct shell commands to iterate through a list of files, the other day I wrote a simple script to download the KEYS file, import it, download all the artefacts, then iterate through them and verify the signature.

Here’s the script. Licensed under the GPL licence.

#!/usr/bin/env bash

url=""

# From: https://blog.mafr.de/2007/08/05/cmdline-options-in-shell-scripts/
USAGE="Usage: `basename $0` [-hv] https://repository.apache.org/.../commons/commons-configuration/2.2/"

# Parse command line options.
while getopts hv: OPT; do
    case "$OPT" in
        h)
            echo $USAGE
            exit 0
            ;;
        v)
            echo "`basename $0` version 0.0.1"
            exit 0
            ;;
        \?)
            # getopts issues an error message
            echo $USAGE >;
    esac
done

# Remove the switches we parsed above.
shift `expr $OPTIND - 1`

# We want at least one non-option argument. 
# Remove this block if you don't need it.
if [ $# -eq 0 ]; then
    echo $USAGE >&2
    exit 1
fi

# Access additional arguments as usual through 
# variables [email protected], $*, $1, $2, etc. or using this loop:
URL=$1

echo "url: ${URL}"

# Use a local temporary directory
BUILD_DIR=$(mktemp -d)
pushd "$BUILD_DIR"

echo "build dir: ${BUILD_DIR}"

# Download KEYS file
KEYS_URL=https://www.apache.org/dist/commons/KEYS

echo "importing KEYS from: ${KEYS_URL}"

wget "$KEYS_URL"
gpg --import KEYS

# Download JARs and signature files
echo "downloading all jars and signature files..."

wget -r -nd -np -e robots=off --wait 1 -R "index.html*" "${URL}"

# Check the files
for x in *.jar; do gpg --verify "${x}".asc; done

# EOF

The script can be found at GitHub too: https://github.com/kinow/dork-scripts/tree/3c519a74f28c08310ce2e65f8e860d61fd0c5c07/gpg/asf-sigs