Menu

Tag

Posts tagged with ‘shell script’

Quickly Verifying jar Signatures For ASF Releases

kinow @ Oct 14, 2017 00:24:54

The release process within the Apache Software Foundation includes a series of steps. Amongst these steps is the voting process. In Apache Commons, the release instructions includes a note on artefact signatures.

During the course of the VOTE, make sure that one or more of the reviewers have verified the signatures and hash files included with the release artifacts. If no one specifically mentions having done that during the VOTE, ask on the dev list and make sure someone does this before you proceed with the release.

Tired of always having to manually check several artefacts, or having to come up with the correct shell commands to iterate through a list of files, the other day I wrote a simple script to download the KEYS file, import it, download all the artefacts, then iterate through them and verify the signature.

Here’s the script. Licensed under the GPL licence.

#!/usr/bin/env bash

url=""

# From: https://blog.mafr.de/2007/08/05/cmdline-options-in-shell-scripts/
USAGE="Usage: `basename $0` [-hv] https://repository.apache.org/.../commons/commons-configuration/2.2/"

# Parse command line options.
while getopts hv: OPT; do
    case "$OPT" in
        h)
            echo $USAGE
            exit 0
            ;;
        v)
            echo "`basename $0` version 0.0.1"
            exit 0
            ;;
        \?)
            # getopts issues an error message
            echo $USAGE >;
    esac
done

# Remove the switches we parsed above.
shift `expr $OPTIND - 1`

# We want at least one non-option argument. 
# Remove this block if you don't need it.
if [ $# -eq 0 ]; then
    echo $USAGE >&2
    exit 1
fi

# Access additional arguments as usual through 
# variables $@, $*, $1, $2, etc. or using this loop:
URL=$1

echo "url: ${URL}"

# Use a local temporary directory
BUILD_DIR=$(mktemp -d)
pushd "$BUILD_DIR"

echo "build dir: ${BUILD_DIR}"

# Download KEYS file
KEYS_URL=https://www.apache.org/dist/commons/KEYS

echo "importing KEYS from: ${KEYS_URL}"

wget "$KEYS_URL"
gpg --import KEYS

# Download JARs and signature files
echo "downloading all jars and signature files..."

wget -r -nd -np -e robots=off --wait 1 -R "index.html*" "${URL}"

# Check the files
for x in *.jar; do gpg --verify "${x}".asc; done

# EOF

The script can be found at GitHub too: https://github.com/kinow/dork-scripts/tree/3c519a74f28c08310ce2e65f8e860d61fd0c5c07/gpg/asf-sigs

Removing Javadoc SVN Id Tags with Shell Script

kinow @ Sep 13, 2017 16:49:26

Subversion supports Keyword Substitution, which performs substitution of some keywords such as Author, Date, and Id. The Id is the date, time, and user that last modified the file.

It used to be common to all Apache Commons components to have a line as follows in the header of each Java class.

/**
 * SomeClass class.
 *
 * @version $Id$
 */
public class SomeClass {

}

Then the generated Javadoc would contain the date of when the class was altered. Although useful, with proper versioning, it becomes obsolete. It is much more important to know what is the version of the software, not the last time it was modified or by whom. In case you have a problem with that specific file, you can always check the history of the file using git log, or git bisect, or …

Apache Commons components that are migrated to git need to have these lines removed. git does not support these Subversion Keywords so it is never properly rendered. And as every time I have to remove these lines I come up with some shell script snippet, I decided to document the last one I wrote, so that it can save me some time ‐ and perhaps for somebody else too?

find . -name "*.java" -exec sed -i '/^.*\*\s*@version\s*\$Id\$.*$/d' {} \;

And then push a commit with the change :-) In case you know some regex, you can change it and use the same command syntax to remove comments, specific configuration lines, etc.

That’s all. Happy scripting!

♥ Open Source

Changing Spring Boot environment variables in the command line

kinow @ Nov 21, 2016 21:26:03

This week while helping developers and testers to experiment with a backend application, some of them found useful to learn a simple trick to change Spring Boot properties when you can run the application locally (our testers build, compile, change the code, how cool is that?).

Here’s how it works. Say you have the following settings in your application’s application.properties:

my.application.database.username=sa
my.application.database.password=notasimplepassword

And that you want to change these parameters in order to, for instance, create an application error, so that you can code and test what happens to the frontend application in that situation.

You replace dots by underscores, and put all your words in upper case. So the variables above would be: MY_APPLICATION_DATABASE_USERNAME and MY_APPLICATION_DATABASE_PASSWORD.

Furthermore, you do not need to edit your application.properties file, if you are on Linux or Mac OS. You can start the application and override environment variables at the same time with the following syntax.

$ MY_APPLICATION_DATABASE_USERNAME=olivei MY_APPLICATION_DATABASE_PASSWORD=7655432222a mvn clean spring-boot:run

This way your application will start with the new values.

Happy hacking!

—EDIT—

As pointed by Stéphane Nicoll (thanks!), you could change the property values without having to use the upper case syntax.

mvn -Dmy.application.database.username=anotheruser clean spring-boot:run

And he even included a link to docs! ♥ the Internet and Open Source!

Add a header to a file with Shell script (sed)

kinow @ Nov 12, 2016 01:55:03

Today I was re-generating the documentation for a REST API written in PHP, with Laravel. To generate the documentation, one would have to call a Laravel command first. That command would create a Markdown page. And since in this project I am using Jekyll for the project site, the final step was adding a header to the file, so that Jekyll can recognize that content as a blog post.

Laravel allows you to add custom commands to your project, so I decided to write a command that would call the other command that generates the documentation, and add an extra step of adding the header to the Markdown file.

Here’s the shell script part, that allows you to add a header to a file, in place (i.e. it will alter and save the change your file).

sed -i 1i"----\nlayout: page\ntitle: API Installation\n----\n\n" ./docs/documentation/api/api.md

Here the first argument to sed, -i is for in place. Then that strange 1i means that it will insert something before the first line, once. Then we have our header, and finally the file.

Happy hacking!

Checking the operating system type in shell script

kinow @ Nov 05, 2016 23:27:03

Last week I learned about a tool called ShellCheck, a tool for static analysis of shell scripts. It reports errors like missing double quotes, use of deprecated syntax, etc.

I decided to check some projects I contribute to, and the first issue I found was in Apache Jena:

kinow@localhost:~/Development/java/jena/jena/apache-jena/bin$ shellcheck arq

In arq line 8:
    case "$OSTYPE" in
          ^-- SC2039: In POSIX sh, OSTYPE is not supported.

So, in summary, the OSTYPE variable should not be available in POSIX shell. The case in question, where OSTYPE is being used, checks for the Darwin OS type (i.e. Mac OS). Knowing how things get weird when you use different operating systems, I decided to check and learn how OSTYPE works. Here’s what I found.

  • In Ubuntu, with /bin/bash, OSTYPE works fine (linux-gnu)
  • In Ubuntu, with /bin/sh, OSTYPE is not set
  • In Mac OS, with /bin/sh, OSTYPE is set (darwin15)

I checked the shells to make sure they were not pointing to symbolic links - some distributions use a different default shell, and replace /bin/bash and/or /bin/sh by a link to another shell. Looks like Mac OS has a POSIX shell that behaves different than Ubuntu’s.

Instead of trying to find a way to use OSTYPE, I decided to spend some time looking at how other projects do the same thing. And the best example I could find was git.

Instead of relying on OSTYPE, git uses uname.

I will spend some time during the next days working on a proposal to replace the OSTYPE from Apache Jena scripts, but then may have to submit more changes for the other issues found by ShellCheck.

Happy hacking!