Menu

 

Blog Taxonomy

Home

Simulating less memory with ulimit

kinow @ Mar 26, 2017 11:14:03

These days I was trying to reproduce a bug in Krita where it would crash when a user copied group layers between windows. It appeared that the user was getting a segmentation fault due to the user’s computer running out of memory.

I could not reproduce the issue, but my computer has 16 GB. The first thing that came to my mind was to create a virtual machine with less memory to reproduce the issue. But I decided to spend some time looking for a simpler way of doing it.

Searching the web I found some suggestions that ulimit could work. After playing for a while with ulimit and htop, and verifying the amount of memory necessary to open two files in two windows in Krita, I came up with the following settings.

# 2550 mb in kb
ulimit -v 2550000

# 2 gb in kb
ulimit -m 2000000

# Confirm limits
ulimit -a

gdb $HOME/Development/cpp/workspace/krita_install/bin/krita

Then after copying a few layers from one window to another, I successfully reproduced the issue, and could include a backtrace in the Krita issue tracking system.

♥ Open Source

Using Google Natural Language API in an AWS Elastic Beanstalk application

kinow @ Mar 15, 2017 13:18:03

Besides providing an API for users and developers, Google also provides a series of (very well-written) client implementation, in several programming languages. That’s the case for Google Storage, Google Vision, and for Google Natural Language API’s.

I recently had to use the latter in a POC at work, and ran into an interesting issue with our AWS Elastic Beanstalk environment.

Google Language API requires that clients authenticate before sending requests. If you use their google-cloud-java, you can define the GOOGLE_APPLICATION_CREDENTIALS. This environment variable must point to a JSON file with the Google Language API credentials.

The issue is that while Google Language API (or more precisely google-auth-library-java) looks for an environment variable, in Elastic Beanstalk you are able to specify only system properties (unless you want to try something with ebextensions, maybe some JNI…).

A workaround for this issue in Google Natural Language API, is to create and pass a LanguageServiceSettings to your LanguageServiceClient. This settings object, when created, must be given a channel provider with a FixedCredentialsProvider.

Of course reading code is way easier than reading this workaround description.

// File: GoogleNaturalLanguageService.java
// ...
    // envvar or property used to specify the Google Application Credentials
    private final static String GOOGLE_APPLICATION_CREDENTIALS = "GOOGLE_APPLICATION_CREDENTIALS";

    /**
     * Google Natural Language API.
     */
    private LanguageServiceClient languageServiceClient;

    @PostConstruct
    public void init() throws Exception {
        // Elastic Beanstalk supports Properties, not Environment Variables.
        // Google credentials library will load
        // the JSON location for the service to authenticate from an envVar. So
        // we need to fix that here.
        String googleApplicationCredentials = System.getenv(GOOGLE_APPLICATION_CREDENTIALS);
        LOGGER.info(String.format("GOOGLE_APPLICATION_CREDENTIALS in environment variable: %s", googleApplicationCredentials));
        if (StringUtils.isBlank(googleApplicationCredentials)) {
            googleApplicationCredentials = System.getProperty(GOOGLE_APPLICATION_CREDENTIALS);
            LOGGER.info(String.format("GOOGLE_APPLICATION_CREDENTIALS in JVM property: %s", googleApplicationCredentials));
        }

        if (googleApplicationCredentials == null) {
            throw new RuntimeException("Could not locate GOOGLE_APPLICATION_CREDENTIALS variable!");
        }

        final LanguageServiceSettings languageServiceSettings;
        try (InputStream is = new FileInputStream(new File(googleApplicationCredentials))) {
            final GoogleCredentials myCredentials = GoogleCredentials
                    .fromStream(is)
                    .createScoped(
                            Collections.singleton("https://www.googleapis.com/auth/cloud-platform")
                    );
            final CredentialsProvider credentialsProvider = FixedCredentialsProvider.create(myCredentials);

            final InstantiatingChannelProvider channelProvider = LanguageServiceSettings
                    .defaultChannelProviderBuilder()
                    .setCredentialsProvider(credentialsProvider)
                    .build();
            languageServiceSettings = LanguageServiceSettings
                    .defaultBuilder()
                    .setChannelProvider(channelProvider)
                    .build();
        } catch (IOException ioe) {
            LOGGER.error(String.format("IO error creating Google NLP settings: %s", ioe.getMessage()), ioe);
            throw ioe;
        }

        // Create Google API client
        this.languageServiceClient = LanguageServiceClient.create(languageServiceSettings);
    }

    @PreDestroy
    public void destroy() throws Exception {
        if (LOGGER.isDebugEnabled()) {
            LOGGER.debug("Destroying Google NLP API client");
        }
        // Close Google API executors and channels
        this.languageServiceClient.close();
    }
// ...

That way you should be able to use the API with AWS Elastic Beanstalk without having to hack your environment to provide the GOOGLE_APPLICATION_CREDENTIALS environment variable.

An alternative would be google-auth-library-java to look for an environment variable and a system property. Or maybe Amazon AWS add a way to provide environment variables.

Note also that I included the @PostConstruct and @PreDestroy annotated methods. The API will start an executor thread pool, so if you do not want to risk to have problems re-deploying your application, then remember to close your streams.

♥ Open Source

Order of containers in Docker Compose

kinow @ Mar 14, 2017 21:22:03

In Docker Compose you are able to control the startup order of the containers via the depends_on statement. This is documented in Controlling startup order in Compose.

If you have a simple setup, with Tomcat and Postgres, sometimes Postgres will start first, but Compose will initialize Tomcat before Postgres has fully booted. When that happens, you may receive 401, 404, or other application errors.

You can fix it by combining depends_on with a healthcheck. For example:

# File: docker-compose.yml
version: '2.1'
services:
  db:
    container_name: twpg
    build:
      context: .
      dockerfile: Dockerfile.postgres
    restart: always
    ports:
      - "5432:5432"
    healthcheck:
      test: "pg_isready -h localhost -p 5432 -q -U postgres"
      interval: 10s
      timeout: 5s
      retries: 5

  web:
    container_name: twtc
    build:
      context: .
      dockerfile: Dockerfile.tomcat
    restart: always
    depends_on:
      db:
        condition: service_healthy
    links:
      - db
    ports:
      - "80:8080"

In the example docker-compose.yml, there are two containers, db and web. web is running a Tomcat, and db is running Postgres. Web depends on db (see depends_on), and uses a condition service_healthy. Which indicates it depends that that container is healthy.

The healthcheck entry under the db container settings define how to check whether Postgres is running or not. In this case, we are using pg_isready, which is available in the vanilla Postgres 9 container.

It will try 5 times, with a 10 seconds interval, and will time out after 5 seconds. You may have to tune these parameters for your application.

This code snippet is from a pull request submitted to Foxoncz/docker-thingworx.

♥ Open Source

Using AWS MFA without a mobile phone

kinow @ Feb 28, 2017 00:47:03

If you use AWS, the chances are that you use MFA - Multi-factor Authentication - to authenticate. I don’t like to install apps in my mobile phone, unless I need to, so having bought a new phone recently, I decided to find a replacement for Google Authenticator.

There are several command line utilities, browser extensions, libraries, and tools (free and paid) that implement the TOTP - time-based one-time password -, the standard required by Amazon for MFA authentication.

I decided to use a Go tool for the first time: gauth. Note that you won’t be able to use it from home, in case you don’t bring your laptop home. You can have one MFA device linked to your AWS account, so you may have to remove an existing one. Follow these instructions with care :^)

Install Go

sudo apt update && sudo apt upgrade -y
cd tmp/ && wget https://storage.googleapis.com/golang/go1.8.linux-amd64.tar.gz
tar -zxvf go1.*.tar.gz
sudo mv go /usr/local
vim ~/.bashrc

Add the following at the bottom of the file.

GOROOT=/usr/local/go
GOPATH="$HOME/go"
PATH="$GOPATH/bin:$GOROOT/bin:$PATH"

And you can test it with . ~/.bashrc && go version.

Install gauth

Given your environment is correctly set up, you should be able to use the following command to install gauth, and have it available in your $PATH.

go get github.com/pcarrier/gauth

Edit ~/.config/gauth.csv adding a value for the AWS MFA key.

Getting the AWS MFA key

To get the value that you must place in your gauth.csv file, you must add a new MFA device. When asked to scan a QR code, look for an option to enter the manual value. That will give you a long string. That’s the value you are looking for.

Extra: Auto copy-paste from command line

If you would like to quickly copy and paste, try creating an alias as described on this gist.

I used these instructions, and can now run one command line, that will put the next MFA code in my clipboard. Then just paste into my browser, and that’s that!

Happy hacking!

References

Proposed logos for OpenNLP

kinow @ Jan 15, 2017 20:42:03

A couple of logos submitted to OPENNLP-6. Made with Inkscape, fonts from Google Fonts.

Proposed logos for OpenNLP
Proposed logos for OpenNLP

For more, check out my DeviantArt page.