Menu

Tag

Posts tagged with ‘aws’

Using Google Natural Language API in an AWS Elastic Beanstalk application

kinow @ Mar 15, 2017 13:18:03

Besides providing an API for users and developers, Google also provides a series of (very well-written) client implementation, in several programming languages. That’s the case for Google Storage, Google Vision, and for Google Natural Language API’s.

I recently had to use the latter in a POC at work, and ran into an interesting issue with our AWS Elastic Beanstalk environment.

Google Language API requires that clients authenticate before sending requests. If you use their google-cloud-java, you can define the GOOGLE_APPLICATION_CREDENTIALS. This environment variable must point to a JSON file with the Google Language API credentials.

The issue is that while Google Language API (or more precisely google-auth-library-java) looks for an environment variable, in Elastic Beanstalk you are able to specify only system properties (unless you want to try something with ebextensions, maybe some JNI…).

A workaround for this issue in Google Natural Language API, is to create and pass a LanguageServiceSettings to your LanguageServiceClient. This settings object, when created, must be given a channel provider with a FixedCredentialsProvider.

Of course reading code is way easier than reading this workaround description.

// File: GoogleNaturalLanguageService.java
// ...
    // envvar or property used to specify the Google Application Credentials
    private final static String GOOGLE_APPLICATION_CREDENTIALS = "GOOGLE_APPLICATION_CREDENTIALS";

    /**
     * Google Natural Language API.
     */
    private LanguageServiceClient languageServiceClient;

    @PostConstruct
    public void init() throws Exception {
        // Elastic Beanstalk supports Properties, not Environment Variables.
        // Google credentials library will load
        // the JSON location for the service to authenticate from an envVar. So
        // we need to fix that here.
        String googleApplicationCredentials = System.getenv(GOOGLE_APPLICATION_CREDENTIALS);
        LOGGER.info(String.format("GOOGLE_APPLICATION_CREDENTIALS in environment variable: %s", googleApplicationCredentials));
        if (StringUtils.isBlank(googleApplicationCredentials)) {
            googleApplicationCredentials = System.getProperty(GOOGLE_APPLICATION_CREDENTIALS);
            LOGGER.info(String.format("GOOGLE_APPLICATION_CREDENTIALS in JVM property: %s", googleApplicationCredentials));
        }

        if (googleApplicationCredentials == null) {
            throw new RuntimeException("Could not locate GOOGLE_APPLICATION_CREDENTIALS variable!");
        }

        final LanguageServiceSettings languageServiceSettings;
        try (InputStream is = new FileInputStream(new File(googleApplicationCredentials))) {
            final GoogleCredentials myCredentials = GoogleCredentials
                    .fromStream(is)
                    .createScoped(
                            Collections.singleton("https://www.googleapis.com/auth/cloud-platform")
                    );
            final CredentialsProvider credentialsProvider = FixedCredentialsProvider.create(myCredentials);

            final InstantiatingChannelProvider channelProvider = LanguageServiceSettings
                    .defaultChannelProviderBuilder()
                    .setCredentialsProvider(credentialsProvider)
                    .build();
            languageServiceSettings = LanguageServiceSettings
                    .defaultBuilder()
                    .setChannelProvider(channelProvider)
                    .build();
        } catch (IOException ioe) {
            LOGGER.error(String.format("IO error creating Google NLP settings: %s", ioe.getMessage()), ioe);
            throw ioe;
        }

        // Create Google API client
        this.languageServiceClient = LanguageServiceClient.create(languageServiceSettings);
    }

    @PreDestroy
    public void destroy() throws Exception {
        if (LOGGER.isDebugEnabled()) {
            LOGGER.debug("Destroying Google NLP API client");
        }
        // Close Google API executors and channels
        this.languageServiceClient.close();
    }
// ...

That way you should be able to use the API with AWS Elastic Beanstalk without having to hack your environment to provide the GOOGLE_APPLICATION_CREDENTIALS environment variable.

An alternative would be google-auth-library-java to look for an environment variable and a system property. Or maybe Amazon AWS add a way to provide environment variables.

Note also that I included the @PostConstruct and @PreDestroy annotated methods. The API will start an executor thread pool, so if you do not want to risk to have problems re-deploying your application, then remember to close your streams.

♥ Open Source

Using AWS MFA without a mobile phone

kinow @ Feb 28, 2017 00:47:03

If you use AWS, the chances are that you use MFA - Multi-factor Authentication - to authenticate. I don’t like to install apps in my mobile phone, unless I need to, so having bought a new phone recently, I decided to find a replacement for Google Authenticator.

There are several command line utilities, browser extensions, libraries, and tools (free and paid) that implement the TOTP - time-based one-time password -, the standard required by Amazon for MFA authentication.

I decided to use a Go tool for the first time: gauth. Note that you won’t be able to use it from home, in case you don’t bring your laptop home. You can have one MFA device linked to your AWS account, so you may have to remove an existing one. Follow these instructions with care :^)

Install Go

sudo apt update && sudo apt upgrade -y
cd tmp/ && wget https://storage.googleapis.com/golang/go1.8.linux-amd64.tar.gz
tar -zxvf go1.*.tar.gz
sudo mv go /usr/local
vim ~/.bashrc

Add the following at the bottom of the file.

GOROOT=/usr/local/go
GOPATH="$HOME/go"
PATH="$GOPATH/bin:$GOROOT/bin:$PATH"

And you can test it with . ~/.bashrc && go version.

Install gauth

Given your environment is correctly set up, you should be able to use the following command to install gauth, and have it available in your $PATH.

go get github.com/pcarrier/gauth

Edit ~/.config/gauth.csv adding a value for the AWS MFA key.

Getting the AWS MFA key

To get the value that you must place in your gauth.csv file, you must add a new MFA device. When asked to scan a QR code, look for an option to enter the manual value. That will give you a long string. That’s the value you are looking for.

Extra: Auto copy-paste from command line

If you would like to quickly copy and paste, try creating an alias as described on this gist.

I used these instructions, and can now run one command line, that will put the next MFA code in my clipboard. Then just paste into my browser, and that’s that!

Happy hacking!

References

Using the AWS API with Python

kinow @ Oct 04, 2016 21:15:03

Amazon Web Services provides a series of cloud services. When you access the web interface, most - if not all - of the actions you do there are actually translated into API calls.

They also provide SDK’s in several programming languages. With these SDK’s you are able to call the same API used by the web interface. Python has boto (or boto3) which lets you to automate several tasks in AWS.

But before you start using the API, you will need to set up your access key.

It is likely that with time you will have different roles, and may have different permissions with each role. You have to configure your local environment so that you can either use the command line Python utility (installed via pip install awscli) or with boto.

The awscli is a dependency for using boto3. After you install it, you need to run aws configure. It will create the ~/.aws/config and ~/.aws/credentials files. You can tweak these files to support multiple roles.

I followed the tutorials, but got all sorts of different issues. Then after debugging some locally installed dependencies, in special awscli files, I found that the following settings work for my environment.

# File: config
[profile default]
region = ap-southeast-2

[profile profile1]
region = ap-southeast-2
source_profile = default
role_arn = arn:aws:iam::123:role/Developer

[profile profile2]
region = ap-southeast-2
source_profile = default
role_arn = arn:aws:iam::456:role/Sysops
mfa_serial = arn:aws:iam::789:mfa/user@domain.blabla

and

# File: credentials
[default]
aws_access_key_id = YOU_KEY_ID
aws_secret_access_key = YOUR_SECRET

And once it is done you can, for example, confirm it is working with some S3 commands in Python.

#!/usr/bin/env python3

import os
import boto3

session = boto3.Session(profile_name='profile2')
s3 = session.resource('s3')

found = False

name = 'mysuperduperbucket'

for bucket in s3.buckets.all():
    if bucket.name == name:
        found = True

if not found:
    print("Creating bucket...")
    s3.create_bucket(Bucket=name)

file_location = os.path.dirname(os.path.realpath(__file__)) + os.path.sep + 'samplefile.txt'
s3.meta.client.upload_file(Filename=file_location, Bucket=name, Key='book.txt')

The AWS files in this example are using MFA too, the multi-factor authentication. So the first time you run this code you may be asked to generate a token, which will be cached for a short time.

That’s it for today.

Happy hacking!