Menu

Posts tagged with ‘jenkins’

PermGen errors and java.lang.ClassCastException: com.sun.crypto.provider.AESCipher cannot be cast to javax.crypto.CipherSpi running Jenkins plug-in tests with PowerMock

kinow @ Dec 19, 2016 19:20:03

Recently while working on a Jenkins plugin some tests were failing with PermGen errors. Even though it worked in my notebook at home (with Java 8, thus no Permgen), it failed in a CloudBees hosted Jenkins job, and also on my Mac (with Java 7) at work.

As I could not change the settings in the CloudBees hosted Jenkins, I decided to spent some time investigating why these tests would require so much memory. Then I found this blog post about PowerMock.

I was not using the @PrepareForTest annotation, but after adding it the issue was gone. So I assume it either prevents PowerMock from trying to dynamically load several classes, or instructs it to unload classes after the tests. But in anyway after adding it the issue was gone.

Then I got the following exception.

WARNING: Failed to instantiate Key[type=hudson.security.csrf.DefaultCrumbIssuer$DescriptorImpl, annotation=[none]]; skipping this component
com.google.inject.ProvisionException: Guice provision errors:

1) Error injecting constructor, java.lang.ClassCastException: com.sun.crypto.provider.AESCipher cannot be cast to javax.crypto.CipherSpi
  at hudson.security.csrf.DefaultCrumbIssuer$DescriptorImpl.<init>(DefaultCrumbIssuer.java:127)

1 error
    at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:52)

And then thanks to this issue I understood that PowerMock was mocking javax.crypto classes. Turns out it is quite easy to tell PowerMock to ignore certain classes from being mocked, with the @PowerMockIgnore annotation.

// snip
@RunWith(PowerMockRunner.class)
@PowerMockIgnore({"javax.crypto.*" })
public class TestAbstractUnoChoiceParameter {
//
}
// snip

Added a couple of notes to this Jenkins Wiki page so that users facing similar issues can try these possible workarounds.

Hope that helps someone!

<p class="postmetadata">

    <small>tags 
     [&nbsp;<a
        href="/tag/jenkins"
        rel="tag">jenkins</a>&nbsp;]&nbsp;

     [&nbsp;<a
        href="/tag/software%20quality"
        rel="tag">software quality</a>&nbsp;]&nbsp;

    </small>


    <br/>
    <small>posted in 
    [&nbsp;<a
        href="/blog"
        title="View all posts in blog"
        rel="category tag">blog</a>&nbsp;]&nbsp;
    category </small>

</p>

Performance problems in Jenkins TAP Plug-in — part 1

kinow @ Sep 03, 2016 23:28:03

JENKINS-17887 reports performance problems in the Jenkins TAP Plug-in. It also lists a series of suggestions on how to improve the Jenkins TAP Plug-in performance. On this initial post, we will get a general idea of how the plug-in performs for large projects.

BioPerl has over 21K tests. That should be enough for giving an initial idea of CPU, memory and disk usage for the plug-in.

git clone https://github.com/bioperl/bioperl-live.git
cd bioperl-live
sudo cpanm  -vv --installdeps --notest .
sudo cpanm Set::Scalar Graph::Directed XML::LibXML XML::SAX \
    SVG XML::Parser::PerlSAX Convert::Binary::C XML::SAX::Writer \
    XML::DOM::XPath Spreadsheet::ParseExcel XML::SAX::Writer \
    XML::DOM HTML::TableExtract XML::Simple Test::Pod DBI
prove -r t/ -a tests.tar.gz

All tests successful.
Files=325, Tests=21095, 94 wallclock secs ( 2.47 usr  0.55 sys + 88.29 cusr  3.85 csys = 95.16 CPU)
Result: PASS

When the test results are parsed, the plug-in also copies TAP files over to the master, in a folder called tap-master-files.

The BioPerl tests are not really big, just 1.7M. It gets doubled as there will be the workspace copy, and the tap-master-files directory copy, so 3.4M.

But several objects get created in memory, and persisted into the build.xml job file. BioPerl generates a build.xml file with 11M. So less than 15M. But the build.xml contains objects that are read via XStream by Jenkins and into the memory.

The build page with the graph, and the other two test result pages are rendering in more than 10 seconds in my computer. But the CPU load is OK, so a closer look at the memory use would probably be more interesting.

JENKINS-17887 YourKit profiler

The image shows one of the screens in YourKit profiler, where it is possible to see that org.tap4j.plugin.model.TapTestResultResult has over 6 million objects.

One build.xml for the BioPerl project gets over 80K entries for the TestResult object.

grep "org.tap4j.model.TestResult" builds/1/build.xml -o | wc -l
84522

This happens because each TAP file may contain multiple test results (lines with test results). Each of these test results gets turned into a Java object and loaded by the plug-in. So when loading the test result pages, Jenkins needs to wait until all these objects have been parsed, deserialized and read into the memory.

The next post will continue on code improvements, and another benchmark.

Happy profiling!

<p class="postmetadata">

    <small>tags 
     [&nbsp;<a
        href="/tag/programming"
        rel="tag">programming</a>&nbsp;]&nbsp;

     [&nbsp;<a
        href="/tag/jenkins"
        rel="tag">jenkins</a>&nbsp;]&nbsp;

     [&nbsp;<a
        href="/tag/software%20quality"
        rel="tag">software quality</a>&nbsp;]&nbsp;

    </small>


    <br/>
    <small>posted in 
    [&nbsp;<a
        href="/blog"
        title="View all posts in blog"
        rel="category tag">blog</a>&nbsp;]&nbsp;
    category </small>

</p>

Using Active Choices with Role Strategy Plug-in

kinow @ Apr 24, 2016 16:05:03

Having worked in Open Source for a few years, one of my favorite things is when you can share experience with other people that you meet. Andrew Gray has worked with .NET and Jenkins for years, and we met through Open Source. He has helped me in the past with Jenkins and .NET, and also maintains the blog Jenkins.NET.

A couple of days ago he sent me an interesting question. He asked me if that would be possible to use Active Choices Plug-in with the Role Strategy Plug-in. This plug-in lets you define roles, define which permissions a role has, and then assign users to the roles.

The idea was to show a set of parameters that would vary according to the user roles. So, say that the user is a tester. Then s/he would see certain parameters. But if s/he was a manager, other parameters would be displayed.

To the code

import hudson.model.User
import hudson.model.Hudson
import hudson.security.AuthorizationStrategy
import hudson.security.Permission
import com.michelin.cio.hudson.plugins.rolestrategy.RoleBasedAuthorizationStrategy
import com.michelin.cio.hudson.plugins.rolestrategy.RoleMap

AuthorizationStrategy strategy = Hudson.getInstance().getAuthorizationStrategy();

jobs = []
user = User.current()
userId = user.getId()

if (strategy != null 
    // not very straightforward to get the groups for a given user
    roles = roleStrategy.getGrantedRoles("globalRoles")
    for (entry in roles) {
        role = entry.key
        users = entry.value
        if (role.getName().equals("tester")) {
            if (userId in users) {
                jobs = ["PROJECT_FOR_TESTERS1", "PROJECT_FOR_TESTERS2"]
                break
            }
        } else if (role.getName().equals("admin")) {
            if (userId in users) {
                jobs = ["PROJECT_FOR_ADMINS1", "PROJECT_FOR_ADMINS2"]
                break
            }
        }
    }
}

return jobs
// TODO: handle anonymous user ;-)

And now let’s dissect the code.

import hudson.model.User
import hudson.model.Hudson
import hudson.security.AuthorizationStrategy
import hudson.security.Permission
import com.michelin.cio.hudson.plugins.rolestrategy.RoleBasedAuthorizationStrategy
import com.michelin.cio.hudson.plugins.rolestrategy.RoleMap

You start by importing the classes that you need.

AuthorizationStrategy strategy = Hudson.getInstance().getAuthorizationStrategy();

jobs = []
user = User.current()
userId = user.getId()

The first line gets the current AuthorizationStrategy used in Jenkins.

Then we create an empty array of jobs, which is the value returned by default. And get the current logged in user ID.

if (strategy != null 
    // not very straightforward to get the groups for a given user
    roles = roleStrategy.getGrantedRoles("globalRoles")
    for (entry in roles) {
        role = entry.key
        users = entry.value
        if (role.getName().equals("tester")) {
            if (userId in users) {
                jobs = ["PROJECT_FOR_TESTERS1", "PROJECT_FOR_TESTERS2"]
                break
            }
        } else if (role.getName().equals("admin")) {
            if (userId in users) {
                jobs = ["PROJECT_FOR_ADMINS1", "PROJECT_FOR_ADMINS2"]
                break
            }
        }
    }
}

return jobs
// TODO: handle anonymous user ;-)

The final part simply iterates through each existing role, and then through the users in that role. I could not find a more ellegant way of doing that, but in case you would like to maybe optimize the code, here are the main classes that I used from Jenkins to write the script.

Normally I either use my local working copy of Jenkins project imported in Eclipse, or use the “Find File” feature in GitHub. Then remember a little bit about Jenkins Java API to walk through the classes and write the Groovy code (I made a video about writing Groovy with Auto Complete in Eclipse, but that’s in Portuguese only).

What the result looks like in Jenkins

In this section we have just simple screenshots, showing the script in the job configuration.

Screenshot 01
Screenshot 01

And the resulting screen. The parameter DeployAPP will be available during the build, and can be used to trigger other jobs or pipelines.

Screenshot 02
Screenshot 02

Where can you find this script?

This blog post has a copy of the script, but I have also submitted a pull request to a repository maintained by another friend, Ioannis Moutsatsos. Ioannis maintains a Jenkins installation in Novartis, and is probably the most skillful user of the Active Choices Plug-in.

Another advantage of this other copy, is that it may be updated with time, in case there are bugs or improvements. So watch Ioannis’ repository for updates!

ps: I’ve also met Ioannis through Open Source code, and have learned a lot by working with him.

♥ Open Source!

<p class="postmetadata">

    <small>tags 
     [&nbsp;<a
        href="/tag/jenkins"
        rel="tag">jenkins</a>&nbsp;]&nbsp;

    </small>


    <br/>
    <small>posted in 
    [&nbsp;<a
        href="/blog"
        title="View all posts in blog"
        rel="category tag">blog</a>&nbsp;]&nbsp;
    category </small>

</p>

Trying SaltStack with Docker

kinow @ Apr 17, 2016 22:06:03

Some weeks ago I started learning SaltStack for a project at work. But I couldn’t find a good Docker image for that and I had to ask the Ops team for some VM’s. We are having a rainy weekend in Auckland, so I decided to have another look at the Jenkins SaltStack Plug-in.

But now since I was at home, I couldn’t use the VM’s that I had access to at work. So decided to look again at Docker or Vagrant images. After playing with a few images, I found bbinet/salt-master. It not only sets up a master, but also provides an easy way to enable the cherrypy API (necessary for the Jenkins plug-in).

This post describes the steps that I took to have a running Salt Master with the API enabled. First you need to create some directories and files to use with the image.

shell$ mkdir ~/master && cd ~/master
shell$ mkdir -p config/master.d/
shell$ vim config/master.d/api.conf

The api.conf contains the SaltStack API configuration. You can change port, user and other settings if necessary. Just remember to add a credential in Jenkins for the plug-in.

# File: api.conf
external_auth:
  pam:
    saltapiuser:
      - .*
      - '@runner'
      - '@wheel'
      - '@jobs'
rest_cherrypy:
  port: 8000
  host: 0.0.0.0
  disable_ssl: True
  static: /opt/molten
  static_path: /assets
  app: /opt/molten/index.html
  app_path: /molten

The image also conveniently provides a script that is executed before the entry point (if provided). So we can also create a user for the API automatically when the image is created.

shell$ vim config/before-exec.sh
#!/bin/bash
# File: before-exec.sh
useradd saltapiuser
echo -e "nosecret\nnosecret\n" | passwd saltapiuser
exit 0

Also make the script executable.

chmod +x config/before-exec.sh

And finally start the container.

docker run --name salt-master -v $PWD/config:/config \
    -p 4505:4505 -p 4506:4506 -p 443:443 -p 8000:8000 \
    bbinet/salt-master

Once the container is running, you can go to http://localhost:8000 and log in as saltapiuser:nosecret, and also configure your plug-in in Jenkins.

Happy hacking!

Deploying WAR files to Tomcat with Jenkins

kinow @ Mar 20, 2016 15:29:03

Table of Contents

A co-worker asked me this week about how to deploy a WAR file to Tomcat with Jenkins. In my team we are currently maintaining and deploying about 10 Java web systems, but we have no consistent way of deploying the applications to Tomcat yet. In the past I used Ant, Maven, Cargo, Grunt, and Jenkins, so I decided to write this short post to show a few different ways it can be achieved, à la Perl’s TMTOWTDI motto.

#1 Deploying with custom scripts

At first you may be tempted to write your own script to deploy to Tomcat with some Shell, Perl, Python or Java. But I think I would choose this option only because either I needed some feature that is not available in the other options, or in order to call other tasks or debug some problem.

Example:

$ docker run -d -p 8888:8080 jeanblanchard/tomcat:8
$ git clone https://github.com/spring-projects/spring-petclinic.git && cd spring-petclinic && mvn package
$ curl --upload-file target/petclinic.war "http://admin:admin@localhost:8888/manager/text/deploy?path=/spring-petclinic&update=true"
OK - Deployed application at context path /spring-petclinic

( Read more … )