Menu

Tag

Posts tagged with ‘jenkins’

Troubleshooting a Jenkins Plug-in compatibility issue

kinow @ Apr 17, 2017 21:01:03

This post is probably different from others. I will give a TL;DR, but will then give you a copy of a comment of a Jenkins JIRA issue. Hope you have fun reading it, specially if you maintain Jenkins servers or plug-ins.

TL;DR: there was an issue in Jenkins Job DSL Plug-in, that caused jobs created to have an invalid script. The fix had not been released, but was already in the master branch in GitHub.

Well, that was fun bug reproduced, I believe I know why that’s happening. Not so complicated to fix… but no fast way to fix it. Here’s the issue analysis (grab a coffee to read it).

  • Downloaded jenkins.war (2.32.3.war)
  • mkdir /tmp/123
  • JENKINS_HOME=/tmp/123 java -jar jenkins.war
  • Entered secret into form and submitted
  • Installed suggested plugins (boy that takes a while)
  • Created temp user
  • Installed (without restart) active-choices-plugin 1.5.3
  • Installed (without restart) job-dsl-plugin
  • Manually stopped Jenkins, and started it again with same command #3
  • Log in with user, all looking good
  • Created Freestyle job JENKINS-42655 (see attached config.xml)
  • Executed job, and found new job JENKINS-42655-1
  • Never opened the job configuration, clicked on the “Generated Items link to JENKINS-42655-1” to open in a new tab
  • Clicked on Build with Parameters
  • Looked at logs, and noticed the security-script-plugin exceptions
  • Went to “Manage Jenkins” / “In-process Script Approval” and approved scripts
  • Went back to the JENKINS-42655-1 build with parameters screen, and everything worked as expected

*Hummm. Issue more or less reproduced. Let’s investigate more.

  • Restarted Jenkins again
  • Changed the JENKINS-42655 seed job configuration to use a different script
  • Copied the config.xml file to another location
  • Went to build with parameters, and now it was broken again
  • Saved the job manually
  • Copied the config.xml file to yet another location
  • Went to build with parameters, and now it worked as reported in this issue

Now comes the interesting part. Looking at the diff. Attaching a screen shot so that others can have fun looking at it too. I installed Kompare as it has some cool features such as disabling diff for white spaces, blank lines, etc. The whole file changes as you save it. But if you ignore the number of white spaces… Then you can see that the Job DSL Plug-in is creating a <script> tag, as we used to do before 1.5 I think.

Screen shot
Screen shot

Now we use the script-security-plugin. So we need to wrap that around the script-security-plugin’s tags. Will report an issue for job-dsl-plugin, and will probably submit a pull request in the next days too. There’s not much left we can change in the plug-in code for that Piotr Tempes, so I’m afraid you will have to:

  • keep saving the job
  • perhaps work on the fix for Job DSL if you feel like doing it (as you could probably be faster than me in submitting the PR)
  • use an older version of the active-choices-plugin that doesn’t use security-script-plugin, though you could be bitten by other old bugs
  • write some script to replace the <script> tag and wrap it by the secureScript (doing what the pull request will do automatically later)

Sorry for not being able to quickly provide any workaround, nor to cut a quick bugfix release.

Cheers Bruno

ps: When I started getting involved in Open Source, I always felt extremely happy when people would spend their time troubleshooting my issues, or just educating me on how to behave in an Open Source community, or even how I should have troubleshooted initially the issue myself. So whenever I have time, I try to write detailed and polite replies. The person posting the comment is my past. The person posting the comment is my future :-) Kindness begets kindness.

♥ Open Source

PermGen errors and java.lang.ClassCastException: com.sun.crypto.provider.AESCipher cannot be cast to javax.crypto.CipherSpi running Jenkins plug-in tests with PowerMock

kinow @ Dec 19, 2016 19:20:03

Recently while working on a Jenkins plugin some tests were failing with PermGen errors. Even though it worked in my notebook at home (with Java 8, thus no Permgen), it failed in a CloudBees hosted Jenkins job, and also on my Mac (with Java 7) at work.

As I could not change the settings in the CloudBees hosted Jenkins, I decided to spent some time investigating why these tests would require so much memory. Then I found this blog post about PowerMock.

I was not using the @PrepareForTest annotation, but after adding it the issue was gone. So I assume it either prevents PowerMock from trying to dynamically load several classes, or instructs it to unload classes after the tests. But in anyway after adding it the issue was gone.

Then I got the following exception.

WARNING: Failed to instantiate Key[type=hudson.security.csrf.DefaultCrumbIssuer$DescriptorImpl, annotation=[none]]; skipping this component
com.google.inject.ProvisionException: Guice provision errors:

1) Error injecting constructor, java.lang.ClassCastException: com.sun.crypto.provider.AESCipher cannot be cast to javax.crypto.CipherSpi
  at hudson.security.csrf.DefaultCrumbIssuer$DescriptorImpl.<init>(DefaultCrumbIssuer.java:127)

1 error
    at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:52)

And then thanks to this issue I understood that PowerMock was mocking javax.crypto classes. Turns out it is quite easy to tell PowerMock to ignore certain classes from being mocked, with the @PowerMockIgnore annotation.

// snip
@RunWith(PowerMockRunner.class)
@PowerMockIgnore({"javax.crypto.*" })
public class TestAbstractUnoChoiceParameter {
//
}
// snip

Added a couple of notes to this Jenkins Wiki page so that users facing similar issues can try these possible workarounds.

Hope that helps someone!

Performance problems in Jenkins TAP Plug-in — part 1

kinow @ Sep 03, 2016 23:28:03

JENKINS-17887 reports performance problems in the Jenkins TAP Plug-in. It also lists a series of suggestions on how to improve the Jenkins TAP Plug-in performance. On this initial post, we will get a general idea of how the plug-in performs for large projects.

BioPerl has over 21K tests. That should be enough for giving an initial idea of CPU, memory and disk usage for the plug-in.

git clone https://github.com/bioperl/bioperl-live.git
cd bioperl-live
sudo cpanm  -vv --installdeps --notest .
sudo cpanm Set::Scalar Graph::Directed XML::LibXML XML::SAX \
    SVG XML::Parser::PerlSAX Convert::Binary::C XML::SAX::Writer \
    XML::DOM::XPath Spreadsheet::ParseExcel XML::SAX::Writer \
    XML::DOM HTML::TableExtract XML::Simple Test::Pod DBI
prove -r t/ -a tests.tar.gz

All tests successful.
Files=325, Tests=21095, 94 wallclock secs ( 2.47 usr  0.55 sys + 88.29 cusr  3.85 csys = 95.16 CPU)
Result: PASS

When the test results are parsed, the plug-in also copies TAP files over to the master, in a folder called tap-master-files.

The BioPerl tests are not really big, just 1.7M. It gets doubled as there will be the workspace copy, and the tap-master-files directory copy, so 3.4M.

But several objects get created in memory, and persisted into the build.xml job file. BioPerl generates a build.xml file with 11M. So less than 15M. But the build.xml contains objects that are read via XStream by Jenkins and into the memory.

The build page with the graph, and the other two test result pages are rendering in more than 10 seconds in my computer. But the CPU load is OK, so a closer look at the memory use would probably be more interesting.

JENKINS-17887 YourKit profiler

The image shows one of the screens in YourKit profiler, where it is possible to see that org.tap4j.plugin.model.TapTestResultResult has over 6 million objects.

One build.xml for the BioPerl project gets over 80K entries for the TestResult object.

grep "org.tap4j.model.TestResult" builds/1/build.xml -o | wc -l
84522

This happens because each TAP file may contain multiple test results (lines with test results). Each of these test results gets turned into a Java object and loaded by the plug-in. So when loading the test result pages, Jenkins needs to wait until all these objects have been parsed, deserialized and read into the memory.

The next post will continue on code improvements, and another benchmark.

Happy profiling!

Using Active Choices with Role Strategy Plug-in

kinow @ Apr 24, 2016 16:05:03

Having worked in Open Source for a few years, one of my favorite things is when you can share experience with other people that you meet. Andrew Gray has worked with .NET and Jenkins for years, and we met through Open Source. He has helped me in the past with Jenkins and .NET, and also maintains the blog Jenkins.NET.

A couple of days ago he sent me an interesting question. He asked me if that would be possible to use Active Choices Plug-in with the Role Strategy Plug-in. This plug-in lets you define roles, define which permissions a role has, and then assign users to the roles.

The idea was to show a set of parameters that would vary according to the user roles. So, say that the user is a tester. Then s/he would see certain parameters. But if s/he was a manager, other parameters would be displayed.

To the code

import hudson.model.User
import hudson.model.Hudson
import hudson.security.AuthorizationStrategy
import hudson.security.Permission
import com.michelin.cio.hudson.plugins.rolestrategy.RoleBasedAuthorizationStrategy
import com.michelin.cio.hudson.plugins.rolestrategy.RoleMap

AuthorizationStrategy strategy = Hudson.getInstance().getAuthorizationStrategy();

jobs = []
user = User.current()
userId = user.getId()

if (strategy != null && strategy instanceof com.michelin.cio.hudson.plugins.rolestrategy.RoleBasedAuthorizationStrategy) {
    roleStrategy = (RoleBasedAuthorizationStrategy) strategy;
    // not very straightforward to get the groups for a given user
    roles = roleStrategy.getGrantedRoles("globalRoles")
    for (entry in roles) {
        role = entry.key
        users = entry.value
        if (role.getName().equals("tester")) {
            if (userId in users) {
                jobs = ["PROJECT_FOR_TESTERS1", "PROJECT_FOR_TESTERS2"]
                break
            }
        } else if (role.getName().equals("admin")) {
            if (userId in users) {
                jobs = ["PROJECT_FOR_ADMINS1", "PROJECT_FOR_ADMINS2"]
                break
            }
        }
    }
}

return jobs
// TODO: handle anonymous user ;-)

And now let’s dissect the code.

import hudson.model.User
import hudson.model.Hudson
import hudson.security.AuthorizationStrategy
import hudson.security.Permission
import com.michelin.cio.hudson.plugins.rolestrategy.RoleBasedAuthorizationStrategy
import com.michelin.cio.hudson.plugins.rolestrategy.RoleMap

You start by importing the classes that you need.

AuthorizationStrategy strategy = Hudson.getInstance().getAuthorizationStrategy();

jobs = []
user = User.current()
userId = user.getId()

The first line gets the current AuthorizationStrategy used in Jenkins.

Then we create an empty array of jobs, which is the value returned by default. And get the current logged in user ID.

if (strategy != null && strategy instanceof com.michelin.cio.hudson.plugins.rolestrategy.RoleBasedAuthorizationStrategy) {
    roleStrategy = (RoleBasedAuthorizationStrategy) strategy;
    // not very straightforward to get the groups for a given user
    roles = roleStrategy.getGrantedRoles("globalRoles")
    for (entry in roles) {
        role = entry.key
        users = entry.value
        if (role.getName().equals("tester")) {
            if (userId in users) {
                jobs = ["PROJECT_FOR_TESTERS1", "PROJECT_FOR_TESTERS2"]
                break
            }
        } else if (role.getName().equals("admin")) {
            if (userId in users) {
                jobs = ["PROJECT_FOR_ADMINS1", "PROJECT_FOR_ADMINS2"]
                break
            }
        }
    }
}

return jobs
// TODO: handle anonymous user ;-)

The final part simply iterates through each existing role, and then through the users in that role. I could not find a more elegant way of doing that, but in case you would like to maybe optimize the code, here are the main classes that I used from Jenkins to write the script.

Normally I either use my local working copy of Jenkins project imported in Eclipse, or use the “Find File” feature in GitHub. Then remember a little bit about Jenkins Java API to walk through the classes and write the Groovy code (I made a video about writing Groovy with Auto Complete in Eclipse, but that’s in Portuguese only).

What the result looks like in Jenkins

In this section we have just simple screenshots, showing the script in the job configuration.

Screenshot 01
Screenshot 01

And the resulting screen. The parameter DeployAPP will be available during the build, and can be used to trigger other jobs or pipelines.

Screenshot 02
Screenshot 02

Where can you find this script?

This blog post has a copy of the script, but I have also submitted a pull request to a repository maintained by another friend, Ioannis Moutsatsos. Ioannis maintains a Jenkins installation in Novartis, and is probably the most skillful user of the Active Choices Plug-in.

Another advantage of this other copy, is that it may be updated with time, in case there are bugs or improvements. So watch Ioannis’ repository for updates!

ps: I’ve also met Ioannis through Open Source code, and have learned a lot by working with him.

♥ Open Source!

Trying SaltStack with Docker

kinow @ Apr 17, 2016 22:06:03

Some weeks ago I started learning SaltStack for a project at work. But I couldn’t find a good Docker image for that and I had to ask the Ops team for some VM’s. We are having a rainy weekend in Auckland, so I decided to have another look at the Jenkins SaltStack Plug-in.

But now since I was at home, I couldn’t use the VM’s that I had access to at work. So decided to look again at Docker or Vagrant images. After playing with a few images, I found bbinet/salt-master. It not only sets up a master, but also provides an easy way to enable the cherrypy API (necessary for the Jenkins plug-in).

This post describes the steps that I took to have a running Salt Master with the API enabled. First you need to create some directories and files to use with the image.

shell$ mkdir ~/master && cd ~/master
shell$ mkdir -p config/master.d/
shell$ vim config/master.d/api.conf

The api.conf contains the SaltStack API configuration. You can change port, user and other settings if necessary. Just remember to add a credential in Jenkins for the plug-in.

# File: api.conf
external_auth:
  pam:
    saltapiuser:
      - .*
      - '@runner'
      - '@wheel'
      - '@jobs'
rest_cherrypy:
  port: 8000
  host: 0.0.0.0
  disable_ssl: True
  static: /opt/molten
  static_path: /assets
  app: /opt/molten/index.html
  app_path: /molten

The image also conveniently provides a script that is executed before the entry point (if provided). So we can also create a user for the API automatically when the image is created.

shell$ vim config/before-exec.sh
#!/bin/bash
# File: before-exec.sh
useradd saltapiuser
echo -e "nosecret\nnosecret\n" | passwd saltapiuser
exit 0

Also make the script executable.

chmod +x config/before-exec.sh

And finally start the container.

docker run --name salt-master -v $PWD/config:/config \
    -p 4505:4505 -p 4506:4506 -p 443:443 -p 8000:8000 \
    bbinet/salt-master

Once the container is running, you can go to http://localhost:8000 and log in as saltapiuser:nosecret, and also configure your plug-in in Jenkins.

Happy hacking!