Jenkins Configuration

Summary

Mantid uses the Jenkins automation server to support the continuous integration requirements of Mantid. This document aims to describe the general setup of the system.

Introduction

Jenkins works on a ‘master->slave’ principle. The master node is responsible for orchestrating jobs and managing the slave nodes, where the work is actually performed. The master node is located at http://builds.mantidproject.org and each facility is responsible for providing hardware to act as slaves for the various required configurations.

General Setup

The master node is set to a fixed TCP port for JNLP slave agents under http://builds.mantidproject.org/configureSecurity.

The anonymous jenkins user has the following rights: Overall Read, Slave Connect, Job Read, View Read.

Setting up a New Slave

Machine Setup

Set up a local builder account that will be used by the slave.

Install the required prerequisites for the relevant OS.

Note

For Windows the Command line Visual C++ build tools may be used in place of a full Visual Studio install from version 2017 onwards (the 2015 tools contain a broken vcvarsall.bat).

Windows

  • Ensure that the location of msbuild.exe (C:\Windows\Microsoft.NET\Framework64\v4.0.30319) is on the PATH

Slave Connection

There are following additional steps required to be able to connect a Windows slave using JNLP as a Windows service :

  1. Setup the slave on the master node using “New Node” under http://builds.mantidproject.org/computer/. If “New Node” is not visible then you do not have the required permissions - ask an admin for help. It is recommended that you copy from an existing node of a similar type.
  2. Once configured on the master, remote desktop to the slave, open a browser and connect to the webpage of the slave, .e.g. http://builds.mantidproject.org/computer/ornl-pc73896/
  3. Click on the connect button to launch the JNLP client.
  4. Once the client is launched, you can select “Install as Windows Service” from the clients menu. If you have a proxy then see the section below for further configuration steps.
  5. Once installed change the recovery behaviour for the service. Do this by right-clicking on the Jenkins service->Properties->Recovery. Change the first, second, subsequent failures to restart the service.
  6. Change the account that the service uses to log on by right-clicking on the Jenkins service->Properties->Log On and enter the details in the builder account
  7. Change the “Startup type:” of the service to be “Automatic (Delayed Start)”
  8. Ensure the msbuild directory is on the PATH
  9. Finally reboot the slave (this is the easiest way for the Jenkins to start trying to reconnect to it)

Note that changes to PATH require restarting the Jenkins service in order to be reflected in the build environment.

Connecting Through a Proxy Server

It is a little more tricky to windows slaves connected through a proxy.To do this you must modify the java arguments that are used to start the jenkins-slave process. Once the “Install as a Windows Service” has completed you should

  1. Find a directory on the machine such as C:\Jenkins` or whatever was configured in the slave config.
  2. Open the jenkins-slave.xml file
  3. Edit the tag and add -Dhttp.proxyHost=PROXYHOST -Dhttp.proxyPort=PROXYPORT to the list
  4. Save the file and restart the service (or machine)

Linux

Install an ssh server.

Install ccache. After installing run ccache --max-size=20G from the builder account.

Install a vnc server and from the builder account run vncpasswd to set a password on the VNC server. It can be any password.

Ensure curl is installed

Any machines acting as performance test servers will require mysqldb to be installed.

Ubuntu

Configure automatic security updates.

Install gdebi-core package to allow installing .deb files.

The builder account must be setup to be able to run gdebi non-interactively. Use visudo to add the following exception got builder:

# Allow no password for gdebi
builder       ALL=(ALL)NOPASSWD:/usr/bin/gdebi, /usr/bin/dpkg
## Disable tty requirement for gdebi and dpkg command
Defaults!/usr/bin/gdebi !requiretty
Defaults!/usr/bin/dpkg  !requiretty

Red Hat

The builder account must be setup to be able to run yum non-interactively. Use visudo to add the following exception got builder:

## Allow no password for yum
builder       ALL = NOPASSWD: /usr/bin/yum,/bin/rpm
## Disable tty requirement for yum command
Defaults!/bin/rpm       !requiretty
Defaults!/usr/bin/yum       !requiretty

Mac OS

Enable SSH (“Remote Login”) and VNC (“Remote Management”). If you have connection issues from a non-OS X client then try adjusting your color depth settings (True Color 32bpp works on Remmina).

Install cppcheck from brew.

The builder account must be setup to be able to run gdebi non-interactively. Use visudo to add the following exception got builder:

# Allow builder to install packages without a password
builder  ALL=(ALL)NOPASSWD:/usr/sbin/installer, /bin/rm
# Disable tty requirement
Defaults!/usr/sbin/installer    !requiretty
Defaults!/bin/rm        !requiretty

In order to run the MantidPlot tests, which require a connection to the windowing system, the user that is running the jenkins slave must have logged in. This is most easily done by VNC - connect, log in, then disconnect. If you see errors such as:

_RegisterApplication(), FAILED TO establish the default connection to the WindowServer,
_CGSDefaultConnection() is NULL.

then no one is logged in to the system.

Linux/Mac Connection Notes

The jenkins JNLP connections are maintained by a crontab entry. The script is in the mantid repository.

The comments at the top describe a typical crontab entry for the script. This needs to be manually set for each slave. Ensure the script is marked executable after downloading it. Also ensure the entry in the crontab has the correct PATH setting (by default cron uses a reduced PATH entry). On macOS latex and sysctl should be available.

Post-Connection Setup - All Systems

Ensure the new machine is added to the relevant ParaView build job and build ParaView. Set the PARAVIEW_DIR & PARAVIEW_NEXT_DIR variables (it’s easiest to just look at the configuration for one of the other nodes of a similar type.

Misc Groovy Scripts

The following is a collection of groovy scripts that can be run either at http://builds.mantidproject.org/script (for master node) or on a given node, e.g isis-mantidx3. You must have admin privileges to run them.

https://github.com/jenkinsci/jenkins-scripts/tree/master/scriptler was helpful for coming up with some of these.

Update ParaView variables on nodes

After running this script the variables look like they are updated but are in fact cached on the slaves so the new values don’t take effect without disconnecting and forcing each slave to reconnect

import jenkins.model.*
import hudson.model.*
import hudson.slaves.*

VARIABLE_NAME = "PARAVIEW_NEXT_DIR"
VERSION = "ParaView-5.1.2"

jenkins = Jenkins.instance
nodes = jenkins.getNodes()
println("Displaying values of " + VARIABLE_NAME + " on all nodes")
println()
for(node in nodes) {
  node_props = node.nodeProperties.getAll(hudson.slaves.EnvironmentVariablesNodeProperty.class)
  if(node_props.size() == 1) {
  env_vars = node_props[0].getEnvVars()
  if(env_vars.containsKey(VARIABLE_NAME)) {
    def pv_dir = node.createPath(env_vars.get(VARIABLE_NAME, ""));
    if(pv_dir) {
      def pv_build_dir = pv_dir.getParent();
      def pv_dir_new = pv_build_dir.child(VERSION);
      println(node.getDisplayName() + ": Updating $VARIABLE_NAME from '" + pv_dir.toString() + "' to '" + pv_dir_new.toString() + "'");
      env_vars.put(VARIABLE_NAME, pv_dir_new.toString());
    }
    else {
      println(node.getDisplayName() + " has variable set but " + env_vars.get(VARIABLE_NAME, "") + " does not exist");
    }
  } else {
    println(node.getDisplayName() + ": $VARIABLE_NAME " +  "not set.")
  }
  } else {
    println(node.getDisplayName() + ": $VARIABLE_NAME " +  "not set.")
  }
}
jenkins.save();

Check existence of ParaView builds

import hudson.model.*

nodes = Jenkins.instance.slaves

PV_VERSION = "5.1.2"

for (node in nodes) {
  FilePath root = node.getRootPath();
  if(root) {
    FilePath fp = root.getParent();
    // assume this is $HOME on osx/linux & drive: on Windows
    if(fp.toString().startsWith("C:")) {
      fp = fp.child("Builds")
    } else {
      fp = fp.child("build");
    }
    fp = fp.child("ParaView-$PV_VERSION");
    if(!fp.exists()) {
      println(node.getDisplayName() + " does not have PV 5.1.2")
    }
  }
}

Remove directories across multiple nodes

It is advised to ensure nothing is running and pause the build queue.

Master Incremental

import hudson.model.*

nodes = Jenkins.instance.slaves

JOBNAME = "master_incremental"


for (node in nodes) {
  labels = ["osx-10.10-build", "rhel6-build", "rhel7-build", "ubuntu-14.04-build", "ubuntu-16.04-build", "win7"];
  for (nodeLabel in labels) {
    FilePath fp = node.createPath(node.getRootPath().toString() + File.separator + "workspace" + File.separator + JOBNAME + File.separator + "label" + File.separator + nodeLabel + File.separator + "build");
    if(fp!=null && fp.exists()) {
      println(fp.toString())
      fp.deleteRecursive()
    }
  }
}

Pull Requests

import hudson.model.*

nodes = Jenkins.instance.slaves

JOB_PREFIX = "pull_requests-"
suffixes = ["win7", "osx", "ubuntu", "ubuntu-python3", "rhel7"];

for (node in nodes) {
  for (suffix in suffixes) {
    FilePath fp = node.createPath(node.getRootPath().toString() + File.separator + "workspace" + File.separator + JOB_PREFIX + suffix + File.separator +  "build");
    if(fp!=null && fp.exists()) {
      println(fp.toString())
      fp.deleteRecursive()
    }
  }
}

Update Branches For Jobs

import hudson.plugins.git.GitSCM
import hudson.plugins.git.BranchSpec
import static com.google.common.collect.Lists.newArrayList;

def NEW_BRANCH = "*/release-next"

// Access to the Hudson Singleton
def jenkins = jenkins.model.Jenkins.instance;

// Retrieve matching jobs
def allItems = jenkins.items
def chosenJobs = allItems.findAll{job -> job.name =- /release_/};

println "Updating branch for chosen jobs to $NEW_BRANCH"
println ""
// Do work
chosenJobs.each { job ->
    def scm = job.scm;
    if (scm instanceof GitSCM && job.name != "release_nightly_deploy" ) {
      //def newScm = scm.clone()
      println "Updating branch for " + job.name
      scm.branches = newArrayList(new BranchSpec(NEW_BRANCH))
      println "Branch for " + job.name + ": " + scm.branches
      println ""
    }
}

List All SCM Urls

import jenkins.model.*;
import hudson.model.*;
import hudson.tasks.*;
import hudson.plugins.git.*;
import org.eclipse.jgit.transport.RemoteConfig;
import org.eclipse.jgit.transport.URIish;

for(project in Hudson.instance.items) {
  scm = project.scm;
  if (scm instanceof hudson.plugins.git.GitSCM) {
    for (RemoteConfig cfg : scm.getRepositories()) {
      for (URIish uri : cfg.getURIs()) {
        println("SCM " + uri.toString() + " for project " + project);
      }
    }
  }
}

Run a Process On a Single Node

Process p = "cmd /c dir".execute()
println "${p.text}"

// kill process on windows slave
Process p = "cmd /c Taskkill /F /IM MantidPlot.exe".execute()
println "${p.text}"

Run a Process Across All Nodes

import hudson.util.RemotingDiagnostics;

for (slave in hudson.model.Hudson.instance.slaves) {
   println slave.name;
   // is it connected?
   if(slave.getChannel()) {
    println RemotingDiagnostics.executeGroovy("println \"ls\".execute().text", slave.getChannel());
  }
}

Update default values for job parameters

import hudson.model.*

def SUFFIX_VARIABLE = "PACKAGE_SUFFIX"
def NEW_SUFFIX = "nightly"

// Access to the Hudson Singleton
def jenkins = jenkins.model.Jenkins.instance;

// Retrieve matching jobs
def chosenJobs = ["release_clean-rhel7"] //, "release_clean-ubuntu-16.04", "release_clean-ubuntu"]

println "Updating default package suffix for chosen jobs to ${NEW_SUFFIX}"
println ""
// Do work
chosenJobs.each { jobName ->
  job = jenkins.getItem(jobName)
  println(job)
  paramsDef = job.getAction(ParametersDefinitionProperty)
  params = paramsDef.getParameterDefinitions()
  params.each { it ->
    if(it.getName() == SUFFIX_VARIABLE) {
      println("Updating default value of '${SUFFIX_VARIABLE}' variable to '${NEW_SUFFIX}'")
      it.setDefaultValue(NEW_SUFFIX)
    }
  }

}