Packaging¶

This page gives an overview of the different packaging mechanisms used to deliver Mantid to users.

Conda ¶

Mantid provides packages to be consumed by the conda package manager. Even though it is a single codebase the repository is split across four conda packages for users, and a metapackage used to create a development environment:

mantid: Provides Python access to the non-GUI elements of mantid and allows users to import mantid as a Python library for use in other programs. This package is the base package that can be installed on its own without the requirement for any other packages to be installed.
mantidqt: Provides Python access to Qt-widgets library customized for use with the mantid framework, e.g. instrumentview, sliceviewer, file finder widget etc.
mantiddocs: Provides the built-in help pages for the Qt Help viewer in workbench.
mantidworkbench: The graphical application produced by the Mantid project that brings together all of the above packages.
mantid-developer: The metapackage used to create a development environment. It contains all the required packages for building and testing our software. See Getting Started for more detail on how to setup a development environment.

Each package is built separately for x64 Windows, Linux and macOS with the exception of mantiddocs where a single, noarch package is created for all operating systems as it only contains static HTML.

All mantid conda packages are available to users through Mantid’s conda organization.

PyStoG is now an optional dependency, so the neutrons conda channel is required when using PyStoG.

Building Conda Packages ¶

Conda requires recipes as input for producing packages. The recipes are stored in the main codebase so that they can evolve as the code develops. The packages are built as part of the CI pipeline using a script to encapsulate the steps.

To build the packages locally it is recommended that you use a separate clone of the repository as the build copies the repository content to a temporary working directory and any additional build directories can interfere with the packaging. The following steps should produce a set of conda packages in a conda-bld directory relative to the current working directory (on Windows run this from Git bash):

git clone https://github.com/mantidproject/mantid.git mantid-conda-build
mantid-conda-build/buildconfig/Jenkins/Conda/package-conda $PWD \
  --build-mantid --build-qt --build-docs --build-workbench 2>&1 | tee package.log
# wait quite a while ...
# packages will appear in a conda-bld directory

You can build the workbench without the documentation by:

mantid-conda-build/buildconfig/Jenkins/Conda/package-conda $PWD \
  --build-mantid --build-qt --build-workbench-without-docs 2>&1 | tee package.log

To create a new test environment with packages from the local build:

mamba create -n local-package-test -c $PWD/conda-bld mantidworkbench
mamba activate local-package-test
workbench

Standalone ¶

Mantid also provides a set of bundled packages that provide a complete install of MantidWorkbench and all of its dependencies without a user having to first setup conda and then install the relevant conda packages. The standalone packages are available to users through Mantid’s download page.

Each installer is simply a conda environment, with mantidworkbench installed within it followed by the appropriate installer technology wrapped around it. Installers are provided for:

Windows: A NSIS installer package providing a wizard-style interface for installing/uninstalling MantidWorkbench.
macOS: A disk image providing a facility to drag-and-drop MantidWorkbench to the /Applications folder
Linux: An xz-compressed tarball of the package files and directories that can be extracted to any location.

Building Standalone Installers ¶

The installers are built as part of the CI pipeline using a script to encapsulate the build steps.

To build an installer locally first build a set of conda packages using the instructions above. Once you have the conda-bld directory run the following from the same working directory:

mantid-conda-build/buildconfig/Jenkins/Conda/package-standalone \
  $PWD --package-suffix Unstable 2>&1 | tee standalone-package.log
# wait some time and the installer will appear in the working directory

The --package-suffix argument is an optional string to append to the name of the final package. We generally pick Unstable for installers not built by the CI pipeline to indicate it has been built outside of the standard process.

Build packages from branch using Jenkins ¶

Developers can build packages to test branches using the build_branch Jenkins job. This job provides the ability to,

Run system tests on Windows, Mac, Linux, or all three
Build a packages on Windows, Mac, Linux, or all three
Publish the package(s) to the mantid-test Anaconda channel
Publish the package(s) to a given Github repository under a specified tag

for a given branch of mantid. The branch can be from the main mantid repo or from a remote.

Options ¶

DESCRIPTION: A description to help catalogue the build. Please make use of this!
BUILD_DEVEL [none, all, linux-64, win-64, osx-arm64]: Run the system tests for this OS.
BUILD_PACKAGE [none, all, linux-64, win-64, osx-arm64]: Build a package on this OS.
PACKAGE_SUFFIX: String to append onto the standalone package name, useful for distinguishing builds. By default this is Unstable.
PUBLISH_TO_ANACONDA: Set true to publish to the given Anaconda channel and label.
PUBLISH_TO_GITHUB: Set true to publish to the Github repository using the specified tag.
ANACONDA_CHANNEL: Anaconda channel to upload the package to. For this job the only option is mantid-test.
ANACONDA_CHANNEL_LABEL: Label attached to the uploaded package. By default this is unstable. Please give it a meaningful label.
GITHUB_RELEASES_REPO: Repository to store the release. By Default this is mantidproject/mantid.
GITHUB_RELEASES_TAG: Name of the tag for the release; only to be used for release candidate builds.
GH_ORGANIZATION_OR_USERNAME: Name of the organisation or Github user name who owns the repository with the code to build. By default this is mantidproject, if you are building from a fork this will need to change to your username.
BRANCH_NAME: Name of the branch to build the packages from.

Example ¶

Say I’ve implemented a new file searching method on a branch 1234_new_file_search and I want to test this on IDAaaS, one of the easiest ways to do this would be to build the packages and upload them to Anaconda using the pipeline. These are the steps I’d take to do this:

Go to the build_branch Jenkins job.
If needed click login in the top right of the window.
Go to Build with parameters in the side bar.
Fill out the following options:
- DESCRIPTION = Upload Linux test packages for new file searching method
- BUILD_DEVEL = none
- BUILD_PACKAGE = linux-64
- PUBLISH_TO_ANACONDA = true
- ANACONDA_CHANNEL_LABEL = new_file_search_test
- BRANCH_NAME = 1234_new_file_search
Click Build. This will take you back to the main job page, the build just set off will be the most recent (highest number) build on the left hand side. It is a good idea to make note of the build number / copy the link somewhere safe. If the build is for testing a pr, make sure to add the link to the testing instructions.
Once the job has successfully completed, check the Mantid Anaconda page to make sure it has uploaded.
Head to IDAaaS (or any linux system) and run mamba install -c mantid-test/label/new_file_search_test mantidworkbench in a new environment to install the test package.

Most often, you won’t need to upload the packages to Anaconda, this is most useful in cases where installing standalone packages is inconvenient. Standalone package builds created by the jenkins job can be found under the jenkins job build artifacts, this is near the top of the page. Say you built a package for Windows using the jenkins job, you should find a mantidworkbench exe file in the build artifacts.

Building a custom `mantid-developer` environment ¶

This is useful if you need to change a pinned version of one of Mantid’s dependencies and test the change locally.

One can create a new environment specifically for building the package (with conda-build, conda-index, rattler-build, and versioningit), but the build itself happens in a separate sandbox which makes this unneccesary.
Make your changes to the conda recipe files.
Change directory to mantid/conda/recipes
With mantid_dev_builder active, run
```
MANTID_VERSION=$(versioningit ../../) rattler-build build -r ./mantid-developer -m ./conda_build_config.yaml --output-dir=/home/me/tmp/rattler
```
This will build a local version of mantid-developer with your changes and place it in the directory /home/me/tmp/rattler. This must be outside of the buildtree because rattler will recursively copy everything.
Index the new packages
```
python -m conda_index /home/me/tmp/rattler/
```
This will allow conda to install the package by name rather than by file. At this time, creating an environment using the filename will skip installing the dependencies and not use any channels.

Create a new environment with the new developer package

mamba create -n test -c /home/me/tmp/rattler/ -c neutrons mantid-developer

You will need to re-run cmake --fresh ... with this new environment.

Exploring package contents ¶

For this example, assume that the package is named mantid and the version number is 1.2.3 and that one is using linux. After creating the package there will be a file named mantid-1.2.3-<buildinfo>.conda. Since this is actually a zip file, it can be unpacked

mkdir tmp && cd tmp
unzip ../<mantid-1.2.3-<buildinfo>.conda

This will unpack 3 files

info-mantid-1.2.3-<buildinfo>.tar.zst
metadata.json
pkg-mantid-1.2.3-<buildinfo>.tar.zst

The info-mantid-1.2.3-<buildinfo>.tar.zst file contains all of the metadata and can be unpacked using

tar --zstd -xf info-mantid-1.2.3-<buildinfo>.tar.zst
ls info
recipe  about.json  git         hash_input.json  paths.json
test    files       has_prefix  index.json       run_exports.json

Similarly, pkg-mantid-1.2.3-<buildinfo>.tar.zst contains all of the actual files to be installed. Generally a listing of the contents is what is desired

tar --zstd -tf pkg-mantid-1.2.3-<buildinfo>.tar.zst