Core Testing¶

These tests are performed immediately after code-freeze in preparation for a release.

The purpose of these tests is to highlight issues that the individual tests of PRs might have overlooked, or side effects to previous functionality that may have gone undetected. While performing these tests you should also report areas of poor usability and cosmetic defects.

Rules¶

The tests must be performed on the installer versions of the final release candidate. Not on local compiled code.
Serious errors involving loss of functionality, crashes, regression etc. should be raised as High Priority tickets to the current iteration and an email sent to the project manager immediately.
Minor and cosmetic issues should be raised as tickets against the next development sprint.
First try things that should work, then try to break Mantid, e.g. entering invalid values, unexpected characters etc.

Environments¶

The following environments must be tested, ideally at all host facilities.

Windows 10 64bit
Red Hat Enterprise 6 & 7 (IDAaaS at ISIS)
Ubuntu 18.04
Mac OS X Catalina

Grouped Tasks¶

In order to test a wide range of functionality and avoid repeating work, areas to test are split into groups. These tasks are for testing how Workbench links to the Framework, through the different Toolboxes. The links here are to give you an idea, but please mess around a little too and try to break Mantid!

This was last updated for Release 6.2. Check if the organiser has considered if the Core testers can test on different environments.

Smaller tasks¶
Area	Description and Links
Install/Uninstall	Windows only: install Mantid, go to the Windows ‘Apps & features’ list and verify the icon is there for Mantid. Uninstall Mantid from within the ‘Apps & features’ list and verify that the desktop and start menu shortcuts are removed.
Project Save/Load	Test the small areas below and do not delete workspaces or plots you produce and manipulate, maybe open a few interfaces. Then File > Save Project and try to reload it! Diagnose any problems.
About / First Time Setup Menu	Under Help > About (On MAC Mantid > About). Check Opens Successfully + all buttons and links work
Help Documentation	Open all Help menu bar options. Load a few algorithm dialogs and click the ?. Produce some plots (1D, Waterfall,Colorfill,3D Surface) and check ? links.
SliceViewer	Please only test basic SliceViewer functionality, such as outlined here, and please use SV with unusual data and in unusual ways! If you want extra inspiration, here are the advanced SliceViewer Manual Testing instructions. Do NOT complete these advanced tests, however you may find the initial Data section useful.
GUI Plotting	Produce 1D (Individual, Waterfall, Tiled), 2D (Colorfill and Contour) and 3D (Surface and Wireframe) plots with different data types. Displaying 1D Data 3D and volumetric data
Manipulating Plots	See Plot Help Docs for info on Toolbar ? buttons, Click Menus, Plots Toolbox, Settings
Plot Workspace Rename	Load some data, make a plot, rename the workspace, check the window and plot titles have updated
Scripting Plots	For some ideas: Basic 1D, 2D and 3D Plots Formatting Plots with the User Interface Matplotlib in Mantid + run any MPL code and check if Figure Options and Generate a script work.
Settings (Preferences on Mac)	Check all options work. Some may need restart.
Script Editor / Interpreter	Check basic functionality, such as checking the buttons/options at the top of SE, running algorithms, accessing workspace properties in both environments. Also, check that closing a tab, while it’s script is running, aborts the script properly. Does autocompletion work for mantid algorithms/numpy/matplotlib? Working with Python
Access mantid framework in Python and Jupyter Notebook	Follow the instructions on Access Mantid in Python and Notebook and check you can import mantid.simpleapi and run a script/notebook.
System Memory Usage Widget	Check that the System Memory Usage Widget updates at a frequency of no more than once every two seconds, ideally on a system with limited resourses. Check that the default position of the widget is in the top right, in the same column as the Messages widget.
Script Repository	Test downloading and uploading scripts to the Script Repository. Try to move the Script Repository to another folder. Check out this page for more information.

Larger tasks¶
Area	Sub-section	Description and Links
Data and Workspace Menus
	Data Loading	Load different data types from different facilities
	Save Nexus/Ascii	Save a few appropriate workspaces (both the algorithm and from the Save button at top of Workspaces Toolbox)
	Show Data	Display the Data for different workspaces: Workspace2D (Histogram), EventWorkspace, TableWorkspace AND plotBin and plotSpectrum from the Data table
	Instrument Viewer	Open Instrument viewer for instruments in different facilities.
	SliceViewer	Open and make some small changes (full test is separate)
	Show Detectors
	Sample Logs
	Workspace History	Check Script generation from History works in various cases
Running Script
	Random scripts	e.g. from Solutions to last 2 induction courses, Script Repo
	Workspace Algebra	See bottom of this page: Matrix Workspace
	Error handling in Python	Get creative, e.g. run code without imports, wrong filepath in Manage User directories, indent errors. Are Error Messages Useful??
Algorithms
	Execution	Check 5-10 algorithms
	Input Validation	Input invalid values and see if a USEFUL * tooltip appears or after executing there is a useful error
	Progress and Cancelling	Show Algorithm Toolbox + run some algorithms. Is the Progress Reported helpfully, under Details do algorithms appear and does cancelling work? e.g. Pause
Fitting
	Normal	Plot a spectrum, click ‘Fit’ Toolbar button. Add different functions, add ties. A more Complex case
	User Defined Function	Halfway down this page: Fit Model Choices
	Sequential	This uses the PlotPeakByLogValue v1 algorithm. Load a workspace with multiple spectra. Plot one spectrum and click the ‘Fit’ Toolbar button. Add an appropriate fit function and click Fit>Sequential Fit. If only one ws is selelcted then all spectra are fit. Try with a range of spectra (e.g. 1:10). Fit spectra from a WorkspaceGroup.
	Fit Algorithm	See Fit v1, check ties work