Core Testing¶
These tests are performed immediately after code-freeze in preparation for a release.
The purpose of these tests is to highlight issues that the individual tests of PRs might have overlooked, or side effects to previous functionality that may have gone undetected. While performing these tests you should also report areas of poor usability and cosmetic defects.
Rules¶
The tests must be performed on the installer versions of the final release candidate. Not on local compiled code.
Serious errors involving loss of functionality, crashes, regression etc. should be raised as
High Priority
tickets to the current iteration and an email sent to the project manager immediately.Minor and cosmetic issues should be raised as tickets against the next development sprint.
First try things that should work, then try to break Mantid, e.g. entering invalid values, unexpected characters etc.
Environments¶
The following environments must be tested, ideally at all host facilities.
Windows 10 64bit
Red Hat Enterprise 6 & 7 (IDAaaS at ISIS)
Ubuntu 18.04
Mac OS X Catalina
Grouped Tasks¶
In order to test a wide range of functionality and avoid repeating work, areas to test are split into groups. These tasks are for testing how Workbench links to the Framework, through the different Toolboxes. The links here are to give you an idea, but please mess around a little too and try to break Mantid!
This was last updated for Release 6.2. Check if the organiser has considered if the Core testers can test on different environments.
Area |
Description and Links |
---|---|
Install/Uninstall |
Windows only: install Mantid, go to the Windows ‘Apps & features’ list and verify the icon is there for Mantid. Uninstall Mantid from within the ‘Apps & features’ list and verify that the desktop and start menu shortcuts are removed. |
Project Save/Load |
Test the small areas below and do not delete workspaces or plots you produce and manipulate, maybe open a few interfaces. Then File > Save Project and try to reload it! Diagnose any problems. |
About / First Time Setup Menu |
Under Help > About (On MAC Mantid > About). Check Opens Successfully + all buttons and links work |
Help Documentation |
Open all Help menu bar options. Load a few algorithm dialogs and click the ?. Produce some plots (1D, Waterfall,Colorfill,3D Surface) and check ? links. |
SliceViewer |
Please only test basic SliceViewer functionality, such as outlined here, and please use SV with unusual data and in unusual ways! If you want extra inspiration, here are the advanced SliceViewer Manual Testing instructions. Do NOT complete these advanced tests, however you may find the initial Data section useful. |
GUI Plotting |
Produce 1D (Individual, Waterfall, Tiled), 2D (Colorfill and Contour) and 3D (Surface and Wireframe) plots with different data types. Displaying 1D Data 3D and volumetric data |
Manipulating Plots |
See Plot Help Docs for info on Toolbar ? buttons, Click Menus, Plots Toolbox, Settings |
Plot Workspace Rename |
Load some data, make a plot, rename the workspace, check the window and plot titles have updated |
Scripting Plots |
For some ideas: Basic 1D, 2D and 3D Plots Formatting Plots with the User Interface Matplotlib in Mantid + run any MPL code and check if Figure Options and Generate a script work. |
Settings (Preferences on Mac) |
Check all options work. Some may need restart. |
Script Editor / Interpreter |
Check basic functionality, such as checking the buttons/options at the top of SE, running algorithms, accessing workspace properties in both environments. Also, check that closing a tab, while it’s script is running, aborts the script properly. Does autocompletion work for mantid algorithms/numpy/matplotlib? Working with Python |
Access mantid framework in Python and Jupyter Notebook |
Follow the instructions on Access Mantid in Python and Notebook and check you can import mantid.simpleapi and run a script/notebook. |
System Memory Usage Widget |
Check that the System Memory Usage Widget updates at a frequency of no more than once every two seconds, ideally on a system with limited resourses. Check that the default position of the widget is in the top right, in the same column as the Messages widget. |
Script Repository |
Test downloading and uploading scripts to the Script Repository. Try to move the Script Repository to another folder. Check out this page for more information. |
Area |
Sub-section |
Description and Links |
---|---|---|
Data and Workspace Menus |
||
Data Loading |
Load different data types from different facilities |
|
Save Nexus/Ascii |
Save a few appropriate workspaces (both the algorithm and from the Save button at top of Workspaces Toolbox) |
|
Show Data |
Display the Data for different workspaces: Workspace2D (Histogram), EventWorkspace, TableWorkspace AND plotBin and plotSpectrum from the Data table |
|
Instrument Viewer |
Open Instrument viewer for instruments in different facilities. |
|
SliceViewer |
Open and make some small changes (full test is separate) |
|
Show Detectors |
||
Sample Logs |
||
Workspace History |
Check Script generation from History works in various cases |
|
Running Script |
||
Random scripts |
e.g. from Solutions to last 2 induction courses, Script Repo |
|
Workspace Algebra |
See bottom of this page: Matrix Workspace |
|
Error handling in Python |
Get creative, e.g. run code without imports, wrong filepath in Manage User directories, indent errors. Are Error Messages Useful?? |
|
Algorithms |
||
Execution |
Check 5-10 algorithms |
|
Input Validation |
Input invalid values and see if a USEFUL * tooltip appears or after executing there is a useful error |
|
Progress and Cancelling |
Show Algorithm Toolbox + run some algorithms. Is the Progress Reported helpfully, under Details do algorithms appear and does cancelling work? e.g. Pause |
|
Fitting |
||
Normal |
Plot a spectrum, click ‘Fit’ Toolbar button. Add different functions, add ties. A more Complex case |
|
User Defined Function |
Halfway down this page: Fit Model Choices |
|
Sequential |
This uses the PlotPeakByLogValue v1 algorithm. Load a workspace with multiple spectra. Plot one spectrum and click the ‘Fit’ Toolbar button. Add an appropriate fit function and click Fit>Sequential Fit. If only one ws is selelcted then all spectra are fit. Try with a range of spectra (e.g. 1:10). Fit spectra from a WorkspaceGroup. |
|
Fit Algorithm |
See Fit v1, check ties work |