EasyBuild – SDU eScience

What is EasyBuild ?

EasyBuild is a software build and installation framework written in Python, which allows to deploy and manage (scientific) applications on High Performance Computing systems. EasyBuild v3.6.2 is currently installed on ABACUS2.0 and can be loaded as a module:

testuser@fe1:~$ module load easybuild

How to search for a particular software

The shell command eb -S is used to search for easyconfig files included in the EasyBuild online repository. Easyconfig files are used to specify the version, dependencies and build parameters of the software. For example:

testuser@fe1:~$ eb -S '^Python-3.6.4-*'
...
  * $CFGS1/Python-3.6.4-foss-2017a.eb
  * $CFGS1/Python-3.6.4-foss-2018a.eb
  * $CFGS1/Python-3.6.4-golf-2018a.eb
  * $CFGS1/Python-3.6.4-intel-2018a.eb
  * $CFGS1/Python-3.6.4-iomkl-2018.02.eb
  * $CFGS1/Python-3.6.4-iomkl-2018a.eb

The output shows the list of well-tested easyconfig files which include the instructions to configure, build and install the application with a particular compiler toolchain. The full list of toolchains implemented in the EasyBuild environment is printed with the command: eb --list-toolchains. Currently, the following toolchains are installed on ABACUS2.0 and can be loaded together with the easybuild module.

foss    date       binutils   GCC     Open MPI   OpenBLAS    LAPACK                 ScaLAPACK    FFTW
2017b   Jul 2017   2.28       6.4.0   2.1.1	 0.2.20     (incl. with OpenBLAS)   2.0.20   	3.3.6
2018a   Jan 2018   2.28       6.4.0   2.1.2	 0.2.20     (incl. with OpenBLAS)   2.0.20  	3.3.7
2018b   Jul 2018   2.30       7.3.0   3.1.1	 0.3.1      (incl. with OpenBLAS)   2.0.20   	3.3.8

goolfc	date	   binutils   GCC     Open MPI	 OpenBLAS    LAPACK      	   ScaLAPACK	FFTW	CUDA
2018a   Jan 2018   2.28       6.4.0   2.1.2	 0.2.20     (incl. with OpenBLAS)   2.0.20   	3.3.7	9.1.85

intel	date       binutils   GCC     Intel compilers	    Intel MPI	    Intel MKL
2018a	Jan 2018   2.28	      6.4.0   2018.1.163	    2018.1.163	    2018.1.163
2018b	Jul 2018   2.30	      7.3.0   2018.3.222	    2018.3.222	    2018.3.222

iomkl	date       binutils   GCC     Intel Compilers	    Open MPI	    Intel MKL
2018a	Jan 2018   2.28	      6.4.0   2018.1.163 	    2.1.2 	    2018.1.163
2018b	Jul 2018   2.30	      7.3.0   2018.3.222            3.1.1   	    2018.3.222

How to install a software using the online repository

It is possible to check all the dependencies defined in a particular software easyconfig file by executing the eb command with the option --dry-run (or -D). For example:

testuser@fe1:~$ eb GROMACS-2018-foss-2018a.eb -D
Dry run: printing build status of easyconfigs and dependencies
...
 * [x] $CFGS/m/M4/M4-1.4.18.eb (module: M4/1.4.18)
 * [x] $CFGS/z/zlib/zlib-1.2.11.eb (module: zlib/1.2.11)
 * [x] $CFGS/h/help2man/help2man-1.47.4.eb (module: help2man/1.47.4)
 * [x] $CFGS/m/M4/M4-1.4.17.eb (module: M4/1.4.17)
 * [x] $CFGS/b/Bison/Bison-3.0.4.eb (module: Bison/3.0.4)
 * [x] $CFGS/f/flex/flex-2.6.3.eb (module: flex/2.6.3)
 * [x] $CFGS/b/binutils/binutils-2.28.eb (module: binutils/2.28)
 * [x] $CFGS/g/GCCcore/GCCcore-6.4.0.eb (module: GCCcore/6.4.0)
 * [x] $CFGS/m/M4/M4-1.4.18-GCCcore-6.4.0.eb (module: M4/1.4.18-GCCcore-6.4.0)
 * [x] $CFGS/z/zlib/zlib-1.2.11-GCCcore-6.4.0.eb (module: zlib/1.2.11-GCCcore-6.4.0)
 * [x] $CFGS/h/help2man/help2man-1.47.4-GCCcore-6.4.0.eb (module: help2man/1.47.4-GCCcore-6.4.0)
 * [x] $CFGS/b/Bison/Bison-3.0.4-GCCcore-6.4.0.eb (module: Bison/3.0.4-GCCcore-6.4.0)
 * [x] $CFGS/f/flex/flex-2.6.4-GCCcore-6.4.0.eb (module: flex/2.6.4-GCCcore-6.4.0)
 * [x] $CFGS/b/binutils/binutils-2.28-GCCcore-6.4.0.eb (module: binutils/2.28-GCCcore-6.4.0)
 * [x] $CFGS/g/GCC/GCC-6.4.0-2.28.eb (module: GCC/6.4.0-2.28)
 * [x] $CFGS/o/OpenBLAS/OpenBLAS-0.2.20-GCC-6.4.0-2.28.eb (module: OpenBLAS/0.2.20-GCC-6.4.0-2.28)
 * [x] $CFGS/a/Autoconf/Autoconf-2.69-GCCcore-6.4.0.eb (module: Autoconf/2.69-GCCcore-6.4.0)
 * [x] $CFGS/a/Automake/Automake-1.15.1-GCCcore-6.4.0.eb (module: Automake/1.15.1-GCCcore-6.4.0)
 * [x] $CFGS/n/ncurses/ncurses-6.0-GCCcore-6.4.0.eb (module: ncurses/6.0-GCCcore-6.4.0)
 * [ ] $CFGS/c/CMake/CMake-3.10.2-GCCcore-6.4.0.eb (module: CMake/3.10.2-GCCcore-6.4.0)
 * [x] $CFGS/l/libtool/libtool-2.4.6-GCCcore-6.4.0.eb (module: libtool/2.4.6-GCCcore-6.4.0)
 * [x] $CFGS/a/Autotools/Autotools-20170619-GCCcore-6.4.0.eb (module: Autotools/20170619-GCCcore-6.4.0)
 * [x] $CFGS/n/numactl/numactl-2.0.11-GCCcore-6.4.0.eb (module: numactl/2.0.11-GCCcore-6.4.0)
 * [x] $CFGS/h/hwloc/hwloc-1.11.8-GCCcore-6.4.0.eb (module: hwloc/1.11.8-GCCcore-6.4.0)
 * [x] $CFGS/o/OpenMPI/OpenMPI-2.1.2-GCC-6.4.0-2.28.eb (module: OpenMPI/2.1.2-GCC-6.4.0-2.28)
 * [x] $CFGS/g/gompi/gompi-2018a.eb (module: gompi/2018a)
 * [x] $CFGS/f/FFTW/FFTW-3.3.7-gompi-2018a.eb (module: FFTW/3.3.7-gompi-2018a)
 * [x] $CFGS/s/ScaLAPACK/ScaLAPACK-2.0.2-gompi-2018a-OpenBLAS-0.2.20.eb (module: ScaLAPACK/2.0.2-gompi-2018a-OpenBLAS-0.2.20)
 * [x] $CFGS/f/foss/foss-2018a.eb (module: foss/2018a)
 * [ ] $CFGS/g/GROMACS/GROMACS-2018-foss-2018a.eb (module: GROMACS/2018-foss-2018a)
== Temporary log file(s) /tmp/eb-tXC0pK/easybuild-smjQ9R.log* have been removed.
== Temporary directory /tmp/eb-tXC0pK has been removed.

All the dependencies marked with [x] are already installed on ABACUS2.0 and can be loaded together with the EasyBuild module. The user can enforce the installation of the missing dependencies using --robot/-r, e.g.

 testuser@fe1:~$ eb GROMACS-2018-foss-2018a.eb -d --robot

The additional option --debug/-d enables debug logging, which includes all the details of the software build and installation in the log file.

By default, EasyBuild will install the software in $HOME/easybuild/software and the corresponding Lua module file in $HOME/easybuild/modules/all. The path of the module file is automatically added to $MODULEPATH, once the EasyBuild module is loaded. The user can also specify a different installation path for a particular software and the corresponding module using the option --installpath (or, more specifically, --installpath-software and --installpath-modules). In this case the new module path must be added with the command: module use <new_module_path>.

An overview of the EasyBuild configuration settings is given by eb --show-config.

How to create new easyconfig files

The user can define new easyconfig files to install applications which are not currently reported on the online repository. EasyBuild provides templates to implement procedures for building and installing software, called easyblocks. For example, the following easyconfig file can be used to install a particular release of the RStudio software:

easyblock = 'PackedBinary'
name = 'RStudio'
version = '1.1.456'
homepage = 'https://www.rstudio.com
description = "RStudio is a set of integrated tools designed to help you be more productive with R. It includes a console, syntax-highlighting editor that supports direct code execution, as well as tools for plotting, history, debugging and workspace management."

toolchain = {'name': 'dummy', 'version': ''}
sources = ['rstudio-%(version)s-amd64-debian.tar.gz']
source_urls = ['http://download1.rstudio.org']

skipsteps = ['build']

sanity_check_paths = {
    'files': ['bin/rstudio'],
    'dirs': ['R']
}

moduleclass = 'tools'

In this case EasyBuild leverages the PackedBinary easyblock, which provides the support for installing packed binary software. Since the software is not compiled, it is used in the dummy toolchain.

How to include external dependencies

Sometimes it can be useful to build a new software where some of the dependencies are already installed on ABACUS2.0. In this case, any of these dependencies must be indicated as EXTERNAL_MODULE and added as separate metadata. For example, in the following the repository easyconfig file for the Intel C/C++ compilers in the intel-2018b toolchain is tweaked in order to use the preinstalled Intel compiler module:

# new easyconfig file icc-2018.3.222-GCC-7.3.0-2.30.eb
name = 'icc'
version = '2018.3.222'

homepage = 'http://software.intel.com/en-us/intel-compilers/'
description = "Intel C and C++ compilers"

toolchain = {'name': 'dummy', 'version': 'dummy'}
gccver = '7.3.0'
binutilsver = '2.30'
versionsuffix = '-GCC-%s-%s' % (gccver, binutilsver)

dependencies = [
    ('GCCcore', gccver),
    ('binutils', binutilsver, '', ('GCCcore', gccver)),
    ('intel/2018.05', EXTERNAL_MODULE)
]

skipsteps = ['configure', 'build', 'install']

dontcreateinstalldir = 'True'

path_compiler = '/opt/sys/apps/intel/2018.05/bin'
sanity_check_paths = {
    'files': [path_compiler+"/icc", path_compiler+"/icpc"],
    'dirs': [],
}

license_file = '/opt/sys/apps/intel/licenses/USE_SERVER.lic'

moduleclass = 'compiler'

Metadata for the external module intel/2018.05 must be declared in a separate file, in this case called ext_module_metadata.cfg, as follows:

[intel/2018.05]
name = icc, icpc, ifort
version = 18.0.3, 18.0.3, 18.0.3
prefix = /opt/sys/apps/intel/2018.05/bin/

The location of the file with the external module metadata can be specified with the configuration option --external_module_metadata. For example, if both icc-2018.3.222-GCC-7.3.0-2.30.eb and ext_module_metadata.cfg are located in the same directory, then the following command is used to install the icc compiler and define the environmental variables to load the external module intel/2018.05:

testuser@fe1:~$ eb icc-2018.1.163-GCC-6.4.0-2.28.eb --external-modules-metadata=ext_module_metadata.cfg --robot

Similarly, the Intel Fortran compiler in intel/2018.05 can be linked to a modified ifort-2018.3.222-GCC-7.3.0-2.30.eb easyconfig file of the intel-2018b toolchain. In this case, it is not necessary to run the eb command with the option for the metadata file, because the external module is already configured.

What is EasyBuild ?

How to search for a particular software

How to install a software using the online repository

How to create new easyconfig files

How to include external dependencies

Further information: