Dedication

HPC doc.

The cluster

Access

See here Access Baobab

Acknowledgments

If you publish any data computed using Baobab cluster, we kindly ask you to add the following acknowledgement to your paper:

The computations were performed at University of Geneva on the Baobab cluster.

Resources

The cluster is mainly composed of a head node (baobab.unige.ch) and a bunch of computation nodes. Each node provides 16 cores, 12 cores or 8 cores. More details here: https://plone.unige.ch/distic/pub/hpc/baobab_en. When you submit a job to the cluster, you allocate resources for your work.

You can either request exclusives resources (allocation by node) or request shared resources (allocation by cores). See Partitions and limits for the choice of a partition.

Since the energy consumption of the cluster is huge, we have enabled a power saving mode. What does it mean is that if a node is idle for more than 10 minutes it will be powered off (it will have a ~idle as state name in http://baobab.unige.ch/slurm/ status page). The nodes that are powered off are automatically started if needed when you submit a new job. This means that a small delay could occur between the time you submit your job and the node is ready (they have an # in their state during the boot time which should be less than two minutes).

Compute nodes

Since the cluster has evolved, not all nodes are of the same generation. You can see the details in the following table.

generation model freq nb cores architecture node extra flag
V1 X5550 2.67GHz 8 cores “Gainestown” (45 nm) node[072-075]  
V2 X5650 2.67GHz 12 cores “Westmere-EP” (32 nm) Efficient Performance node[076-153]  
V3 E5-2660V0 2.20GHz 16 cores “Sandy Bridge-EP” (32 nm) Efficient Performance node[001-056,058]  
V3 E5-2670V0 2.60GHz 16 cores “Sandy Bridge-EP” (32 nm) Efficient Performance node[059-062,067-070]  
V3 E5-4640V0 2.40GHz 32 cores “Sandy Bridge-EP” (32 nm) Efficient Performance node057  
V4 E5-2650V2 2.60GHz 16 cores “Ivy Bridge-EP” (22 nm) Efficient Performance node[063-066,071,154-172]  
V5 E5-2643V3 3.40GHz 12 cores “Haswell-EP” (22 nm) Efficient Performance gpu[002-003]  
V6 E5-2630V4 2.20GHz 20 cores “Broadwell-EP” (14 nm) node[173-201],gpu[004-006]  
V6 E5-2643V4 3.40GHz 12 cores “Broadwell-EP” (14 nm) node[202] HIGH_FREQUENCY
V6 E5-2680V4 2.40GHz 28 cores “Broadwell-EP” (14 nm) node[203]  

The “generation” column is just a way to classify the nodes on Baobab. In the following table you can see the features of each architecture.

  MMX SSE SSE2 SSE3 SSSE3 SSE4.1 SSE4.2 AVX F16C AVX2 FMA3
Gainestown YES YES YES YES YES YES YES NO NO NO NO
Westmere-EP YES YES YES YES YES YES YES NO NO NO NO
Sandy Bridge-EP YES YES YES YES YES YES YES YES NO NO NO
Ivy Bridge-EP YES YES YES YES YES YES YES YES YES NO NO
Haswell-EP YES YES YES YES YES YES YES YES YES YES NO
Broadwell-EP YES YES YES YES YES YES YES YES YES YES YES

Specify the cpu type you want

It’s normally not important on which type of node your job is running. But there are cases where it’s important to be able to stick to a given kind of cpu or generation. You can request for example to have only nodes with cpu E5-2660V0:

srun --constraint=E5-2660V0
or
#SBATCH --constraint=E5-2660V0

Or you can specify that you want a node of generation V3:

srun --constraint=V3
or
#SBATCH --constraint=V3

You can specify as well multiple contraints using OR. For example if you don’t want to use nodes of generation V1:

srun --constraint="V2|V3|V4|V5|V6"
or
#SBATCH --constraint="V2|V3|V4|V5|V6"

Partitions and limits

There are various partitions available to everybody on the cluster:

  • debug with a max execution time of 15 minutes and a maximum number of cores of 32.
  • bigmem with a max execution time of 4 days and a maximum number of cores of 16. This partition provides 256Gb of ram.
  • parallel with a max execution time of 4 days and a maximum number of cores per job of 400.
  • shared with a max execution time of 12 hours and a maximum number of cores per job of 400.
  • mono-shared same as shared but allocation of resources is by core.
  • mono same as parallel but allocation of resources is by core.

If you need to access GPUs, please see Nvida CUDA for details.

There are others partitions only available to their owner:

  • cui
  • wesolowski
  • dpt
  • gsem
  • pawlowski
  • kruse
  • biosc
  • cisa
  • lehmann
  • askja
  • montet
  • kruse

Have a look on the web page baobab.unige.ch at the “status” tab to see the detail of a given partition. If you belongs to one of this group, please request to have access to the correct partition as it’s not made automatically.

The partitions shared and mono-shared contain all Baobab’s nodes including the private nodes. When a private node isn’t being used by its owner, it is available in this partition. These two partitions are heterogeneous. There are two nodes having 32 cores and 512GB of ram (node57 and 186) and the other nodes are Intel 8 to 20 cores with 3-4GB per core.

If you want to launch a job that requires less than a full node, please use one of the mono* partitions. If you don’t, you will monopolize the cluster which turns out to be very inefficient.

Important

The default partition is debug.

To see more details about the partitions (default time, etc.):

scontrol show part

The time formats accepted by SLURM are as follows:

minutes
minutes:seconds
hours:minutes:seconds
days-hours
days-hours:minutes
days-hours:minutes:seconds

Backup

There is a backup of your home directory that takes place every day. As some of your temporary (or generated) files could be very big and it would be a waste of resources to back them up, we ask you to proceed as follows:

  • Create a directory named “scratch” at the root of your home directory.
  • Use it as your working place to store all the big files that don’t need a backup. We thank you for your cooperation.

File transfer

There is no need to transfer files between compute nodes and the master as they share the same storage space.

Anyway, if you need specifically to transfer something from the login to the local storage of a node, you can use sbcast (but you probably shouldn’t).

Compilation with MPI support

You have the choice between openMPI for gcc or intel. It’s very important not to compile using directly gcc, icc or similar commands, but rather rely on the wrappers mpicc, mpic++, mpicxx or similar ones.

All the newer versions of mpi will be available through the use of Module - lmod.

Example for the latest version of openMPI for gcc:

module load foss

Example for the latest version of openMPI for Intel:

module load openmpi/intel

If needed, you can stick to a particular version:

module load foss/2016a

Applications

Several applications are installed on the cluster. We describe how to use them or install them if this should be done by the user. See Applications.

Module - lmod

The easiest way to use an application is to use the “module” command. By using “module”, you don’t need to know where the softwares are physically located (the full path), but instead you can just type the application name as if its path were in your PATH (this is indeed what “module” does). The “module” command can as well set other important variables for an application, so it’s always a good idea to use it.

Search for an application:

module spider R

The output of this command will indicate that you need two dependencies:

GCC/4.9.3-2.25  OpenMPI/1.10.2

To Load R:

module load GCC/4.9.3-2.25  OpenMPI/1.10.2 R/3.2.3

By using a module without specifying a version, you will always get the latest version of the software. If you want to stick to a particular version, you need to specify it.

Then you can juste invoke for example R by typing R (instead of the full path). Of course you are still required to use SLURM to launch your software.

You can see the help for a particular module:

module help R

To load some modules at login, you can add something like this in your .bashrc:

if [ -z "$BASHRC_READ" ]; then
  export BASHRC_READ=1
  # Place any module commands here
  # module load git
fi

Migrating from module tcl to module lmod.

You need to patch your .bash_profile and .bashrc files. I have created a patch for this purpose. To apply it:

cd ~/
patch < /opt/lmod/cleanupmodule.patch

You should see:

patching file .bash_profile
patching file .bashrc

If you see something like that:

patching file .bash_profile
Reversed (or previously applied) patch detected!  Assume -R? [n]

Please cancel the process (ctrl-c) and contact hpc@unige.ch for help.

Activate the new module:

touch .lmod

To take effects, you need to logout and login again.

To test if it’s working, do as follow:

module --version

The output should be more or less this one (version may vary):

Modules based on Lua: Version 6.4  2016-06-01 17:42
   by Robert McLay mclay@tacc.utexas.edu

Important

If you were using module in your sbatch scripts, you need to clean them by removing the following line:

source /etc/profile.modules