Architecture

ALICE is a large system built from a number of components.

 

HPC system overview

Login nodes

The ALICE login nodes provide the command line access to the cluster. When users log in to alice.le.ac.uk, they will be connected to one of the two login nodes; internally these are known as alice03 and alice04. They are completely identical in use.

It is from the login nodes that users perform any actions with the cluster: compiling code, transferring data in and out of the cluster, and submitting jobs to the scheduler. The login nodes are not to be used for running jobs, the scheduler should always be used, even for testing. Any processor-intensive jobs running on the login nodes are likely to be summarily killed by the cluster administrators. This is to ensure that the login nodes remain as responsive as possible for all users.

ALICE users can also login to SPECTRE login nodes should they require a remote desktop. All commands will also work from the SPECTRE login nodes.

Compute nodes

There are 170 standard compute nodes available for job execution. Once a job is submitted to the job queue, the scheduler will decide which compute node(s) the job should run on, depending on a combination of the job's requirements and available resources.

A user requesting to run an interactive job will be automatically logged in to an available compute node in order to run the job.

Standard Compute Nodes

Each standard compute node has a pair of 14-core Intel Xeon skylake CPUs running at 2.6GHz, and 128GB of RAM. There is approximately 428GB of local storage available on each node, available in /local.

Standard compute nodes are named nodeNNN where NNN is between 001 and 170.

Large Memory Nodes

There are two large-memory compute nodes. Each of these has 4 Intel Xeon CPUs (48 cores in total) running at 3.00GHz, and 1TB of RAM. There is approximately 800GB of local storage in /local.

Large memory nodes are named lmem001 and lmem002. 6 new large mem nodes have been purchased and will replace the above nodes in February 2017.

GPU Nodes

There are 4 GPU nodes.

Each have 64GB RAM, two Ivy Bridge CPUs at 2.50GHz (20 cores in total) and 2 x NVIDIA Tesla K40m GPU cards.

New Nvidia Tesla P100 processing nodes will be purchased in 2017Q1.

Xeon Phi Nodes

There are no Xeon Phi nodes available on ALICE at present. A Xeon Phi development server is available on request for those interested in developing MIC codes.

Management nodes

There are two nodes whose task is to run the essential services for the cluster, such as the job scheduler and management tools. These are inaccessible to users.

Storage

A high performance distributed storage system provides approximately 900TB of storage for user data, and is accessible to all nodes on the cluster. The organisation of the file systems on this storage is described in the Filesystems section.

Interconnect

Each component is connected to all the others via a high-bandwidth low-latency FDR Infiniband interconnect for both MPI parallel communication and storage data.

Share this page:

Contact us

IT Service Desk

Report an issue or make a request

Online:

In person:

Opening hours:
Mon - Fri 8.30am - 5.00pm

Phone:
0116 252 2253
Escalation:
Ask for Ismail Patel

 

Teaching room help

Instant help:
0116 252 2919
Borrow:
Search IT Help