Using the huff cluster

Using the huff cluster

Table of Contents

System Information

Name: huff.hprc.vcu.edu

The huff cluster runs Centos 7.9 and comprises 28 standard nodes, 5 GPU nodes, 9 large-memory nodes. The storage includes 3.1 PB of shared space.

Nodes

Partition

CPU

Total nodes

Total CPU

Total GPU

RAM (per node)

local /tmp

Nodes

Partition

CPU

Total nodes

Total CPU

Total GPU

RAM (per node)

local /tmp

Standard compute

basic

2x Epyc2 64 core, 128 real cores, 256 vcores

12 - huff9-20

3072 vcores

none

256 GB

1.2 TB

Upgraded compute

basic

2x Epyc3 64 core, 128 real cores, 256 vcores

16 - huff24-39

4096 vcores

none 

512 GB

768 GB

GPU compute

GPU

2x Epyc2 64 core, 128 real cores, 256 vcores

3 - huff21-23

768 vcores

6x V100, 32GB

512 GB

1.2 TB

GPU compute 

GPU 

2x Epyc3 64 core, 128 real cores, 256 vcores

1- huff44,45

512 Vcore

8x A100, 80GB

1TB

14 TB

Large memory

LGMEM

2x Epyc2 64 core, 128 real cores, 256 vcores

 

 

2x Epyc4, 48  core, 96 real cores, 192 vcores

4 - huff5-8

 

 

1- huff4

1024 vcores

 

 

192 vcores

none

 

 

none

2x 768 GB

1x 1 TB

1x 2 TB

1x 3 TB

3.2 TB

 

 

10 TB

Upgraded large memory

LARGE

2x Epyc3 64 core, 128 real cores, 256 vcores

4- huff 40-43

1024 vcores

none

1 TB

768 GB

Software

All huff nodes provide modules allowing users to set up the following software. Use the command "module avail" to see more. Please read the documentation on how to use modules to load your software version.

Software

Versions

GCC

7, 8, 9, 10, 11

CUDA (GPU nodes)

11.3, 11.4, 12.1

Python

3.6.9

R

4.0.5, 4.1.2

ncbi-blast

2.11.0

MPI

4.1.1, 4.1.5

Intel compilers

2021.2

Data Classification Definitions

Huff may contain only Category II and Category III data according to the VCU Data Classification Standard.

  • Category III: non-propriety data.

  • Category II: propriety data, de-identified research information.

  • Category I: sensitive data protected by federal and state law, except ITAR and export-controlled. 

Access

All VCU faculty, staff, and graduate students are authorized to have accounts on the "basic" partition of huff. Access to the GPU and LGMEM partitions requires special requests on demand. 
Users may create their accounts using the Account Creation on the HPRC website.

Logging in

Huff may be accessed via SSH or SFTP from Windows, Mac, and Linux computers. For Windows, we recommend downloading Putty. Putty should work with both Windows 10 and 11.  For Mac or Linux, one can open a terminal window and connect using the command "ssh yourvcueid@huff.hprc.vcu.edu" and your password. Please note that you will need to use your VCU eid regardless of your connection method. Note that all access to huff from off-campus requires the use of the VCU VPN. 

Logging in with Putty on Windows

Logging in with a terminal window on Mac/Linux

Logging in with Putty on Windows

Logging in with a terminal window on Mac/Linux

Setting up X-Windows in Putty

In general HPRC clusters are not set up for interactive use with X-Windows. But sometimes there are reasons to make a graphical connection to the head node. Below are the settings for X-WIndows with PUTTY. 

Common Slurm Commands

Commands

Syntax

Description

Commands

Syntax

Description

sbatch

sbatch <scriptname>

Submit a batch script to Slurm for processing. ex. sbatch myjob.sh

squeue

squeue -u <username>

Show information about your job(s) in the queue. The command when run without the -u flag, shows a list of your job(s) and all other jobs in the queue. ex squeue -u yourvcueid

srun

srun <resource-parameters>

Run jobs interactively on the cluster. ex. srun --nodes=1 --ntasks-per-node=1 --mem=4G --pty bash -i

scancel

scancel <job-id>

Cancel a job. ex. scancel 58794

sinfo

sinfo

Get information about the resources on available nodes that make up the HPC cluster.

Running jobs on huff using Slurm

Job type

Description

Single-node interactive job

The simplest option is opening an interactive connection to a compute node and then running a job. The command below will open an interactive connection to a huff node and allow you to run a command on one processor. 

By default, if you do not add a memory amount the job will need to wait for a completely empty node to run. 

srun --nodes=1 --ntasks-per-node=1 --mem=4G --pty bash -i

Single-node simple job

Another type of simple job running the sleep command on a compute node. The output of a sleep job provides important information to troubleshoot your jobs and learn job submission. Click the link below for more information.

Example of a job that runs on a single node

Single-node parallel job

Sometimes one wants to run parallel jobs on a single compute node. This works very well for threaded software as opposed to intensive software that requires message passing. Click the link below for more information.

Example of a parallel job that runs on one node

Multi-node parallel job

Many HPC users have a need to run jobs that take multiple nodes and use message passing (MPI) to provide communication. Click the link below for more information.

Example of a parallel job that runs on multiple nodes

Please read the Slurm hints that can help you improve the performance of your memory or CPU-intensive jobs.

 

Using Containers at HPRC

There are a couple of types of containers available in the Research Computing Space. These are Docker and Apptainer (formerly Singularity).  

Apptainer (Singularity) is a Secure Alternative to Docker

Docker images are not secure because they provide a means to gain root access to the system they are running on. For this reason Docker is not available on the HPRC clusters (neither is nvidia-docker). This is not a problem because we offer Apptainer (Singularity), which is an alternative to Docker that is both secure and designed for high-performance computing. Singularity is compatible with all Docker images and it can be used with GPUs and MPI applications. Learn about the differences between virtual machines, Docker and Singularity. Be aware of Apptainer by reading about the history of singularity.

(Note: If you are more familiar with Docker commands, this conversion table is very useful)