Transitioning from huff to athena

Transitioning from huff to athena

How do I access athena?

Users with an existing account on huff already have an account on athena. Note that all access from off-campus requires the use of the VCU VPN. You should connect via SSH using your VCU eID and VCU password (not the huff password) on the command line prompt on Windows, Mac, or Linux. There are two login nodes: athena1.hprc.vcu.edu and athena2.hprc.vcu.edu. You can SSH into either of them by running:

ssh yourvcueid@athena.hprc.vcu.edu

If you wish you can set up a passwordless access using SSH keys. While most operations in a cluster are based on the command line, there is also an option to using X11 Forwarding to Run Remote Graphical Applications.

You can also see the new web interface based on Open OnDemand, which is a web portal to the Athena cluster providing a graphical user interface for the most commonly used applications including Matlab, RStudio, Jupyter lab, ParaView, SAS, VS Code, and generative AI services.

Do I need to copy my files from huff to athena?

No. Huff users do not need to copy any files because athena uses the same file system as huff. Your files are already accessible.

We recommend research groups sharing files and folders to have a shared group folder among multiple users, please reach out to tsresearch@vcu.edu and we'll create one for your group.

Do I need to make any changes to my Slurm scripts?

Yes. There are few changes you need to make in your Slurm scripts.

Huff

Athena

Explanation

Huff

Athena

Explanation

--partition=basic

--partition=cpu

Basic partition renamed to "cpu". See "sinfo" to list all partitions.

--partition=GPU
--qos=gpu

--partition=gpu
--gres gpu:1

GPU partition renamed to "gpu". The partition no longer requires a separate QoS.

When using the "gpu" partition you must specify the number of GPUs you need (e.g. --gres gpu:1 for one GPU and --gres gpu:2 for two GPUs).

Do not select more than one GPU unless your code really uses more than one GPU.

See "sinfo" to list all partitions. You may select a specific partition for a GPU generation if needed.

--cpus-per-task=256

(any number >= 127)

--cpus-per-task=126

(any number <= 128)

Nodes have 2 CPUs reserved for the operating system.

The maximum number of CPUs per node is 26, 30, 126 for nodes with 28, 32, and 128 CPU cores, respectively.

Do not use more CPUs than the number of threads your code is able to use (default 1).

Hyperthreading is disabled. The performance of your previous code running on 256 logical cores will be similar to now running on 126 physical cores.

--mem=128G

(any number = mem size)

--mem=117G

(any number < mem size)

Nodes have a reserved amount of memory for the operating system.

The maximum memory available per node is 117G, 242G, 494G for 128GB, 256GB and 512GB nodes, respectively.

Memory limits are enforced. Your job may not use more memory than requested, if it does, it may be killed.

Optimize the memory allocation to the job needs. Do not overallocate in excess.

--time=XX:XX:XX

nothing

Runtime limits are not enforced. Your job may take as much time as it needs to complete.

Do I need to recompile my source code for applications like MPI?

Yes. If your code needs to be compiled (typically using "make"), you need to recompile the code on athena. The reason is that any compiled code on huff may not work on athena because the linking to different versions of shared libraries.

If you receive an error like "error while loading shared libraries: libXXXXX.so: cannot open shared object file: No such file or directory" when running a program that worked on huff, you need to recompile your code on athena. If you still get an error regarding shared libraries after recompiling your code, please reach out to tsresearch@vcu.edu The library may need to be installed on athena.

I use anaconda / miniconda / miniforge, what do I need to do?

Installations of anaconda / miniconda / miniforge on the user's home folder are strongly discouraged because they slow down significantly the user's session (did you notice your ssh to the cluster takes long to resolve?). Therefore, we recommend that you remove it and use the anaconda / miniconda / miniforge we installed as modules. To disable your home's conda environment, edit your .bashrc file and comment the conda section by adding ### to the lines contained within the "conda initialize" lines.

(base) [user@athena1 ~]$ nano .bashrc ### Add ### to the conda section ### >>> conda initialize >>> ### !! Contents within this block are managed by 'conda init' !! ###__conda_setup="$('/lustre/home/user/anaconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)" ###if [ $? -eq 0 ]; then ### eval "$__conda_setup" ###else ### if [ -f "/lustre/home/user/anaconda3/etc/profile.d/conda.sh" ]; then ### . "/lustre/home/user/anaconda3/etc/profile.d/conda.sh" ### else ### export PATH="/lustre/home/user/anaconda3/bin:$PATH" ### fi ###fi ###unset __conda_setup ### <<< conda initialize <<<

Exit the ssh session and reconnect to the cluster to load the new settings. The (base) prefix to your command line prompt will have disappeared. At this point, you can load anaconda3 / miniconda3 / miniforge3, list, and activate your own environments.

[user@athena1 ~]$ module load miniconda3 [user@athena1 ~]$ conda info --envs # conda environments: pytorch-venv /lustre/home/user/.conda/envs/pytorch-venv mycondaenv /lustre/home/user/miniconda3/envs/mycondaenv base * /opt/miniconda3-py39_23.9.0 [user@athena1 ~]$ conda activate mycondaenv (mycondaenv) [user@athena1 ~]$

You can also create and activate a new environment using:

[user@athena1 ~]$ module load miniconda3 [user@athena1 ~]$ conda create --name myvenv python=3.11 [user@athena1 ~]$ conda activate myvenv (myvenv) [user@athena1 ~]$

Will huff be discontinued?

Yes. All nodes in huff will transition to athena and huff will be shut down.

Common errors

srun/sbatch: error: invalid partition specified: basic / GPU

The names of the partitions on athena are "cpu" and "gpu". Replace "basic" with "cpu" or "GPU" with "gpu" (case sensitive).

srun/sbatch: error: Unable to allocate resources: Invalid qos specification

Remove "--qos=gpu" from your script.

srun/sbatch: error: Unable to allocate resources: Requested node configuration is not available

This error means that the resources requested to run the job are not available. Please check that the number of CPUs (--cpus-per-task), memory (--mem), nodes (--nodes), and GPUs (--gres gpu) are within the specifications. Please review the hardware specifications of each node and consider the following:

  • The maximum number of CPUs on any node is equal to the number of cores minus two (reserved for the operating system). The effective maximum number of CPUs for any node on athena is 126. If your job has --cpus-per-task > 126 it won't ever run.

  • The maximum memory on any node is equal to the RAM size minus 8 GB (reserved for the operating system). The maximum memory available per node is 117G, 242G, 494G for 128GB, 256GB and 512GB nodes, respectively.

  • The number of GPUs vary per partition (e.g. gpu-v100 has 2 GPUs per node, gpu-a100 has 4 GPUs per node). You may not request a number of GPUs larger than the number of GPUs available in one node. You may not request GPUs in any of the CPU partitions.

Lmod has detected the following error: The following module(s) are unknown:

The software modules on athena have been updated to newer versions. You can run "module avail" to list all the modules available and their versions. Edit your script to load the correct version of the software module.

error while loading shared libraries: libXXXXX.so: cannot open shared object file: No such file or directory

You need to recompile your code on athena. If you still get an error regarding shared libraries after recompiling your code, please reach out to tsresearch@vcu.edu The library may need to be installed on athena.