Transitioning from teal to athena

Transitioning from teal to athena

$userHow do I access athena?

Users with an existing account on teal already have an account on athena. Note that all access from off-campus requires the use of the VCU VPN. You should connect via SSH using your VCU eID and VCU password (not the teal password) on the command line prompt on Windows, Mac, or Linux. There are two login nodes: athena1.hprc.vcu.edu and athena2.hprc.vcu.edu. You can SSH into either of them by running:

ssh yourvcueid@athena.hprc.vcu.edu

If you wish you can set up a passwordless access using SSH keys. While most operations in a cluster are based on the command line, there is also an option to using X11 Forwarding to Run Remote Graphical Applications.

You can also see the new web interface based on Open OnDemand, which is a web portal to the Athena cluster providing a graphical user interface for the most commonly used applications including Matlab, RStudio, Jupyter lab, ParaView, SAS, VS Code, and generative AI services.

Do I need to copy my files from teal to athena?ncpus

Yes. The files on teal that you still need for running jobs need to be copied to athena. Files on teal will not be removed. There are two options to copy files:

  • If the size of the data is relatively small. You can download the files from teal to your computer and then upload them to athena. See this page about transferring files to/from the cluster.

  • If the size of the data is large or you don't feel comfortable doing it by yourself, please reach out to tsresearch@vcu.edu indicating the files and locations that you want to copy. It is important to agree on a cutoff date to guarantee that the data on teal is not modified while/after is copied to athena.

We recommend research groups sharing files and folders to have a shared group folder among multiple users, please reach out to tsresearch@vcu.edu and we'll create one for your group or copy it from teal.

Do I need to make any changes to run my scripts?

Yes. teal uses Sun Grid Engine (SGE) and athena uses Slurm, which is a new job scheduler. In the following, we'll show the main differences to help you convert your SGE scripts to Slurm.

Commands

User Command

SGE

SLURM

User Command

SGE

SLURM

Interactive Login

qlogin

srun -p $partition_name --pty bash

Job submission

qsub $script_file

sbatch $script_file

Job deletion

qdel $job_id

scancel $job_id

Job status by job

qstat -u \* -j $job_id

squeue $job_id

Job status by user

qstat -u $user_name

squeue -u $user_name

Queue list

qstat -f

squeue

List nodes

qhost

scontrol show nodes

Environment Variables

Information

SGE

SLURM

Information

SGE

SLURM

Job ID

$JOB_ID

$SLURM_JOBID

Submit directory

$SGE_O_WORKDIR

$SLURM_SUBMIT_DIR

Submit host

$SGE_O_HOST

$SLURM_SUBMIT_HOST

Node list

$PE_HOSTFILE

$SLURM_JOB_NODELIST

Job Array Index

$SGE_TASK_ID

$SLURM_ARRAY_TASK_ID

Job Specification

Information

SGE

SLURM

Information

SGE

SLURM

Script directive

#$

#SBATCH

queue/partition

-q $queue

-p $partition

count of nodes

N/A

-N [min[-max]]

CPU count

-pe smp <count> OR -pe orte <count>

-n <count>

stdout file

-o <file_name>

-o <file_name>

stderr file

-e <file_name>

-e <file_name>

copy environment

-V

--export=(ALL/NONE/<variables>)

event notification

-m abe

--mail-type=<events>

send notification email

-M <address>

--mail-user=<address>

job name

-N <name>

--job-name=<name>

Restart job

-r [yes|no]

--requeue OR --no-requeue

set working directory

-wd <dir_name>

--workdir=<dir_name>

Resource sharing

-l exclusive

--exclusive OR —shared

Memory size

-l mem_free=[memory][K|M|G]

--mem=[mem][M|G|T] OR --mem-percpu=[mem][M|G|T]

Charge to an account

-A <account>

--account=<account>

Tasks per node

fixed rule in parallel environment (PE)

--tasks-per-node=<count>

Job dependency

-hold_jid <job_id> OR -hold_jid=<job_name>

--depend=[<state>:]<job_id>

Job project

-P <name>

--wckey=<name>

Job host preferences

-q <queue>@<node>

--nodelist=<nodes> AND/OR —exclude=<nodes>

Job arrays

-t $array_spec

--array=$array_spec

Generic Resources

-l $resource=$value

--gres=<resource_spec> (e.g. --gres gpu:1 )

Submit Scripts

SGE

SLURM

SGE

SLURM

#!/bin/bash

#$ -N test
#$ -j y
#$ -o test.output
#$ -cwd
#$ -M MYUSER@vcu.edu
#$ -m bea
#$ -P cpu
#$ -l mem_free=8G
#$ -pe smp 1

myprogram inputfile.txt

#!/bin/bash

#SBATCH --job-name test
#SBATCH --output test."%j".out
#SBATCH --error test."%j".err
#SBATCH --mail-user MYUSER@vcu.edu
#SBATCH --mail-type=ALL
#SBATCH --partition cpu
#SBATCH --mem 8G
#SBATCH --ntasks 1
#SBATCH --cpus-per-task 1

myprogram inputfile.txt

Do I need to recompile my source code for applications like MPI?

Yes. If your code needs to be compiled (typically using "make"), you need to recompile the code on athena. The reason is that any compiled code on teal may not work on athena because the linking to different versions of shared libraries.

If you receive an error like "error while loading shared libraries: libXXXXX.so: cannot open shared object file: No such file or directory" when running a program that worked on teal, you need to recompile your code on athena. If you still get an error regarding shared libraries after recompiling your code, please reach out to tsresearch@vcu.edu The library may need to be installed on athena.

I use anaconda / miniconda / miniforge, what do I need to do?

Installations of anaconda / miniconda / miniforge on the user's home folder are strongly discouraged because they slow down significantly the user's session. Therefore, we recommend using the anaconda / miniconda / miniforge we installed as modules. You can use module load anaconda3 / miniconda3 / miniforge3, and then create and activate your own environments.

[user@athena1 ~]$ module load miniconda3 [user@athena1 ~]$ conda create --name myvenv python=3.11 [user@athena1 ~]$ conda activate myvenv (myvenv) [user@athena1 ~]$

Will teal be discontinued?

No. However, the majority of the compute nodes in teal will move to athena. There are a few older nodes that can't be moved to athena. Therefore, teal will continue running for a little bit longer but with significantly smaller compute capacity. All data in teal will be preserved.

Common errors

srun/sbatch: error: invalid partition specified: basic / GPU

The names of the partitions on athena are "cpu" and "gpu" (case sensitive).

srun/sbatch: error: Unable to allocate resources: Invalid qos specification

Remove any "--qos" from your script.

srun/sbatch: error: Unable to allocate resources: Requested node configuration is not available

This error means that the resources requested to run the job are not available. Please check that the number of CPUs (--cpus-per-task), memory (--mem), nodes (--nodes), and GPUs (--gres gpu) are within the specifications. Please review the hardware specifications of each node and consider the following:

  • The maximum number of CPUs on any node is equal to the number of cores minus two (reserved for the operating system). The effective maximum number of CPUs for any node on athena is 126. If your job has --cpus-per-task > 126 it won't ever run.

  • The maximum memory on any node is equal to the RAM size minus 8 GB (reserved for the operating system). The maximum memory available per node is 117G, 242G, 494G for 128GB, 256GB and 512GB nodes, respectively.

  • The number of GPUs vary per partition (e.g. gpu-v100 has 2 GPUs per node, gpu-a100 has 4 GPUs per node). You may not request a number of GPUs larger than the number of GPUs available in one node. You may not request GPUs in any of the CPU partitions.

Lmod has detected the following error: The following module(s) are unknown:

The software modules on athena have been updated to newer versions. You can run "module avail" to list all the modules available and their versions. Edit your script to load the correct version of the software module.

error while loading shared libraries: libXXXXX.so: cannot open shared object file: No such file or directory

You need to recompile your code on athena. If you still get an error regarding shared libraries after recompiling your code, please reach out to tsresearch@vcu.edu The library may need to be installed on athena.