Transitioning from teal to athena
$userHow do I access athena?
Users with an existing account on teal already have an account on athena. Note that all access from off-campus requires the use of the VCU VPN. You should connect via SSH using your VCU eID and VCU password (not the teal password) on the command line prompt on Windows, Mac, or Linux. There are two login nodes: athena1.hprc.vcu.edu and athena2.hprc.vcu.edu. You can SSH into either of them by running:
ssh yourvcueid@athena.hprc.vcu.eduIf you wish you can set up a passwordless access using SSH keys. While most operations in a cluster are based on the command line, there is also an option to using X11 Forwarding to Run Remote Graphical Applications.
You can also see the new web interface based on Open OnDemand, which is a web portal to the Athena cluster providing a graphical user interface for the most commonly used applications including Matlab, RStudio, Jupyter lab, ParaView, SAS, VS Code, and generative AI services.
Do I need to copy my files from teal to athena?ncpus
Yes. The files on teal that you still need for running jobs need to be copied to athena. Files on teal will not be removed. There are two options to copy files:
If the size of the data is relatively small. You can download the files from teal to your computer and then upload them to athena. See this page about transferring files to/from the cluster.
If the size of the data is large or you don't feel comfortable doing it by yourself, please reach out to tsresearch@vcu.edu indicating the files and locations that you want to copy. It is important to agree on a cutoff date to guarantee that the data on teal is not modified while/after is copied to athena.
We recommend research groups sharing files and folders to have a shared group folder among multiple users, please reach out to tsresearch@vcu.edu and we'll create one for your group or copy it from teal.
Do I need to make any changes to run my scripts?
Yes. teal uses Sun Grid Engine (SGE) and athena uses Slurm, which is a new job scheduler. In the following, we'll show the main differences to help you convert your SGE scripts to Slurm.
Commands
User Command | SGE | SLURM |
|---|---|---|
Interactive Login |
|
|
Job submission |
|
|
Job deletion |
|
|
Job status by job |
|
|
Job status by user |
|
|
Queue list |
|
|
List nodes |
|
|
Environment Variables
Information | SGE | SLURM |
|---|---|---|
Job ID |
|
|
Submit directory |
|
|
Submit host |
|
|
Node list |
|
|
Job Array Index |
|
|
Job Specification
Information | SGE | SLURM |
|---|---|---|
Script directive |
|
|
queue/partition |
|
|
count of nodes | N/A |
|
CPU count |
|
|
stdout file |
|
|
stderr file |
|
|
copy environment |
|
|
event notification |
|
|
send notification email |
|
|
job name |
|
|
Restart job |
|
|
set working directory |
|
|
Resource sharing |
|
|
Memory size |
|
|
Charge to an account |
|
|
Tasks per node | fixed rule in parallel environment (PE) |
|
Job dependency |
|
|
Job project |
|
|
Job host preferences |
|
|
Job arrays |
|
|
Generic Resources |
|
|
Submit Scripts
SGE | SLURM |
|---|---|
#$ -N test
|
#SBATCH --job-name test myprogram inputfile.txt |
Do I need to recompile my source code for applications like MPI?
Yes. If your code needs to be compiled (typically using "make"), you need to recompile the code on athena. The reason is that any compiled code on teal may not work on athena because the linking to different versions of shared libraries.
If you receive an error like "error while loading shared libraries: libXXXXX.so: cannot open shared object file: No such file or directory" when running a program that worked on teal, you need to recompile your code on athena. If you still get an error regarding shared libraries after recompiling your code, please reach out to tsresearch@vcu.edu The library may need to be installed on athena.
I use anaconda / miniconda / miniforge, what do I need to do?
Installations of anaconda / miniconda / miniforge on the user's home folder are strongly discouraged because they slow down significantly the user's session. Therefore, we recommend using the anaconda / miniconda / miniforge we installed as modules. You can use module load anaconda3 / miniconda3 / miniforge3, and then create and activate your own environments.
[user@athena1 ~]$ module load miniconda3
[user@athena1 ~]$ conda create --name myvenv python=3.11
[user@athena1 ~]$ conda activate myvenv
(myvenv) [user@athena1 ~]$Will teal be discontinued?
No. However, the majority of the compute nodes in teal will move to athena. There are a few older nodes that can't be moved to athena. Therefore, teal will continue running for a little bit longer but with significantly smaller compute capacity. All data in teal will be preserved.
Common errors
srun/sbatch: error: invalid partition specified: basic / GPU
The names of the partitions on athena are "cpu" and "gpu" (case sensitive).
srun/sbatch: error: Unable to allocate resources: Invalid qos specification
Remove any "--qos" from your script.
srun/sbatch: error: Unable to allocate resources: Requested node configuration is not available
This error means that the resources requested to run the job are not available. Please check that the number of CPUs (--cpus-per-task), memory (--mem), nodes (--nodes), and GPUs (--gres gpu) are within the specifications. Please review the hardware specifications of each node and consider the following:
The maximum number of CPUs on any node is equal to the number of cores minus two (reserved for the operating system). The effective maximum number of CPUs for any node on athena is 126. If your job has
--cpus-per-task> 126 it won't ever run.The maximum memory on any node is equal to the RAM size minus 8 GB (reserved for the operating system). The maximum memory available per node is 117G, 242G, 494G for 128GB, 256GB and 512GB nodes, respectively.
The number of GPUs vary per partition (e.g. gpu-v100 has 2 GPUs per node, gpu-a100 has 4 GPUs per node). You may not request a number of GPUs larger than the number of GPUs available in one node. You may not request GPUs in any of the CPU partitions.
Lmod has detected the following error: The following module(s) are unknown:
The software modules on athena have been updated to newer versions. You can run "module avail" to list all the modules available and their versions. Edit your script to load the correct version of the software module.
error while loading shared libraries: libXXXXX.so: cannot open shared object file: No such file or directory
You need to recompile your code on athena. If you still get an error regarding shared libraries after recompiling your code, please reach out to tsresearch@vcu.edu The library may need to be installed on athena.