Skip to content

Software

Note

The latest software available on ASPIRE 2A can be found here.

Software Development

Cray Programming Environment

  • AMD AOCC

  • GNU GCC

  • HPE CCE

  • Intel oneAPI

  • NVHPC

ASPIRE 2A is an HPE Cray system that features the Cray Programming Environment (CPE). It includes a set of compiler wrappers (cc, CC, and ftn for C, C++, and Fortran, respectively) that automatically apply optimal flags and link necessary libraries.

To switch from CPE to another programming environment (e.g., GNU GCC), use:

module swap PrgEnv-cray PrgEnv-gnu

To verify the current compiler, run:

cc --version

Example output:

gcc (GCC) 11.2.0 20210728 (Cray Inc.)
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Refer to the mapping below to see which module and compilers to use for different toolchains.

Toolchain Module Compiler Compiler Wrapper
GNU GCC PrgEnv-gnu gcc cc
g++ CC
gfortran ftn
Intel oneAPI PrgEnv-intel

icc (Intel classic)

icx (Intel oneAPI)

 cc

icpc (Intel classic)

icpx (Intel oneAPI)

 CC

ifort (Intel classic)

ifx (Intel oneAPI)

 ftn
NVHPC PrgEnv-nvhpc nvc cc
nvc++ CC
nvfortran ftn
HPE CCE PrgEnv-cray craycc cc
crayCC CC
crayftn ftn

When using CPE to compile applications that follow the configure-make procedure, you often need to manually specify the compilers. This is because most software packages are not pre-configured for CPE.

For example, you can manually specify them as follows:

./configure CC=cc CXX=CC FC=ftn

Programming

  • Nvidia CUDA

  • Go

  • Python

  • Java

  • Julia

  • Octave

  • R

Libraries

MPI libraries

  • Cray MPICH

  • Open MPI

Scientific Libraries

  • AMD AOCL

  • BLIS

  • Cray LibSci

  • FFTW

  • GSL

  • Intel MKL

  • PETSc

I/O Libraries

  • HDF5

  • NetCDF

Profiling & Debugging

  • ARM Forge

  • Cray Stat

  • gdb4hpc

  • perftools

  • valgrind4hpc

Performance Monitoring

Monitoring GPU Activity with nvidia-smi

When your job is running on GPU nodes, you can monitor GPU activity using the nvidia-smi command by following these steps:

  1. Find your job ID and compute node hostname. Use the qstat command:

    qstat -n1

    Example output:

                                                                     Req'd  Req'd   Elap
Job ID               Username Queue    Jobname    SessID NDS TSK Memory Time  S Time
-------------------- -------- -------- ---------- ------ --- --- ------ ----- - -----
<job-id>      ***      g3       ***         ***   1  64  440gb 02:00 R 00:01 <hostname>/0*64

    Note down the <job-id> and the <hostname>.

  2. Set the job environment variable.

    export PBS_JOBID=<job-id>

  3. Connect to the compute node.

    ssh <hostname>

  4. Run nvidia-smi in the compute node to check GPU activity.

    nvidia-smi

    Example output:

    Fri Jun  6 15:47:43 2025
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.154.05             Driver Version: 535.154.05   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA A100-SXM4-40GB          On  | **************** Off |                    0 |
| N/A   63C    P0             291W / 400W |   8010MiB / 40960MiB |     98%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA A100-SXM4-40GB          On  | **************** Off |                    0 |
| N/A   56C    P0             285W / 400W |   8082MiB / 40960MiB |    100%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
|   2  NVIDIA A100-SXM4-40GB          On  | **************** Off |                    0 |
| N/A   62C    P0             306W / 400W |   8082MiB / 40960MiB |    100%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
|   3  NVIDIA A100-SXM4-40GB          On  | **************** Off |                    0 |
| N/A   58C    P0             285W / 400W |   8010MiB / 40960MiB |     98%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A   3475239      C   ...c0-8de7-bc7e8d4ef790/files/vasp_std     8002MiB |
|    1   N/A  N/A   3475240      C   ...c0-8de7-bc7e8d4ef790/files/vasp_std     8074MiB |
|    2   N/A  N/A   3475241      C   ...c0-8de7-bc7e8d4ef790/files/vasp_std     8074MiB |
|    3   N/A  N/A   3475242      C   ...c0-8de7-bc7e8d4ef790/files/vasp_std     8002MiB |
+---------------------------------------------------------------------------------------+

    • The Processes section at the bottom lists the processes currently using each GPU.

    • The Volatile GPU-Util column shows the GPU utilization percentage — higher values indicate more efficient usage.

    • The Memory-Usage column shows how much GPU memory is being used.

    Monitoring these metrics helps assess whether your job is efficiently using GPU resources.

  5. Optional: Continuous monitoring. To refresh the GPU status every few seconds, use:

    watch -n 2 nvidia-smi

    Replace 2 with your preferred refresh interval in seconds. Press Ctrl+C to exit.