Orion Network Cabinets

[Vangele Ioannidis]

Tohelp you make an informed choice we have independently tested all the Quiet Rack ranges side by side enabling us to recommend the right Quiet Rack to meet your needs. You are also welcome to compare the racks for yourself in our unique Quiet Rack Demo Room.

Orion's Acoustic Wall Mount Comms Rack provides a cost effective solution for noisy network equipment in a wide range of heights. These Orion Quiet Racks are competitively priced and deliver a surprisingly effective noise reduction performance.

Please contact us on +44 (0)121 725 1900 and we will be happy to help with any technical questions on this range of quiet rack cabinets. Our staff have many years experience in the IT market, and in particular with soundproofed rackmount enclosures - we have compared and tested every 19" rack that we supply. Contact us for a quick response.

"The level of service was above the competitors - quick reply-time in all our inquiries made us choose Acousti Products. And our new noise product solution has improved our working environment a lot."

Petri Halonen, Atomi Advertising Ltd, Finland

Frontier is a HPE Cray EX supercomputer located at the Oak Ridge Leadership Computing Facility. With a theoretical peak double-precision performance of approximately 2 exaflops (2 quintillion calculations per second), it is the fastest system in the world for a wide range of traditional computational science applications. The system has 74 Olympus rack HPE cabinets, each with 128 AMD compute nodes, and a total of 9,408 AMD compute nodes.

The 8 GCDs contained in the 4 MI250X will show as 8 separate GPUs according to Slurm, ROCR_VISIBLE_DEVICES, and the ROCr runtime, so from this point forward in the quick-start guide, we will simply refer to the GCDs as GPUs.

By default, Frontier reserves the first core in each L3 cache region. Frontier uses low-noise mode,which constrains all system processes to core 0. Low-noise mode cannot be disabled by users.In addition, Frontier uses SLURM core specialization (-S 8 flag at job allocation time, e.g., sbatch)to reserve one core from each L3 cache region, leaving 56 allocatable cores. Set -S 0 at job allocation to override this setting.

On Frontier, there are two major types of nodes you will encounter: Login and Compute. While these aresimilar in terms of hardware (see: Frontier Compute Nodes), they differ considerably in their intendeduse.

Each Frontier compute node contains 4 AMD MI250X. The AMD MI250X has a peak performance of 53 TFLOPS in double-precision for modeling and simulation. Each MI250X contains 2 GPUs, where each GPU has a peak performance of 26.5 TFLOPS (double-precision), 110 compute units, and 64 GB of high-bandwidth memory (HBM2) which can be accessed at a peak of 1.6 TB/s. The 2 GPUs on an MI250X are connected with Infinity Fabric with a bandwidth of 200 GB/s (in each direction simultaneously).

By default, connecting to Frontier will automatically place the user on a random login node. If you need to access a specific login node, you will ssh to that node after your intial connection to Frontier.

On Alpine, there was no user-exposed concept of file striping, the process of dividing a file between the storage elements of the filesystem. Orion uses a feature called Progressive File Layout (PFL) that changes the striping of files as they grow. Because of this, we ask users not to manually adjust the file striping. If you feel the default striping behavior of Orion is not meeting your needs, please contact he...@olcf.ornl.gov.

The OLCF provides a wrapper for the lfs setstripe command that simplifies the process of striping files. The wrapper will enforce that certain settings are used to ensure that striping is done correctly. This will help to ensure good performance for users as well as prevent filesystem issues that could arise from incorrect striping practices. The wrapper is accessible via the lfs-wrapper module and will soon be added to the default environment on Frontier.

Orion is different than other Lustre filesystems that you may have used previously. To make effective use of Orion and to help ensure that the filesystem performs well for all users, it is important that you do the following:

Please note that the HPSS is not mounted directly onto Frontier nodes. There are two main methods for accessing and moving data to/from the HPSS. The first is to use the command line utilities hsi and htar. The second is to use the Globus data transfer service. See HPSS Data Archival System for more information on both of these methods.

To use the NVMe, users must request access during job allocation using the -C nvme option to sbatch, salloc, or srun. Once the devices have been granted to a job, users can access them at /mnt/bb/. Users are responsible for moving data to/from the NVMe before/after their jobs. Here is a simple example script:

Globus has restriction of 8 active transfers across all the users. Each user has a limit of 3 active transfers, so it is required to transfer a lot of data on each transfer than less data across many transfers.

If a folder is constituted with mixed files including thousands of small files (less than 1MB each one), it would be better to tar the smallfiles. Otherwise, if the files are larger, Globus will handle them.

The AMD Instinct MI200 is built on advanced packaging technologiesenabling two Graphic Compute Dies (GCDs) to be integratedinto a single package in the Open Compute Project (OCP) Accelerator Module (OAM)in the MI250 and MI250X products.Each GCD is build on the AMD CDNA 2 architecture.A single Frontier node contains 4 MI250X OAMs for the total of 8 GCDs.

The Slurm workload manager and the ROCr runtime treat each GCD as a separate GPUand visibility can be controlled using the ROCR_VISIBLE_DEVICES environment variable.Therefore, from this point on, the Frontier guide simply refers to a GCD as a GPU.

The 110 CUs in each GPU deliver peak performance of 26.5 TFLOPS in double precision.Also, each GPU contains 64 GB of high-bandwidth memory (HBM2) accessible at a peakbandwidth of 1.6 TB/s.The 2 GPUs in an MI250X are connected with [4x] GPU-to-GPU Infinity Fabric linksproviding 200+200 GB/s of bandwidth.(Consult the diagram in the Frontier Compute Nodes section for informationon how the accelerators are connected to each other, to the CPU, and to the network.

The MI250X has different denormal handling for FP16 and BF16 datatypes, which is relevant for ML training. Prefer using the BF16 over the FP16 datatype for ML models as you are more likely to encounter denormal values with FP16 (which get flushed to zero, causing failure in convergence for some ML models). See more in Using reduced precision (FP16 and BF16 datatypes).

HIP has two kinds of memory allocations, coarse grained and fine grained, with tradeoffs between performance and coherence. Particularly relevant if you want to ues the hardware FP atomic instructions. See more in Floating-Point (FP) Atomic Operations and Coarse/Fine Grained Memory Allocations.

Modules with dependencies are only available when the underlying dependencies, such as compiler families, are loaded. Thus, module avail will only display modules that are compatible with the current state of the environment. To search the entire hierarchy across all possible dependencies, the spider sub-command can be used as summarized in the following table.

Cray, AMD, and GCC compilers are provided through modules on Frontier. The Cray and AMD compilers are both based on LLVM/Clang. There is also a system/OS versions of GCC available in /usr/bin. The table below lists details about each of the module-provided compilers. Please see the following Compiling section for more detailed inforation on how to compile using these modules.

Cray provides PrgEnv- modules (e.g., PrgEnv-cray) that load compatible components of a specific compiler toolchain. The components include the specified compiler as well as MPI, LibSci, and other libraries. Loading the PrgEnv- modules also defines a set of compiler wrappers for that compiler toolchain that automatically add include paths and link in libraries for Cray software. Compiler wrappers are provided for C (cc), C++ (CC), and Fortran (ftn).

Use the -craype-verbose flag to display the full include and link information used by the Cray compiler wrappers. This must be called on a file to see the full output (e.g., CC -craype-verbose test.cpp).

Cray, AMD, and GCC compilers are provided through modules on Frontier. The Cray and AMD compilers are both based on LLVM/Clang. There is also a system/OS versions of GCC available in /usr/bin. The table below lists details about each of the module-provided compilers.

The gcc-native compiler module was introduced in the December 2023 release of the HPE/Cray Programming Environment (CrayPE) and replaces gcc.gcc provides GCC installations that were packaged within CrayPE, while gcc-native provides GCC installations outside of CrayPE.

When loading non-default versions of Cray-provided components, please see Understanding the Compatibility of Compilers, ROCm, and Cray MPICH for information about loading a set of compatible Cray modules.

If you need to add the tools and libraries related to ROCm, the framework for targeting AMD GPUs, to your path, you will need to use a version of ROCm that is compatible with your programming environment.ROCm can be loaded with: module load rocm/X.Y.Z, or to load the default ROCm version, module load rocm.

3a8082e126