TACC Taps Dell And 63K Intel Xeon CPU Max Cores For 10-Petaflop Stampede3 Supercomputer

Modern scientific breakthroughs aren't solely relegated to a lab these days, but also come by way of the massive compute resources that modern data centers full of servers can muster. For researchers, maintaining open access to these invaluable complexes has been vital. The Stampede and Stampede2 supercomputer systems, for example, have helped lead this charge for the U.S. National Science Foundation’s (NSF) ACCESS ecosystem over the past decade. Now the Texas Advanced Computing Center (TACC) has announced its latest incarnation, Stampede3.

Nearly 63,000 Xeon CPU Max Cores Crunching HPC Compute

This supercomputer is driven by 1,858 compute nodes from Dell Technologies that contribute a combined total of more than 140,000 cores and over 330 terabytes of system memory. This configuration includes 560 Intel Xeon CPU Max Series processors, which it says contributes nearly 63,000 of the cores and around 4 petaflops of compute. We can extrapolate here to presume these are all the flagship Xeon 9480 56-core parts, and that total threads are being used for that figure. The Xeon CPU Max Series includes 64GB of on-package high bandwidth memory (HBM2e) which particularly benefits the kinds of data-intensive high-performance compute (HPC) workloads that this system will run.

Dan Stanzione, executive director of TACC, remarked, “We will continue our long partnership with Dell and Intel and leverage the NSF investments in Stampede2 for this new science resource using the latest technology processors with high bandwidth memory, and making Intel graphics processing units widely available to the NSF open science community.”

Intel Data Center GPU Max Graphics And AI Subsystem

Stampede3 also adds a new graphics and AI processing subsystem. This is comprised of ten Dell PowerEdge XE9640 servers which host 40 Intel Data Center GPU Max Series processors. Formerly code-named Ponte Vecchio, these graphics processors are based on Xe-HPC architecture, supporting 128 Xe-cores in a 2-stack Intel GPU Max Series configuration. The GPU Max Series can achieve 104 TFLOPS of FP16 compute, or half that for FP32 and FP64 workloads, but also drives tensor operations with its XMX units to 839 TFLOPS in BF16 and FP16 formats, or to 1678 TOPS INT8.

“We’re proud to contribute the technologies for TACC’s Stampede3 system, which will empower the research community through improved performance with no code changes through our Max Series CPUs, and offer a path forward for open, accelerated computing through our Max Series GPUs,” said Mark Hirsch, Intel vice president, general manager of super compute product management, customer architecture and engineering. “The combination of the two processors will forge a path for scientists and engineers to gain unprecedented productivity.”

The Stampede3 supercomputer is made possible thanks to an NSF award for $10M to fund the new server hardware, but the system will also reintegrate parts from Stampede2. These include over 1,000 existing 2nd Gen Intel Xeon Scalable Processor nodes to “support high-throughput computing, interactive workloads, and other smaller workloads.”

Linking Stampede3 Together And Research Aspirations

All these nodes are linked together via Intel’s 400Gb/s Omni-Path Fabric technology. This provides a 24TB/s backplane to make low-latency connections possible, allowing applications to scale with greater efficiency. All told, the system is projected to achieve nearly 10 petaflops of peak compute capacity, but we will have to see where it ultimately lands on rankings like the TOP500 list where Stampede2 slots in at #56 as of June 2023.

Stampede3 will continue to support the open science community once it enters full production in early 2024. Dave Lincoln, vice president of compute systems and solutions for Dell Technologies, said, “Stampede3 will be a key to discovery as its researchers use the system to study everything from the creation of distant galaxies to finding treatments for once thought incurable diseases. This Dell Technologies-powered system will help TACC continue our long-shared history of deploying the latest technologies to enable curiosity and advance society.”

According to the press release, Stampede2 has enabled more than 11,000 users to run over 11 million simulations and data analysis jobs for over 3,000 funded projects during its lifespan. We look forward to Stampede3 expanding this legacy and enabling future breakthroughs across a broad range of scientific disciplines.