Red Hat Puts OpenShift LLM Inference Through STAC-AI Audit

Red Hat has put OpenShift into a financial-services LLM benchmark with an independent STAC-AI LANG6 audit, arguing that the Kubernetes layer is not the bottleneck for inference workloads that firms would otherwise be tempted to run close to bare metal.

In a Red Hat Blog post, senior performance and scalability engineer Sebastian Jug says Red Hat worked with NVIDIA and Supermicro to run the full STAC-AI LANG6 inference-only suite on OpenShift. The public STAC result page lists the system as audited and identifies the stack as a Supermicro SYS-222C-TN server with two NVIDIA RTX PRO 6000 Blackwell Series GPUs managed by Red Hat OpenShift.

The result matters because financial-services AI teams are under pressure to prove both performance and operating discipline. If a regulated firm can keep LLM inference inside a Kubernetes platform it already governs, it can reuse the same policy, audit, scheduling, and lifecycle controls that apply to other production workloads. If the platform adds visible latency or throughput penalties, that argument falls apart quickly.

What STAC audited

STAC-AI LANG6 focuses on LLM inference for financial-services workloads. NVIDIA’s technical backgrounder explains that the benchmark uses Llama 3.1 8B and 70B Instruct models with EDGAR-based datasets that model medium- and long-context summarization from SEC 10-K filings. It measures both batch mode, where requests are submitted together, and interactive mode, where requests arrive over time and latency becomes part of the result.

The STAC page says the audited stack used Red Hat OpenShift Container Platform 4.20, Red Hat Enterprise Linux CoreOS 9.6, NVIDIA TensorRT-LLM 1.2.0rc2 with the PyTorch backend, TensorRT 10.13.3.9, NVIDIA Model Optimizer 0.37.0 for NVFP4 quantization, and two RTX PRO 6000 Blackwell GPUs with 96 GiB of memory each. Those details are important: this was not a generic “AI on Kubernetes” claim, but a specific software and hardware configuration.

Among the public results, STAC reports 32.9 inferences per second and 5,549 words per second on the Llama-3.1-8B EDGAR4a batch workload. It also reports 5.28 inferences per second and 834 words per second on the Llama-3.1-70B EDGAR4b batch workload. For interactive mode, the same page lists a 30.0 inferences-per-second arrival rate on EDGAR4a and 5.00 inferences per second on EDGAR4b at the highest sustained arrival rates shown publicly.

Why the platform claim matters

Red Hat’s most pointed claim is operational rather than cosmetic. The company says these are the first audited STAC-AI results produced on a containerized Kubernetes platform, and says the audit did not show the OpenShift runtime or scheduler as the limiting factor in the inference pipeline. Red Hat quotes STAC commentary saying the container and orchestration layers did not appear to introduce material performance limitations in practice.

That is exactly the claim platform teams want benchmark evidence for. A financial institution evaluating LLM inference has to weigh throughput and latency against patching, access control, observability, tenant separation, model rollout, and reproducibility. A narrow bare-metal benchmark can be fast while still leaving the operations team with a second stack to secure and audit. Red Hat is trying to show that OpenShift can stay in the path without becoming the performance story.

The supporting pieces are familiar to OpenShift GPU operators. NVIDIA’s GPU Operator documentation for OpenShift covers the Node Feature Discovery and GPU Operator path that exposes GPUs to the platform. Red Hat’s post says that, after those operators are installed, GPUs are scheduled as nvidia.com/gpu resources and benchmark work can be driven declaratively through Kubernetes custom resources.

What this does and does not prove

The audit should not be read as a universal promise that every LLM application will behave the same way on every cluster. The public result covers a specific two-GPU Supermicro system, specific NVIDIA software versions, specific model sizes, and STAC’s financial-document workloads. Production teams still need to test their own prompts, context lengths, quantization choices, concurrency targets, retrieval stack, and governance controls.

It does, however, give enterprise AI teams a more concrete starting point. Instead of debating Kubernetes overhead in the abstract, they can compare their own inference service against an audited configuration with published throughput, latency, and stack details. For financial-services buyers, that shifts the conversation from “can Kubernetes run this?” toward “can our platform reproduce, monitor, and govern this class of workload?”

Bottom line

Red Hat’s OpenShift benchmark is a useful signal for regulated AI infrastructure: LLM inference performance is becoming a platform-engineering question, not just a GPU procurement question. The fastest accelerator still matters, but the winning deployment pattern also has to be reproducible, auditable, and maintainable by the teams that already run production systems.

Sources: Red Hat Blog, STAC SMCI260303 result page, NVIDIA Technical Blog on STAC-AI LANG6, and NVIDIA GPU Operator on OpenShift documentation.

Featured image: Tokyo Stock Exchange Market Center photograph by ehnmark, licensed CC BY 2.0 via Wikimedia Commons/Flickr; cropped, resized, and converted to WebP.