OpenAI Launches Training Spec to Boost Large-Scale AI
The protocol is designed to improve GPU performance as AI compute ramps up.
InfoWorld AI·
Late last year I got pulled into a capacity planning exercise for a global retailer that had wired a 70B model into their product search and recommendation pipeline. Every search query triggered an inference call. During holiday traffic their cluster was burning through GPU-hours at a rate that made their cloud finance team physically uncomfortable. They had already scaled from 24 to 48 H100s and latency was still spiking during peak hours. I was brought in to answer a simple question: Do we need 96 GPUs for the January sale or is something else going on? I started where I always start with these engagements: profiling. I instrumented the serving layer and broke the utilization data down by inference phase. What came back changed how I think about GPU infrastructure. During prompt processing — the phase where the model reads the entire user input in parallel — the H100s were running at 92% compute utilization. Tensor cores fully saturated. Exactly what you want to see on a $30K GPU. Bu
Read full articleThe protocol is designed to improve GPU performance as AI compute ramps up.
Why reasoning models dramatically increase token usage, latency, and infrastructure costs in production systems The post Inference Scaling (Test-Time Compute): Why Reasoning Models Raise Your Compute Bill appeared first on Towards Data Science.
The ongoing shift from generative AI (genAI) to agentic AI provides an opportunity for enterprises to move to more nimble and less expensive forms of computing, according to analysts. Early AI models were largely built on expensive GPUs from Nvidia and AMD that offered raw processing power. But newer agentic AI tools, rooted in business process and workflow management, can run on more efficient, cost-effective hardware. As a result, IT decision-makers who still think they require GPUs for anything AI-related need to reconsider their hardware options in terms of both cost and capabilities, analysts said. “A better way of thinking about this is the cost of AI compute and now agentic AI platform services or systems,” said Leonard Lee, principal analyst at Next Curve. “’AI computing’ or ‘accelerated computing’ has clearly transcended the GPU as an inference accelerator.” The new hardware options include CPUs and specialized AI chips, also known as ASICs in semiconductor parlance. Although
Learn how to eranscribe audio locally using Faster‑Whisper and Python, with an emphasis on privacy‑first and CPU/GPU‑ready.
In this tutorial, we explore kvcached, a dynamic KV-cache implementation on top of vLLM, to understand how dynamic KV-cache allocation transforms GPU memory usage for large language models. We begin by setting up the environment and deploying lightweight Qwen2.5 models through an OpenAI-compatible API, ensuring a realistic inference workflow. We then design controlled experiments where […] The post A Coding Implementation on kvcached for Elastic KV Cache Memory, Bursty LLM Serving, and Multi-Model GPU Sharing appeared first on MarkTechPost.
Astronomers are turning to GPUs to find needles in the galactic haystack.
In this tutorial, we build an end-to-end implementation around Qwen 3.6-35B-A3B and explore how a modern multimodal MoE model can be used in practical workflows. We begin by setting up the environment, loading the model adaptively based on available GPU memory, and creating a reusable chat framework that supports both standard responses and explicit thinking […] The post A Coding Implementation on Qwen 3.6-35B-A3B Covering Multimodal Inference, Thinking Control, Tool Calling, MoE Routing, RAG, and Session Persistence appeared first on MarkTechPost.
When a major insurer’s AI system takes months to settle a claim that should be resolved in hours, the problem usually isn’t the model in isolation. It’s the system around the model and the latency that system introduces at every step. Speed in enterprise AI isn’t about impressive benchmark numbers. It’s about whether AI can... The post AI latency is a business risk. Here’s how to manage it appeared first on DataRobot.