Addressing the Inference Bottleneck

Agent-orchestrated inference stacks eliminate the human engineering bottleneck and ship clean, performant code from day one.

INT21’s agent swarms have built inference stacks for video, music, and speech generation, and these engineering samples are now live. Start generating your own videos, music, and text-to-speech content—no API key required.

Key Takeaways

• Agent-orchestrated inference stacks directly address the talent bottleneck of finding and hiring highly specialized inference engineers.
• Production code matters as much as production performance. AI-generated infrastructure typically accumulates cleanup debt. INT21’s video, music, and speech stacks require no human review and outperform state-of-the-art inference solutions in our tests.
• This is the second category proof for self-improving infrastructure, scaling from GPU kernels in PTX Kernel Factory to sophisticated inference systems.
• Try the free live demos for video, music, and speech generation.

The Bottleneck

A new AI model seems to be unveiled each week, multiplying faster than the number of people needed to run them. Every model added to an inference stack requires an engineer with deep expertise in GPU and framework optimization, and these specialists are increasingly hard to find. For token factories and inference providers, this talent constraint is one of the biggest bottlenecks to scaling AI infrastructure.

The core question is whether AI agents can handle this dilemma. That raises a second question: does agent-generated infrastructure ship clean, or does it contain invisible technical debt?

GitClear analyzed 211 million lines of code and found that duplicated code blocks increased roughly 10x while refactoring dropped from 24.1% to 9.5%. In production repositories, more than 15% of AI-generated commits introduce issues that often survive for months. GitHub Copilot’s command-injection vulnerability in Hysteria 2, for example, was not caught for three weeks. AI-generated code can ship faster, but technical debt can accumulate invisibly and erode trust in what is being built.

What the Inference Engine Factory Does

INT21 is addressing this problem by taking the first step toward eliminating the need for a handmade inference engine. Our self-improving agent swarms orchestrate the entire inference stack end to end: CUDA and PTX kernels, megakernels, a Rust CUDA driver, and serving integration. The stack is designed and optimized from first principles, with no PyTorch, kernel-library abstractions, or inference frameworks.

Video, music, and speech generation are highly complex inference workloads. They involve multistage, multiscale network architectures and demand sophisticated kernels and runtimes. Proving that agents can build clean production stacks for these workloads validates that they can handle broader infrastructure optimization challenges. When a 768p video can be generated and returned immediately, that is self-improving infrastructure in action.

The underlying models are MiniMax H3, which holds the first three positions across the Artificial Analysis video generation leaderboards, MiniMax Music 3 for music generation, and Fish Audio S2 Pro for text-to-speech. The agent-orchestrated stack produces top-tier performance without requiring human expert tuning.

These are the first engineering samples from Inference Engine Factory, part of INT21’s self-improving infrastructure. It builds on our first product, PTX Kernel Factory. PTX demonstrated that agents could understand and build low-level GPU instructions. Inference Engine Factory shows that agents can orchestrate entire production systems.

Kernels are isolated problems with clear boundaries. Inference stacks are systems with complex interactions and architectural decisions that ripple across layers. This is where one would expect AI-generated code to break down and require cleanup.

Code Quality Over AI Slop

INT21’s video, music, and speech generation stacks show that code built by self-improving agent swarms requires no human code review and outperforms state-of-the-art inference solutions at low-latency inference under the same concurrency settings. We tested across MiniMax H3, MiniMax Music 3, and Fish Audio S2 Pro to demonstrate the Factory’s breadth.

MiniMax H3 — 8x B200

The benchmark uses the official BF16 checkpoint merged with Turbo LoRA in four steps, generating 15 seconds of H.264/AAC media. The result is the average inference time across four different prompts.

MiniMax Music 3 — 2x H100

The benchmark uses 14 requests across two sessions, all producing equivalent, valid 32-kHz stereo WAV files lasting exactly 10 seconds.

Highlight: Cooperative megakernel.

Fish Audio S2 Pro — 1x H100

The benchmark uses the full Seed-TTS-Eval English test set across 1,088 samples.

Highlights: Slow and fast autoregressive megakernels, plus a 316-node CUDA graph for codec operations.

All INT21 engineering-sample engines are built directly on the CUDA driver and can be deployed without additional dependencies. CUDA Runtime containers are not required. Each engine contains roughly 20,000 lines of Rust and 10,000 lines of CUDA and PTX kernel code.

What This Means for Self-Improving Infrastructure

The inference-engineer bottleneck is significant: every new model, hardware target, and architectural optimization demands rare specialist time. INT21’s video, music, and speech stacks show that these constraints should not hold infrastructure back. Agents can orchestrate entire production systems beyond kernels and ship code that is correct from day one.

This is what self-improving infrastructure looks like. Humans set the objectives, define the tests, and authorize what ships. The optimization work itself happens autonomously, faster and better than manual tuning.

For infrastructure teams competing on AI, the advantage goes to whoever runs the best autonomous optimization loop. What we are announcing today is an engineering sample of the Factory that demonstrates self-improving infrastructure works: compute improves infrastructure faster and better than human expertise alone.

Try the live demos to generate video, music, and speech instantly and see self-improving infrastructure handle complex inference work. Video, music, and text-to-speech generations are limited to 15 seconds, 3 minutes, and 30 seconds respectively. Each user receives two free generations per 24 hours.

We especially thank the NVIDIA Inception Program for sponsoring eight H100 GPUs for SwarmOS development.

Learn more: What Is Self-Improving Infrastructure? Defining a New Category