700 Tokens per Second: New Chip Speeds Challenge Nvidia Dominance

For developers and enterprises relying on large language models, the options for running AI workloads are expanding beyond the traditional hardware ecosystem. A new software integration announced this week allows Google’s custom chips to support PyTorch, a widely used framework, without requiring significant code rewrites. This shift means that organizations can now deploy their existing PyTorch-based applications on Google’s tensor processing units, or TPUs, as a native option. The development signals a notable change in how major tech companies approach hardware compatibility for artificial intelligence tasks.

The core of this announcement is the release of new, PyTorch-native versions of the SGLang inference engine and the open-source vLLM engine. Previously, running these engines on TPUs required the use of Google’s own JAX framework. The new versions, known as SGLang-TorchTPU and vLLM-TorchTPU, eliminate that dependency. According to Zongwei Zhou, Meta’s PyTorch-TPU lead, the SGLang-TorchTPU system is already running in production at Meta. This deployment resulted from a three-way collaboration between Meta, Google, and the SGLang team, a group now operating under the startup RadixArk. Zhou described the new engine as providing a remarkably mature serving stack for Meta’s needs.

Meta’s involvement in this transition is strategic rather than incidental. Jana van Greunen, director of PyTorch engineering at Meta, stated that the company cannot be constrained by a single hardware architecture. She explained that TorchTPU enables Meta to deploy its PyTorch workloads directly to Google TPUs. Google confirmed that it co-developed this solution with Meta’s PyTorch team. This marks a significant step forward from earlier attempts, such as the PyTorch/XLA bridge, which Google now terms a predecessor. Reports from last year indicated that Meta research teams used TPUs through that older bridge between 2020 and 2023 before those contracts were canceled. The current push appears to be a more direct response to Meta’s preference for avoiding a migration to JAX.

While the specific location of Meta’s production TPUs remains unclear, industry talks reportedly included plans for Google Cloud TPU rentals in 2026, with purchases expected in 2027. Meta is also reported to have signed a multi-year rental agreement. The timing of this announcement is particularly noteworthy as it came one day after Amazon announced it would open-source TorchNeuron, its own native PyTorch backend for its Trainium2 and Trainium3 chips. Both moves appear designed to loosen the software lock-in associated with Nvidia’s hardware, offering developers more flexible and cost-effective alternatives for inference workloads.

Performance metrics suggest that the new TPU setup can compete with leading Nvidia hardware. A tuned vLLM configuration serving Moonshot AI’s Kimi K3 model, which has 2.8 trillion parameters, reportedly exceeds 700 tokens per second for a single user on the seventh-generation Ironwood TPU. This figure was provided by Inferact, a startup founded by the creators of vLLM that assisted Google in building the new integration. The test utilized 16 Ironwood chips with speculative decoding, a technique that speeds up inference by using a smaller draft model. In comparison, a similar setup on 16 Nvidia GB200s reached 452 tokens per second.

Beyond peak single-user speeds, the new kernels for Kimi K3 and Qwen 3.8 27B deliver throughput that is roughly 1.4 to 2 times higher than the GB200 baseline at small batch sizes, even without speculative decoding. These results indicate that TPUs are becoming a viable high-performance option for serving large open-weight models. The collaboration plans to continue adding support for new models, starting with DeepSeek V4.1 Flash and Reflection Beam. RadixArk has also stated that it is working toward Day 0 TPU support for new open models, meaning compatibility on the day of release, by the first quarter of 2027.

Further details on the technology and the broader partnership are expected to be shared at the SGLang Summit 2026. This event is scheduled for November 12 and 13 in San Francisco, where Google, Meta, and RadixArk plan to present more information. The convergence of these major players on a common software stack for alternative hardware suggests a maturing ecosystem for AI inference. As the industry moves away from a single dominant supplier, the focus on software compatibility and performance parity becomes critical for companies looking to optimize their AI infrastructure.