Featured Summary:
- Microsoft is preparing Maia 300 for long-context reasoning and agents
- Meta is redesigning chip communication around large recommendation workloads
- Amazon and Google are scaling custom accelerators across training and inference
- AI development is moving deeper into hardware, networking and system design
Microsoft plans to deploy Maia at multi-gigawatt scale beginning in 2027 and expects the system to carry a growing share of inference across its products and services.
Maia 300 is being developed for long-context reasoning, low-latency conversations and agents that stay active through multiple steps. Microsoft says operating Maia 200 in production informed the next chip’s design.
Meta is attacking a different problem. Its MTIA 300 places dedicated networking beside the compute silicon to move data through large recommendation models without consuming the same resources used for training.
Amazon has made Trainium3 available through EC2 for reasoning, multimodal and agent workloads.
Google, meanwhile, has separated its latest TPU generation into TPU 8t for training and TPU 8i for inference.
None of those systems removes Nvidia from the picture. Nvidia remains deeply embedded in AI training and deployment, but its biggest cloud customers are taking responsibility for more of the machinery around their models.
AI development is beginning to reach further into the hardware itself. Decisions about memory, data movement and accelerator design are being made earlier, as part of building and running the model rather than after the software is finished.
Maia 300 Is Being Designed Around Longer Reasoning
Microsoft says Maia 300 is aimed at memory-heavy inference, long context and agent workloads that remain active over extended tasks.
The company expects the system to deliver about 40% more tokens per megawatt than leading GPU-based systems and up to seven times the rack-scale token throughput of Maia 200. Those are Microsoft’s own performance estimates ahead of production deployment.
The chip uses separate compute, memory-controller and I/O chiplets. Microsoft is also tying the accelerator more closely to the networking required to move data across larger deployments, rather than leaving those functions to a separate layer of the system.
Maia 200 has already given Microsoft production data on how its models behave under real workloads.
Maia 300 is being built from that experience, with the silicon shaped around the memory and communication demands Microsoft is seeing in inference.
MTIA 300 Moves Communication Off the Compute Cores
Meta designed MTIA 300 to reduce the time large recommendation models spend exchanging data between accelerators.
On conventional GPU clusters, that traffic can compete with the work being carried out on the compute cores. MTIA 300 shifts more of the exchange into dedicated networking hardware inside the package.
Meta says that on a 150-billion-parameter production recommendation model, communication ran 3.9 times faster than on the GPU cluster used for comparison.
When communication and computation overlapped, compute throughput fell by less than 0.5%.
More of the data exchange is now being handled outside the cores performing the training.
Amazon and Google Are Splitting Compute by Workload
AWS is using Trainium3 for training and serving reasoning, mixture-of-experts and long-context models.
Its latest UltraServers add dedicated communication cores and higher bandwidth between chips. Trainium3 is already available through EC2.
Google has separated its eighth-generation TPU line between two systems. TPU 8t is built for training, while TPU 8i is designed for inference.
Google says TPU 8i is aimed at lower-latency serving and can support large numbers of concurrent agents.
Amazon is tuning one accelerator across several demanding workloads, while Google is dividing training and inference between different chips. Both approaches move away from treating AI compute as a single hardware category.
Nvidia Is Responding by Turning the Rack Into the Computer
Vera Rubin combines Nvidia’s Rubin GPU, Vera CPU, NVLink 6, networking and infrastructure processors in a rack-scale platform.
Nvidia says NVLink 6 delivers 3.6 TB/s of bidirectional bandwidth per GPU and 260 TB/s across the rack.
The company is selling more of the machinery around the accelerator as its largest cloud customers design more silicon themselves.
Vera Rubin puts the GPU, processor links and networking inside one Nvidia system rather than leaving those pieces to be assembled separately.
The competition is no longer confined to the chip. More of the AI system is now being designed around the models it will run.
Recent Comments