Microsoft Maia 300: AI Inference Specs and 2027 Rollout

Microsoft announced Azure Maia 300 on October 12, 2026, as the next generation of its custom AI inference system. The company says the system is designed for long-context reasoning, fast conversations and multi-step AI agents, with production deployments planned for 2027. Its headline claims include up to seven times the rack-scale token throughput of Maia 200 in a full 96-accelerator configuration. These are Microsoft’s published figures, not independent benchmark results.

What Microsoft announced

Maia 300 is the third generation of Microsoft’s Azure Maia accelerator program. Rather than presenting it as a chip alone, Microsoft describes a complete system that combines silicon, software, memory, networking, racks and data-center operations. The company plans to expand Azure Maia to multi-gigawatt deployments starting in 2027, with Maia 300 entering production during that year.

Microsoft also says Maia 200 is already serving production workloads, including Microsoft Copilot. The company expects GitHub Copilot and Microsoft Foundry workloads to follow on Maia 200 by the end of 2026. Maia 300 is intended to support later workloads that need more context, more concurrent sessions and longer agent tasks.

Maia 300 specifications at a glance

Specifications can change before a product enters production. For capacity planning, use the published details as an early design signal rather than a final quote, availability promise or guaranteed workload result.

Why the design focuses on memory and data movement

Generating a token requires more than raw arithmetic. A serving system must move model weights, attention state and intermediate data between memory and compute. For long prompts and agent workflows, data movement and memory bandwidth can limit throughput even when the accelerator has plenty of arithmetic capacity.

Microsoft says Maia 300 uses a multi-chiplet design, HBM4 memory, low-precision formats such as enhanced MXFP4, and a software-defined dataflow architecture. It also changes memory organization across chiplets so software can place data closer to the compute that uses it. These choices aim to reduce unnecessary movement and support more useful work within a fixed power budget.

The network is another part of the design. Microsoft describes a two-tier, Ethernet-based scale-up fabric that connects accelerators within a rack and across racks. The stated goal is to let software coordinate larger groups of accelerators without treating every rack as a separate island. This matters for large models and workloads that divide prefill, decoding or other steps across different workers.

What the performance claims do and do not mean

Microsoft reports up to seven times the rack-scale token throughput of Maia 200 in a full 96-accelerator Maia 300 configuration. It also reports more than 2.5 times the performance per watt compared with Maia 200 in its technical architecture post. The separate claim of 40% more tokens per megawatt compares Maia 300 with leading GPU-based systems.

Keep those comparisons separate. They use different baselines, and “up to” figures describe a best-case configuration rather than every model or request. Real results depend on model architecture, precision, prompt length, output length, concurrency, batching, software kernels, networking and how much of the system a workload can use.

The public announcement does not provide an independent third-party benchmark suite, customer pricing or a general-purpose hardware purchasing path. Until production systems are available and tested with representative workloads, teams should treat the numbers as vendor claims. For a practical view of how serving speed affects applications, see our guide to diagnosing LLM API latency.

What AI developers should watch next

For most developers, Maia 300 is not a chip to install in a workstation. It is a cloud infrastructure development that may affect the capacity, efficiency and cost of AI services built on Microsoft’s platforms. The practical impact will become clearer when Microsoft publishes production availability, supported deployment options and workload measurements.

• Model availability: Watch for which model families and serving endpoints use Maia 300, and whether customers can select the hardware or only the service.
• Measured latency: Look for time to first token, output tokens per second and p95/p99 latency, not only aggregate rack throughput.
• Concurrency: Check how performance changes when requests have different context lengths and arrive at the same time.
• Cost: Compare the price of a successful task, including input tokens, output tokens, retries and tool calls.
• Reliability: Check how the service handles overloaded capacity, model updates and regional availability.

If you are running models on your own hardware, the problem is different: you need to check model weights, context memory and runtime overhead against the memory you have available. Our local LLM GPU memory guide explains that calculation.

What the system-level design means in practice

The numbers are easier to understand when they are read together. Microsoft says each Maia 300 accelerator has eight HBM4 stacks with 288 GB of memory and 18 TB/s of peak bandwidth. A full rack contains 96 accelerators, so the rack-level design combines a large memory pool with a high-speed network. Microsoft lists 27.6 TB of HBM capacity and 1.7 PB/s of aggregate HBM bandwidth across the rack. These are system specifications, not the amount of memory available to one model process without any sharing or overhead.

The 167 kW rack power figure is also important. A data-center accelerator is not deployed like a desktop graphics card. Power delivery, liquid cooling, network switches, maintenance access and failure handling all affect how many systems a provider can run in a facility. Microsoft says liquid cooling removes more than 80% of the rack’s thermal load from the accelerators, voltage regulators and switch chips, while air cooling handles other components. This shows why the company describes Maia 300 as a complete system rather than a chip alone.

The network design is meant to reduce the cost of moving data between accelerators. Microsoft describes direct links within trays, switched connections across a rack and an optical network between racks. The technical post says the superpod can scale to 12,288 accelerators. That does not mean every request uses all of them. It means the system is designed to support workloads that need a larger group of accelerators and a consistent way to communicate across them.

Why long-context and agent workloads need a different balance

Many AI applications have two broad inference phases. During prefill, the server processes the input prompt and builds the attention state needed for generation. During decoding, it generates output tokens and continues to read and update that state. Long prompts and long-running conversations increase the amount of state that must be stored and moved. For some workloads, memory capacity and bandwidth can become as important as the accelerator’s raw arithmetic rate.

Agent workflows add another pattern. An agent may reason, call a search or database tool, read the result, decide on another action and repeat. The service must handle tool results, changing context, and many separate requests without letting one long task block the rest. Microsoft says Maia 300 is designed for sustained, multi-step agent tasks as well as latency-sensitive conversations. That is a design target, not proof that every agent framework will run faster automatically.

Low-precision formats such as enhanced MXFP4 can reduce the amount of data used for weights and some state, but precision is a quality decision as well as a memory decision. Teams still need to test accuracy, instruction following, structured output and tool use with the specific model and workload. The best format is the one that meets quality and reliability needs while using resources efficiently.

How to compare the performance claims fairly

When independent results become available, compare systems with the same model, model version, precision, prompt length, output length, concurrency and serving settings. A test that generates short answers from a small prompt does not represent a long-context agent that calls tools several times. Likewise, a rack-level throughput number is not the same as the time a single user waits for the first token.

For an application team, a useful benchmark should report at least the following:

• Time to first token: how long the user waits before the answer starts.
• Inter-token latency: how consistently the response continues after it starts.
• Tokens per second: output speed for the tested model and request size.
• Throughput under load: how many requests finish per second at a stated concurrency level.
• Tail latency: p95 or p99 response time, which can reveal slow requests hidden by an average.
• Cost and energy: cost per successful task and energy use for the same amount of useful output.

Record quality alongside speed. A faster answer that needs more retries, makes more factual errors or fails a tool call may cost more in the full workflow. For API-based products, our guide to calculating LLM API cost per task covers the wider cost calculation.

What teams should prepare before the 2027 rollout

There is no need to redesign an application around Maia 300 before a supported service is available. Instead, make workloads easier to compare across providers. Keep a repeatable evaluation set, record the model and inference settings, and measure response quality, latency and cost on real request shapes. If a provider later offers a Maia-backed endpoint, the same test set can show whether it improves your use case.

Teams using Microsoft services should watch for details on supported models, regions, availability dates, deployment choices and pricing. A provider may route a model to custom silicon behind the scenes, or it may expose a choice of deployment profile. The announcement does not yet settle those details for every customer. Ask whether a service guarantees a particular hardware path, what happens during capacity shortages, and whether the service reports enough usage data to compare performance over time.

For organizations evaluating their own infrastructure, include cooling, rack power, network topology, utilization and operations in the business case. Accelerator purchase price alone is not a useful comparison when one design can deliver more completed work per rack but requires a different power and cooling setup. These system-level costs are one reason cloud providers design inference hardware around the data center as a whole.

Questions developers may have

Can I buy an Azure Maia 300 card for a desktop?

Microsoft has announced a data-center system and a planned production rollout in 2027. It has not announced a consumer or workstation card for direct purchase.

Does seven times the throughput mean every request will finish seven times faster?

No. It is a rack-scale throughput claim for a specified configuration. Individual request latency depends on the model, request shape, concurrency and serving software.

When will Maia 300 enter production?

Microsoft says production deployments will begin in 2027. It has not published a specific customer availability date in the announcement.

Sources

• Microsoft Official Blog: Microsoft to expand Azure Maia system to multi-gigawatt deployments (October 12, 2026).
• Microsoft Azure Blog: Deep dive into Azure Maia 300 architecture (October 12, 2026).
• NVIDIA NIM: Troubleshooting GPU memory errors.