Explainer · AI infrastructure

Unified memory, explained

Why Apple's chip design suits running AI, and NVIDIA's suits training it

Written 1 September 2026 · All explainers

How to read the markers. [D] — verified from a source retrieved for this note, listed under Sources. [I] — industry-understood; not formally disclosed by a named party. [S] — our own inference or judgment. Every figure is either sourced or marked as an estimate.

The short version

Running an AI model is mostly a memory problem, not a thinking-speed problem. Apple builds one large pool of memory that every part of the chip can reach directly. NVIDIA’s data-centre design keeps two separate pools and copies data between them, and each pool is small — which is a large part of why you need a whole building full of cards to hold one big model.

That difference genuinely favours Apple for using models and NVIDIA for training them. But the popular framing — a widely shared video titled “Apple Just Killed AI Data Centers” prompted this note — is too absolute in two ways: NVIDIA now sells shared-memory designs of its own, and Apple’s advantage is in how much memory you get for your money, not in how fast it is. Apple’s machines are far slower per unit of work than a data-centre card.

So the defensible claim is narrower than “Apple killed AI data centres.” It is: a large-memory Mac can take over the small-and-medium-business inference slice of what data centres do today. Training and frontier-scale serving stay where they are.

1. The three chips, and what each is for

A modern Apple chip is not one processor. It is several different kinds of processor on one piece of silicon, each good at a different shape of work.

CPU — the Central Processing Unit. The general-purpose one. It does a small number of things at a time, but it can do anything, and it does each step very fast, one after another. Think of one extremely quick, versatile person working through a list. Almost all ordinary software runs here.

GPU — the Graphics Processing Unit. Originally built to draw video-game images, where you must work out the colour of two million pixels at once and each pixel’s answer is independent of the others. So a GPU is thousands of much simpler workers doing the same arithmetic simultaneously on different data. It is bad at complicated one-after-another logic and superb at doing one simple sum across enormous quantities of numbers.

That turned out to matter enormously, because an AI model is, at bottom, a very large pile of numbers and a lot of multiplication. The maths that draws a game frame and the maths that runs a language model are close enough that the same hardware does both. This is the whole reason a graphics-card company became the most important AI company.

Neural Engine — Apple’s name for a third kind of unit (the industry term is NPU, Neural Processing Unit). It is narrower than a GPU: it can only do the specific operations that neural networks need, but it does them using much less electricity. Apple uses it for the always-on, small-model work on iPhones and Macs — dictation, photo recognition, text prediction — where battery life matters more than raw speed. [I]

The important structural point for this note is simply that there are several processors, and they all need to look at the same data. How they are wired to that data is the whole subject.

2. What “memory” means here, and why it is the bottleneck

Two different things both get called “memory”, and mixing them up is the most common confusion.

  • Storage (the SSD or hard disk) is where files live when the machine is off. Large and slow.
  • Memory (RAM) is the working surface — what the processor can actually reach right now. Much smaller, and hundreds of times faster.

The workbench image is the right one. Storage is the cupboard; memory is the bench top. If the job does not fit on the bench, you cannot do it — no matter how fast your hands are.

For AI there are exactly two numbers that matter, and they are different questions:

  1. Capacity — how many gigabytes (GB) of memory there are. This decides whether a model fits at all. Below the threshold, the answer is not “slow”, it is “no”.
  2. Bandwidth — how many gigabytes per second (GB/s) can be moved between memory and the processor. This decides how fast text comes out once the model does fit.

Here is the part that makes AI unusual, and it is worth being precise about because everything else follows from it.

To produce a single word, the machine must read the model’s entire set of numbers. Not a relevant portion of them — all of them, every time. A model with 30 billion numbers in it reads all 30 billion to produce one word, then reads all 30 billion again for the next word. It is closer to reading a whole reference book cover to cover to write each individual word than to looking something up in it.

The consequence: generating text is limited by how fast you can move numbers, not by how fast you can multiply them. The arithmetic units spend most of their time waiting for data to arrive. In the trade this is called being memory-bound, and it is why memory bandwidth, rather than raw processing power, sets the speed of a local model. [I]

3. Two ways to wire memory to processors

The traditional way: separate memory, and a bridge between

In a conventional computer, and in an NVIDIA data-centre server, there are two separate pools of memory:

  • the system memory (RAM), attached to the CPU;
  • the graphics-card memory (VRAM), which lives on the plug-in card with the GPU.

The GPU can only work on data that is in its own memory. So anything the CPU has must be copied across to the card, over a connection called PCIe — the physical slot the card plugs into. That bridge is far slower than either memory pool it connects. [I]

Two rooms with a letterbox between them. Each room is fast to work in; passing things through the letterbox is not.

That copy has two costs. The obvious one is time. The less obvious one is that the graphics card’s memory is small and expensive, because it uses a specialised high-speed type of memory built for bandwidth rather than capacity. So a model that does not fit in one card’s memory has to be split across several cards, wired together with fast interconnects, in a server, in a rack, in a building with the power and cooling to run it.

Put plainly: a large part of what an AI data centre physically is, is an expensive machine for stitching many small fast memories into one big one. [S — our framing, but it follows directly from the capacity constraint above]

Apple’s way: one pool everything shares

Apple Silicon puts the memory on the same package as the processor, and the CPU, GPU and Neural Engine all address that single pool directly. [D — Apple’s own description; see Sources]

Nothing is copied. If the CPU prepares data and the GPU needs it, the GPU simply reads it where it already sits. One room, one bench, several people working at it.

Two consequences follow:

  1. No copying tax. Work that alternates between the general-purpose processor and the number-crunching one does not pay a transfer cost at each handover. This is a bigger deal for real workflows than for benchmarks, because real workflows alternate constantly.
  2. Capacity is bought at ordinary memory prices. Because Apple uses one pool for everything rather than specialised graphics memory, you can have a lot of it — vastly more than any consumer graphics card offers, for far less money.

Point 2 is the one that actually threatens the data centre. The reason to rent a data centre for inference is usually that the model does not fit on your own machine. Remove that, and much of the reason goes with it.

Two ways to wire memory to processors Left: a conventional design with system memory attached to the CPU and separate graphics memory attached to the GPU, joined by a narrow PCIe bridge across which data is copied. Right: Apple Silicon, where CPU, GPU and Neural Engine all read one shared pool of memory directly. Conventional · NVIDIA data centre Two pools, one bridge Apple Silicon One pool, everything shares System memory RAM · large, ordinary CPU Card memory VRAM · small, costly GPU PCIe bridge copy 80GB 80GB 80GB 80GB … A model larger than one card is split across many cards, in a server, in a rack, in a building. Built for training · compute-bound Unified memory · up to 512GB one pool, on the same package as the chip CPU GPU Neural Engine all three read the same data where it sits · nothing is copied Capacity bought at ordinary memory prices, so a 400GB model fits on a desk. Built for using models · memory-bound
Figure 1. Two ways to wire memory to processors. Left, the conventional design copies data over a narrow bridge between two pools, and a large model must be split across many small card memories. Right, Apple Silicon gives every processor direct access to one large pool.

4. Why this favours using models and not training them

These are two genuinely different jobs, and they stress different parts of the machine.

Training is building the model: showing it enormous quantities of text and adjusting billions of numbers, over and over, for weeks. It is compute-bound — the limit is how much arithmetic you can do per second — and it is done in large batches, which is exactly the shape of work a GPU is built for. Every ounce of raw processing power converts directly into finishing sooner. [I]

Inference is using the finished model to answer something. As set out in section 2, it is memory-bound: the constraint is whether the model fits, and how fast its numbers can be streamed. Extra arithmetic capacity mostly sits idle waiting. [I]

So the architectures line up with the jobs:

  • NVIDIA’s design maximises arithmetic throughput and accepts small, expensive, separate memory — right for training.
  • Apple’s design maximises memory capacity per pound and accepts far less arithmetic throughput — right for single-user inference.

This is the substance behind the popular claim, and on this point it is correct. [S]

One important qualification. “Inference is memory-bound” is true of generating text. Reading a long prompt — a big document, a long conversation — is compute-bound, and here Apple’s weaker arithmetic throughput shows. In practice this is why a local model on a Mac can feel quick to chat with but slow to start on a large document. [I]

A second qualification, and it is the commercially decisive one. A data-centre GPU serves many users at once by batching their requests together, which reuses each pass through the weights across all of them. A Mac has no such advantage. So the Apple design suits one team’s workload, not a thousand tenants — which is precisely why the honest version of the thesis is about small and mid-sized businesses running their own systems, and not about hyperscale serving. [S]

5. Where the popular case overstates

Unified memory is not Apple-exclusive. NVIDIA has recognised the same gap and now builds parts where the CPU and GPU share a coherent memory address space, and sells a small unified-memory desk machine of its own. Apple got there first at consumer prices and at consumer capacities, but the architectural idea is no longer a point of difference. [D — see Sources]

Apple wins on capacity, and loses on speed. Both memory bandwidth and raw compute on Apple’s best chips are far below a data-centre GPU. The right way to state the advantage is capacity per pound, not performance. [D for the figures below; S for the framing]

“Killing data centres” does not follow. Training must still happen somewhere, and frontier models are still far better than anything that runs locally. What large local memory genuinely threatens is one slice: a mid-sized company running an open-weight model on its own premises instead of renting inference by the token. That is a real erosion of marginal demand, and a useful thing to hold in mind against the data-centre capital-spending narrative. It is not the collapse of the category. [S]

6. The numbers

All figures below were retrieved from the manufacturers’ own pages on 1 September 2026. [D]

Machine Memory capacity Memory bandwidth Built for
Mac Studio, M5 Max up to 128GB 614 GB/s (460 GB/s on the base configuration) a desk
Mac Studio, M5 Ultra up to 512GB 1.2 TB/s a desk
Mac Studio, M3 Ultra (previous generation) up to 512GB 819 GB/s a desk
NVIDIA DGX Spark (GB10 Grace Blackwell) 128GB unified 273 GB/s a desk
NVIDIA H100 SXM (one card) 80GB HBM3 3.35 TB/s a data centre

Three readings, and each one corrects a different part of the popular story:

  1. Capacity: Apple wins by a lot. An M5 Ultra holds 6.4 times the memory of a single H100 card. That is the number the whole local-inference argument rests on. An eight-card H100 server reaches 640GB in total, but only by being a data-centre machine.
  2. Bandwidth: NVIDIA wins by a lot. One H100 moves memory about 2.8 times faster than an M5 Ultra. Since bandwidth sets generation speed, the same model on a Mac produces text materially more slowly than on a data-centre card — before you even count the data centre’s ability to serve many users at once from a single pass through the weights.
  3. Unified memory by itself is not the advantage. NVIDIA’s own desktop machine has unified memory too, and it is not competitive with Apple here: a quarter of the M5 Ultra’s capacity and less than half the M5 Max’s bandwidth. What Apple actually has is a large, fast pool at a consumer price — which is an execution advantage, not an architectural one. That is the sharpest correction to the popular framing.

On the 1.5TB machine the popular story is built around

The premise is a rumoured 1.5TB Mac Studio. What the cited Tom’s Hardware piece (13 July 2026) actually says: a rumoured M7 Ultra is designed to support up to 1.5TB of unified memory, in 2028, relaying a Bloomberg report by Mark Gurman, with the article stating plainly that none of the dates or specifications are confirmed by Apple, and that the configuration is tied to memory availability. [D — that is what the article says; the underlying claim is a rumour]

Two things follow, and the second matters more:

  • The article’s own arithmetic was already wrong. It describes 1.5TB as roughly twice the M5 Ultra’s planned capacity, implying about 768GB. Apple then shipped the M5 Ultra at 512GB. Treat the 1.5TB figure as a direction, not a specification. [D]
  • The argument does not need the rumour. The best evidence offered for the thesis is that at a $1bn-revenue Australian manufacturer with 80,000 products, the entire business planning model sat in about 400GB of memory. A Mac Studio you can order today holds 512GB. If that anecdote is representative, the hardware precondition has already arrived and the 2028 chip is beside the point. [S]

7. Caveats

  • The figures in section 6 are point-in-time and move quickly. Treat them as the right order of magnitude for an argument, not as a basis for a purchase.
  • “Memory-bound” is a simplification of a real spectrum. Batch size, quantisation, context length and the specific model architecture all shift where the bottleneck sits. The direction of the argument holds; any single number does not.
  • The claim that local inference erodes small-and-mid-business data-centre demand is [S] — our inference, not a measured effect. No series exists that would show it.
  • No price-per-gigabyte comparison is made in this note, because H100 pricing is not published and the Mac Studio 512GB configuration price was not retrieved. The “capacity per pound” claim in section 5 is therefore directional [S], not a costed one.
  • No measurement was found of the context length at which inference stops being memory-bound and becomes compute-bound on Apple Silicon. That crossover decides whether the local thesis survives real analytical documents, and it is the most important unmeasured quantity in this note.

Sources [D]

All retrieved and confirmed to load on 1 September 2026.

Apple — unified memory architecture

  • Apple unleashes M1 (Nov 2020) — the clearest statement of the mechanism: the unified memory architecture brings memory “into a single pool within a custom package”, which “allows all of the technologies in the SoC to access the same data without copying it between multiple pools of memory”.
  • Apple unleashes M5 (Oct 2025) — “the entire chip” accesses “a large single pool of memory”, fuelling “the faster CPU, GPU, and Neural Engine”. Wording caveat: Apple says “all of the technologies in the SoC” and names the CPU, GPU and Neural Engine in an adjacent sentence rather than in the same one. The three-way claim as popularly phrased is a fair paraphrase of Apple, not a verbatim Apple sentence.

Apple — Mac Studio specifications

The 1.5TB rumour

NVIDIA — coherent and unified memory

NVIDIA — H100