How Accurate Are LLMCheck's Mac Speed Estimates? 67 Runs Checked

The scorecard

Each run is compared with what the formula gives for the same model on the same chip at the same quantization, with no help from any measured row — the figure the index would print for a model nobody had run yet. “Run vs formula” is the measured speed over that figure: +30% means the Mac ran 30% faster than estimated, −20% that it ran slower. The 24 measured rows the index already held on 5 October are left out. Some of them set the formula’s constants, so they can’t test it.

Measured runs added to the LLMCheck index since 5 October 2026 (data version 2026-10-17), against the formula at the same model, chip and quantization. Miss = |run ÷ formula − 1|.

The formulas are on the methodology page. In short: decode speed is weight bytes read per token over memory bandwidth, times an efficiency of 0.783, plus a fixed cost per token. Dense models read every weight. Mixture-of-experts (MoE) models read only their active parameters, and pay more fixed cost for routing.

Dense models: within 18%

The dense formula’s two constants (0.783 efficiency, 1.46 ms per token) were solved in August from two vendor-published M5 Max runs. Against 16 new dense runs on 5 models it missed by 10% at the median and by no more than 18%; 9 of the 16 landed within 10%.

Dense models: new runs vs the formula at the same chip and quant.

Two results stand out. Apple’s own MLX benchmarks (published 15 October) put Qwen3.8-27B at BF16 at 19.69 tok/s on an M5 Ultra and 12.46 on an M3 Ultra (both published), against ~17 and ~11 estimated. Mistral Medium 3.5 at FP8 ran 7.7 and 4.95 (published) against ~7.0 and ~4.7 estimated. At 8 and 16 bits per weight, where decode is almost entirely a memory problem, the formula runs slightly low. Llama 3.3 70B at 4-bit MLX told the same story on an M5 Max: 13.1 tok/s published in tbreak’s review, +11% against the formula.

The second is how much the runtime matters. On the M5 Ultra, across three reviewers’ Macs, Qwen3.8-27B at about 4 bits ran 46.27 tok/s in llama.cpp (Tom’s Hardware, published), 48 in oMLX after a 6,091-token prompt (MacStories, published) and 54.7 in mlx_lm.benchmark (NetworkChuck’s published test repo, community-measured). From slowest to fastest that is a spread of 18% on the same chip with roughly the same weights, and the formula’s misses sit inside it. Below that level, the runtime you pick matters more than the estimate.

The Mac mini M6 is the slowest chip in the set, and the formula held there too: Qwen3.8-27B at 4-bit QAT ran 9.42 tok/s community-measured in LM Studio on a 24 GB M6, against ~8.4 estimated (+12%).

Small MoEs: right on average, not per model

For MoEs up to 40B total parameters the index uses the dense formula on active-parameter bytes with a ~5 ms per-token overhead, refit on 5 October to six runs. Across 27 new runs on 10 models the median run came in at 0.97× the formula, so the overhead is about right on average. Individual runs scattered: the median miss was 16%, 18 of 27 landed within 20%, and the extremes were −34% and +49%.

Small MoEs (≤40B total): new runs vs the formula at the same chip and quant.

On the M5 Max at 4-bit, independent setups agree with the formula. The comparison chart on Liquid AI’s LFM2.5-24B-A2B model card (llama.cpp, Q4_K_M, a 4K-token input; published) puts Qwen 3.6-35B-A3B at 118 tok/s, Gemma 4 26B-A4B at 107 and Nemotron 3.5 Lightning at 114, against ~117, ~106 and ~117 estimated. A community-measured oMLX run of North Mini Code 1.0 on the M5 Max measured 119.7 against ~117, and tbreak’s published Ollama figure for GPT-oss 20B was 107.51 against ~111. tbreak’s Mac mini M6 review published a figure for Qwen 3 30B-A3B, 62.01 tok/s in MLX, +11% against the formula.

KAT-Coder-V2.5 is the miss that changed the leaderboard. Its first fully specified run came from oMLX on a 40-core M5 Max, with the OptiQ 4-bit build and MTP off. It measured 89.3 tok/s (community-measured; oMLX database), −26% against the formula’s ~121 at that build’s 4.51 bits per weight. The leaderboard had ranked it #1 on an estimate of 117. MiniCPM-V 4.7 also came in under: −16% on the M5 Max (100.2 tok/s, community-measured) and −31% on an M4 Pro at 5-bit (48.2 tok/s, community-measured).

The misses don’t share a direction, which is why the index has not refit the overhead again. On the base M5, North Mini Code 1.0 ran −34% against the formula (34.2 tok/s, community-measured) while LFM2.5-24B-A2B ran +23% (74.2, community-measured). A constant that fixed one would make the other worse. LFM2.5-24B-A2B’s own runs also disagree with each other. Liquid’s published 184 tok/s on an M5 Max is +45% against the formula, while a community-measured 5-bit run on an M4 Max is −21%. Its estimates on other chips are therefore not a clean bandwidth curve.

Large MoEs: the 0.42 factor undershoots

Above 40B total parameters the index multiplies the active-bytes formula by 0.42. That factor was fitted in September to one 2-bit DeepSeek V4 Flash run and kept on purpose as conservative. The new runs show how conservative. Across 24 runs on 12 models the median came in at 1.17× the estimate, with a range of 0.4× to 2.1×; 15 of the 24 ran faster than estimated.

Large MoEs (>40B total): new runs vs the formula (with the 0.42 factor) at the same chip and quant.

MiMo-V2.6-Flash is the case that matters most. Two independent runs on a 256 GB M3 Ultra measured 59.4 tok/s in oMLX and 59 in Rapid-MLX (both community-measured, 4-bit, no MTP). That is about twice the formula’s ~28–32 estimated for those two builds. Nemotron 3.5 Bolt 74B-A9B also runs at about twice its estimate: 71.7 tok/s community-measured at 4-bit on an M5 Max, through a community MLX port; stock llama.cpp still crashes on it on Metal, and the fix (PR #30699) was still open on 18 October. GPT-oss 120B in Ollama measured 75.65 tok/s on an M5 Max (published, tbreak), +30% against the ~58 estimated at its native MXFP4. Ling-3.0 Flash got its first figure of any kind: 68 tok/s community-measured on an M5 Max in oMLX, +20% against the formula.

Three models fell below the factor, all on young runtimes. Qwen3.8-Flash-Next ran 47 tok/s on an M5 Max in MTPLX with MTP off, but 28.5 in oMLX on the same chip and 26.5 through a custom MLX port on an M3 Ultra (all community-measured). DeepSeek V4.1 Flash measured 20.23 on an M3 Ultra in oMLX’s own V4.1 support (community-measured), and Kolibri-1 52 on an M1 Max with a loader bundled in its repo (community-measured). The custom port and Kolibri’s bundled loader are unmerged code, so those two runs are displayed but do not anchor estimates.

The methodology page measures the spread a second way. It takes every large-MoE row allowed to anchor an estimate, from any date, and compares it with the formula without the factor:

Large MoEs: every measured plain-decode row allowed to anchor an estimate (22 rows, 12 models, all dates), over the formula without the 0.42 factor. Median per model.

Individual runs land at 0.26–0.88 of the unscaled formula. Per model the medians run from 0.33 to 0.87 (0.8×–2.1× the 0.42 estimate), and the median across models is 0.54. The factor stays because it sits below that median: for 10 of these 12 models it was a floor, and for 2 it was not. But it only applies until the first run, and any model with a measured row is scaled from that row instead (see the rules). Read a large-MoE estimate as “about this fast or faster on a mature runtime”. A new architecture on a first-week port can fall below it.

1-bit and ternary: measured only

1-bit and ternary formats don’t read weights the way the formula assumes. Each weight is a sign bit or a trit unpacked by a dedicated kernel, so decode is limited by compute, not bandwidth. Until this update the index carried a bandwidth estimate for Bonsai 27B: 30 tok/s on an M5 Max. It should never have existed, and it is retracted. PrismML’s model card reports 66.4 tok/s on an M5 Max, 44.2 on an M5 Pro and 26 on an M4 Pro (published; llama.cpp tg128 in PrismML’s fork, Q1_0_g128, 3.9 GB).

1-bit and ternary runs added since 5 October. No formula figure: these formats are measured-only.

These models are now measured-only. The leaderboard shows the best published figure with the chip it was taken on, and a chip with no published run shows no figure at all rather than an estimate.

Why the leaderboard moved

The leaderboard prints one reference figure per model, and the speed part of the LLMCheck Score (tok/s × 0.25, capped at 25) uses it. Where a new run landed on a model’s reference chip, it replaced the estimate:

Reference figures that changed because of a run added since 5 October. 5 Oct = the deployed index of 5 October 2026; ranks also moved because 20 models joined the catalog.

The top changed hands. On its first measurement KAT-Coder-V2.5 dropped from #1 (Score 82) to #3 (79). MiMo-V2.6-Flash rose from #6 to #1 (82) because its M5 Ultra figure is now scaled from the two M3 Ultra runs instead of the 0.42 factor. It went from 42 to ~79 tok/s at MXFP4, still an estimate and still marked with a hollow diamond. It needs a Mac Studio with at least 192 GB of unified memory. On a Mac with 32 GB or less, the best pick is Qwen 3.6-35B-A3B (Score 81; 64 tok/s community-measured on a Mac mini M6 32 GB; needs 24 GB). The current figures are on the leaderboard and the key numbers page.

The rules: reference row and anchors

Two rules decide which number you see. Both are documented under which figure the leaderboard shows, and one function in the build computes them for every page.

The reference row

• Reference chip. The M5 Max, or the Mac Studio M5 Ultra for a model that needs more than 128 GB.
• A measurement wins. Among measured plain-decode rows on that chip at 5 bits or fewer, the index takes the lower median. That is always a real run with its own quant and provenance, never simply the fastest one anyone posted. A run whose quant the source didn’t state counts only when no stated-quant run qualifies.
• Otherwise the estimate, at Q4_K_M or the model’s native format (MXFP4 for GPT-oss). When Q4_K_M doesn’t fit, it uses the quant that sets the model’s minimum RAM.
• 1-bit and ternary models show their best measured figure and the chip it came from.

Example: Qwen 3.6-35B-A3B has 3 measured rows on the M5 Max: 126 tok/s (MLX 4-bit, community-measured); 118 tok/s (llama.cpp Q4_K_M, published); 92.9 tok/s (llama.cpp Q8_0, community-measured; excluded: more than 5 bits). The leaderboard shows 118 tok/s published, the lower median of the two eligible runs, not the faster one. GPT-oss 120B’s only M5 Max run, tbreak’s Ollama figure, did not state a quant. With no stated-quant run to defer to, it is the reference.

Anchors

• A model’s other chips are scaled from its nearest-bandwidth measured row using its class formula. When several runs sit at that distance, a run at the target’s own bytes per weight is used alone; otherwise the index takes the lower median of the scaled results.
• Some rows are displayed but never anchor and never become the reference figure: rows flagged as a runtime patched with unmerged code (a fork, a custom port or a loader from an open pull request), an approximate figure or one its own source doubts; SSD-streamed runs; and binned-GPU configurations. Pre-release builds of a released runtime (oMLX .dev and rc builds, mlx-vlm from its main branch) count as released.
• For one model at one quant, an estimate is capped at the estimate for any higher-bandwidth chip, so more bandwidth never shows a lower number.

That is how MiMo-V2.6-Flash gets ~79 tok/s on an M5 Ultra without an M5 Ultra run. It is also why a patched-llama.cpp run of Nemotron 3.5 Bolt on an M3 Ultra appears in the dataset but does not drive Bolt’s other figures. When a model goes from estimate to measurement, the leaderboard can move by several points in either direction. The table above shows both.

What stays out of the table

Some published figures measure a different workload, and comparing them with a plain-decode formula would flatter or punish it unfairly. They are cited here, labelled, and never entered as plain-decode rows:

• Multi-token prediction (MTP). KAT-Coder-V2.5 reached 124.5 tok/s on a 40-core M5 Max with oMLX’s MTP on (community-measured, oMLX database), against 89.3 with it off. Nemotron 3.5 Super VL ran 57.1 on an M5 Max with its MTP head against 30.6 plain (community-measured, mlx-vlm PR #2528). GLM-5.3-Flash ran 61 on an M5 Ultra with a 3-token MTP head against 39.7 without (one community report, quant unstated).
• Long prompts. Qwen 3.6-35B-A3B decoded at 60.7 tok/s on a 24 GB Mac mini M6 after a 16K-token prompt (community-measured, oMLX). NetworkChuck’s 32K-prompt runs put Qwen 3.5 122B-A10B at 74.7 on an M5 Ultra in mlx_lm.benchmark (community-measured). Decode slows as the context grows, so these sit below short-prompt rows.
• SSD-streamed experts. DeepSeek V4.1 Flash ran 29.6 tok/s on a 128 GB M5 Max by streaming a 341 GiB Q2 GGUF from SSD through an 82 GB expert cache (community-measured, a ds4 fork). It is in the dataset, labelled, but a model that doesn’t fit in memory can’t anchor estimates for Macs where it does. MiniCPM-V 4.7 managed 7.36 tok/s on a 16 GB Mac mini M4 the same way (community-measured, lilting.ch).
• Clusters. A community report of GLM-5.3-Flash at 36.8 tok/s used MTP on two M5 Ultras linked together. That is two Macs, not one.
• Unplaceable chips. PrismML’s “up to 179 tok/s” claim for Ternary Bonsai 2 9B names a chip Apple doesn’t make (“M5 Max Pro”) and states no runtime, so it can’t be placed.

How to read a speed on LLMCheck now

• Look at the shape first. A filled square is published (a vendor or a press review that documents its setup). A hollow circle is one person’s linked run. A hollow diamond is the formula.
• Dense estimate: in this check, every run landed within 18%. Your runtime can move the result by as much.
• Small-MoE estimate (40B total or less): right on average, but any single model can miss by a third or more.
• Large-MoE estimate with no run of its own: usually a floor. On mature runtimes, runs reached up to 2.1× the estimate; first-week ports fell below it.
• 1-bit and ternary: measured figures only, on the chip they were taken on.
• Read the quant and runtime under every figure. On the M5 Max, Qwen3.8-Flash-Next ran 1.65× faster in MTPLX (flat 4-bit) than in oMLX (oQ4e), in two different users’ runs.

Every measured row is in the open dataset (CSV and JSON) and on the benchmarks page. Have a run the index doesn’t list? Submit it with the chip, RAM, quant, runtime and version, prompt length, and whether MTP was off.

Frequently Asked Questions