LLMCheck Index Methodology
How the LLMCheck index ranks 130 local and frontier LLMs for Mac — a published estimation model, linked third-party benchmarks, and community submissions. Fully transparent, fully reproducible.
LLMCheck is an independent index of local-LLM performance on Apple Silicon — not a benchmark lab. Every figure is either a transparent estimate from the published model below, a sourced third-party benchmark (linked per row), or a community submission. Each data point is labeled with its provenance, and no figure is claimed as a first-party lab measurement.
Where Every Number Comes From
Every row in the leaderboard, the benchmarks table, and the open dataset carries one of three provenance labels. This mirrors the provenance_policy field published in benchmarks.json:
Derived from the LLMCheck estimation model — memory-bandwidth scaling plus quantization arithmetic, documented in full below. Estimates are useful for planning ("will this model fit and feel usable on my Mac?") but are not measurements. Every estimated speed figure on the site links back to this page.
A published figure from an accountable source, with the URL attached to the row — a vendor’s model card or announcement, an independent press review that documents its setup, Arena AI ELO ratings, MMLU/HumanEval/SWE-Bench results from official model cards and papers, or independent hardware test data. We link, we don't re-host or re-run.
One person’s run on a real Mac — a blog post, forum thread or benchmark submission — with its URL as the source. Community numbers are sanity-checked against the estimation model and known baselines before inclusion. Own a Mac? Submit a benchmark — real runs beat estimates every time.
Why an index and not a lab? A single lab machine can only test one chip, one runtime, one macOS version. An index that publishes its estimation math, links its sources, and accepts community runs covers the entire Apple Silicon range — and you can audit every step. When we're estimating, we say so.
How We Estimate Performance
Local LLM performance on Apple Silicon is unusually predictable, because two hardware numbers dominate everything: unified memory capacity (can the model fit?) and memory bandwidth (how fast can weights stream through the GPU?). The LLMCheck estimation model is built on those two numbers. How its estimates compared with the runs later measured on the same chips is checked run by run in How accurate are Mac LLM speed estimates? (October 2026).
Step 1 — Will it fit? (RAM model)
+ KV cache: grows with every token of context, by an amount that differs between models of the same size by ten times or more — computed per model from its config.json (RAM at 64K, below); minimum RAM covers about the first 4K tokens only
+ macOS overhead: ~2–3 GB for the OS and the inference runtime
Fit rule: a model is listed for a Mac when its minimum RAM is no more than the Mac’s unified memory
+ GPU memory limit: macOS lets the GPU use only part of that memory by default, so a listed model that needs more than the default is tagged raise GPU limit, or over GPU limit when even a raised limit is short (the macOS GPU memory limit, below)
Until 19 November 2026 this step said macOS gives the GPU about three-quarters of the memory on every Mac. That is the macOS 15 figure from 36 GB up; below that macOS 15 allows two-thirds, and macOS 26 allows more at every size from 16 to 256 GB. The rule, its sources and the command that raises the limit are below.
Minimum RAM rule. A model’s minimum RAM is the size of its smallest build of at least 3 bits that has a working Mac runtime, plus about 2 GB for the runtime and a short (about 4K-token) context, leaving headroom within its memory tier; a longer context adds its KV cache on top (RAM at 64K). A 2-bit build sets the minimum only where a published Mac run used it.
Step 2 — How fast? (Speed model)
Token generation is memory-bandwidth-bound: for every token, the runtime must stream essentially all active model weights from RAM through the GPU. Compute is rarely the bottleneck on Apple Silicon. That gives a simple ceiling:
model_bytes_GB: weight bytes read per token — total size for dense models, active-parameter bytes for MoE
0.783: runtime efficiency on the bandwidth term — inside the ~0.6–0.8 range typical of llama.cpp / MLX-class runtimes. Mixture-of-experts models use the same form with their own constants (below)
0.00146: fixed per-token cost in seconds (~1.46 ms) — kernel launches and sampling, which do not shrink as the model does
Cross-chip scaling: tok/s scales with the bandwidth ratio only while the bandwidth term dominates — below roughly 4 GB of weights the fixed cost takes over, and doubling bandwidth stops doubling speed
They are solved, not chosen. The two unknowns were fixed in August 2026 by the two vendor-published M5 Max figures the dataset then held: LFM2.5-2.6B at 220 tok/s and Muse Glimmer 30B at 27 tok/s.
The result was then checked against a figure it had never seen — Muse Glimmer 30B on M4 Max, vendor-published at 24 tok/s. The formula predicts 24.1, an error of +0.5%. The single community figure on an M4 Pro comes out about 23% high, which is what a thermally-limited laptop under sustained load should do.
5 October 2026: the fit was first solved with the M5 Max at 600 GB/s, the figure the index carried for its MacBook Pro configurations. Apple’s figure is 614 GB/s (and 307 GB/s for the M5 Pro, which the index had at 273), so the efficiency is 0.783 rather than 0.801. Bandwidth × efficiency on the M5 Max is unchanged, so every M5 Max estimate is unchanged; other chips’ estimates moved about 2% lower, M5 Pro estimates up to 12% higher, and the held-out M4 Max check improved from +3% to +0.5%.
One efficiency multiplier cannot fit both ends of the range. At 30B, decode really is bandwidth-bound; at 2–3B the fixed per-token cost dominates, and a pure bandwidth model predicts speeds no small model actually reaches. Every estimated figure in the index was recomputed with the formula above on 15 August 2026. Before that, 63 of them implied a model reading its weights faster than the memory bus could deliver them — not achievable on any hardware. Measured figures, the filled squares and hollow circles, were not touched by this and never are.
Mixture-of-experts models — refit 5 October 2026
A mixture-of-experts (MoE) model stores all of its parameters but reads only the active ones for each token, so its bandwidth term uses active-parameter bytes: a 35B MoE with 3B active reads about 1.7 GB per token instead of 20 GB. It also pays more fixed cost per token than a dense model — routing, gathering experts, more kernel launches per layer. Until October the index scaled the dense formula by a flat 0.42, fitted to a single 2-bit DeepSeek V4 Flash run. Six fully specified plain-decode measurements of small MoEs then came in at 2–4× those estimates:
| Model | Chip | Quant / runtime | Measured | Source |
|---|---|---|---|---|
| Qwen 3 30B-A3B | M4 Max | 4-bit, MLX | 109.7 tok/s | arXiv 2601.19139 (sourced) |
| Qwen 3 30B-A3B | M4 Max | 4-bit, llama.cpp | 89.9 tok/s | arXiv 2601.19139 (sourced) |
| Qwen 3.6-35B-A3B | M5 Max | 4-bit, MLX, MTP off | 125.8 tok/s | oMLX database (community) |
| Qwen 3.6-35B-A3B | Mac mini M6 | 4-bit, MLX | 63.8 tok/s | HDZucht, raw data (community) |
| Nemotron 3.5 Lightning | Mac mini M6 | Q4_K_M | 45.1 tok/s | HDZucht (community) |
| Gemma 4 26B-A4B | Mac mini M6 | QAT | 48.2 tok/s | HDZucht (community) |
0.42: the large-MoE factor, fitted in September 2026 to the then only fully specified large-MoE runs (DeepSeek V4 Flash at 2-bit). As of 18 November 2026 the index holds 71 measured plain-decode rows, for 18 large MoEs, that may anchor an estimate. Against the formula without the factor they land at 0.26–0.88 (median 0.57 by model; that is 0.6–2.1× the 0.42 estimate). Per model: DeepSeek V4.1 Flash 0.33, Qwen3.8-Flash-Next 0.40, Mistral Large 4 0.41, DeepSeek V4 Flash 0.42, MiniMax-M3.1-Flash 0.43, Ling-3.1-flash 0.45, Ling-3.0 Flash 0.50, Muse Spark Open 0.54, GLM 5.2 0.56, Step-3.7-Flash 0.58, Qwen 3.5 122B-A10B 0.61, GPT-oss 120B 0.61, MiniMax-M3 0.63, Qwen3-235B-A22B 0.67, Nemotron 3.5 Super VL 120B-A12B 0.68, Qwen 3.5 397B-A17B 0.71, MiMo-V2.6-Flash 0.75, Nemotron 3.5 Bolt 74B-A9B 0.79. 0.42 is kept as a deliberately conservative default, below that median, but 3 of the 18 measured lower still (DeepSeek V4.1 Flash, Qwen3.8-Flash-Next, Mistral Large 4), so for a large MoE with no run of its own the estimate is a rough figure in either direction. Any model with a measured row is anchored on that row instead of the factor (computed from benchmarks.json on every build)
November 2026 runs: as of 14 November 2026 (the line above is recomputed on every build), the new measured rows for Qwen 3.5 397B-A17B (one 4-bit run on a 256 GB M5 Ultra) and Qwen3-235B-A22B (one 4-bit M5 Ultra run) land at 0.71 and 0.67 of the unscaled formula, and Step-3.7-Flash’s four runs at 0.42–0.64 by row (0.42 for its 2-bit oQ2e build and 0.60 for oQ3e, both on an M5 Max). 0.42 stays the default for a large MoE with no run of its own. It is a conservative default, not a guaranteed floor: three measured models sit below it
Own measurements first: a model with its own measured rows has every other chip scaled from its nearest-bandwidth measured row, using the same class model
Not bandwidth-bound: ternary and sub-2-bit formats decode through unpacking kernels; the index quotes only their measured figures and scales those, labelled estimated
The refit moved the small MoE models up the leaderboard: on an M5 Max several now estimate above 100 tok/s, where speed points cap at 25. On 5 October the index’s #1, KAT-Coder-V2.5, ranked on an estimate from this model (~117 tok/s); the first linked run measured 89.3 tok/s (community, oMLX), and since 17 October the reference-row rule ranks it on that measurement. The full 35B of weights still has to be resident — about 20 GB — which is why it is a 24 GB-Mac model despite generating like a 3B one.
Which figure the leaderboard shows — the reference row (17 October 2026, extended 24 October and 14 November)
Each model has one reference figure: the number the leaderboard prints, the LLMCheck Score uses, and every answer capsule, llms.txt line and quotable stat repeats. One rule picks it, and one function in the build computes it for every page:
- Reference chip. The M5 Max (614 GB/s); for a model that needs more than 128 GB, which no M5 Max can hold, the Mac Studio M5 Ultra (1,228 GB/s), tagged on the leaderboard.
- A measurement wins. Among the model’s measured plain-decode rows on that chip (sourced or community) at 5 bits or fewer, or with an unstated quant, the index takes the lower median. The figure shown is always a real run, with its own quant, runtime and provenance, and never simply the best run anyone posted.
- Otherwise the estimate, at Q4_K_M (or the model’s native format, such as MXFP4). When Q4_K_M does not fit the reference chip’s largest configuration, the estimate is costed at the quant that sets the model’s minimum RAM, and the row names it.
- Sub-2-bit models (ternary, 1-bit, PQ2_0, Q1_0) are measured-only: their decode is kernel-bound, the formula does not describe them, and the leaderboard shows their best measured figure with the chip it was taken on.
Which runs may anchor an estimate. A model’s other chips are scaled from its nearest-bandwidth measured row. A row whose quant the source did not state is displayed, but anchors only when the model has no stated-quant measurement — unless the model is natively quantized (below). Rows flagged anchor: false (a runtime patched with unmerged code — a fork, a custom port or a loader from an open pull request — or a run its own source doubts), SSD-streamed runs and runs on binned configurations are displayed, but never anchor and never become the reference figure (a binned run can still set a floor under the full chip’s estimate; below). Binned means a configuration with less memory bandwidth than the full chip: the 32-core-GPU M5 Max in the 36 GB Mac Studio (460 GB/s against 614), or the 16 GB Mac mini M6 (153 against 170). Fewer GPU cores alone does not make a part binned: the 64-core M5 Ultra and the 60-core M3 Ultra have their chip’s full 1,228 and 819 GB/s, so their runs anchor and count for the reference figure like any other. Pre-release builds of a released runtime (oMLX .dev and rc builds, mlx-vlm from its main branch) are treated as released. For one model at one quant, an estimate is capped at the estimate for any higher-bandwidth chip, so anchoring two chips on two different runs can never make more bandwidth look slower. Speeds measured with multi-token prediction (MTP) or speculative decoding, and long-prompt averages, are a different workload and are never entered as rows: since 19 November 2026 a speculative figure is kept on the plain row of the same session (below) and decode at depth as that row’s long-context field (below), and neither can become a reference figure.
One row per measurement session (14 November 2026). A session is one sitting by one person or publisher: one Mac, one runtime build and one checkpoint, however many runs it logs. The index enters each session as one row, at the session’s shortest context (the median where that context was run several times), leaving out any run the source itself documents as a cold start or warm-up. Back-to-back repeats are one session, and so is one benchmark logged at 1K and at 4K; a different day, machine, runtime build or checkpoint is a different session. The lower median is then taken over sessions, not over individual runs, so one prolific submitter cannot set a model’s figure: a single 21-run series of MiMo-V2.6-Flash on one 80-core M5 Ultra on 26 September is one row (61.7 tok/s at 1K), and the model’s M5 Ultra reference figure is the lower median of its 15 sessions. This was already how the dataset was built; it is written down here because the #1 can turn on it.
Natively quantized models (24 October 2026). GPT-oss 20B and GPT-oss 120B ship their weights in MXFP4, so a run whose source did not state a quant is a run of MXFP4. The index costs such a run at MXFP4 and treats it as a stated-quant run: it anchors estimates alongside runs labelled MXFP4, and it counts for the reference row on the same terms. Without this rule, the first run labelled MXFP4 (GPT-oss 20B on an M5 Ultra, 184 tok/s community-measured) would have hidden the model’s two unlabelled runs, 45 tok/s on a Mac mini M6 (community) and 107.5 on an M5 Max (published), and every other chip would have been rescaled from the M5 Ultra run, where the fixed per-token cost dominates. The 16 GB Mac mini M6 estimate would have gone from 41 to 115 tok/s, and the M5 Max estimate to 170, above the 107.5 measured on that chip.
Coherence within a chip generation (24 October 2026). Scaling from the nearest-bandwidth run can rank a cut-down chip above the full one. On 24 October, Recursive Flint’s nearest anchor to the M5 Max is an M3 Ultra run, which scaled to 77–95 tok/s on the full 614 GB/s M5 Max, while a community run on the binned 460 GB/s M5 Max (the 36 GB Mac Studio) measured 131. The M3 generation is simply slower per GB/s on this model than the M5. Two rules now hold within one chip generation, where a generation is the chip family (M5, M5 Pro, M5 Max and M5 Ultra are one) and binned parts count as configurations of their chip:
- Floor. An estimate is never below a measured plain-decode run of the same model on a slower configuration of the same generation at equal or more bits per weight. The full M5 Max cannot be estimated slower than the binned M5 Max was measured, nor the M5 Ultra slower than either. Where a slower configuration has several qualifying runs (5 bits or fewer, not
anchor: false, not SSD-streamed), its lower median counts. Builds within about 10% of each other in bytes per weight count as the same width, so Q4_K_M, MXFP4, INT4 and 4-bit MLX builds compare directly. - Cap. An estimate is never above a measured run on a faster configuration of the same generation, unless that run used a build so much heavier that the bandwidth gap cannot explain the difference. When the two rules conflict, the floor wins.
- Per-configuration figures never decrease with bandwidth. The figures a Best-by-Mac page prints for one model rise, or stay level, as bandwidth rises within a generation. The build applies the same two rules to them, and validation fails if an estimate makes a figure fall as bandwidth rises.
Across generations the floor never applies: a newer chip family’s run says nothing about an older chip’s minimum. The cap does apply when the faster chip’s figure comes from a build of the same or fewer bytes per weight, because more bandwidth reading the same weights bounds a slower chip whatever its family. Recursive Flint’s M4 Max estimate (546 GB/s) is therefore held at or below the 4-bit run measured on the faster M3 Ultra (819 GB/s, community91 tok/s), at estimated~89 tok/s, instead of a figure scaled up from the M5 generation. Where a floor and a cross-generation cap conflict, the floor wins: on 24 October that held Flint’s full-M5 Max estimate at the binned M5 Max’s measured 131 tok/s although the M3 Ultra measured less, because both are real results from different chips, not an error to correct. Measured rows are never changed; only estimates move.
Update, 14 November 2026. The full chip has since been measured. Five qualifying community runs on 40-core M5 Max machines, 25 October to 1 November (llama.cpp b11871 and oMLX 0.7.3 pre-releases, 77.4 to 132.5 tok/s across five 4-bit-class builds), replace the estimate, and Flint’s leaderboard figure is now their lower median, community123 tok/s on the M5 Max (oMLX’s plain 4-bit build). That is below the binned part’s single 131 tok/s run. Neither row is edited: the floor and the cap bound estimates, not measurements, so two measurements that disagree stay as published, the build reports the pair as a warning, and the 36 GB Mac Studio page keeps quoting its own measured run.
Which model a Best-by-Mac page picks — 18 October 2026
Each Best-by-Mac page ranks every model that fits the Mac’s memory by suitability: capability (0–50) plus a speed bonus of tok/s × 0.12 on that Mac, capped at 15. The speed is the figure the page prints for that Mac, with its provenance. The headline pick is the best-ranked model that:
- reaches 15 tok/s on that Mac. Slower models stay in the table, tagged slow, and the most capable of them can take the “Most capable (slow)” card. Without this floor, dense 27B models at under 10 tok/s headlined the Mac mini M6 pages.
- has a Mac runtime. Models with none are never ranked: Solar Open 2 250B, Step 5 Preview.
- carries no runtime caveat. A model that runs on a Mac only through a community port, a runtime still in an open pull request, or Metal kernels nobody has validated is ranked and listed, but never headlined; the page names it and says why. Until 18 October 2026 Nemotron 3.5 Bolt headlined the Mac Studio M5 Ultra 96 GB page although stock llama.cpp then crashed on it (fixed in b11755, 20 October). Caveated models as of 18 November 2026:
- Beam 1.0 — Metal unvalidated: llama.cpp b11701+ loads Beam, but no real-weight run on Apple Silicon has been published; there is no MLX port, and Ollama offers it cloud-only.
- Hy4 preview — llama.cpp has its hy_v4 architecture from b10813, but it was validated only on CUDA and no Mac run has been published; MLX needs an unmerged mlx-lm branch.
- Inkling — llama.cpp support is the unmerged PR #25731 and stock mlx-lm cannot load it; the MLX builds bundle their own loader.
- Inkling 2 Small — Runs on llama.cpp b12406+ (PR #32217) and oMLX 0.8.0 (18 Nov 2026, PR #5229), but no plain-decode Mac run on a released runtime is in the index yet; stock mlx-lm cannot load it (#2061).
- Inkling-Small — llama.cpp support is the unmerged PR #25731 and stock mlx-lm cannot load it; the MLX builds bundle their own loader.
- Maple Preview 20B-A1B — Runs only through DeepGrove's own forks (mlx-lm-deepgrove, deepgrove-ai/llama.cpp); its cards name no stock llama.cpp, mlx-lm, LM Studio or Ollama route.
- Nemotron 3.5 Bolt 74B-A9B — Stock llama.cpp runs it from b11755, but NVIDIA's 24 GB GGUF falls back to the CPU on its NVFP4 layers (use a Q4_K_M file); Ollama does not support it on Apple Silicon (PR #19229) and stock mlx-lm cannot load it.
RAM tiers. /best-llm/by-ram/ groups the 54 configurations into 8 tiers — 8 GB, 16 GB, 24 GB, 32 GB, 64 GB, 128 GB, 192–256 GB, 512 GB. A Mac belongs to the tier whose floor is the highest at or below its memory (a 36 or 48 GB Mac sits in the 32 GB tier, a 96 GB Mac in 64 GB, a 192 GB Mac in 192–256 GB). Each tier’s picks are computed at its smallest Mac’s memory, so every pick fits every Mac listed in it, and quoted on one representative Mac (8 GB: MacBook Air M1 8 GB; 16 GB: Mac mini M6 16 GB; 24 GB: Mac mini M6 24 GB; 32 GB: Mac mini M6 32 GB; 64 GB: Mac mini M5 Pro 64 GB; 128 GB: Mac Studio M5 Max 128 GB; 192–256 GB: Mac Studio M5 Ultra 256 GB; 512 GB: Mac Studio M5 Ultra 512 GB). The homepage RAM strip and FAQ, the leaderboard FAQ, llms.txt and the key-numbers page quote exactly these picks and figures, and the “best pick at 32 GB or less” that accompanies every #1 sentence is the 32 GB tier’s pick on its own Mac — no M5 Max ships with 32 GB or less, so the M5 Max reference figure is never quoted for it.
Step 3 — A worked example
▶ Llama 3.3 70B (dense) on an M5 Max, Q4_K_M
- Weights:
70 × 0.57 ≈ 40 GBon disk and in RAM. - The model’s need:
40 + ~2 (runtime and a short context’s KV cache) ≈ 42 GB. macOS’s own few gigabytes sit outside the GPU limit, so they are not added here. - Fit check: by default macOS 15 lets the GPU use
64 × 0.75 = 48 GBof a 64 GB Mac (macOS 26: about 52) — the 42 GB fits. A 48 GB Mac allows36 GB(macOS 26: about 37) — short of 42, and even the index’s raised limit for 48 GB, 42 GB, leaves no margin. So a 64 GB Mac is the practical floor; the catalog records the model’s own need, about 44 GB (GPU memory limit). - Speed ceiling:
614 GB/s ÷ 40 GB ≈ 15.4 tok/stheoretical maximum. - Apply the formula:
1 ÷ (40 ÷ (614 × 0.783) + 0.00146) ≈ 12 tok/s— the estimated figure the index shows.
What the estimates can't capture
Honest limits of the model: thermals (a fanless MacBook Air throttles on long generations; a Mac Studio doesn't), context length (speed degrades as the KV cache grows — long chats get slower; the memory it takes is computed, below, but decode at depth is shown only where a run measured it), quantization variants (Q5, Q8, and MLX 4-bit all shift size and speed), and runtime differences (Ollama, LM Studio, llama.cpp, and MLX can differ 10–20% on the same hardware). Real numbers will vary from the estimates — that's exactly why every estimated row is labeled, and why community submissions are invited to replace estimates with real runs.
How far off the estimates are: the typical error (19 November 2026)
Every measured plain-decode run the speed formula can describe is compared with what the formula alone predicts for that model, chip and quant, without the anchoring on a model’s own runs: the figure a model gets when it has no run of its own. The typical error of a class is the median of |formula ÷ measured − 1| over its runs, and the signed median shows the direction. /best-llm/ and model pages print the class figure beside estimated speeds. It describes the formula, not one model: half the runs in a class sit further from the formula than its typical error. A measured run on the reference chip replaces the estimate altogether.
| Model class | Typical error (median |formula ÷ measured − 1|) | Median signed error | Runs | Models |
|---|---|---|---|---|
| Dense | ±9% | −6% | 75 | 20 |
| Small MoE (≤40B total) | ±15% | −7% | 88 | 15 |
| Large MoE (>40B total) | ±34% | −33% (the formula usually undershoots) | 70 | 18 |
233 measured runs counted (dataset version 2026-11-18): sourced and community plain-decode rows at a stated quant of 8 bits or fewer, not sub-2-bit, on a full-bandwidth chip, not flagged anchor: false, SSD-streamed or binned. Computed on every build into /data/estimator_error.json.
Time to first token. The estimator models decode only. Since 5 October 2026 estimated rows carry no time-to-first-token; a TTFT appears only where a source published one. Prompt processing is where the M5 generation’s Neural Accelerators made their biggest gains, so a decode estimate says little about it.
Context length and memory: RAM at 64K (19 November 2026)
A model’s minimum RAM covers its weights and about 2 GB for the runtime at a short context. A long context needs more. For every token in the context, each attention layer keeps a key and a value vector, the KV cache, and the cache grows with every token the prompt and the reply add. A coding agent that carries tool definitions, files and a long conversation in its prompt fills tens of thousands of tokens, so for every model whose layout it has read, the index computes what that context costs in memory:
bytesPerToken: over every full-attention layer, 2 (a key and a value) × KV heads × head dimension × 2 bytes, summed. The inputs come from the model’s own config.json (layer count, layer types,
num_key_value_heads, head_dim), which data/models.json links per model in its kv.source field16-bit by default: llama.cpp, Ollama, LM Studio and mlx-lm keep the cache at 16 bits unless told otherwise, so every figure the index prints assumes 16-bit
8-bit (q8): an 8-bit cache halves the cache a longer context adds: llama.cpp
--cache-type-k q8_0 --cache-type-v q8_0, Ollama OLLAMA_KV_CACHE_TYPE=q8_0 (GGUF models on Ollama’s llama.cpp engine only, where its FAQ says this needs Flash Attention, which it turns on where supported; its MLX engine never reads it), mlx-lm --kv-bits 8. The leaderboard gives the 8-bit total beside the 16-bit one; a 4-bit cache quarters the added part at a larger quality costSliding windows: models that alternate full-attention layers with sliding-window layers (GPT-oss and Gemma 3, for example) keep at most window tokens in each sliding layer, so that share stops growing once the context passes the window
MLA: multi-head latent attention (DeepSeek V3, Kimi K2) caches one compressed latent per layer instead of full keys and values, (
kv_lora_rank + qk_rope_head_dim) × 2 bytes per layer per token; the index costs the latent as the config defines it, and a runtime that expands it needs moreLinear attention and state-space layers: hybrids that mix full attention with Gated DeltaNet (the Qwen3-Next design) or Mamba layers keep a fixed-size state in those layers that does not grow with the context. They add nothing per token, and only their full-attention layers count. Each model’s
kv.note says which case applies
Agent-ready at 64K. A model is agent-ready at 64K on a Mac when its own context window is at least 64K tokens and its RAM at 65,536 tokens, with a 16-bit KV cache, is no more than the GPU memory the Mac can use with the limit raised (the GPU memory limit, below; on an 8 GB Mac, which has no safe raise, the default). Until 19 November 2026 it was compared with the whole unified memory, which macOS never gives the GPU. A context longer than the model’s window never fits, however much memory the Mac has. 64K is a common setting for local coding-agent tools, which send tool definitions, files and the whole conversation with every request. It is a statement about memory only: it does not say how fast the model runs that deep. Decode slows as the context fills, because each new token reads the cache as well as the weights, and a 64K prompt has to be processed (prefilled) before the first token appears. Where a run measured decode at depth or prefill speed, the benchmarks table shows it beside the run, and the leaderboard prints prefill under the speed. The index estimates neither: prefill is compute-bound, and the bandwidth model does not describe it.
Why the cache is added to the minimum, not folded into it. The minimum RAM is the floor for running a model at a short context. The cache beyond that is the price of a long context, and between models of similar size it differs by ten times or more: a hybrid with few full-attention layers, or an MLA model, holds 64K in a few gigabytes, while a dense model with many KV heads can need more memory for the cache than for its weights. Folding one fixed allowance into the minimum would misstate both. The runtime holds the cache beside the weights (llama.cpp reserves it for the whole context window when it loads a model), and macOS lets the GPU use only part of unified memory by default (the GPU memory limit, below), so a model at the edge of the fit may need a shorter context, an 8-bit cache or a raised limit. The figures are labelled estimated: they are arithmetic on the published architecture, not a reading of any Mac’s memory.
Worked examples, computed on every build
| Model | Attention layout (from config.json) | KV added at 64K, 16-bit / 8-bit | Min RAM | RAM at 64K | Agent-ready from |
|---|---|---|---|---|---|
| MiMo-V2.6-Flash | 9 full layers (K 192 + V 128 per kv head) + 39 sliding layers (window 128) config.json | 1.5 / 0.8 GB | 170 GB | estimated172 GB | 192 GB |
| Qwen 3.6-35B-A3B | hybrid: full attention on 10 of 40 layers; Gated DeltaNet layers keep a constant state config.json | 1.3 / 0.7 GB | 24 GB | estimated26 GB | 32 GB |
| GLM-4.7-Flash | MLA on all 47 layers (kv_lora_rank 512 + rope 64); runtimes that expand MLA to per-head K/V use far more config.json | 3.4 / 1.7 GB | 18 GB | estimated22 GB | 32 GB |
| K2-Horizon-3.7B | GQA on all 36 layers (8 kv heads x 128) config.json | 9.1 / 4.6 GB | 4 GB | estimated14 GB | 24 GB |
117 of the 130 catalog models have a recorded layout as of 18 November 2026 (computed on every build); the rest print — until their config.json is read.
The macOS GPU memory limit (19 November 2026)
Unified memory is shared, but macOS does not let the GPU use all of it. Metal reports the ceiling as recommendedMaxWorkingSetSize, which Apple describes as an approximation of how much memory the GPU can allocate without affecting its performance. llama.cpp prints it at startup (ggml_metal_device_init: recommendedMaxWorkingSetSize = … MB), Ollama logs it as the Metal total, and MLX treats it as the system’s wired limit. The limit is a performance threshold, not a hard cap. A model whose weights and cache fit the Mac’s memory but not this limit runs fully on the GPU only after the limit is raised; otherwise, depending on the runtime, it loads with a shorter context or part of the model on the CPU (llama.cpp, Ollama), runs slowly (MLX), or stops with an out-of-memory error.
How the rule was read. Apple publishes no formula. The index reads it off the startup logs people post with their Mac’s memory and macOS version, keeping only values that are a clean fraction of the installed memory and recur across reporters (a raised limit is a round number someone typed). Every memory size Apple sells is covered, but not every report states the memory: for some sizes it is inferred from the chip and the value, and for 18, 48, 64 and 96 GB no cited report states the memory. macOS 26 changed the rule at every size from 16 to 256 GB. At 512 GB the reported 464 GB cannot be told apart from a raised limit (it is exactly 475,136 MiB, and it also appears on macOS 15.7), so the index uses the 256 GB band’s 87%. The two macOS 27 values found, at 24 and 32 GB, match macOS 26.
Which figure a page uses. The lowest default that any macOS the Mac can run allows. An M5 or M6 Mac shipped with macOS 26 and cannot run an older one, so it gets the macOS 26 figure. An M1–M4 Mac may still run macOS 15, so it gets the macOS 15 figure, which is never higher; a tag that applies only there says (macOS 15). Requirements and limits are compared in the catalog’s gigabytes, which errs on the safe side.
What changes on the pages. Not the ranking and not the fit rule: a model is listed for a Mac when its minimum RAM fits the Mac’s memory, as before. A listed model whose minimum RAM is above the Mac’s default limit is tagged raise GPU limit, with the exact command in the tag’s tooltip and in the page’s GPU memory limit note; one that needs more than even the raised limit is tagged over GPU limit: expect heavy swapping, a CPU split or an out-of-memory error. Measured runs outrank the formula: a model with a published or community run on a Mac with no more memory is never tagged over the limit, only raise GPU limit, and the tag says so. An over-limit model stays listed and ranked but is never a page’s headline pick, as with a runtime caveat. Model pages mark the same for each memory tier in their memory-by-context table, where “fits” means within the GPU memory a Mac of that tier can use with the limit raised; agent-ready at 64K means the same (above).
Figures per configuration. A measured run is shown on a configuration of its chip only if its build fits there: the Mac it ran on had no more memory, or the build (parameters × the run’s bits per weight, plus about 2 GB) fits the GPU memory that configuration can use with the limit raised. Otherwise the configuration shows the estimate for the build it can hold, labelled estimated, and never above the figure measured on that chip. Until 19 November 2026 a 24 GB M5 Pro page showed Recursive Flint’s community run at Q4_1, a build of about 26 GB measured on 48 GB Macs.
Raising it. sudo sysctl iogpu.wired_limit_mb=N sets the limit to N MiB until the Mac restarts (macOS 13 used debug.iogpu.wired_limit, in bytes). Apple’s MLX documentation says the wired limit should remain strictly less than the total memory, and the mlx-lm README that N should be larger than the model and smaller than the machine’s memory; neither names a reserve for macOS, and a whole-system stall was reported on a 128 GB Mac with the limit raised to within 6.4 GB of its memory (the report blames the app’s own memory budget, not the limit). So the index’s command keeps a reserve, stated under the table. Quit other apps before loading a model near the limit.
| Unified memory | Default GPU limit, macOS 26 and later | macOS 15 and earlier | Raise it (until the next restart) | Raised limit / left for macOS |
|---|---|---|---|---|
| 8 GB | 5.3 GB (67%) | 5.3 GB (67%) | — | |
| 16 GB | 11.8 GB (74%) | 10.7 GB (67%) | sudo sysctl iogpu.wired_limit_mb=12288 | 12 GB / 4 GB |
| 18 GB | 13.3 GB (74%) | 12 GB (67%) | sudo sysctl iogpu.wired_limit_mb=14336 | 14 GB / 4 GB |
| 24 GB | 17.8 GB (74%) | 16 GB (67%) | sudo sysctl iogpu.wired_limit_mb=20480 | 20 GB / 4 GB |
| 32 GB | 25 GB (78%) | 21.3 GB (67%) | sudo sysctl iogpu.wired_limit_mb=28672 | 28 GB / 4 GB |
| 36 GB | 28.1 GB (78%) | 27 GB (75%) | sudo sysctl iogpu.wired_limit_mb=32256 | 31.5 GB / 4.5 GB |
| 48 GB | 37.4 GB (78%) | 36 GB (75%) | sudo sysctl iogpu.wired_limit_mb=43008 | 42 GB / 6 GB |
| 64 GB | 51.8 GB (81%) | 48 GB (75%) | sudo sysctl iogpu.wired_limit_mb=57344 | 56 GB / 8 GB |
| 96 GB | 77.8 GB (81%) | 72 GB (75%) | sudo sysctl iogpu.wired_limit_mb=86016 | 84 GB / 12 GB |
| 128 GB | 107.5 GB (84%) | 96 GB (75%) | sudo sysctl iogpu.wired_limit_mb=114688 | 112 GB / 16 GB |
| 192 GB | 161.3 GB (84%) | 144 GB (75%) | sudo sysctl iogpu.wired_limit_mb=180224 | 176 GB / 16 GB |
| 256 GB | 222.7 GB (87%) | 192 GB (75%) | sudo sysctl iogpu.wired_limit_mb=245760 | 240 GB / 16 GB |
| 512 GB | 445.4 GB (87%) | 384 GB (75%) | sudo sysctl iogpu.wired_limit_mb=507904 | 496 GB / 16 GB |
MB is what llama.cpp prints (106 bytes); GB is Apple’s memory unit, in which the catalog states every requirement. The raised limit leaves max(4 GB, one eighth of the memory) for macOS, at most 16 GB, and the command is printed only where it exceeds the macOS 26 default, so it never lowers the limit on any macOS. sudo sysctl iogpu.wired_limit_mb=0 restores the default without a restart. Computed on every build from context_fit.py.
Sources: Apple’s and MLX’s documentation, then a startup log for each memory size, with its macOS version where the report states one. Some sizes are inferred from the chip and the value; for 18, 48, 64 and 96 GB no cited report states the memory. The 512 GB row uses the 256 GB band’s 87%: the 464 GB that 512 GB Macs report equals a typed 475,136 MiB raise.
- Apple: MTLDevice.recommendedMaxWorkingSetSize
- MLX: mlx.core.set_wired_limit (the wired limit, iogpu.wired_limit_mb)
- mlx-lm README: large models and iogpu.wired_limit_mb
- iogpu.wired_limit_mb=0 is the default split; Ventura used debug.iogpu.wired_limit (llama.cpp #4406)
- macOS 26.6, 8 GB M1: 5726.63 MB (unchanged from macOS 15)
- macOS 26.5, 16 GB M4 (16.0 GiB): 12713.12 MB
- 18 GB M3 Pro: 14302.25 MB (memory inferred from the chip / value)
- macOS 27.2, 24 GB M6: 18186 MiB (Ollama log)
- macOS 26.4, 32 GB M5 MacBook Air: 26800.60 MB
- macOS 26.1, 36 GB M3 Pro: 30150.67 MB (memory inferred from the chip / value)
- macOS 26.2, 48 GB M4 Pro: 40200.90 MB (memory inferred from the chip / value)
- macOS 26.5.2, 64 GB M3 Max: 55662.79 MB (memory inferred from the chip / value)
- 96 GB: 83494.17 MB (memory inferred from the value)
- 128 GB M5 Max (“M5 with 128 GB”): 115448.73 MB
- 192 GB M2 Ultra (192.0 GiB): 173173.08 MB
- 256 GB M3 Ultra (256.0 GiB): 239143.78 MB
- 512 GB: 498216.21 MB = 475,136 MiB, indistinguishable from a raised limit (not used)
- macOS 15.8.1, 36 GB M3 Pro: 28991 MB (the older 3/4 rule)
- macOS 15.3.1, 128 GB M4 Max: 103079.22 MB (3/4)
- A raised limit: iogpu.wired_limit_mb=114688 gives 120259.08 MB
- A stall on a 128 GB Mac with the limit raised to within 6.4 GB of its memory (the report blames the app’s own memory budget, not the limit)
Speculative decoding and MTP: shown, never ranked (19 November 2026)
Speculative decoding has a cheap drafter propose several tokens ahead, and the full model checks them all in one pass. The drafter can be the model’s own multi-token-prediction (MTP) heads, a DFlash or EAGLE head, a small draft model, or n-gram lookup. Accepted tokens are tokens the model would have produced anyway, so the output is the model’s own; but the speed now depends on how many drafts are accepted, and that depends on the text.
Plain decode stays the ranked figure for three reasons:
- The gain depends on the content. In one llama.cpp report on an M2 Ultra (Qwen3.6-27B on Metal), MTP made code 1.35× faster but creative writing 0.91× as fast as plain decode (llama.cpp #23052, community). Another, on an M1 Max with Qwen3.5-9B, found MTP slower than plain decode in every setting tried (#23752, community). No single multiplier describes a model.
- It depends on the runtime and its settings: the drafter, how many tokens it drafts, the sampling temperature. These differ between runtimes and change between releases.
- The estimator cannot model it. The bandwidth model prices one read of the weights per token, and speculative decoding changes how many tokens each read yields. Plain decode is the figure that compares across chips, models and runtimes, so it alone sets the reference figure, the Score, the estimates and their anchors.
The rules for speculative figures. Since 19 November 2026 the dataset carries them in a separate spec field on a measured plain-decode row, never as rows of their own:
- Measured only. No speculative figure is ever estimated, or scaled to another chip.
- Paired. One is entered only on the plain-decode row from the same session (the same Mac, runtime build, checkpoint and quant), so the plain figure beside it is its baseline. A source that publishes only a drafted figure is cited in prose instead.
- Labelled. Each figure names its method (MTP, DFlash, EAGLE, a named draft model, n-gram, or the runtime’s default) and its workload (code, chat, or unstated). The benchmarks table prints it as a multiplier on the plain figure and lists every one in its speculative-decoding table.
- Never ranked. A speculative figure never becomes a reference figure or an anchor, and it never enters the Score, an estimate, the coherence floor or cap, or a RAM-tier pick. It never sorts a table.
Ollama turns it on by default for some models. Since v0.31.1 (30 June 2026), Ollama runs Gemma 4 on Apple Silicon with MTP drafting on by default, with no setting to change; its release notes put the gain at “nearly 90% faster on average across a coding-agent benchmark”. Since v0.32.6 (4 August 2026) its MLX engine also uses Qwen 3.5’s MTP head automatically. A Gemma 4 run on Ollama 0.31.1 or later therefore counts as drafted unless its source says MTP was off: it can enter the dataset only as a speculative figure beside a paired plain run, never as a plain-decode row. This is one reason Ollama can show a higher number than the index for the same model on the same Mac.
The LLMCheck Score Formula
The estimation model above feeds one of four components in the composite 0–100 LLMCheck Score used to rank the catalog:
Speed (0–25): the displayed tok/s on the reference chip × 0.25, capped at 25 — the lower-median measured run where one exists, otherwise estimated via the model above (reference row)
Accessibility (0–15): RAM tier: ≤8 GB = 15, ≤16 GB = 12, ≤24 GB = 10, ≤32 GB = 9, ≤48 GB = 6, ≤64 GB = 5, above 64 GB = 3
License (0–10): MIT = 10, GLM-5.3 / Modified MIT = 9, Apache 2.0 / OpenMDW = 8, Gemma / LFM Open / NVIDIA Open / Solar License = 6, Meta Custom / Mistral Ecosystem Open-Weight / Mistral Modified MIT / Moonshot Custom / Muse Model License / Qwen Custom / StepFun Model License = 5, MiniMax Community / xAI Custom = 4, CC-BY-NC / MiniMax Non-Commercial / N/A = 2
Dimension 1: Capability (50 points) — sourced
Raw model intelligence — reasoning, knowledge, coding, instruction following. Sourced from published third-party benchmark systems, linked per model.
The capability score (capScore) is the most heavily weighted dimension because the primary value of an LLM is output quality. capScore is sourced, not estimated: the index normalizes it from three public benchmark systems:
- Arena AI ELO Rating (40% weight) — Human preference ranking from lmarena.ai. Measures how often real users prefer a model's response over competitors in blind A/B tests. The gold standard for "does this model feel good to use."
- MMLU Score (35% weight) — Massive Multitask Language Understanding. Tests knowledge across 57 academic subjects. Sourced from official model cards on HuggingFace and arXiv papers.
- Coding Benchmarks (25% weight) — HumanEval (code generation) and SWE-Bench Verified (real bug fixing). Sourced from llm-stats.com and official model publications.
When official benchmarks are unavailable for a model (common for very new releases), the index extrapolates from related models in the same family and marks the score as estimated — same provenance rules as everything else. These estimates are replaced with sourced scores within 2–4 weeks of release.
Why 50 points? A very fast model that gives poor answers is less useful than a slower model with excellent reasoning. Capability is weighted highest because it determines whether the model actually solves your problem. Speed and RAM determine whether you can run it — but there's no point running a model that can't help you.
Capability Score Table (All 130 Models)
| Model | Params | capScore | License | Key published evidence | Source |
|---|---|---|---|---|---|
| DeepSeek V4 Pro | 1.6T MoE | 50 | MIT | SWE-bench Verified 80.6% (V4-Pro-Max, vendor) | HF |
| DeepSeek V4.1 Flash | 552B MoE | 50 | MIT | 1017 snapshot: DeepSWE v1.1 77.0 (Mini-SWE-V2 harness), Terminal-Bench 4.0 42.3 (Claude Code harness), HLE 42.6 (model card) | HF |
| GLM 5.2 | 753B MoE | 50 | MIT | SWE-Bench Pro 62.1% (SEAL standardized) / 68.5% vendor scaffold | HF |
| GLM-5.3 | 753B MoE | 50 | GLM-5.3 | Terminal-Bench 2.1 88.2, DeepSWE v1.1 66.9 (model card); post-training upgrade of GLM-5.2 | HF |
| Kimi K3 | 2.8T MoE | 50 | Moonshot Custom | AA Intelligence Index #3; Frontend Code Arena #1 (1,679) | HF |
| Kimi K3.1 Pro | 2.8T MoE | 50 | Moonshot Custom | AA's highest-ranked open-weights model (independent) | HF |
| MiMo-V2.6-Pro | 1.02T MoE | 50 | MIT | RL-checkpoint card figures: Terminal-Bench 2.1 89.9, DeepSWE v1.1 71.9; the MOPD checkpoint publishes no new table | HF |
| Qwen3.8-2.4T-A95B | 2.4T MoE | 50 | Qwen Custom | SWE-Bench Pro 67.7, GPQA 92.6 (model card) | HF |
| MiMo-V2.6-Flash | 309B MoE | 49 | MIT | RL-checkpoint card figures: Terminal-Bench 2.1 87.6, DeepSWE v1.1 67.9; the MOPD checkpoint publishes no new table | HF |
| Muse Spark Open | 816B MoE | 49 | Muse Model License | Artificial Analysis Intelligence Index 39 (independent); GPQA Diamond 89.0, Terminal-Bench 2.1 76.4 (Terminus 2), DeepSWE 1.1 60 (preferred harness) (model card) | HF |
| Step 5 Preview | 600B MoE | 49 | StepFun Model License | Artificial Analysis Intelligence Index 44; DeepSWE v1.1 67.7 (SWE-agent harness), Terminal-Bench 4.0 33.3 (card) | HF |
| GLM-5.1 | 754B MoE | 48 | MIT | Led open SWE-Bench Pro before GLM 5.2 | HF |
| GLM-5.3-Flash | 320B MoE | 48 | MIT | Terminal-Bench 2.1 84.3 (model card); vendor claims above GLM-5.2 — no SWE-Bench Pro figure on the card as of 7 Sep 2026 | HF |
| Hy4 preview | 770B MoE | 48 | Apache 2.0 | SWE-bench Pro 65.7 (harness not stated; GLM-5.3 64.6 on the same table), GPQA 92.3; the card calls it an early version with known issues | HF |
| Inkling 2 Small | 594B MoE | 48 | Apache 2.0 | AA Intelligence Index v5 30.6 (independent); Terminal-Bench 2.1 81.3, GPQA Diamond 93.0 (model card) | HF |
| Kimi K2.6 | 1T MoE | 48 | Modified MIT | 1T-A32B multimodal agent flagship | HF |
| Kimi K2.7 Code | 1T MoE | 48 | Modified MIT | Coding-focused K2.6 update; the card's benchmarks are charts only, no SWE-bench figure published | HF |
| MiniMax-M3 | 428B MoE | 48 | MiniMax Community | SWE-bench Verified 80.5, SWE-bench Pro 59 (harness not stated), MMMU-Pro 78.1 (model card) | HF |
| MiniMax-M3.1-Flash | 428B MoE | 48 | MiniMax Community | — | HF |
| Beam 1.0 | 501.3B MoE | 47 | Apache 2.0 | SWE-bench Verified 81.6 (harness not stated), SWE-Bench Pro 66.1 (vendor scaffold), Terminal-Bench 2.1 80.1, GPQA 90.6 (model card); AA Index 30 | HF |
| DeepSeek V4 Flash | 284B MoE | 47 | MIT | AA Index 50 (top open tier); Terminal-Bench 2.1 82.7 (vendor) | HF |
| DeepSeek V4 Flash Vision-Exp | 305B MoE | 47 | MIT | Inherits V4 Flash text evidence; vision experimental, no independent benchmark verified | HF |
| Inkling | 975B MoE | 47 | Apache 2.0 | AA Index 41; SWE-bench Verified 77.6 | HF |
| Kimi K2.5 | 1T MoE | 47 | Modified MIT | — | HF |
| Ling-3.1-flash | 560B MoE | 47 | MIT | SWE-bench Pro 68.74 (LingCode vendor scaffold), Terminal-Bench 2.1 81.18, HLE 37.44 (HF eval results) | HF |
| Qwen3.8-Flash-Next | 125B MoE | 47 | Qwen Custom | SWE-Bench Pro 62.5 (vendor scaffold), LiveCodeBench v6 91.9, GPQA 91.7 (model card) | HF |
| Inkling-Small | 276B MoE | 46 | Apache 2.0 | SWE-bench Verified 80.2 (bash-only harness); GPQA 89.5 | HF |
| Nanbeige4.3-33B | 33.1B | 46 | Apache 2.0 | AA ranks it level with Qwen3.8-27B at the top of small open models (independent); vendor: SWE-Bench Pro 60.3 (mini-swe-agent, not SEAL), GPQA 89.5, Terminal-Bench 2.1 80.9 | HF |
| Qwen3-235B-A22B | 235B MoE | 46 | Apache 2.0 | — | HF |
| Qwen3.8-27B | 27.8B | 46 | Apache 2.0 | SWE-Bench Pro 61.7 (vendor scaffold), Terminal-Bench 2.1 73.0, LiveCodeBench v6 90.3, GPQA 89.2 (model card) | HF |
| DeepSeek V3.2-Speciale | 685B MoE | 45 | MIT | High-reasoning sibling of V3.2 | HF |
| Mistral Large 4 | 1.05T MoE | 45 | Mistral Ecosystem Open-Weight | AA Intelligence Index v5 26 (independent); AA Coding Agent Index level with GLM-5.3 | HF |
| Qwen 3.6-27B | 27B | 44 | Apache 2.0 | SWE-bench Verified 77.2 | HF |
| dots3-note preview | 280B MoE | 44 | Apache 2.0 | Vendor bar chart: SWE-bench Pro 61.0, Terminal-Bench 2.1 75.1 (harness not stated); preview checkpoint | HF |
| Laguna S 2.1 | 118B MoE | 43 | OpenMDW | Terminal-Bench 2.1 70.2 (max thinking); SWE-Pro public 59.4 | HF |
| Mistral Medium 3.5 | 128B | 43 | Mistral Modified MIT | SWE-Bench Verified 77.6 (harness not stated), τ³-Telecom 91.4 (model card and Mistral announcement) | HF |
| DeepSeek V3.2 | 685B MoE | 42 | MIT | — | HF |
| Ling-3.0-flash-VL | 124B MoE | 42 | MIT | AA ranks it with Nemotron 3.5 Super VL among the two strongest medium open models (independent); vendor AA v4.1.1 42 vs Ling-3.0 Flash 38 | HF |
| MiniMax-M2.7 | 228.7B MoE | 42 | MiniMax Non-Commercial | SWE-Pro 56.2 (harness not stated), Terminal Bench 2 57.0, SWE Multilingual 76.5 (vendor) | HF |
| Muse Glimmer 30B | 30B | 42 | Apache 2.0 | SWE-Bench Pro 51.2; AIME 94.7; GPQA 83.5 (vendor) | HF |
| Nemotron 3 Ultra 550B-A55B | 550B MoE | 42 | OpenMDW | SWE-bench Verified 70.7 (harness not stated), Terminal-Bench 2.1 56.4, GPQA 87.0, LiveCodeBench v6 89.0 (model card) | HF |
| Nemotron 3.5 Super VL 120B-A12B | 120B MoE | 42 | OpenMDW | SWE-bench Verified 71.1, SWE-bench Pro 51.8 (vendor scaffold), GPQA 89.5 (model card); AA Index 26 (independent) | HF |
| Ripple-65B Preview | 64.8B | 42 | Apache 2.0 | Ricursive Intelligence, self-reported: SWE-bench Verified 76.5 (OpenCode harness), Terminal-Bench 2.1 80.5, GPQA Diamond 85.2; new lab, preview checkpoint (model card) | HF |
| Solar Open 2 250B | 250B MoE | 42 | Solar License | MMLU-Pro 86.2; LiveCodeBench v6 92.4 (vendor) | HF |
| Hunyuan Hy3 | 295B MoE | 41 | Apache 2.0 | AA Index 41 | HF |
| Step-3.7-Flash | 198B MoE | 41 | Apache 2.0 | SWE-Bench Pro 56.3 (vendor scaffold), Terminal-Bench 2.1 59.5 (model card) | HF |
| Gemma 4 31B | 31B | 40 | Apache 2.0 | — | HF |
| MiniMax M2.5 | 230B MoE | 40 | Modified MIT | SWE-bench Verified 80.2 (launch coverage) | HF |
| Nemotron 3.5 Bolt 74B-A9B | 74B MoE | 40 | OpenMDW | BF16: GPQA Diamond 88.8, Terminal-Bench 2.1 53.9, BrowseComp 70.0 (model card); no SWE-bench figure | HF |
| KAT-Coder-V2.5 | 35B MoE | 39 | Apache 2.0 | SWE-bench Verified 69.4 | HF |
| Ling-3.0 Flash | 124B MoE | 38 | MIT | AA Index 38 | HF |
| Qwen 3.5 122B-A10B | 122B MoE | 38 | Apache 2.0 | SWE-bench Verified 72.0, GPQA 86.6, MMLU-Pro 86.7 (model card) | HF |
| Qwen 3.5 397B-A17B | 397B MoE | 38 | Apache 2.0 | — | HF |
| Qwen 3.6-35B-A3B | 35B MoE | 38 | Apache 2.0 | SWE-bench 73.4 | HF |
| DeepSeek R1 | 671B MoE | 37 | MIT | — | HF |
| DeepSeek V3 | 685B MoE | 37 | MIT | — | HF |
| Recursive Flint | 39.8B MoE | 37 | Apache 2.0 | SWE-bench Verified 74.2, Terminal-Bench 2.1 54.9 (Recursive). No Arena, MMLU or HumanEval figure: capability estimated from vendor coding benchmarks only | HF |
| Ternary Bonsai 2 27B | 27.36B | 37 | Apache 2.0 | Vendor: 98.2% of FP16 quality retained across 14 benchmarks (model card) | HF |
| Kolibri-1 | 78.1B MoE | 36 | Apache 2.0 | SWE-Bench Verified 66.4 (harness not stated), GPQA Diamond 84.3, LiveCodeBench v6 85.9 (model card) | HF |
| Llama 4 Maverick | 400B MoE | 36 | Meta Custom | — | HF |
| Mistral Large 3 | 675B MoE | 36 | Apache 2.0 | — | HF |
| Nemotron 3.5 Lightning | 30B MoE | 36 | OpenMDW | AA accuracy-vs-speed Pareto claim (vendor) | HF |
| North Mini Code 1.0 | 30.5B MoE | 36 | Apache 2.0 | SWE-bench Verified 67.6 (SWE-agent v1.1.0), SWE-bench Pro 40.2 (SWE-agent vendor scaffold) (Cohere eval results) | HF |
| GLM-4.7-Flash | 31B MoE | 35 | MIT | — | HF |
| Gemma 4 26B-A4B | 26B MoE | 35 | Apache 2.0 | — | HF |
| LFM2.5-24B-A2B | 24B MoE | 35 | LFM Open | SWE-Bench Verified 51.0 (OpenCode harness), IFEval 96.8, GPQA Diamond 74.4 (model card) | HF |
| Llama 3.1 405B | 405B | 35 | Meta Custom | — | HF |
| MiniCPM-V 4.7 35B-A3B | 35.2B MoE | 35 | Apache 2.0 | Card tables are images; third-party 4-bit eval (mlx-optiq): MMLU 86.2, HumanEval 90.9 | HF |
| Qwen3-Coder-Next | 80B MoE | 35 | Apache 2.0 | SWE-bench Verified 70.6 | HF |
| GLM-4.7 | 355B MoE | 34 | MIT | — | HF |
| Mistral Small 4 | 119B MoE | 34 | Apache 2.0 | — | HF |
| Raon-1 mini preview | 28.8B MoE | 34 | Apache 2.0 | — | HF |
| Bonsai 27B | 27B | 33 | Apache 2.0 | ~90% of Qwen3.6-27B FP16 quality (vendor) | HF |
| GLM-4.5-Air | 106B MoE | 33 | MIT | — | HF |
| Laguna XS 2.1 | 33B MoE | 33 | OpenMDW | — | HF |
| Step-3.5-Flash | 196B MoE | 33 | Apache 2.0 | — | HF |
| GPT-oss 120B | 117B MoE | 32 | Apache 2.0 | — | HF |
| MiMo-V2-Flash | 309B MoE | 32 | MIT | — | HF |
| Granite 4.2 30B | 29.28B | 31 | Apache 2.0 | SWE Bench Verified 57 (harness not stated), SWE Bench Pro 33.3, LiveCodeBench v6 75.8 (model card) | HF |
| DeepSeek R1 70B | 70B | 30 | MIT | — | HF |
| Gemma 4 12B | 12B | 30 | Apache 2.0 | MMLU-Pro 77.2, GPQA Diamond 78.8, LiveCodeBench v6 72.0 (model card) | HF |
| Llama 4 Scout | 109B MoE | 30 | Meta Custom | — | HF |
| Nemotron-Cascade 2 | 30B MoE | 30 | NVIDIA Open | AIME / LiveCodeBench golds at 3B active (vendor) | HF |
| Apertus 1.5 70B | 70B | 29 | Apache 2.0 | — | HF |
| Hermes 4 70B | 70B | 28 | Meta Custom | — | HF |
| Mixtral 8x22B | 141B MoE | 28 | Apache 2.0 | — | HF |
| Qwen 2.5 72B | 72B | 28 | Apache 2.0 | — | HF |
| Llama 3.3 70B | 70B | 27 | Meta Custom | — | HF |
| Qwen 3.5 35B-A3B | 35B MoE | 27 | Apache 2.0 | — | HF |
| QwQ 32B | 32B | 26 | Apache 2.0 | — | HF |
| Mistral Small 3.2 24B | 24B | 25 | Apache 2.0 | — | HF |
| Qwen 3 32B | 32B | 25 | Apache 2.0 | — | HF |
| DeepSeek R1 32B | 32B | 24 | MIT | — | HF |
| Devstral Small 24B | 24B | 24 | Apache 2.0 | — | HF |
| GPT-oss 20B | 21B MoE | 22 | Apache 2.0 | SWE-bench Verified 53.2, GPQA Diamond 58.6 at medium reasoning (model card) | HF |
| Gemma 3 27B | 27B | 22 | Gemma | — | HF |
| K2-Horizon-3.7B | 5.1B | 22 | Apache 2.0 | AA ranks it the strongest open model of 4B or less, ahead of MiniCPM5-2B (independent); vendor: SWE-bench Verified 65.6 (harness not stated), Terminal-Bench 2.1 39.7, GPQA Diamond 68.9 | HF |
| Qwen 3 30B-A3B | 30B MoE | 22 | Apache 2.0 | — | HF |
| MiMo-V2.6-Distill-Qwen-9B | 9.41B | 21 | MIT | SWE Verified 61.1 (avg@3), SWE Pro 44.6, Terminal Bench 2.1 37.1 (model card) | HF |
| MiniCPM5-2B | 2.52B | 21 | Apache 2.0 | AA ranks it among the two strongest open models of 4B or less (independent); vendor GPQA 70.2, MMLU-Pro 70.8 | HF |
| Qwen 3.5 27B | 27B | 21 | Apache 2.0 | — | HF |
| Maple Preview 20B-A1B | 20B MoE | 20 | MIT | IMO-level claims (vendor, unverified) | HF |
| Nanbeige4.2-3B | 3B | 20 | Apache 2.0 | SWE-bench Verified 63.6 (vendor) | HF |
| Qwen 3 14B | 14B | 20 | Apache 2.0 | — | HF |
| Spark-X2.6-4B | 4.11B | 20 | Apache 2.0 | — | HF |
| Phi-4 14B | 14B | 19 | MIT | — | HF |
| Mellum2.1-12B-A2.5B-Thinking | 12.15B MoE | 18 | Apache 2.0 | SWE-bench Verified 47.0 and SWE-bench Pro 28.0 (Pi v0.73.1 harness, not SEAL), LiveCodeBench v6 82.0 (JetBrains) | HF |
| Ministral 3 14B | 14B | 18 | Apache 2.0 | — | HF |
| Qwen 2.5 14B | 14B | 18 | Apache 2.0 | — | HF |
| Qwen 3.5 9B | 9B | 18 | Apache 2.0 | — | HF |
| Gemma 3 12B | 12B | 17 | Gemma | — | HF |
| LFM3-1.7B | 1.73B | 17 | LFM Open | SWE-bench Verified 43.4 (OpenCode harness), IFEval 96.7, GPQA 63.4 (model card) | HF |
| Nemotron 3.5 Bolt E4B | 4B | 17 | OpenMDW | — | HF |
| Apertus 1.5 8B | 8B | 16 | Apache 2.0 | — | HF |
| DeepSeek R1 8B | 8B | 16 | MIT | — | HF |
| Gemma 4 E4B | 4B | 16 | Apache 2.0 | — | HF |
| Granite 4.2 8B | 8.79B | 16 | Apache 2.0 | SWE-Bench Verified 47.7 (harness not stated), LiveCodeBench v6 73.2, GPQA 64.1 (IBM) | HF |
| Ternary Bonsai 2 9B | 9.41B | 16 | Apache 2.0 | Vendor, vs Qwen3.5-9B: BFCLv3 71.8 (73.8), GPQA Diamond 69.9 (82.5), IFEval 90.0 | HF |
| Ling-3.0-tiny | 7.9B MoE | 15 | MIT | Only vendor-quoted AA indices (v4.1.1 25, Agentic 16, in an image): held below Gemma 4 E4B until reproduced | HF |
| Qwen 3 8B | 8B | 15 | Apache 2.0 | — | HF |
| LFM2.5-2.6B | 2.6B | 14 | LFM Open | Size-class instruction-following leader (vendor) | HF |
| Ministral 8B | 8B | 14 | Apache 2.0 | — | HF |
| Phi-4 Mini | 3.8B | 14 | MIT | — | HF |
| Gemma 4 E2B | 2B | 13 | Apache 2.0 | — | HF |
| Mistral 7B | 7B | 13 | Apache 2.0 | — | HF |
| Llama 3.1 8B | 8B | 12 | Meta Custom | — | HF |
| Qwen 3.5 4B | 4B | 12 | Apache 2.0 | — | HF |
| Qwen 3 4B | 4B | 11 | Apache 2.0 | — | HF |
| Gemma 3 4B | 4B | 10 | Gemma | — | HF |
| SmolLM3 3B | 3B | 10 | Apache 2.0 | — | HF |
Dimension 2: Speed on Apple Silicon (25 points) — estimated
Tokens per second on the reference chip: the M5 Max — or, for models that need more than 128 GB, which no M5 Max can hold, the Mac Studio M5 Ultra, tagged on the leaderboard. The figure is the reference row: the lower-median measured run on that chip where one exists, otherwise the estimate from the model above. LLMCheck runs no lab.
Speed points are calculated as: displayed tok/s × 0.25, capped at 25 points — exactly the figure the leaderboard shows. Any model at 100+ tok/s on the reference configuration receives full speed points. The reference conditions:
- Quantization: 4-bit (Q4_K_M for estimates) unless the row names another quant — a measured row keeps the quant it was run at, and the leaderboard prints it
- Reference hardware: M5 Max, 128 GB Unified Memory (614 GB/s bandwidth); Mac Studio M5 Ultra (1,228 GB/s) for models above 128 GB
- Runtime class: llama.cpp / MLX-class engines (Ollama, LM Studio) with dense efficiency ~0.6–0.8
- Context: short-context generation — long contexts reduce speed as the KV cache grows; decode at depth and prefill are shown beside a run where it measured them, and never scored
- Plain decode only: speculative-decoding (MTP) figures are shown beside their paired plain run and never enter the Score (why)
- Provenance: rows are upgraded from estimated to sourced or community as linked third-party tests and user submissions come in
Models that fit no Mac (more than 512 GB) or have no Mac runtime yet receive 0 speed points. The full dataset — with a provenance label on every row — is available for download at /data/.
Dimension 3: Accessibility (15 points)
How many Mac users can actually run this model? Lower RAM requirements = higher accessibility score.
Accessibility is a step function based on minimum RAM required at 4-bit quantization, derived from the RAM model above (weights + KV cache + macOS overhead, within the ~75% unified-memory budget). Since a large share of Mac users have 16 GB or less, accessibility is crucial for real-world impact:
| Min RAM | Points | Top-ranked models in the tier |
|---|---|---|
| ≤ 8 GB | 15 | Bonsai 27B, K2-Horizon-3.7B, Maple Preview 20B-A1B |
| ≤ 16 GB | 12 | Ternary Bonsai 2 27B, GPT-oss 20B, Gemma 4 12B |
| ≤ 24 GB | 10 | Qwen 3.6-35B-A3B, GLM-4.7-Flash, Recursive Flint |
| ≤ 32 GB | 9 | — (no current model) |
| ≤ 48 GB | 6 | Nemotron 3.5 Bolt 74B-A9B, Qwen3-Coder-Next, Ripple-65B Preview |
| ≤ 64 GB | 5 | Ling-3.0-flash-VL, GPT-oss 120B, Kolibri-1 |
| > 64 GB | 3 | MiMo-V2.6-Flash, DeepSeek V4 Flash, DeepSeek V4 Flash Vision-Exp |
Dimension 4: License Openness (10 points)
How freely can you use, modify, and distribute the model? More open = higher score.
| License | Points | Can Modify? | Commercial Use? | Models |
|---|---|---|---|---|
| MIT | 10 | Yes | Unrestricted | MiMo-V2.6-Flash, GLM-4.7-Flash, DeepSeek V4 Flash, DeepSeek V4 Flash Vision-Exp |
| GLM-5.3 | 9 | Yes | Yes (custom GLM-5.3 terms) | GLM-5.3 |
| Modified MIT | 9 | Yes | Yes (attribution clauses) | Kimi K2.6, Kimi K2.7 Code, Kimi K2.5, MiniMax M2.5 |
| Apache 2.0 | 8 | Yes | Yes (with notice) | Qwen 3.6-35B-A3B, Recursive Flint, KAT-Coder-V2.5, North Mini Code 1.0 |
| OpenMDW | 8 | Yes | Yes | Nemotron 3.5 Lightning, Laguna XS 2.1, Nemotron 3.5 Bolt 74B-A9B, Nemotron 3.5 Bolt E4B |
| Gemma | 6 | Yes | Yes (use restrictions) | Gemma 3 4B, Gemma 3 12B, Gemma 3 27B |
| LFM Open | 6 | Yes | Yes (revenue threshold) | LFM2.5-24B-A2B, LFM3-1.7B, LFM2.5-2.6B |
| NVIDIA Open | 6 | Yes | Yes (NVIDIA terms) | Nemotron-Cascade 2 |
| Solar License | 6 | Yes | Yes (attribution / terms) | Solar Open 2 250B |
| Meta Custom | 5 | Limited | Yes (<700M monthly users) | Llama 3.1 8B, Llama 4 Maverick, Llama 4 Scout, Llama 3.1 405B |
| Mistral Ecosystem Open-Weight | 5 | Yes | Yes, except Model-as-a-Service — API / hosted-inference businesses need a separate Mistral licence | Mistral Large 4 |
| Mistral Modified MIT | 5 | Yes | Yes, revenue-gated — large companies need a separate licence | Mistral Medium 3.5 |
| Moonshot Custom | 5 | Limited | Yes (revenue thresholds) | Kimi K3, Kimi K3.1 Pro |
| Muse Model License | 5 | Yes | Yes — “Built with Muse Spark Open” credit; API hosts above $20M revenue need a separate agreement | Muse Spark Open |
| Qwen Custom | 5 | Limited | Yes (revenue thresholds / product gates) | Qwen3.8-Flash-Next, Qwen3.8-2.4T-A95B |
| StepFun Model License | 5 | Yes | Yes — device makers and API hosts above $20M need a separate licence | Step 5 Preview |
| MiniMax Community | 4 | Yes | Yes, with attribution, below a revenue threshold | MiniMax-M3, MiniMax-M3.1-Flash |
| xAI Custom | 4 | Limited | Yes (custom terms) | (none currently listed) |
| CC-BY-NC | 2 | Yes | Non-commercial only | (none currently listed) |
| MiniMax Non-Commercial | 2 | Yes | Non-commercial only | MiniMax-M2.7 |
| N/A | 2 | No | API only | (none currently listed) |
Licence points are editorial calls, stated here so they can be audited. Licences that gate commercial use by revenue, require a product credit or need a separate agreement for API hosting score below Apache 2.0: the Muse Model License, the StepFun Model License and Mistral Modified MIT (revenue-gated, so deliberately not the 9 of plain Modified MIT) score 5, MiniMax Community 4 (commercial use permitted with attribution below a revenue threshold) and MiniMax Non-Commercial 2.
The Mistral Ecosystem Open-Weight License (Mistral Large 4, weights 29 October 2026) also scores 5. Its grant is MIT-style with an attribution notice, but any company running a Model-as-a-Service business — giving third parties inference or post-training access with meaningful control over inputs, parameters or training data — needs a separate commercial licence from Mistral, with no revenue threshold. Internal use, products that embed the model in their own features, and relaying requests to models hosted elsewhere are exempt. It gates a whole business category, which no 6-point licence does, so it sits with the other licences that need a separate agreement for API hosting, not with Apache 2.0.
Note: CC-BY-NC (non-commercial) scores 2 — usable for research but not commercial deployment. Meta / xAI community licenses score 5 / 4, reflecting modification rights with commercial caps.
Score Examples
Computed from the live data on every build, using exactly the figure the leaderboard displays.
MiMo-V2.6-Flash (Score: 81) = capScore 49 + speed 19 (min(25, 77 tok/s × 0.25); community-measured on an M5 Ultra at MXFP4) + accessibility 3 (170 GB) + license 10 (MIT) = 81.
KAT-Coder-V2.5 (Score: 79) = capScore 39 + speed 22 (min(25, 89 tok/s × 0.25); community-measured on an M5 Max at OptiQ-4bit) + accessibility 10 (24 GB) + license 8 (Apache 2.0) = 79.
Kimi K2.5 (Score: 66) = capScore 47 + speed 7 (min(25, 28 tok/s × 0.25); estimated on an M5 Ultra at 3.5 bpw) + accessibility 3 (417 GB) + license 9 (Modified MIT) = 66.
Fine-tunes and derivatives: which ones the index lists (14 November 2026)
A derivative is a checkpoint whose model card names a catalog model as its base, or whose architecture is identical to one (the same layers, experts and hidden size, with or without an extended vocabulary). That covers continued pre-training, supervised and reinforcement-learning fine-tunes, merges, abliteration and domain specialisation. A derivative is not listed as a separate model unless all four of these hold:
- An accountable publisher and a licence file. An identifiable organisation publishes it, with a licence file in the repository; a licence named only in the card’s metadata is not enough.
- A measured capability gain. A third-party evaluation (Artificial Analysis, LMArena, or a press or independent test that documents its setup), or the publisher’s own table against its base with the harness stated, shows a gain worth at least 2 capScore points over the base model.
- A Mac build on a stock runtime. A downloadable build that released llama.cpp, mlx-lm or mlx-vlm, LM Studio or Ollama runs.
- More than a behaviour change. A checkpoint whose only change is removed safety training, a persona or a writing style does not qualify, however it scores.
Three cases are not treated as derivatives. A lab’s updated checkpoint of its own model replaces the catalog entry instead of adding one (DeepSeek V4.1 Flash 1017, the MiMo-V2.6 MOPD checkpoints, Ling-3.1-flash-1019). A lab’s official distillation of its own model into a different architecture is a new model (MiMo-V2.6-Distill-Qwen-9B). A quantized build is a build of its base, not a model. And a derivative’s speed runs never stand in for its base: runs of an abliterated Recursive Flint build on an M4 Pro were not entered as Flint rows.
Considered in November 2026 and not listed (the condition each one misses):
- Arete-27B (Nous Research, 6 November; a supervised fine-tune of Qwen3.8-27B) — 2 and 4: its card reports a propaganda and factuality evaluation and claims no meaningful regression, with no standard-benchmark table against the base.
- Sage-A1.5 (InternScience; built on Qwen3.6-35B-A3B) — 2: its results are agentic-science evaluations with no comparison the index can score against its base, and its only Mac run used multi-token prediction.
- Thinklite-27B (Sparset; a fine-tune of Qwen3.8-27B) — 2: it is an efficiency tune, using about half the reasoning tokens while keeping 99.6% of the base’s average accuracy (89.8 against 90.2), by its own table.
- Clover-35B-A3B-Preview (DiffuSpace; base Qwen3.6-35B-A3B) — 2: a diffusion language model whose results are vendor-run and published as charts on its launch blog.
- Nex-N2.6 (Nex-AGI, 10 November; built on GLM-5.3-Flash) — 1: the repository has no licence file; Apache 2.0 appears only in the card metadata.
- Hydrogen-1.0 (Deep Principle, 31 October; a gated science assistant built on GLM-5.3) — 2: its card publishes no evaluation against its base.
- Atria Dawn (internlm, final weights 23 October; a 753B model built on GLM-5.2) — 1 and 3: the README says MIT but the repository has no licence file, and no Mac build of the final weights exists.
- Ornith-1.5 9B, 35B-A3B and 122B (ornith-ai; “developed on top of” Qwen 3.5 and Gemma 4, per the card) — 1 and 2: the licence file the card links returns 404, and every evaluation is vendor-run.
- Intern-S2-397B (Shanghai AI Laboratory; its configuration is Qwen 3.5 397B-A17B’s with the vocabulary extended) — 2: its benchmarks are published only as an image, with no comparison against the base the index can transcribe.
- Ling-3.0-flash-Fin (inclusionAI; finance continued training of Ling-3.0 Flash) — 1 and 2: no licence file in the repository, and finance-only benchmarks.
- webAI Air-26B-A4B — not a derivative but a quantized build of Gemma 4 26B-A4B (“nothing was trained”, per its card), under a non-commercial licence; its own table puts it a point below the base.
Derivatives already in the catalog. Six entries predate this policy and are derivatives under its definition: KAT-Coder-V2.5 (post-trained from Qwen3.6-35B-A3B), Nemotron 3.5 Bolt E4B (from Gemma 4 E4B), Hermes 4 70B (a fine-tune of Llama 3.1 70B, the architecture of the catalog’s Llama 3.3 70B), and Prism ML’s low-bit rebuilds Bonsai 27B (of Qwen3.6-27B), Ternary Bonsai 2 27B (of Qwen3.8-27B) and Ternary Bonsai 2 9B (of Qwen 3.5 9B). They stay listed while the policy is new. Each will be checked against the four conditions in the December 2026 update, and any change will be recorded, with its reason, in the changelog. The low-bit rebuilds trade capability for memory rather than adding capability, a case condition 2 does not describe, and the December review will say how they are treated.
Limitations & Known Issues
- Arena ELO volatility: Ratings shift weekly as new votes come in. The index snapshots monthly and notes the capture date.
- Estimated capScores: ~15% of capScores are extrapolated from model-family performance when official benchmarks are pending. These are marked "est." in the table above.
- Estimates are not measurements: speed figures come from the bandwidth model, which cannot capture thermals, long-context slowdown, or runtime-specific optimizations. Expect real-world variance of 10–20% either way — sometimes more on fanless machines.
- Quantization baseline: estimates are costed at each row’s own labelled quant (Q4_K_M for the reference estimate); measured rows keep the quant they were run at, and every leaderboard row prints its quant. Formats differ in size and speed, so compare like with like.
- No multimodal scoring: The current score does not reward multimodal capabilities (image/audio input). Models like Gemma 4 E4B have multimodal features not captured in the 0–100 score.
Corrections and real-world benchmark data are always welcome. If you have runs that differ from the index's estimates, submit them through the community benchmark process — verified community rows replace estimates.
Frequently Asked Questions
Does LLMCheck run its own benchmarks?
No. LLMCheck is an independent index, not a benchmark lab. Every figure on the site is one of three things: an estimate from the published LLMCheck estimation model (memory-bandwidth math and quantization arithmetic, fully documented on this page), a sourced number from a linked third-party benchmark, or a community-submitted run. Each row is labeled with its provenance, and no figure is claimed as a first-party lab measurement.
How does LLMCheck calculate its scores?
The LLMCheck Score is a 0-100 composite: Capability (50 pts) normalized from published third-party evaluations, Speed (25 pts) from the reference figure (the lower-median measured run on the M5 Max, or the M5 Ultra for models above 128 GB, where one exists, otherwise the published estimate; tok/s x 0.25, capped), Accessibility (15 pts) stepping down with minimum RAM, and License Openness (10 pts) where MIT scores 10 and restrictive licenses score lower. The formula is fully transparent and reproducible.
Where does LLMCheck get its capability scores?
Capability scores are derived from three public benchmark sources: Arena AI ELO ratings (human preference, weighted 40%), MMLU scores from official model cards (knowledge breadth, weighted 35%), and coding benchmarks like HumanEval and SWE-Bench (weighted 25%). All sources are linked per model. When official benchmarks are unavailable, the index extrapolates from related models in the same family and marks the score as 'estimated'.
How does LLMCheck estimate tokens per second on Apple Silicon?
Token generation on Apple Silicon is memory-bandwidth-bound, so the index estimates it with a two-term model: tok/s = 1 / (model GB / (bandwidth x 0.783) + 1.46 ms), where model GB is the parameter count times the bytes per parameter at the row's own quant (0.57 GB per billion at Q4_K_M). Mixture-of-experts models use active-parameter bytes. Since the 5 October 2026 refit, small MoEs (40B total or less) use a 4.97 ms per-token overhead fitted to six fully specified plain-decode runs; only large MoEs (above 40B) keep the deliberately conservative 0.42 factor. A model with its own measured runs has its other chips scaled from the nearest-bandwidth measured row, and a measured run on the reference chip replaces the estimate (the lower median where there are several). Every estimated figure is labelled; the full model is at llmcheck.net/methodology#estimation.
Why does LLMCheck weight capability at 50% of the total score?
Capability receives the highest weight because the primary value of an LLM is the quality of its outputs. A very fast model that gives poor answers is less useful than a slower model with excellent reasoning. However, speed (25%) and accessibility (15%) ensure that models which actually run well on consumer Macs score higher than server-only models with superior capability but no practical local use.
Why does Ollama show a faster speed than LLMCheck for some models?
Usually because of speculative decoding. Since v0.31.1 (30 June 2026) Ollama runs Gemma 4 on Apple Silicon with multi-token prediction (MTP) drafting on by default, and since v0.32.6 its MLX engine uses Qwen 3.5’s MTP head automatically. Drafting can speed up code substantially and slow other text down, so the index ranks plain decode only: a measured speculative figure is shown beside the plain run from the same session, labelled with its method and workload, and never enters the Score or an estimate. Details: llmcheck.net/methodology#speculative.
How often are LLMCheck scores updated?
Scores are updated within 48–72 hours of major model releases. The full leaderboard is refreshed monthly with the latest Arena AI ELO ratings and community benchmark submissions. Speed estimates are recomputed as new Apple Silicon hardware specifications become available. All updates are timestamped in the open dataset at llmcheck.net/data/.
Benchmark Sources
- Arena AI / LMSYS Chatbot Arena — Human preference ELO ratings
- HuggingFace Model Hub — Official model cards with MMLU and benchmark data
- LLM Stats — Aggregated benchmark scores (HumanEval, SWE-Bench)
- Onyx Open LLM Leaderboard — Tier rankings and benchmark aggregation
- Google DeepMind Gemma 4 Blog — Official Gemma 4 benchmarks
- GEO: Generative Engine Optimization (Princeton/Georgia Tech) — Research methodology reference
LLMCheck. (2026). LLMCheck Index Methodology. llmcheck.net. https://llmcheck.net/methodology (updated ). CC BY 4.0.
See the Full Leaderboard
130 verified models ranked by LLMCheck Score. Filter by your Mac's RAM, sort by speed or capability — every figure labeled with its provenance.
View Leaderboard →