LLMCheck Index Methodology

How the LLMCheck index ranks 130 local and frontier LLMs for Mac — a published estimation model, linked third-party benchmarks, and community submissions. Fully transparent, fully reproducible.

LLMCheck is an independent index of local-LLM performance on Apple Silicon — not a benchmark lab. Every figure is either a transparent estimate from the published model below, a sourced third-party benchmark (linked per row), or a community submission. Each data point is labeled with its provenance, and no figure is claimed as a first-party lab measurement.

Where Every Number Comes From

Every row in the leaderboard, the benchmarks table, and the open dataset carries one of three provenance labels. This mirrors the provenance_policy field published in benchmarks.json:

Estimated

Derived from the LLMCheck estimation model — memory-bandwidth scaling plus quantization arithmetic, documented in full below. Estimates are useful for planning ("will this model fit and feel usable on my Mac?") but are not measurements. Every estimated speed figure on the site links back to this page.

Sourced

A published figure from an accountable source, with the URL attached to the row — a vendor’s model card or announcement, an independent press review that documents its setup, Arena AI ELO ratings, MMLU/HumanEval/SWE-Bench results from official model cards and papers, or independent hardware test data. We link, we don't re-host or re-run.

Community

One person’s run on a real Mac — a blog post, forum thread or benchmark submission — with its URL as the source. Community numbers are sanity-checked against the estimation model and known baselines before inclusion. Own a Mac? Submit a benchmark — real runs beat estimates every time.

Why an index and not a lab? A single lab machine can only test one chip, one runtime, one macOS version. An index that publishes its estimation math, links its sources, and accepts community runs covers the entire Apple Silicon range — and you can audit every step. When we're estimating, we say so.

How We Estimate Performance

Local LLM performance on Apple Silicon is unusually predictable, because two hardware numbers dominate everything: unified memory capacity (can the model fit?) and memory bandwidth (how fast can weights stream through the GPU?). The LLMCheck estimation model is built on those two numbers. How its estimates compared with the runs later measured on the same chips is checked run by run in How accurate are Mac LLM speed estimates? (October 2026).

Step 1 — Will it fit? (RAM model)

model_size_GB (Q4_K_M) ≈ params_B × 0.57
≈ 4.5 bits/param effective — Q4_K_M is nominally 4-bit but carries scale factors, higher-precision layers, and embedding tables
+ KV cache: grows with every token of context, by an amount that differs between models of the same size by ten times or more — computed per model from its config.json (RAM at 64K, below); minimum RAM covers about the first 4K tokens only
+ macOS overhead: ~2–3 GB for the OS and the inference runtime
Fit rule: a model is listed for a Mac when its minimum RAM is no more than the Mac’s unified memory
+ GPU memory limit: macOS lets the GPU use only part of that memory by default, so a listed model that needs more than the default is tagged raise GPU limit, or over GPU limit when even a raised limit is short (the macOS GPU memory limit, below)

Until 19 November 2026 this step said macOS gives the GPU about three-quarters of the memory on every Mac. That is the macOS 15 figure from 36 GB up; below that macOS 15 allows two-thirds, and macOS 26 allows more at every size from 16 to 256 GB. The rule, its sources and the command that raises the limit are below.

Minimum RAM rule. A model’s minimum RAM is the size of its smallest build of at least 3 bits that has a working Mac runtime, plus about 2 GB for the runtime and a short (about 4K-token) context, leaving headroom within its memory tier; a longer context adds its KV cache on top (RAM at 64K). A 2-bit build sets the minimum only where a published Mac run used it.

Step 2 — How fast? (Speed model)

Token generation is memory-bandwidth-bound: for every token, the runtime must stream essentially all active model weights from RAM through the GPU. Compute is rarely the bottleneck on Apple Silicon. That gives a simple ceiling:

est. tok/s = 1 ÷ ( model_bytes_GB ÷ (bandwidth_GB/s × 0.783) + 0.00146 )
bandwidth_GB/s: Apple’s published memory bandwidth for the chip (e.g. M4 Pro 273 GB/s, M5 Pro 307 GB/s, M5 Max 614 GB/s); binned parts such as the 32-core-GPU M5 Max (460 GB/s) or the 16 GB Mac mini M6 (153 GB/s) use their own figure
model_bytes_GB: weight bytes read per token — total size for dense models, active-parameter bytes for MoE
0.783: runtime efficiency on the bandwidth term — inside the ~0.6–0.8 range typical of llama.cpp / MLX-class runtimes. Mixture-of-experts models use the same form with their own constants (below)
0.00146: fixed per-token cost in seconds (~1.46 ms) — kernel launches and sampling, which do not shrink as the model does
Cross-chip scaling: tok/s scales with the bandwidth ratio only while the bandwidth term dominates — below roughly 4 GB of weights the fixed cost takes over, and doubling bandwidth stops doubling speed
Where those two constants come from — revised 15 August and 5 October 2026

They are solved, not chosen. The two unknowns were fixed in August 2026 by the two vendor-published M5 Max figures the dataset then held: LFM2.5-2.6B at 220 tok/s and Muse Glimmer 30B at 27 tok/s.

The result was then checked against a figure it had never seen — Muse Glimmer 30B on M4 Max, vendor-published at 24 tok/s. The formula predicts 24.1, an error of +0.5%. The single community figure on an M4 Pro comes out about 23% high, which is what a thermally-limited laptop under sustained load should do.

5 October 2026: the fit was first solved with the M5 Max at 600 GB/s, the figure the index carried for its MacBook Pro configurations. Apple’s figure is 614 GB/s (and 307 GB/s for the M5 Pro, which the index had at 273), so the efficiency is 0.783 rather than 0.801. Bandwidth × efficiency on the M5 Max is unchanged, so every M5 Max estimate is unchanged; other chips’ estimates moved about 2% lower, M5 Pro estimates up to 12% higher, and the held-out M4 Max check improved from +3% to +0.5%.

One efficiency multiplier cannot fit both ends of the range. At 30B, decode really is bandwidth-bound; at 2–3B the fixed per-token cost dominates, and a pure bandwidth model predicts speeds no small model actually reaches. Every estimated figure in the index was recomputed with the formula above on 15 August 2026. Before that, 63 of them implied a model reading its weights faster than the memory bus could deliver them — not achievable on any hardware. Measured figures, the filled squares and hollow circles, were not touched by this and never are.

Mixture-of-experts models — refit 5 October 2026

A mixture-of-experts (MoE) model stores all of its parameters but reads only the active ones for each token, so its bandwidth term uses active-parameter bytes: a 35B MoE with 3B active reads about 1.7 GB per token instead of 20 GB. It also pays more fixed cost per token than a dense model — routing, gathering experts, more kernel launches per layer. Until October the index scaled the dense formula by a flat 0.42, fitted to a single 2-bit DeepSeek V4 Flash run. Six fully specified plain-decode measurements of small MoEs then came in at 2–4× those estimates:

ModelChipQuant / runtimeMeasuredSource
Qwen 3 30B-A3BM4 Max4-bit, MLX109.7 tok/sarXiv 2601.19139 (sourced)
Qwen 3 30B-A3BM4 Max4-bit, llama.cpp89.9 tok/sarXiv 2601.19139 (sourced)
Qwen 3.6-35B-A3BM5 Max4-bit, MLX, MTP off125.8 tok/soMLX database (community)
Qwen 3.6-35B-A3BMac mini M64-bit, MLX63.8 tok/sHDZucht, raw data (community)
Nemotron 3.5 LightningMac mini M6Q4_K_M45.1 tok/sHDZucht (community)
Gemma 4 26B-A4BMac mini M6QAT48.2 tok/sHDZucht (community)
small MoE (≤ 40B total): est. tok/s = 1 ÷ ( active_GB ÷ (bandwidth × 0.783) + 0.00497 )
large MoE (> 40B total): est. tok/s = 0.42 ÷ ( active_GB ÷ (bandwidth × 0.783) + 0.00146 )
0.00497: per-token overhead for small MoEs (~5 ms) — the median solved from the six runs above; mean absolute error ~12%, the dense model’s own accuracy class
0.42: the large-MoE factor, fitted in September 2026 to the then only fully specified large-MoE runs (DeepSeek V4 Flash at 2-bit). As of 18 November 2026 the index holds 71 measured plain-decode rows, for 18 large MoEs, that may anchor an estimate. Against the formula without the factor they land at 0.26–0.88 (median 0.57 by model; that is 0.6–2.1× the 0.42 estimate). Per model: DeepSeek V4.1 Flash 0.33, Qwen3.8-Flash-Next 0.40, Mistral Large 4 0.41, DeepSeek V4 Flash 0.42, MiniMax-M3.1-Flash 0.43, Ling-3.1-flash 0.45, Ling-3.0 Flash 0.50, Muse Spark Open 0.54, GLM 5.2 0.56, Step-3.7-Flash 0.58, Qwen 3.5 122B-A10B 0.61, GPT-oss 120B 0.61, MiniMax-M3 0.63, Qwen3-235B-A22B 0.67, Nemotron 3.5 Super VL 120B-A12B 0.68, Qwen 3.5 397B-A17B 0.71, MiMo-V2.6-Flash 0.75, Nemotron 3.5 Bolt 74B-A9B 0.79. 0.42 is kept as a deliberately conservative default, below that median, but 3 of the 18 measured lower still (DeepSeek V4.1 Flash, Qwen3.8-Flash-Next, Mistral Large 4), so for a large MoE with no run of its own the estimate is a rough figure in either direction. Any model with a measured row is anchored on that row instead of the factor (computed from benchmarks.json on every build)
November 2026 runs: as of 14 November 2026 (the line above is recomputed on every build), the new measured rows for Qwen 3.5 397B-A17B (one 4-bit run on a 256 GB M5 Ultra) and Qwen3-235B-A22B (one 4-bit M5 Ultra run) land at 0.71 and 0.67 of the unscaled formula, and Step-3.7-Flash’s four runs at 0.42–0.64 by row (0.42 for its 2-bit oQ2e build and 0.60 for oQ3e, both on an M5 Max). 0.42 stays the default for a large MoE with no run of its own. It is a conservative default, not a guaranteed floor: three measured models sit below it
Own measurements first: a model with its own measured rows has every other chip scaled from its nearest-bandwidth measured row, using the same class model
Not bandwidth-bound: ternary and sub-2-bit formats decode through unpacking kernels; the index quotes only their measured figures and scales those, labelled estimated

The refit moved the small MoE models up the leaderboard: on an M5 Max several now estimate above 100 tok/s, where speed points cap at 25. On 5 October the index’s #1, KAT-Coder-V2.5, ranked on an estimate from this model (~117 tok/s); the first linked run measured 89.3 tok/s (community, oMLX), and since 17 October the reference-row rule ranks it on that measurement. The full 35B of weights still has to be resident — about 20 GB — which is why it is a 24 GB-Mac model despite generating like a 3B one.

Which figure the leaderboard shows — the reference row (17 October 2026, extended 24 October and 14 November)

Each model has one reference figure: the number the leaderboard prints, the LLMCheck Score uses, and every answer capsule, llms.txt line and quotable stat repeats. One rule picks it, and one function in the build computes it for every page:

  1. Reference chip. The M5 Max (614 GB/s); for a model that needs more than 128 GB, which no M5 Max can hold, the Mac Studio M5 Ultra (1,228 GB/s), tagged on the leaderboard.
  2. A measurement wins. Among the model’s measured plain-decode rows on that chip (sourced or community) at 5 bits or fewer, or with an unstated quant, the index takes the lower median. The figure shown is always a real run, with its own quant, runtime and provenance, and never simply the best run anyone posted.
  3. Otherwise the estimate, at Q4_K_M (or the model’s native format, such as MXFP4). When Q4_K_M does not fit the reference chip’s largest configuration, the estimate is costed at the quant that sets the model’s minimum RAM, and the row names it.
  4. Sub-2-bit models (ternary, 1-bit, PQ2_0, Q1_0) are measured-only: their decode is kernel-bound, the formula does not describe them, and the leaderboard shows their best measured figure with the chip it was taken on.

Which runs may anchor an estimate. A model’s other chips are scaled from its nearest-bandwidth measured row. A row whose quant the source did not state is displayed, but anchors only when the model has no stated-quant measurement — unless the model is natively quantized (below). Rows flagged anchor: false (a runtime patched with unmerged code — a fork, a custom port or a loader from an open pull request — or a run its own source doubts), SSD-streamed runs and runs on binned configurations are displayed, but never anchor and never become the reference figure (a binned run can still set a floor under the full chip’s estimate; below). Binned means a configuration with less memory bandwidth than the full chip: the 32-core-GPU M5 Max in the 36 GB Mac Studio (460 GB/s against 614), or the 16 GB Mac mini M6 (153 against 170). Fewer GPU cores alone does not make a part binned: the 64-core M5 Ultra and the 60-core M3 Ultra have their chip’s full 1,228 and 819 GB/s, so their runs anchor and count for the reference figure like any other. Pre-release builds of a released runtime (oMLX .dev and rc builds, mlx-vlm from its main branch) are treated as released. For one model at one quant, an estimate is capped at the estimate for any higher-bandwidth chip, so anchoring two chips on two different runs can never make more bandwidth look slower. Speeds measured with multi-token prediction (MTP) or speculative decoding, and long-prompt averages, are a different workload and are never entered as rows: since 19 November 2026 a speculative figure is kept on the plain row of the same session (below) and decode at depth as that row’s long-context field (below), and neither can become a reference figure.

One row per measurement session (14 November 2026). A session is one sitting by one person or publisher: one Mac, one runtime build and one checkpoint, however many runs it logs. The index enters each session as one row, at the session’s shortest context (the median where that context was run several times), leaving out any run the source itself documents as a cold start or warm-up. Back-to-back repeats are one session, and so is one benchmark logged at 1K and at 4K; a different day, machine, runtime build or checkpoint is a different session. The lower median is then taken over sessions, not over individual runs, so one prolific submitter cannot set a model’s figure: a single 21-run series of MiMo-V2.6-Flash on one 80-core M5 Ultra on 26 September is one row (61.7 tok/s at 1K), and the model’s M5 Ultra reference figure is the lower median of its 15 sessions. This was already how the dataset was built; it is written down here because the #1 can turn on it.

Natively quantized models (24 October 2026). GPT-oss 20B and GPT-oss 120B ship their weights in MXFP4, so a run whose source did not state a quant is a run of MXFP4. The index costs such a run at MXFP4 and treats it as a stated-quant run: it anchors estimates alongside runs labelled MXFP4, and it counts for the reference row on the same terms. Without this rule, the first run labelled MXFP4 (GPT-oss 20B on an M5 Ultra, 184 tok/s community-measured) would have hidden the model’s two unlabelled runs, 45 tok/s on a Mac mini M6 (community) and 107.5 on an M5 Max (published), and every other chip would have been rescaled from the M5 Ultra run, where the fixed per-token cost dominates. The 16 GB Mac mini M6 estimate would have gone from 41 to 115 tok/s, and the M5 Max estimate to 170, above the 107.5 measured on that chip.

Coherence within a chip generation (24 October 2026). Scaling from the nearest-bandwidth run can rank a cut-down chip above the full one. On 24 October, Recursive Flint’s nearest anchor to the M5 Max is an M3 Ultra run, which scaled to 77–95 tok/s on the full 614 GB/s M5 Max, while a community run on the binned 460 GB/s M5 Max (the 36 GB Mac Studio) measured 131. The M3 generation is simply slower per GB/s on this model than the M5. Two rules now hold within one chip generation, where a generation is the chip family (M5, M5 Pro, M5 Max and M5 Ultra are one) and binned parts count as configurations of their chip:

Across generations the floor never applies: a newer chip family’s run says nothing about an older chip’s minimum. The cap does apply when the faster chip’s figure comes from a build of the same or fewer bytes per weight, because more bandwidth reading the same weights bounds a slower chip whatever its family. Recursive Flint’s M4 Max estimate (546 GB/s) is therefore held at or below the 4-bit run measured on the faster M3 Ultra (819 GB/s, community91 tok/s), at estimated~89 tok/s, instead of a figure scaled up from the M5 generation. Where a floor and a cross-generation cap conflict, the floor wins: on 24 October that held Flint’s full-M5 Max estimate at the binned M5 Max’s measured 131 tok/s although the M3 Ultra measured less, because both are real results from different chips, not an error to correct. Measured rows are never changed; only estimates move.

Update, 14 November 2026. The full chip has since been measured. Five qualifying community runs on 40-core M5 Max machines, 25 October to 1 November (llama.cpp b11871 and oMLX 0.7.3 pre-releases, 77.4 to 132.5 tok/s across five 4-bit-class builds), replace the estimate, and Flint’s leaderboard figure is now their lower median, community123 tok/s on the M5 Max (oMLX’s plain 4-bit build). That is below the binned part’s single 131 tok/s run. Neither row is edited: the floor and the cap bound estimates, not measurements, so two measurements that disagree stay as published, the build reports the pair as a warning, and the 36 GB Mac Studio page keeps quoting its own measured run.

Which model a Best-by-Mac page picks — 18 October 2026

Each Best-by-Mac page ranks every model that fits the Mac’s memory by suitability: capability (0–50) plus a speed bonus of tok/s × 0.12 on that Mac, capped at 15. The speed is the figure the page prints for that Mac, with its provenance. The headline pick is the best-ranked model that:

  1. reaches 15 tok/s on that Mac. Slower models stay in the table, tagged slow, and the most capable of them can take the “Most capable (slow)” card. Without this floor, dense 27B models at under 10 tok/s headlined the Mac mini M6 pages.
  2. has a Mac runtime. Models with none are never ranked: Solar Open 2 250B, Step 5 Preview.
  3. carries no runtime caveat. A model that runs on a Mac only through a community port, a runtime still in an open pull request, or Metal kernels nobody has validated is ranked and listed, but never headlined; the page names it and says why. Until 18 October 2026 Nemotron 3.5 Bolt headlined the Mac Studio M5 Ultra 96 GB page although stock llama.cpp then crashed on it (fixed in b11755, 20 October). Caveated models as of 18 November 2026:

RAM tiers. /best-llm/by-ram/ groups the 54 configurations into 8 tiers — 8 GB, 16 GB, 24 GB, 32 GB, 64 GB, 128 GB, 192–256 GB, 512 GB. A Mac belongs to the tier whose floor is the highest at or below its memory (a 36 or 48 GB Mac sits in the 32 GB tier, a 96 GB Mac in 64 GB, a 192 GB Mac in 192–256 GB). Each tier’s picks are computed at its smallest Mac’s memory, so every pick fits every Mac listed in it, and quoted on one representative Mac (8 GB: MacBook Air M1 8 GB; 16 GB: Mac mini M6 16 GB; 24 GB: Mac mini M6 24 GB; 32 GB: Mac mini M6 32 GB; 64 GB: Mac mini M5 Pro 64 GB; 128 GB: Mac Studio M5 Max 128 GB; 192–256 GB: Mac Studio M5 Ultra 256 GB; 512 GB: Mac Studio M5 Ultra 512 GB). The homepage RAM strip and FAQ, the leaderboard FAQ, llms.txt and the key-numbers page quote exactly these picks and figures, and the “best pick at 32 GB or less” that accompanies every #1 sentence is the 32 GB tier’s pick on its own Mac — no M5 Max ships with 32 GB or less, so the M5 Max reference figure is never quoted for it.

Step 3 — A worked example

▶ Llama 3.3 70B (dense) on an M5 Max, Q4_K_M

  1. Weights: 70 × 0.57 ≈ 40 GB on disk and in RAM.
  2. The model’s need: 40 + ~2 (runtime and a short context’s KV cache) ≈ 42 GB. macOS’s own few gigabytes sit outside the GPU limit, so they are not added here.
  3. Fit check: by default macOS 15 lets the GPU use 64 × 0.75 = 48 GB of a 64 GB Mac (macOS 26: about 52) — the 42 GB fits. A 48 GB Mac allows 36 GB (macOS 26: about 37) — short of 42, and even the index’s raised limit for 48 GB, 42 GB, leaves no margin. So a 64 GB Mac is the practical floor; the catalog records the model’s own need, about 44 GB (GPU memory limit).
  4. Speed ceiling: 614 GB/s ÷ 40 GB ≈ 15.4 tok/s theoretical maximum.
  5. Apply the formula: 1 ÷ (40 ÷ (614 × 0.783) + 0.00146) ≈ 12 tok/s — the estimated figure the index shows.

What the estimates can't capture

Honest limits of the model: thermals (a fanless MacBook Air throttles on long generations; a Mac Studio doesn't), context length (speed degrades as the KV cache grows — long chats get slower; the memory it takes is computed, below, but decode at depth is shown only where a run measured it), quantization variants (Q5, Q8, and MLX 4-bit all shift size and speed), and runtime differences (Ollama, LM Studio, llama.cpp, and MLX can differ 10–20% on the same hardware). Real numbers will vary from the estimates — that's exactly why every estimated row is labeled, and why community submissions are invited to replace estimates with real runs.

How far off the estimates are: the typical error (19 November 2026)

Every measured plain-decode run the speed formula can describe is compared with what the formula alone predicts for that model, chip and quant, without the anchoring on a model’s own runs: the figure a model gets when it has no run of its own. The typical error of a class is the median of |formula ÷ measured − 1| over its runs, and the signed median shows the direction. /best-llm/ and model pages print the class figure beside estimated speeds. It describes the formula, not one model: half the runs in a class sit further from the formula than its typical error. A measured run on the reference chip replaces the estimate altogether.

Model classTypical error (median |formula ÷ measured − 1|)Median signed errorRunsModels
Dense±9%−6%7520
Small MoE (≤40B total)±15%−7%8815
Large MoE (>40B total)±34%−33% (the formula usually undershoots)7018

233 measured runs counted (dataset version 2026-11-18): sourced and community plain-decode rows at a stated quant of 8 bits or fewer, not sub-2-bit, on a full-bandwidth chip, not flagged anchor: false, SSD-streamed or binned. Computed on every build into /data/estimator_error.json.

Time to first token. The estimator models decode only. Since 5 October 2026 estimated rows carry no time-to-first-token; a TTFT appears only where a source published one. Prompt processing is where the M5 generation’s Neural Accelerators made their biggest gains, so a decode estimate says little about it.

Context length and memory: RAM at 64K (19 November 2026)

A model’s minimum RAM covers its weights and about 2 GB for the runtime at a short context. A long context needs more. For every token in the context, each attention layer keeps a key and a value vector, the KV cache, and the cache grows with every token the prompt and the reply add. A coding agent that carries tool definitions, files and a long conversation in its prompt fills tens of thousands of tokens, so for every model whose layout it has read, the index computes what that context costs in memory:

KV(n) = bytesPerToken × n + slidingBytesPerToken × min(n, window)
RAM at a context = minimum RAM + ( KV(context) − KV(min(context, 4,096)) ) ÷ 109
Minimum RAM already covers a short context: the weights, the runtime and about the first 4,096 tokens of cache. At 4K or less, RAM at a context is the minimum RAM; a longer context adds only the cache beyond those first 4,096 tokens
bytesPerToken: over every full-attention layer, 2 (a key and a value) × KV heads × head dimension × 2 bytes, summed. The inputs come from the model’s own config.json (layer count, layer types, num_key_value_heads, head_dim), which data/models.json links per model in its kv.source field
16-bit by default: llama.cpp, Ollama, LM Studio and mlx-lm keep the cache at 16 bits unless told otherwise, so every figure the index prints assumes 16-bit
8-bit (q8): an 8-bit cache halves the cache a longer context adds: llama.cpp --cache-type-k q8_0 --cache-type-v q8_0, Ollama OLLAMA_KV_CACHE_TYPE=q8_0 (GGUF models on Ollama’s llama.cpp engine only, where its FAQ says this needs Flash Attention, which it turns on where supported; its MLX engine never reads it), mlx-lm --kv-bits 8. The leaderboard gives the 8-bit total beside the 16-bit one; a 4-bit cache quarters the added part at a larger quality cost
Sliding windows: models that alternate full-attention layers with sliding-window layers (GPT-oss and Gemma 3, for example) keep at most window tokens in each sliding layer, so that share stops growing once the context passes the window
MLA: multi-head latent attention (DeepSeek V3, Kimi K2) caches one compressed latent per layer instead of full keys and values, (kv_lora_rank + qk_rope_head_dim) × 2 bytes per layer per token; the index costs the latent as the config defines it, and a runtime that expands it needs more
Linear attention and state-space layers: hybrids that mix full attention with Gated DeltaNet (the Qwen3-Next design) or Mamba layers keep a fixed-size state in those layers that does not grow with the context. They add nothing per token, and only their full-attention layers count. Each model’s kv.note says which case applies

Agent-ready at 64K. A model is agent-ready at 64K on a Mac when its own context window is at least 64K tokens and its RAM at 65,536 tokens, with a 16-bit KV cache, is no more than the GPU memory the Mac can use with the limit raised (the GPU memory limit, below; on an 8 GB Mac, which has no safe raise, the default). Until 19 November 2026 it was compared with the whole unified memory, which macOS never gives the GPU. A context longer than the model’s window never fits, however much memory the Mac has. 64K is a common setting for local coding-agent tools, which send tool definitions, files and the whole conversation with every request. It is a statement about memory only: it does not say how fast the model runs that deep. Decode slows as the context fills, because each new token reads the cache as well as the weights, and a 64K prompt has to be processed (prefilled) before the first token appears. Where a run measured decode at depth or prefill speed, the benchmarks table shows it beside the run, and the leaderboard prints prefill under the speed. The index estimates neither: prefill is compute-bound, and the bandwidth model does not describe it.

Why the cache is added to the minimum, not folded into it. The minimum RAM is the floor for running a model at a short context. The cache beyond that is the price of a long context, and between models of similar size it differs by ten times or more: a hybrid with few full-attention layers, or an MLA model, holds 64K in a few gigabytes, while a dense model with many KV heads can need more memory for the cache than for its weights. Folding one fixed allowance into the minimum would misstate both. The runtime holds the cache beside the weights (llama.cpp reserves it for the whole context window when it loads a model), and macOS lets the GPU use only part of unified memory by default (the GPU memory limit, below), so a model at the edge of the fit may need a shorter context, an 8-bit cache or a raised limit. The figures are labelled estimated: they are arithmetic on the published architecture, not a reading of any Mac’s memory.

Worked examples, computed on every build

ModelAttention layout (from config.json)KV added at 64K, 16-bit / 8-bitMin RAMRAM at 64KAgent-ready from
MiMo-V2.6-Flash9 full layers (K 192 + V 128 per kv head) + 39 sliding layers (window 128) config.json1.5 / 0.8 GB170 GBestimated172 GB192 GB
Qwen 3.6-35B-A3Bhybrid: full attention on 10 of 40 layers; Gated DeltaNet layers keep a constant state config.json1.3 / 0.7 GB24 GBestimated26 GB32 GB
GLM-4.7-FlashMLA on all 47 layers (kv_lora_rank 512 + rope 64); runtimes that expand MLA to per-head K/V use far more config.json3.4 / 1.7 GB18 GBestimated22 GB32 GB
K2-Horizon-3.7BGQA on all 36 layers (8 kv heads x 128) config.json9.1 / 4.6 GB4 GBestimated14 GB24 GB

117 of the 130 catalog models have a recorded layout as of 18 November 2026 (computed on every build); the rest print — until their config.json is read.

The macOS GPU memory limit (19 November 2026)

Unified memory is shared, but macOS does not let the GPU use all of it. Metal reports the ceiling as recommendedMaxWorkingSetSize, which Apple describes as an approximation of how much memory the GPU can allocate without affecting its performance. llama.cpp prints it at startup (ggml_metal_device_init: recommendedMaxWorkingSetSize = … MB), Ollama logs it as the Metal total, and MLX treats it as the system’s wired limit. The limit is a performance threshold, not a hard cap. A model whose weights and cache fit the Mac’s memory but not this limit runs fully on the GPU only after the limit is raised; otherwise, depending on the runtime, it loads with a shorter context or part of the model on the CPU (llama.cpp, Ollama), runs slowly (MLX), or stops with an out-of-memory error.

How the rule was read. Apple publishes no formula. The index reads it off the startup logs people post with their Mac’s memory and macOS version, keeping only values that are a clean fraction of the installed memory and recur across reporters (a raised limit is a round number someone typed). Every memory size Apple sells is covered, but not every report states the memory: for some sizes it is inferred from the chip and the value, and for 18, 48, 64 and 96 GB no cited report states the memory. macOS 26 changed the rule at every size from 16 to 256 GB. At 512 GB the reported 464 GB cannot be told apart from a raised limit (it is exactly 475,136 MiB, and it also appears on macOS 15.7), so the index uses the 256 GB band’s 87%. The two macOS 27 values found, at 24 and 32 GB, match macOS 26.

Which figure a page uses. The lowest default that any macOS the Mac can run allows. An M5 or M6 Mac shipped with macOS 26 and cannot run an older one, so it gets the macOS 26 figure. An M1–M4 Mac may still run macOS 15, so it gets the macOS 15 figure, which is never higher; a tag that applies only there says (macOS 15). Requirements and limits are compared in the catalog’s gigabytes, which errs on the safe side.

What changes on the pages. Not the ranking and not the fit rule: a model is listed for a Mac when its minimum RAM fits the Mac’s memory, as before. A listed model whose minimum RAM is above the Mac’s default limit is tagged raise GPU limit, with the exact command in the tag’s tooltip and in the page’s GPU memory limit note; one that needs more than even the raised limit is tagged over GPU limit: expect heavy swapping, a CPU split or an out-of-memory error. Measured runs outrank the formula: a model with a published or community run on a Mac with no more memory is never tagged over the limit, only raise GPU limit, and the tag says so. An over-limit model stays listed and ranked but is never a page’s headline pick, as with a runtime caveat. Model pages mark the same for each memory tier in their memory-by-context table, where “fits” means within the GPU memory a Mac of that tier can use with the limit raised; agent-ready at 64K means the same (above).

Figures per configuration. A measured run is shown on a configuration of its chip only if its build fits there: the Mac it ran on had no more memory, or the build (parameters × the run’s bits per weight, plus about 2 GB) fits the GPU memory that configuration can use with the limit raised. Otherwise the configuration shows the estimate for the build it can hold, labelled estimated, and never above the figure measured on that chip. Until 19 November 2026 a 24 GB M5 Pro page showed Recursive Flint’s community run at Q4_1, a build of about 26 GB measured on 48 GB Macs.

Raising it. sudo sysctl iogpu.wired_limit_mb=N sets the limit to N MiB until the Mac restarts (macOS 13 used debug.iogpu.wired_limit, in bytes). Apple’s MLX documentation says the wired limit should remain strictly less than the total memory, and the mlx-lm README that N should be larger than the model and smaller than the machine’s memory; neither names a reserve for macOS, and a whole-system stall was reported on a 128 GB Mac with the limit raised to within 6.4 GB of its memory (the report blames the app’s own memory budget, not the limit). So the index’s command keeps a reserve, stated under the table. Quit other apps before loading a model near the limit.

Unified memoryDefault GPU limit, macOS 26 and latermacOS 15 and earlierRaise it (until the next restart)Raised limit / left for macOS
8 GB5.3 GB (67%) 5,727 MB5.3 GB (67%) 5,727 MBnone: macOS needs the rest—
16 GB11.8 GB (74%) 12,713 MB10.7 GB (67%) 11,453 MBsudo sysctl iogpu.wired_limit_mb=1228812 GB / 4 GB
18 GB13.3 GB (74%) 14,302 MB12 GB (67%) 12,885 MBsudo sysctl iogpu.wired_limit_mb=1433614 GB / 4 GB
24 GB17.8 GB (74%) 19,070 MB16 GB (67%) 17,180 MBsudo sysctl iogpu.wired_limit_mb=2048020 GB / 4 GB
32 GB25 GB (78%) 26,801 MB21.3 GB (67%) 22,906 MBsudo sysctl iogpu.wired_limit_mb=2867228 GB / 4 GB
36 GB28.1 GB (78%) 30,151 MB27 GB (75%) 28,991 MBsudo sysctl iogpu.wired_limit_mb=3225631.5 GB / 4.5 GB
48 GB37.4 GB (78%) 40,201 MB36 GB (75%) 38,655 MBsudo sysctl iogpu.wired_limit_mb=4300842 GB / 6 GB
64 GB51.8 GB (81%) 55,663 MB48 GB (75%) 51,540 MBsudo sysctl iogpu.wired_limit_mb=5734456 GB / 8 GB
96 GB77.8 GB (81%) 83,494 MB72 GB (75%) 77,309 MBsudo sysctl iogpu.wired_limit_mb=8601684 GB / 12 GB
128 GB107.5 GB (84%) 115,449 MB96 GB (75%) 103,079 MBsudo sysctl iogpu.wired_limit_mb=114688112 GB / 16 GB
192 GB161.3 GB (84%) 173,173 MB144 GB (75%) 154,619 MBsudo sysctl iogpu.wired_limit_mb=180224176 GB / 16 GB
256 GB222.7 GB (87%) 239,144 MB192 GB (75%) 206,158 MBsudo sysctl iogpu.wired_limit_mb=245760240 GB / 16 GB
512 GB445.4 GB (87%) 478,288 MB384 GB (75%) 412,317 MBsudo sysctl iogpu.wired_limit_mb=507904496 GB / 16 GB

MB is what llama.cpp prints (106 bytes); GB is Apple’s memory unit, in which the catalog states every requirement. The raised limit leaves max(4 GB, one eighth of the memory) for macOS, at most 16 GB, and the command is printed only where it exceeds the macOS 26 default, so it never lowers the limit on any macOS. sudo sysctl iogpu.wired_limit_mb=0 restores the default without a restart. Computed on every build from context_fit.py.

Sources: Apple’s and MLX’s documentation, then a startup log for each memory size, with its macOS version where the report states one. Some sizes are inferred from the chip and the value; for 18, 48, 64 and 96 GB no cited report states the memory. The 512 GB row uses the 256 GB band’s 87%: the 464 GB that 512 GB Macs report equals a typed 475,136 MiB raise.

Speculative decoding and MTP: shown, never ranked (19 November 2026)

Speculative decoding has a cheap drafter propose several tokens ahead, and the full model checks them all in one pass. The drafter can be the model’s own multi-token-prediction (MTP) heads, a DFlash or EAGLE head, a small draft model, or n-gram lookup. Accepted tokens are tokens the model would have produced anyway, so the output is the model’s own; but the speed now depends on how many drafts are accepted, and that depends on the text.

Plain decode stays the ranked figure for three reasons:

The rules for speculative figures. Since 19 November 2026 the dataset carries them in a separate spec field on a measured plain-decode row, never as rows of their own:

  1. Measured only. No speculative figure is ever estimated, or scaled to another chip.
  2. Paired. One is entered only on the plain-decode row from the same session (the same Mac, runtime build, checkpoint and quant), so the plain figure beside it is its baseline. A source that publishes only a drafted figure is cited in prose instead.
  3. Labelled. Each figure names its method (MTP, DFlash, EAGLE, a named draft model, n-gram, or the runtime’s default) and its workload (code, chat, or unstated). The benchmarks table prints it as a multiplier on the plain figure and lists every one in its speculative-decoding table.
  4. Never ranked. A speculative figure never becomes a reference figure or an anchor, and it never enters the Score, an estimate, the coherence floor or cap, or a RAM-tier pick. It never sorts a table.

Ollama turns it on by default for some models. Since v0.31.1 (30 June 2026), Ollama runs Gemma 4 on Apple Silicon with MTP drafting on by default, with no setting to change; its release notes put the gain at “nearly 90% faster on average across a coding-agent benchmark”. Since v0.32.6 (4 August 2026) its MLX engine also uses Qwen 3.5’s MTP head automatically. A Gemma 4 run on Ollama 0.31.1 or later therefore counts as drafted unless its source says MTP was off: it can enter the dataset only as a speculative figure beside a paired plain run, never as a plain-decode row. This is one reason Ollama can show a higher number than the index for the same model on the same Mac.

The LLMCheck Score Formula

The estimation model above feeds one of four components in the composite 0–100 LLMCheck Score used to rank the catalog:

LLMCheck Score = Capability + Speed + Accessibility + License
Capability (0–50): normalized from Arena AI ELO + MMLU + coding benchmarks — sourced from published third-party evals
Speed (0–25): the displayed tok/s on the reference chip × 0.25, capped at 25 — the lower-median measured run where one exists, otherwise estimated via the model above (reference row)
Accessibility (0–15): RAM tier: ≤8 GB = 15, ≤16 GB = 12, ≤24 GB = 10, ≤32 GB = 9, ≤48 GB = 6, ≤64 GB = 5, above 64 GB = 3
License (0–10): MIT = 10, GLM-5.3 / Modified MIT = 9, Apache 2.0 / OpenMDW = 8, Gemma / LFM Open / NVIDIA Open / Solar License = 6, Meta Custom / Mistral Ecosystem Open-Weight / Mistral Modified MIT / Moonshot Custom / Muse Model License / Qwen Custom / StepFun Model License = 5, MiniMax Community / xAI Custom = 4, CC-BY-NC / MiniMax Non-Commercial / N/A = 2

Dimension 1: Capability (50 points) — sourced

CAPABILITY50 / 100 pts

Raw model intelligence — reasoning, knowledge, coding, instruction following. Sourced from published third-party benchmark systems, linked per model.

The capability score (capScore) is the most heavily weighted dimension because the primary value of an LLM is output quality. capScore is sourced, not estimated: the index normalizes it from three public benchmark systems:

When official benchmarks are unavailable for a model (common for very new releases), the index extrapolates from related models in the same family and marks the score as estimated — same provenance rules as everything else. These estimates are replaced with sourced scores within 2–4 weeks of release.

Why 50 points? A very fast model that gives poor answers is less useful than a slower model with excellent reasoning. Capability is weighted highest because it determines whether the model actually solves your problem. Speed and RAM determine whether you can run it — but there's no point running a model that can't help you.

Capability Score Table (All 130 Models)

ModelParamscapScoreLicenseKey published evidenceSource
DeepSeek V4 Pro1.6T MoE50MITSWE-bench Verified 80.6% (V4-Pro-Max, vendor)HF
DeepSeek V4.1 Flash552B MoE50MIT1017 snapshot: DeepSWE v1.1 77.0 (Mini-SWE-V2 harness), Terminal-Bench 4.0 42.3 (Claude Code harness), HLE 42.6 (model card)HF
GLM 5.2753B MoE50MITSWE-Bench Pro 62.1% (SEAL standardized) / 68.5% vendor scaffoldHF
GLM-5.3753B MoE50GLM-5.3Terminal-Bench 2.1 88.2, DeepSWE v1.1 66.9 (model card); post-training upgrade of GLM-5.2HF
Kimi K32.8T MoE50Moonshot CustomAA Intelligence Index #3; Frontend Code Arena #1 (1,679)HF
Kimi K3.1 Pro2.8T MoE50Moonshot CustomAA's highest-ranked open-weights model (independent)HF
MiMo-V2.6-Pro1.02T MoE50MITRL-checkpoint card figures: Terminal-Bench 2.1 89.9, DeepSWE v1.1 71.9; the MOPD checkpoint publishes no new tableHF
Qwen3.8-2.4T-A95B2.4T MoE50Qwen CustomSWE-Bench Pro 67.7, GPQA 92.6 (model card)HF
MiMo-V2.6-Flash309B MoE49MITRL-checkpoint card figures: Terminal-Bench 2.1 87.6, DeepSWE v1.1 67.9; the MOPD checkpoint publishes no new tableHF
Muse Spark Open816B MoE49Muse Model LicenseArtificial Analysis Intelligence Index 39 (independent); GPQA Diamond 89.0, Terminal-Bench 2.1 76.4 (Terminus 2), DeepSWE 1.1 60 (preferred harness) (model card)HF
Step 5 Preview600B MoE49StepFun Model LicenseArtificial Analysis Intelligence Index 44; DeepSWE v1.1 67.7 (SWE-agent harness), Terminal-Bench 4.0 33.3 (card)HF
GLM-5.1754B MoE48MITLed open SWE-Bench Pro before GLM 5.2HF
GLM-5.3-Flash320B MoE48MITTerminal-Bench 2.1 84.3 (model card); vendor claims above GLM-5.2 — no SWE-Bench Pro figure on the card as of 7 Sep 2026HF
Hy4 preview770B MoE48Apache 2.0SWE-bench Pro 65.7 (harness not stated; GLM-5.3 64.6 on the same table), GPQA 92.3; the card calls it an early version with known issuesHF
Inkling 2 Small594B MoE48Apache 2.0AA Intelligence Index v5 30.6 (independent); Terminal-Bench 2.1 81.3, GPQA Diamond 93.0 (model card)HF
Kimi K2.61T MoE48Modified MIT1T-A32B multimodal agent flagshipHF
Kimi K2.7 Code1T MoE48Modified MITCoding-focused K2.6 update; the card's benchmarks are charts only, no SWE-bench figure publishedHF
MiniMax-M3428B MoE48MiniMax CommunitySWE-bench Verified 80.5, SWE-bench Pro 59 (harness not stated), MMMU-Pro 78.1 (model card)HF
MiniMax-M3.1-Flash428B MoE48MiniMax Community—HF
Beam 1.0501.3B MoE47Apache 2.0SWE-bench Verified 81.6 (harness not stated), SWE-Bench Pro 66.1 (vendor scaffold), Terminal-Bench 2.1 80.1, GPQA 90.6 (model card); AA Index 30HF
DeepSeek V4 Flash284B MoE47MITAA Index 50 (top open tier); Terminal-Bench 2.1 82.7 (vendor)HF
DeepSeek V4 Flash Vision-Exp305B MoE47MITInherits V4 Flash text evidence; vision experimental, no independent benchmark verifiedHF
Inkling975B MoE47Apache 2.0AA Index 41; SWE-bench Verified 77.6HF
Kimi K2.51T MoE47Modified MIT—HF
Ling-3.1-flash560B MoE47MITSWE-bench Pro 68.74 (LingCode vendor scaffold), Terminal-Bench 2.1 81.18, HLE 37.44 (HF eval results)HF
Qwen3.8-Flash-Next125B MoE47Qwen CustomSWE-Bench Pro 62.5 (vendor scaffold), LiveCodeBench v6 91.9, GPQA 91.7 (model card)HF
Inkling-Small276B MoE46Apache 2.0SWE-bench Verified 80.2 (bash-only harness); GPQA 89.5HF
Nanbeige4.3-33B33.1B46Apache 2.0AA ranks it level with Qwen3.8-27B at the top of small open models (independent); vendor: SWE-Bench Pro 60.3 (mini-swe-agent, not SEAL), GPQA 89.5, Terminal-Bench 2.1 80.9HF
Qwen3-235B-A22B235B MoE46Apache 2.0—HF
Qwen3.8-27B27.8B46Apache 2.0SWE-Bench Pro 61.7 (vendor scaffold), Terminal-Bench 2.1 73.0, LiveCodeBench v6 90.3, GPQA 89.2 (model card)HF
DeepSeek V3.2-Speciale685B MoE45MITHigh-reasoning sibling of V3.2HF
Mistral Large 41.05T MoE45Mistral Ecosystem Open-WeightAA Intelligence Index v5 26 (independent); AA Coding Agent Index level with GLM-5.3HF
Qwen 3.6-27B27B44Apache 2.0SWE-bench Verified 77.2HF
dots3-note preview280B MoE44Apache 2.0Vendor bar chart: SWE-bench Pro 61.0, Terminal-Bench 2.1 75.1 (harness not stated); preview checkpointHF
Laguna S 2.1118B MoE43OpenMDWTerminal-Bench 2.1 70.2 (max thinking); SWE-Pro public 59.4HF
Mistral Medium 3.5128B43Mistral Modified MITSWE-Bench Verified 77.6 (harness not stated), τ³-Telecom 91.4 (model card and Mistral announcement)HF
DeepSeek V3.2685B MoE42MIT—HF
Ling-3.0-flash-VL124B MoE42MITAA ranks it with Nemotron 3.5 Super VL among the two strongest medium open models (independent); vendor AA v4.1.1 42 vs Ling-3.0 Flash 38HF
MiniMax-M2.7228.7B MoE42MiniMax Non-CommercialSWE-Pro 56.2 (harness not stated), Terminal Bench 2 57.0, SWE Multilingual 76.5 (vendor)HF
Muse Glimmer 30B30B42Apache 2.0SWE-Bench Pro 51.2; AIME 94.7; GPQA 83.5 (vendor)HF
Nemotron 3 Ultra 550B-A55B550B MoE42OpenMDWSWE-bench Verified 70.7 (harness not stated), Terminal-Bench 2.1 56.4, GPQA 87.0, LiveCodeBench v6 89.0 (model card)HF
Nemotron 3.5 Super VL 120B-A12B120B MoE42OpenMDWSWE-bench Verified 71.1, SWE-bench Pro 51.8 (vendor scaffold), GPQA 89.5 (model card); AA Index 26 (independent)HF
Ripple-65B Preview64.8B42Apache 2.0Ricursive Intelligence, self-reported: SWE-bench Verified 76.5 (OpenCode harness), Terminal-Bench 2.1 80.5, GPQA Diamond 85.2; new lab, preview checkpoint (model card)HF
Solar Open 2 250B250B MoE42Solar LicenseMMLU-Pro 86.2; LiveCodeBench v6 92.4 (vendor)HF
Hunyuan Hy3295B MoE41Apache 2.0AA Index 41HF
Step-3.7-Flash198B MoE41Apache 2.0SWE-Bench Pro 56.3 (vendor scaffold), Terminal-Bench 2.1 59.5 (model card)HF
Gemma 4 31B31B40Apache 2.0—HF
MiniMax M2.5230B MoE40Modified MITSWE-bench Verified 80.2 (launch coverage)HF
Nemotron 3.5 Bolt 74B-A9B74B MoE40OpenMDWBF16: GPQA Diamond 88.8, Terminal-Bench 2.1 53.9, BrowseComp 70.0 (model card); no SWE-bench figureHF
KAT-Coder-V2.535B MoE39Apache 2.0SWE-bench Verified 69.4HF
Ling-3.0 Flash124B MoE38MITAA Index 38HF
Qwen 3.5 122B-A10B122B MoE38Apache 2.0SWE-bench Verified 72.0, GPQA 86.6, MMLU-Pro 86.7 (model card)HF
Qwen 3.5 397B-A17B397B MoE38Apache 2.0—HF
Qwen 3.6-35B-A3B35B MoE38Apache 2.0SWE-bench 73.4HF
DeepSeek R1671B MoE37MIT—HF
DeepSeek V3685B MoE37MIT—HF
Recursive Flint39.8B MoE37Apache 2.0SWE-bench Verified 74.2, Terminal-Bench 2.1 54.9 (Recursive). No Arena, MMLU or HumanEval figure: capability estimated from vendor coding benchmarks onlyHF
Ternary Bonsai 2 27B27.36B37Apache 2.0Vendor: 98.2% of FP16 quality retained across 14 benchmarks (model card)HF
Kolibri-178.1B MoE36Apache 2.0SWE-Bench Verified 66.4 (harness not stated), GPQA Diamond 84.3, LiveCodeBench v6 85.9 (model card)HF
Llama 4 Maverick400B MoE36Meta Custom—HF
Mistral Large 3675B MoE36Apache 2.0—HF
Nemotron 3.5 Lightning30B MoE36OpenMDWAA accuracy-vs-speed Pareto claim (vendor)HF
North Mini Code 1.030.5B MoE36Apache 2.0SWE-bench Verified 67.6 (SWE-agent v1.1.0), SWE-bench Pro 40.2 (SWE-agent vendor scaffold) (Cohere eval results)HF
GLM-4.7-Flash31B MoE35MIT—HF
Gemma 4 26B-A4B26B MoE35Apache 2.0—HF
LFM2.5-24B-A2B24B MoE35LFM OpenSWE-Bench Verified 51.0 (OpenCode harness), IFEval 96.8, GPQA Diamond 74.4 (model card)HF
Llama 3.1 405B405B35Meta Custom—HF
MiniCPM-V 4.7 35B-A3B35.2B MoE35Apache 2.0Card tables are images; third-party 4-bit eval (mlx-optiq): MMLU 86.2, HumanEval 90.9HF
Qwen3-Coder-Next80B MoE35Apache 2.0SWE-bench Verified 70.6HF
GLM-4.7355B MoE34MIT—HF
Mistral Small 4119B MoE34Apache 2.0—HF
Raon-1 mini preview28.8B MoE34Apache 2.0—HF
Bonsai 27B27B33Apache 2.0~90% of Qwen3.6-27B FP16 quality (vendor)HF
GLM-4.5-Air106B MoE33MIT—HF
Laguna XS 2.133B MoE33OpenMDW—HF
Step-3.5-Flash196B MoE33Apache 2.0—HF
GPT-oss 120B117B MoE32Apache 2.0—HF
MiMo-V2-Flash309B MoE32MIT—HF
Granite 4.2 30B29.28B31Apache 2.0SWE Bench Verified 57 (harness not stated), SWE Bench Pro 33.3, LiveCodeBench v6 75.8 (model card)HF
DeepSeek R1 70B70B30MIT—HF
Gemma 4 12B12B30Apache 2.0MMLU-Pro 77.2, GPQA Diamond 78.8, LiveCodeBench v6 72.0 (model card)HF
Llama 4 Scout109B MoE30Meta Custom—HF
Nemotron-Cascade 230B MoE30NVIDIA OpenAIME / LiveCodeBench golds at 3B active (vendor)HF
Apertus 1.5 70B70B29Apache 2.0—HF
Hermes 4 70B70B28Meta Custom—HF
Mixtral 8x22B141B MoE28Apache 2.0—HF
Qwen 2.5 72B72B28Apache 2.0—HF
Llama 3.3 70B70B27Meta Custom—HF
Qwen 3.5 35B-A3B35B MoE27Apache 2.0—HF
QwQ 32B32B26Apache 2.0—HF
Mistral Small 3.2 24B24B25Apache 2.0—HF
Qwen 3 32B32B25Apache 2.0—HF
DeepSeek R1 32B32B24MIT—HF
Devstral Small 24B24B24Apache 2.0—HF
GPT-oss 20B21B MoE22Apache 2.0SWE-bench Verified 53.2, GPQA Diamond 58.6 at medium reasoning (model card)HF
Gemma 3 27B27B22Gemma—HF
K2-Horizon-3.7B5.1B22Apache 2.0AA ranks it the strongest open model of 4B or less, ahead of MiniCPM5-2B (independent); vendor: SWE-bench Verified 65.6 (harness not stated), Terminal-Bench 2.1 39.7, GPQA Diamond 68.9HF
Qwen 3 30B-A3B30B MoE22Apache 2.0—HF
MiMo-V2.6-Distill-Qwen-9B9.41B21MITSWE Verified 61.1 (avg@3), SWE Pro 44.6, Terminal Bench 2.1 37.1 (model card)HF
MiniCPM5-2B2.52B21Apache 2.0AA ranks it among the two strongest open models of 4B or less (independent); vendor GPQA 70.2, MMLU-Pro 70.8HF
Qwen 3.5 27B27B21Apache 2.0—HF
Maple Preview 20B-A1B20B MoE20MITIMO-level claims (vendor, unverified)HF
Nanbeige4.2-3B3B20Apache 2.0SWE-bench Verified 63.6 (vendor)HF
Qwen 3 14B14B20Apache 2.0—HF
Spark-X2.6-4B4.11B20Apache 2.0—HF
Phi-4 14B14B19MIT—HF
Mellum2.1-12B-A2.5B-Thinking12.15B MoE18Apache 2.0SWE-bench Verified 47.0 and SWE-bench Pro 28.0 (Pi v0.73.1 harness, not SEAL), LiveCodeBench v6 82.0 (JetBrains)HF
Ministral 3 14B14B18Apache 2.0—HF
Qwen 2.5 14B14B18Apache 2.0—HF
Qwen 3.5 9B9B18Apache 2.0—HF
Gemma 3 12B12B17Gemma—HF
LFM3-1.7B1.73B17LFM OpenSWE-bench Verified 43.4 (OpenCode harness), IFEval 96.7, GPQA 63.4 (model card)HF
Nemotron 3.5 Bolt E4B4B17OpenMDW—HF
Apertus 1.5 8B8B16Apache 2.0—HF
DeepSeek R1 8B8B16MIT—HF
Gemma 4 E4B4B16Apache 2.0—HF
Granite 4.2 8B8.79B16Apache 2.0SWE-Bench Verified 47.7 (harness not stated), LiveCodeBench v6 73.2, GPQA 64.1 (IBM)HF
Ternary Bonsai 2 9B9.41B16Apache 2.0Vendor, vs Qwen3.5-9B: BFCLv3 71.8 (73.8), GPQA Diamond 69.9 (82.5), IFEval 90.0HF
Ling-3.0-tiny7.9B MoE15MITOnly vendor-quoted AA indices (v4.1.1 25, Agentic 16, in an image): held below Gemma 4 E4B until reproducedHF
Qwen 3 8B8B15Apache 2.0—HF
LFM2.5-2.6B2.6B14LFM OpenSize-class instruction-following leader (vendor)HF
Ministral 8B8B14Apache 2.0—HF
Phi-4 Mini3.8B14MIT—HF
Gemma 4 E2B2B13Apache 2.0—HF
Mistral 7B7B13Apache 2.0—HF
Llama 3.1 8B8B12Meta Custom—HF
Qwen 3.5 4B4B12Apache 2.0—HF
Qwen 3 4B4B11Apache 2.0—HF
Gemma 3 4B4B10Gemma—HF
SmolLM3 3B3B10Apache 2.0—HF

Dimension 2: Speed on Apple Silicon (25 points) — estimated

SPEED25 / 100 pts

Tokens per second on the reference chip: the M5 Max — or, for models that need more than 128 GB, which no M5 Max can hold, the Mac Studio M5 Ultra, tagged on the leaderboard. The figure is the reference row: the lower-median measured run on that chip where one exists, otherwise the estimate from the model above. LLMCheck runs no lab.

Speed points are calculated as: displayed tok/s × 0.25, capped at 25 points — exactly the figure the leaderboard shows. Any model at 100+ tok/s on the reference configuration receives full speed points. The reference conditions:

Models that fit no Mac (more than 512 GB) or have no Mac runtime yet receive 0 speed points. The full dataset — with a provenance label on every row — is available for download at /data/.

Dimension 3: Accessibility (15 points)

ACCESSIBILITY15 / 100 pts

How many Mac users can actually run this model? Lower RAM requirements = higher accessibility score.

Accessibility is a step function based on minimum RAM required at 4-bit quantization, derived from the RAM model above (weights + KV cache + macOS overhead, within the ~75% unified-memory budget). Since a large share of Mac users have 16 GB or less, accessibility is crucial for real-world impact:

Min RAMPointsTop-ranked models in the tier
≤ 8 GB15Bonsai 27B, K2-Horizon-3.7B, Maple Preview 20B-A1B
≤ 16 GB12Ternary Bonsai 2 27B, GPT-oss 20B, Gemma 4 12B
≤ 24 GB10Qwen 3.6-35B-A3B, GLM-4.7-Flash, Recursive Flint
≤ 32 GB9— (no current model)
≤ 48 GB6Nemotron 3.5 Bolt 74B-A9B, Qwen3-Coder-Next, Ripple-65B Preview
≤ 64 GB5Ling-3.0-flash-VL, GPT-oss 120B, Kolibri-1
> 64 GB3MiMo-V2.6-Flash, DeepSeek V4 Flash, DeepSeek V4 Flash Vision-Exp

Dimension 4: License Openness (10 points)

LICENSE10 / 100 pts

How freely can you use, modify, and distribute the model? More open = higher score.

LicensePointsCan Modify?Commercial Use?Models
MIT10YesUnrestrictedMiMo-V2.6-Flash, GLM-4.7-Flash, DeepSeek V4 Flash, DeepSeek V4 Flash Vision-Exp
GLM-5.39YesYes (custom GLM-5.3 terms)GLM-5.3
Modified MIT9YesYes (attribution clauses)Kimi K2.6, Kimi K2.7 Code, Kimi K2.5, MiniMax M2.5
Apache 2.08YesYes (with notice)Qwen 3.6-35B-A3B, Recursive Flint, KAT-Coder-V2.5, North Mini Code 1.0
OpenMDW8YesYesNemotron 3.5 Lightning, Laguna XS 2.1, Nemotron 3.5 Bolt 74B-A9B, Nemotron 3.5 Bolt E4B
Gemma6YesYes (use restrictions)Gemma 3 4B, Gemma 3 12B, Gemma 3 27B
LFM Open6YesYes (revenue threshold)LFM2.5-24B-A2B, LFM3-1.7B, LFM2.5-2.6B
NVIDIA Open6YesYes (NVIDIA terms)Nemotron-Cascade 2
Solar License6YesYes (attribution / terms)Solar Open 2 250B
Meta Custom5LimitedYes (<700M monthly users)Llama 3.1 8B, Llama 4 Maverick, Llama 4 Scout, Llama 3.1 405B
Mistral Ecosystem Open-Weight5YesYes, except Model-as-a-Service — API / hosted-inference businesses need a separate Mistral licenceMistral Large 4
Mistral Modified MIT5YesYes, revenue-gated — large companies need a separate licenceMistral Medium 3.5
Moonshot Custom5LimitedYes (revenue thresholds)Kimi K3, Kimi K3.1 Pro
Muse Model License5YesYes — “Built with Muse Spark Open” credit; API hosts above $20M revenue need a separate agreementMuse Spark Open
Qwen Custom5LimitedYes (revenue thresholds / product gates)Qwen3.8-Flash-Next, Qwen3.8-2.4T-A95B
StepFun Model License5YesYes — device makers and API hosts above $20M need a separate licenceStep 5 Preview
MiniMax Community4YesYes, with attribution, below a revenue thresholdMiniMax-M3, MiniMax-M3.1-Flash
xAI Custom4LimitedYes (custom terms)(none currently listed)
CC-BY-NC2YesNon-commercial only(none currently listed)
MiniMax Non-Commercial2YesNon-commercial onlyMiniMax-M2.7
N/A2NoAPI only(none currently listed)

Licence points are editorial calls, stated here so they can be audited. Licences that gate commercial use by revenue, require a product credit or need a separate agreement for API hosting score below Apache 2.0: the Muse Model License, the StepFun Model License and Mistral Modified MIT (revenue-gated, so deliberately not the 9 of plain Modified MIT) score 5, MiniMax Community 4 (commercial use permitted with attribution below a revenue threshold) and MiniMax Non-Commercial 2.

The Mistral Ecosystem Open-Weight License (Mistral Large 4, weights 29 October 2026) also scores 5. Its grant is MIT-style with an attribution notice, but any company running a Model-as-a-Service business — giving third parties inference or post-training access with meaningful control over inputs, parameters or training data — needs a separate commercial licence from Mistral, with no revenue threshold. Internal use, products that embed the model in their own features, and relaying requests to models hosted elsewhere are exempt. It gates a whole business category, which no 6-point licence does, so it sits with the other licences that need a separate agreement for API hosting, not with Apache 2.0.

Note: CC-BY-NC (non-commercial) scores 2 — usable for research but not commercial deployment. Meta / xAI community licenses score 5 / 4, reflecting modification rights with commercial caps.

Score Examples

Computed from the live data on every build, using exactly the figure the leaderboard displays.

MiMo-V2.6-Flash (Score: 81) = capScore 49 + speed 19 (min(25, 77 tok/s × 0.25); community-measured on an M5 Ultra at MXFP4) + accessibility 3 (170 GB) + license 10 (MIT) = 81.

KAT-Coder-V2.5 (Score: 79) = capScore 39 + speed 22 (min(25, 89 tok/s × 0.25); community-measured on an M5 Max at OptiQ-4bit) + accessibility 10 (24 GB) + license 8 (Apache 2.0) = 79.

Kimi K2.5 (Score: 66) = capScore 47 + speed 7 (min(25, 28 tok/s × 0.25); estimated on an M5 Ultra at 3.5 bpw) + accessibility 3 (417 GB) + license 9 (Modified MIT) = 66.

Fine-tunes and derivatives: which ones the index lists (14 November 2026)

A derivative is a checkpoint whose model card names a catalog model as its base, or whose architecture is identical to one (the same layers, experts and hidden size, with or without an extended vocabulary). That covers continued pre-training, supervised and reinforcement-learning fine-tunes, merges, abliteration and domain specialisation. A derivative is not listed as a separate model unless all four of these hold:

  1. An accountable publisher and a licence file. An identifiable organisation publishes it, with a licence file in the repository; a licence named only in the card’s metadata is not enough.
  2. A measured capability gain. A third-party evaluation (Artificial Analysis, LMArena, or a press or independent test that documents its setup), or the publisher’s own table against its base with the harness stated, shows a gain worth at least 2 capScore points over the base model.
  3. A Mac build on a stock runtime. A downloadable build that released llama.cpp, mlx-lm or mlx-vlm, LM Studio or Ollama runs.
  4. More than a behaviour change. A checkpoint whose only change is removed safety training, a persona or a writing style does not qualify, however it scores.

Three cases are not treated as derivatives. A lab’s updated checkpoint of its own model replaces the catalog entry instead of adding one (DeepSeek V4.1 Flash 1017, the MiMo-V2.6 MOPD checkpoints, Ling-3.1-flash-1019). A lab’s official distillation of its own model into a different architecture is a new model (MiMo-V2.6-Distill-Qwen-9B). A quantized build is a build of its base, not a model. And a derivative’s speed runs never stand in for its base: runs of an abliterated Recursive Flint build on an M4 Pro were not entered as Flint rows.

Considered in November 2026 and not listed (the condition each one misses):

Derivatives already in the catalog. Six entries predate this policy and are derivatives under its definition: KAT-Coder-V2.5 (post-trained from Qwen3.6-35B-A3B), Nemotron 3.5 Bolt E4B (from Gemma 4 E4B), Hermes 4 70B (a fine-tune of Llama 3.1 70B, the architecture of the catalog’s Llama 3.3 70B), and Prism ML’s low-bit rebuilds Bonsai 27B (of Qwen3.6-27B), Ternary Bonsai 2 27B (of Qwen3.8-27B) and Ternary Bonsai 2 9B (of Qwen 3.5 9B). They stay listed while the policy is new. Each will be checked against the four conditions in the December 2026 update, and any change will be recorded, with its reason, in the changelog. The low-bit rebuilds trade capability for memory rather than adding capability, a case condition 2 does not describe, and the December review will say how they are treated.

Limitations & Known Issues

Corrections and real-world benchmark data are always welcome. If you have runs that differ from the index's estimates, submit them through the community benchmark process — verified community rows replace estimates.

Frequently Asked Questions

Does LLMCheck run its own benchmarks?

No. LLMCheck is an independent index, not a benchmark lab. Every figure on the site is one of three things: an estimate from the published LLMCheck estimation model (memory-bandwidth math and quantization arithmetic, fully documented on this page), a sourced number from a linked third-party benchmark, or a community-submitted run. Each row is labeled with its provenance, and no figure is claimed as a first-party lab measurement.

How does LLMCheck calculate its scores?

The LLMCheck Score is a 0-100 composite: Capability (50 pts) normalized from published third-party evaluations, Speed (25 pts) from the reference figure (the lower-median measured run on the M5 Max, or the M5 Ultra for models above 128 GB, where one exists, otherwise the published estimate; tok/s x 0.25, capped), Accessibility (15 pts) stepping down with minimum RAM, and License Openness (10 pts) where MIT scores 10 and restrictive licenses score lower. The formula is fully transparent and reproducible.

Where does LLMCheck get its capability scores?

Capability scores are derived from three public benchmark sources: Arena AI ELO ratings (human preference, weighted 40%), MMLU scores from official model cards (knowledge breadth, weighted 35%), and coding benchmarks like HumanEval and SWE-Bench (weighted 25%). All sources are linked per model. When official benchmarks are unavailable, the index extrapolates from related models in the same family and marks the score as 'estimated'.

How does LLMCheck estimate tokens per second on Apple Silicon?

Token generation on Apple Silicon is memory-bandwidth-bound, so the index estimates it with a two-term model: tok/s = 1 / (model GB / (bandwidth x 0.783) + 1.46 ms), where model GB is the parameter count times the bytes per parameter at the row's own quant (0.57 GB per billion at Q4_K_M). Mixture-of-experts models use active-parameter bytes. Since the 5 October 2026 refit, small MoEs (40B total or less) use a 4.97 ms per-token overhead fitted to six fully specified plain-decode runs; only large MoEs (above 40B) keep the deliberately conservative 0.42 factor. A model with its own measured runs has its other chips scaled from the nearest-bandwidth measured row, and a measured run on the reference chip replaces the estimate (the lower median where there are several). Every estimated figure is labelled; the full model is at llmcheck.net/methodology#estimation.

Why does LLMCheck weight capability at 50% of the total score?

Capability receives the highest weight because the primary value of an LLM is the quality of its outputs. A very fast model that gives poor answers is less useful than a slower model with excellent reasoning. However, speed (25%) and accessibility (15%) ensure that models which actually run well on consumer Macs score higher than server-only models with superior capability but no practical local use.

Why does Ollama show a faster speed than LLMCheck for some models?

Usually because of speculative decoding. Since v0.31.1 (30 June 2026) Ollama runs Gemma 4 on Apple Silicon with multi-token prediction (MTP) drafting on by default, and since v0.32.6 its MLX engine uses Qwen 3.5’s MTP head automatically. Drafting can speed up code substantially and slow other text down, so the index ranks plain decode only: a measured speculative figure is shown beside the plain run from the same session, labelled with its method and workload, and never enters the Score or an estimate. Details: llmcheck.net/methodology#speculative.

How often are LLMCheck scores updated?

Scores are updated within 48–72 hours of major model releases. The full leaderboard is refreshed monthly with the latest Arena AI ELO ratings and community benchmark submissions. Speed estimates are recomputed as new Apple Silicon hardware specifications become available. All updates are timestamped in the open dataset at llmcheck.net/data/.

Benchmark Sources

Cite this page

LLMCheck. (2026). LLMCheck Index Methodology. llmcheck.net. https://llmcheck.net/methodology (updated ). CC BY 4.0.

How to cite LLMCheck →Data: CC BY 4.0Markdown version

See the Full Leaderboard

130 verified models ranked by LLMCheck Score. Filter by your Mac's RAM, sort by speed or capability — every figure labeled with its provenance.

View Leaderboard →