LLMCheck Index Methodology

How the LLMCheck index ranks 79 local and frontier LLMs for Mac — a published estimation model, linked third-party benchmarks, and community submissions. Fully transparent, fully reproducible.

LLMCheck is an independent index of local-LLM performance on Apple Silicon — not a benchmark lab. Every figure is either a transparent estimate from the published model below, a sourced third-party benchmark (linked per row), or a community submission. Each data point is labeled with its provenance, and no figure is claimed as a first-party lab measurement.

Where Every Number Comes From

Every row in the leaderboard, the benchmarks table, and the open dataset carries one of three provenance labels. This mirrors the provenance_policy field published in benchmarks.json:

How We Estimate Performance

Local LLM performance on Apple Silicon is unusually predictable, because two hardware numbers dominate everything: unified memory capacity (can the model fit?) and memory bandwidth (how fast can weights stream through the GPU?). The LLMCheck estimation model is built on those two numbers.

Step 1 — Will it fit? (RAM model)

That 75% rule is where the minimum-RAM guidance across the index comes from: a 16 GB Mac has a ~12 GB working budget, a 24 GB Mac ~18 GB, a 64 GB Mac ~48 GB, and so on.

Step 2 — How fast? (Speed model)

Token generation is memory-bandwidth-bound: for every token, the runtime must stream essentially all active model weights from RAM through the GPU. Compute is rarely the bottleneck on Apple Silicon. That gives a simple ceiling:

Mixture-of-experts models — refit 5 October 2026

A mixture-of-experts (MoE) model stores all of its parameters but reads only the active ones for each token, so its bandwidth term uses active-parameter bytes: a 35B MoE with 3B active reads about 1.7 GB per token instead of 20 GB. It also pays more fixed cost per token than a dense model — routing, gathering experts, more kernel launches per layer. Until October the index scaled the dense formula by a flat 0.42, fitted to a single 2-bit DeepSeek V4 Flash run. Six fully specified plain-decode measurements of small MoEs then came in at 2–4× those estimates:

The refit moved the small MoE models up the leaderboard: on an M5 Max several now estimate above 100 tok/s, where speed points cap at 25. The index’s #1, KAT-Coder-V2.5, ranks on an estimate from this model; #2, Qwen 3.6-35B-A3B, on its community-measured 126 tok/s. The full 35B of weights still has to be resident — about 20 GB — which is why it is a 24 GB-Mac model despite generating like a 3B one.

Step 3 — A worked example

What the estimates can't capture

Honest limits of the model: thermals (a fanless MacBook Air throttles on long generations; a Mac Studio doesn't), context length (speed degrades as the KV cache grows — long chats get slower), quantization variants (Q5, Q8, and MLX 4-bit all shift size and speed), and runtime differences (Ollama, LM Studio, llama.cpp, and MLX can differ 10–20% on the same hardware). Real numbers will vary from the estimates — that's exactly why every estimated row is labeled, and why community submissions are invited to replace estimates with real runs.

Time to first token. The estimator models decode only. Since 5 October 2026 estimated rows carry no time-to-first-token; a TTFT appears only where a source published one. Prompt processing is where the M5 generation’s Neural Accelerators made their biggest gains, so a decode estimate says little about it.

The LLMCheck Score Formula

The estimation model above feeds one of four components in the composite 0–100 LLMCheck Score used to rank the catalog:

Dimension 1: Capability (50 points) — sourced

The capability score (capScore) is the most heavily weighted dimension because the primary value of an LLM is output quality. capScore is sourced, not estimated: the index normalizes it from three public benchmark systems:

• Arena AI ELO Rating (40% weight) — Human preference ranking from lmarena.ai. Measures how often real users prefer a model's response over competitors in blind A/B tests. The gold standard for "does this model feel good to use."
• MMLU Score (35% weight) — Massive Multitask Language Understanding. Tests knowledge across 57 academic subjects. Sourced from official model cards on HuggingFace and arXiv papers.
• Coding Benchmarks (25% weight) — HumanEval (code generation) and SWE-Bench Verified (real bug fixing). Sourced from llm-stats.com and official model publications.

When official benchmarks are unavailable for a model (common for very new releases), the index extrapolates from related models in the same family and marks the score as estimated — same provenance rules as everything else. These estimates are replaced with sourced scores within 2–4 weeks of release.

Capability Score Table (All 91 Models)

Dimension 2: Speed on Apple Silicon (25 points) — estimated

Speed points are calculated as: est. tok/s × 0.25, capped at 25 points. Any model estimated at 100+ tok/s on the reference configuration receives full speed points. The estimates assume these reference conditions:

• Quantization: Q4_K_M (most common consumer quantization)
• Reference hardware: M5 Max, 128 GB Unified Memory (614 GB/s bandwidth); Mac Studio M5 Ultra (1,228 GB/s) for models above 128 GB
• Runtime class: llama.cpp / MLX-class engines (Ollama, LM Studio) with dense efficiency ~0.6–0.8
• Context: short-context generation — long contexts reduce speed as the KV cache grows
• Provenance: rows are upgraded from estimated to sourced or community as linked third-party tests and user submissions come in

Models that fit no Mac (more than 512 GB) or have no Mac runtime yet receive 0 speed points. The full dataset — with a provenance label on every row — is available for download at /data/.

Dimension 3: Accessibility (15 points)

Accessibility is a step function based on minimum RAM required at 4-bit quantization, derived from the RAM model above (weights + KV cache + macOS overhead, within the ~75% unified-memory budget). Since a large share of Mac users have 16 GB or less, accessibility is crucial for real-world impact:

Dimension 4: License Openness (10 points)

Note: CC-BY-NC (non-commercial) scores 2 — usable for research but not commercial deployment. Meta / xAI community licenses score 5 / 4, reflecting modification rights with commercial caps.

Score Examples

Limitations & Known Issues

• Arena ELO volatility: Ratings shift weekly as new votes come in. The index snapshots monthly and notes the capture date.
• Estimated capScores: ~15% of capScores are extrapolated from model-family performance when official benchmarks are pending. These are marked "est." in the table above.
• Estimates are not measurements: speed figures come from the bandwidth model, which cannot capture thermals, long-context slowdown, or runtime-specific optimizations. Expect real-world variance of 10–20% either way — sometimes more on fanless machines.
• Quantization baseline: All estimates assume Q4_K_M. Some models are commonly run at Q5, Q8, or MLX 4-bit, which shift both size and speed; the index standardizes on one baseline for fair comparison.
• No multimodal scoring: The current score does not reward multimodal capabilities (image/audio input). Models like Gemma 4 E4B have multimodal features not captured in the 0–100 score.

Corrections and real-world benchmark data are always welcome. If you have runs that differ from the index's estimates, submit them through the community benchmark process — verified community rows replace estimates.