LLMCheck Index Methodology

How the LLMCheck index ranks 79 local and frontier LLMs for Mac — a published estimation model, linked third-party benchmarks, and community submissions. Fully transparent, fully reproducible.

LLMCheck is an independent index of local-LLM performance on Apple Silicon — not a benchmark lab. Every figure is either a transparent estimate from the published model below, a sourced third-party benchmark (linked per row), or a community submission. Each data point is labeled with its provenance, and no figure is claimed as a first-party lab measurement.

Where Every Number Comes From

Every row in the leaderboard, the benchmarks table, and the open dataset carries one of three provenance labels. This mirrors the provenance_policy field published in benchmarks.json:

How We Estimate Performance

Local LLM performance on Apple Silicon is unusually predictable, because two hardware numbers dominate everything: unified memory capacity (can the model fit?) and memory bandwidth (how fast can weights stream through the GPU?). The LLMCheck estimation model is built on those two numbers.

Step 1 — Will it fit? (RAM model)

That 75% rule is where the minimum-RAM guidance across the index comes from: a 16 GB Mac has a ~12 GB working budget, a 24 GB Mac ~18 GB, a 64 GB Mac ~48 GB, and so on.

Step 2 — How fast? (Speed model)

Token generation is memory-bandwidth-bound: for every token, the runtime must stream essentially all active model weights from RAM through the GPU. Compute is rarely the bottleneck on Apple Silicon. That gives a simple ceiling:

This is also why Mixture-of-Experts models dominate the speed rankings: a 32B MoE with 3B active parameters reads ~1.7 GB per token instead of ~18 GB — a 10× smaller memory bill per token, while still storing the full 32B of knowledge in RAM.

Step 3 — A worked example

Contrast with an MoE model: Qwen 3.6-35B-A3B activates ~3B parameters per token, so it reads only 3 × 0.57 ≈ 1.7 GB per token. On an M4 Pro (273 GB/s) the ceiling is ~160 tok/s; realized MoE efficiency of ~0.4 (routing overhead, expert scatter across memory) lands the estimate near ~62 tok/s — on a mid-range chip. The full 32B of weights still need ~18 GB of RAM to be resident, which is why it's a 24 GB-Mac model despite generating like a 3B.

What the estimates can't capture

Honest limits of the model: thermals (a fanless MacBook Air throttles on long generations; a Mac Studio doesn't), context length (speed degrades as the KV cache grows — long chats get slower), quantization variants (Q5, Q8, and MLX 4-bit all shift size and speed), and runtime differences (Ollama, LM Studio, llama.cpp, and MLX can differ 10–20% on the same hardware). Real numbers will vary from the estimates — that's exactly why every estimated row is labeled, and why community submissions are invited to replace estimates with real runs.

The LLMCheck Score Formula

The estimation model above feeds one of four components in the composite 0–100 LLMCheck Score used to rank the catalog:

Dimension 1: Capability (50 points) — sourced

The capability score (capScore) is the most heavily weighted dimension because the primary value of an LLM is output quality. capScore is sourced, not estimated: the index normalizes it from three public benchmark systems:

• Arena AI ELO Rating (40% weight) — Human preference ranking from lmarena.ai. Measures how often real users prefer a model's response over competitors in blind A/B tests. The gold standard for "does this model feel good to use."
• MMLU Score (35% weight) — Massive Multitask Language Understanding. Tests knowledge across 57 academic subjects. Sourced from official model cards on HuggingFace and arXiv papers.
• Coding Benchmarks (25% weight) — HumanEval (code generation) and SWE-Bench Verified (real bug fixing). Sourced from llm-stats.com and official model publications.

When official benchmarks are unavailable for a model (common for very new releases), the index extrapolates from related models in the same family and marks the score as estimated — same provenance rules as everything else. These estimates are replaced with sourced scores within 2–4 weeks of release.

Capability Score Table (All 81 Models)

Dimension 2: Speed on Apple Silicon (25 points) — estimated

Speed points are calculated as: est. tok/s × 0.25, capped at 25 points. Any model estimated at 100+ tok/s on the reference configuration receives full speed points. The estimates assume these reference conditions:

• Quantization: Q4_K_M (most common consumer quantization)
• Reference hardware: M5 Max, 128 GB Unified Memory (600 GB/s bandwidth)
• Runtime class: llama.cpp / MLX-class engines (Ollama, LM Studio) with dense efficiency ~0.6–0.8
• Context: short-context generation — long contexts reduce speed as the KV cache grows
• Provenance: rows are upgraded from estimated to sourced or community as linked third-party tests and user submissions come in

Models that cannot run on any Mac (server-only, >128 GB RAM) receive 0 speed points. The full dataset — with a provenance label on every row — is available for download at /data/.

Dimension 3: Accessibility (15 points)

Accessibility is a step function based on minimum RAM required at 4-bit quantization, derived from the RAM model above (weights + KV cache + macOS overhead, within the ~75% unified-memory budget). Since a large share of Mac users have 16 GB or less, accessibility is crucial for real-world impact:

Dimension 4: License Openness (10 points)

Note: CC-BY-NC (non-commercial) scores 2 — usable for research but not commercial deployment. Meta / xAI community licenses score 5 / 4, reflecting modification rights with commercial caps.

Score Examples

Limitations & Known Issues

• Arena ELO volatility: Ratings shift weekly as new votes come in. The index snapshots monthly and notes the capture date.
• Estimated capScores: ~15% of capScores are extrapolated from model-family performance when official benchmarks are pending. These are marked "est." in the table above.
• Estimates are not measurements: speed figures come from the bandwidth model, which cannot capture thermals, long-context slowdown, or runtime-specific optimizations. Expect real-world variance of 10–20% either way — sometimes more on fanless machines.
• Quantization baseline: All estimates assume Q4_K_M. Some models are commonly run at Q5, Q8, or MLX 4-bit, which shift both size and speed; the index standardizes on one baseline for fair comparison.
• No multimodal scoring: The current score does not reward multimodal capabilities (image/audio input). Models like Gemma 4 E4B have multimodal features not captured in the 0–100 score.

Corrections and real-world benchmark data are always welcome. If you have runs that differ from the index's estimates, submit them through the community benchmark process — verified community rows replace estimates.