Submit a Benchmark
LLMCheck runs no benchmarks of its own, so every community row comes from someone’s Mac. To add one, run a model with speculative decoding off, note the plain decode tok/s and its prompt depth, the quant, the runtime version, your chip, GPU cores and RAM, and submit it on GitHub or by email.
Reviewed runs join the index as community rows, credited to you, at the next data release. They appear on the benchmarks page, the model’s pages and the open data.
How to contribute (5 steps, about 10 minutes)
Pick a model and a runtime
Any model on the leaderboard helps; the most useful runs are for Macs and models the index has few measurements for (every speed on /benchmarks is marked measured or estimated). Use the runtime you normally use: Ollama, LM Studio, llama.cpp or mlx-lm (the software guide compares them).
Note the exact build you run: the GGUF file name, the Hugging Face repo, or an Ollama tag that names its quant, such as qwen3.6:35b-a3b-q4_K_M.
Measure plain decode
Ollama. Run the model with --verbose, switch drafting off, then send a prompt. The eval rate line is the plain decode speed:
$ ollama run qwen3.6:35b-a3b-q4_K_M --verbose >>> /set parameter draft_num_predict 0 Set parameter 'draft_num_predict' to '0' >>> Write a haiku about unified memory. … prompt eval count: NN token(s) prompt eval rate: NNN.NN tokens/s eval count: NN token(s) eval rate: NN.NN tokens/s ← this number
draft_num_predict 0 switches speculative drafting off on Ollama’s llama.cpp engine, which runs the tags that name a GGUF quant, such as qwen3.6:35b-a3b-q4_K_M: use one of those. It matters: Ollama has drafted Gemma 4 with multi-token prediction by default on Apple Silicon since version 0.31.1, and its -mtp tags carry draft heads. Tags with no quant in the name (gemma4:26b-a4b, qwen3.6:35b-a3b) run on Ollama’s MLX engine on Apple Silicon since 0.40.0, which (as of 0.40.16) drafts whenever the model ships a draft head and ignores draft_num_predict. On a 16 GB Mac, try qwen3.5:9b-q4_K_M.
llama.cpp. llama-bench (in brew install llama.cpp) measures prefill and decode separately, and never drafts:
$ llama-bench -m Qwen3.6-35B-A3B-Q4_K_M.gguf -p 512,8192 -n 0 -r 3 $ llama-bench -m Qwen3.6-35B-A3B-Q4_K_M.gguf -p 0 -n 128 -d 0,8192 -r 3
The first command prints prefill (pp512, pp8192); the second prints plain decode from an empty context (tg128) and with 8,192 tokens already in the context (tg128 @ d8192). Paste both tables and the build: line under each: it is the llama.cpp build number.
MLX. mlx_lm.benchmark (in pip install mlx-lm) times a prompt of the length you give it, then 128 generated tokens:
$ mlx_lm.benchmark --model mlx-community/<repo> -p 512 -g 128 -n 3 $ mlx_lm.benchmark --model mlx-community/<repo> -p 8192 -g 128 -n 3 $ pip show mlx-lm | grep Version
Each Trial line gives prompt_tps (prefill), generation_tps (plain decode at that prompt depth) and peak_memory; report the median of the trials. It downloads the model first if it is not already on disk.
LM Studio. The stats under each response show tokens per second and time to first token; lms chat -p "…" --stats prints the same in Terminal. Leave Speculative Decoding (the draft-model setting) off, and note the prompt length (Prompt Tokens in the stats), LM Studio’s version and its engine version.
Or let the helper script do steps 2 and 4
tools/llmcheck-bench.sh in the llmcheck-data repository runs the llama-bench or mlx_lm.benchmark commands above (3 timed runs each, medians) on a GGUF file or an MLX model you already have:
$ git clone https://github.com/Humanoid-Intel/llmcheck-data.git $ cd llmcheck-data $ tools/llmcheck-bench.sh --dry-run ~/models/Qwen3.6-35B-A3B-Q4_K_M.gguf $ tools/llmcheck-bench.sh ~/models/Qwen3.6-35B-A3B-Q4_K_M.gguf
It reads your chip, GPU cores, RAM and macOS version from the Mac itself, then prints a filled submission and a link that fills in the GitHub form. It makes no network calls, never downloads a model, sends no telemetry and uploads nothing, and its output carries no user name, file path or serial number. It is a short bash script under the MIT licence: read it before you run it.
Note your Mac
→ About This Mac gives the chip and memory. System Information → Graphics/Displays gives the GPU cores (or run system_profiler SPDisplaysDataType | grep Cores), and sw_vers -productVersion the macOS version. GPU cores matter because some configurations of one chip have less memory bandwidth: the 36 GB M5 Max has 460 GB/s, not 614.
Submit it
The GitHub form asks for each field below, and the raw output paste:
→ Submit on GitHub ✉ Or email llmcheck@protonmail.com
No GitHub account? Email the same details, or the helper script’s submission.md.
What makes a row usable
A run becomes a row only when it can be tied to one Mac, one build of the model and one runtime version. Every submission needs:
- Chip, GPU cores and unified memory.
- The exact model build: file name, Hugging Face repo or Ollama tag.
- The quant, as the file names it. Write “unstated” if you cannot tell: the index never guesses one.
- The runtime and its exact version or build: speeds move between releases.
- The prompt depth of each figure: how many tokens were in the context when it was measured. Generation slows as the context fills.
- Plain decode, with speculative decoding off. A drafted speed is welcome alongside it (below).
- The raw output: the tool’s own table or stats, so the numbers can be checked.
One sitting is one row. One Mac, one runtime build and one checkpoint, however many runs, is entered once: at its shortest context, the median where you repeated it, any warm-up run left out. A different day, Mac, build or checkpoint is a new row (methodology).
Runs at more than 5 bits per weight (8-bit, BF16) are published, but the leaderboard quotes runs at 5 bits or fewer (which figure the index quotes).
Speculative decoding (MTP, draft models): report it separately
Speculative decoding drafts several tokens ahead, with the model’s own multi-token-prediction (MTP) heads, a small draft model or n-gram lookups, and keeps the ones the model agrees with. It can speed generation up a lot, but by how much depends on the text: code drafts well, prose less so. So the index ranks and estimates on plain decode only, and records a drafted speed only next to a plain one from the same sitting, with its method and workload. A drafted figure never sets a leaderboard speed.
- Ollama: off is
/set parameter draft_num_predict 0on a tag that names a GGUF quant (the llama.cpp engine); a tag with no quant in its name runs on the MLX engine, which drafts whenever the model ships a draft head and ignores the setting. On is Gemma 4’s default, and the-mtptags, which already setdraft_num_predict 2in their Modelfile. - llama.cpp: off is llama-bench, or
llama-serveras it starts (--spec-type none). On isllama-server -m model.gguf --spec-type draft-mtpfor a model with MTP heads, or-md draft.gguf --spec-type draft-simplewith a draft model (a draft model alone does not draft:--spec-typedefaults to none). Send both the same prompt and copy theeval timeanddraft acceptancelines from the server log. - mlx-lm: off is
mlx_lm.benchmark; on ismlx_lm.generate --model <repo> --draft-model <small repo> -p "…" -m 256(itsGenerationline). - LM Studio: Speculative Decoding off, then on with a draft model selected.
The form’s “Same-sitting speed WITH drafting” field takes the tok/s, the method (MTP, DFlash, EAGLE-3, the draft model’s name, n-gram) and the workload (code or chat).
BEFORE YOU RUN — QUICK GUIDELINES
- Plugged in, with Low Power Mode off: Apple says a MacBook Pro performs the same on battery, but an Energy Mode set to Low Power on battery (System Settings › Battery) slows it.
- Quiet machine: close other heavy apps. Browsers with 50 tabs count; they share the same unified memory.
- Exact versions:
ollama --version, the build number llama-bench prints,pip show mlx-lm, or LM Studio’s About screen. For reference, on 18 November 2026 the newest stable Ollama was 0.40.16, llama.cpp had released v1.0.0 (13 November) and mlx-lm was at 0.32.2. - Repeat it: three timed runs, the median, the first (warm-up) run left out.
- Default settings, or say what you changed: context length, KV-cache type, thinking mode,
iogpu.wired_limit_mb.
What happens next
A maintainer reviews each submission by hand: the paste against the figures, the speed against the memory-bandwidth ceiling for that chip and model, and against the index’s own estimate (a big gap gets a question, not a rejection). Accepted runs join the next data release as community rows, with the GitHub issue as their source and your handle as credit, and appear on the benchmarks page, the model’s pages and the open data downloads. Community rows are labelled apart from estimated and sourced figures; the methodology page explains all three.
FAQ
Do I need to be technical?
Not really. If you can paste one command into Terminal, or read the stats LM Studio prints under a response, and fill in a form, you can contribute. The helper script fills in most of the form for you.
Why does speculative decoding have to be off?
Because how much drafting helps depends on the text being generated: code drafts well, prose less so. Plain decode is the figure that compares across Macs and runtimes, so it is the one the index ranks on. A drafted figure is welcome too, next to a plain figure from the same sitting, with its method and workload.
What data do the form and the script collect?
Only what you choose to submit: chip, GPU cores, RAM, macOS version, the model, quant and runtime, and the speeds. The helper script runs on your Mac with no network calls; it downloads nothing, sends no telemetry and uploads nothing. The site itself uses Google Analytics and Cloudflare Web Analytics (see the privacy policy).
Which runs help the index most?
Any run with all the fields above helps. Most useful are chips and models with few measurements so far (each speed on the benchmarks page is marked measured or estimated), runs at 5 bits per weight or fewer, which can set a leaderboard speed, and plain decode at 8K or 32K context next to the short-context figure.
Can I send a figure someone else published?
Yes: email the link to llmcheck@protonmail.com. A figure from an accountable source, such as a vendor or a review that documents its setup, becomes a sourced row; one person’s post becomes a community row. The GitHub form is for your own runs.
How long until my benchmark appears?
Submissions are reviewed by hand and merged at the next data release. Releases ship every few weeks, and the changelog lists each one. Your GitHub issue is closed with a link to your row.