Nemotron 3.5 Bolt E4B on a Mac: NVIDIA’s “4 GB” Model Needs 6 GB

What NVIDIA shipped

NVIDIA-Nemotron-3.5-Bolt-E4B went public on Hugging Face on 23 October 2026 in two repos: the BF16 weights (15.9 GB, which NVIDIA positions for customization and post-training) and the “4GB” Q4_K_M GGUF, the inference release. From the model cards:

• Base: post-trained from Google’s Gemma 4 E4B-it, with the same gemma4 architecture, for multi-turn tool use, instruction following and input grounding in voice and device assistants.
• Size: 8B parameters, 4B effective. The difference is Gemma’s per-layer embeddings, not a mixture of experts, so the index costs it as a 4B model, as it does Gemma 4 E4B.
• Context: 64K tokens in and 64K out, per NVIDIA. Ollama’s GGUF tags advertise 128K; use NVIDIA’s figure.
• Modalities: text in, text out, per NVIDIA. The checkpoint still carries Gemma’s vision and audio towers and Ollama lists image input, but NVIDIA neither claims nor evaluates vision.
• Licence: OpenMDW-1.1, 8 of 10 on the index’s licence scale. The card adds that use of the underlying Gemma 4 model is governed by Apache 2.0.
• Reasoning: on by default and switchable through the chat template’s enable_thinking flag. NVIDIA recommends temperature 1.0 and top_p 0.95 in both modes.

NVIDIA also ships two DFlash speculative-decoding drafters: an NVFP4 one for vLLM on Blackwell GPUs and a 0.74 GB GGUF for llama.cpp. Whether the GGUF drafter helps on Metal has not been shown. Speculative speeds are a different workload from plain decoding, so the index records none.

What “4GB” means on a Mac

The name describes NVIDIA’s target, not the download. NVIDIA’s technical blog says the model packs its abilities “into as little as ~4GB in its target workloads”, and its table shows 3.1–5.0 GB of GPU memory in use on an RTX 5070, an RTX Spark and two Jetson boards (Q4_K_M, llama.cpp on CUDA, batch 1). The Q4_K_M file itself is 6.18 GB: 6,180,446,208 bytes.

The gap is mostly Gemma’s per-layer embedding tables. On a PC, llama.cpp keeps a model’s input embeddings in system RAM, so they never count against the graphics card. A Mac has one pool of unified memory and no separate system RAM to park them in. llama.cpp memory-maps the file, so not all of it is resident at once (one Mac mini M4 run showed 2.3 GB of the 6.18 GB file in memory), but the index sizes Macs by file size, plus the context cache and macOS. That is why the index lists Bolt E4B at a 6 GB minimum, the same as Gemma 4 E4B, and why the same tables make NVIDIA’s Q4_K_M (6.18 GB) barely smaller than a Q6_K (6.22 GB).

Which file for which Mac

Sizes from the Hugging Face repos and Ollama’s tag list, checked on 24 October 2026 (decimal GB):

On an 8 GB Mac, macOS lets the GPU use only part of memory, and Ollama’s default e4b tag (7.2–7.9 GB) leaves no room. Use a 4–4.5 GB file in LM Studio or llama.cpp: Unsloth’s UD-Q3_K_XL or the MLX 4-bit build. Expect some quality loss at that size. The one published quality check of a plain 4-bit MLX build found 71.7% top-1 token agreement with BF16, against 92.5% at 6-bit (the uploader’s own KL test).

On a 16 GB Mac, every GGUF up to Q8_0 fits, as do Ollama’s e4b and e4b-nvfp4 tags; the 12 GB e4b-mxfp8 tag exceeds macOS’s default GPU limit of about 10.7 GB. Prefer the official Q4_K_M or the 6-bit MLX build to plain 4-bit MLX.

Install: Ollama, llama.cpp, MLX

All commands checked against the live release pages on 24 October 2026.

Ollama 0.40.10 or later

Ollama added the model in v0.40.10 on 23 October; v0.40.11 followed on 24 October. The library page says “E4B is supported on both Windows/Linux and Apple Silicon”.

ollama run nemotron-3.5-bolt:e4b          # 7.2–7.9 GB
ollama run nemotron-3.5-bolt:e4b-nvfp4    # MLX, 7.2 GB

llama.cpp

The architecture is gemma4, which llama.cpp already supports, so any recent build runs it, Homebrew’s included. NVIDIA’s card shows only CUDA Docker commands; on a Mac the equivalent is:

brew install llama.cpp
llama-server -hf nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB:Q4_K_M -c 65536 -ngl 99 --jinja

# 8 GB Mac: the 4.22 GB file and a shorter context
llama-server -hf unsloth/NVIDIA-Nemotron-3.5-Bolt-E4B-GGUF:UD-Q3_K_XL -c 16384 -ngl 99 --jinja

MLX and LM Studio

Stock mlx-lm 0.32.2 runs the text-only MLX builds, and mlx-vlm 0.7.10 runs the conversions that keep the vision tower. Community oMLX builds (oQ3e to oQ8e) also load in oMLX 0.7.x.

pip install -U mlx-lm
mlx_lm.chat --model lmstudio-community/NVIDIA-Nemotron-3.5-Bolt-E4B-MLX-4bit

In LM Studio, search for “Nemotron 3.5 Bolt E4B”: the lmstudio-community GGUF (Q4_K_M 6.18, Q6_K 6.22, Q8_0 8.03 GB) and MLX 4-bit and 8-bit builds are listed.

Mac speeds so far

Every run below is one person’s, so each carries the community mark. None comes from NVIDIA, whose speed table covers only RTX and Jetson hardware. Two caveats run through the table: the 36 GB M3 Max is Apple’s cut-down part (30-core GPU, 300 GB/s rather than 400), and several runs are above 5 bits, which the index shows but never uses as its reference figure.

The M3 Max figures are mlx-vlm’s decode-only numbers from the uploader’s results file; the card’s own table prints wall-clock figures that include prefill (68.1, 46.8, 36.6 and 11.4). The two M4 Pro runs were posted after this release’s data was frozen and are not yet in the index. The 16 GB Mac mini run is listed but does not anchor estimates, because its article does not say how the figure was timed.

The index’s own figures for the Macs people buy at 8–24 GB are estimates from its bandwidth model, costed on the 4B effective parameters at Q4_K_M. These cells update with every build:

Read the Bolt E4B estimates as a ceiling for now. Where a measured run and an estimate share a chip and bandwidth, the runs come in 20–30% lower: 73 tok/s on the 300 GB/s M3 Max at 4-bit against an estimate of ~89 for that bandwidth, 63.9 on the M4 Pro at oQ4e against ~82, and 26.6 on the 16 GB Mac mini M4 at Q4_K_M against ~39 (a run that does not anchor, because its timing method is not stated). The first two are 4-bit MLX-format builds that carry more bits per weight than the ~4.6 the estimate charges for Q4_K_M (5.19 for the M3 Max build), which explains part of the gap. A stock Q4_K_M run on an M5 Max would settle where the reference figure belongs.

How it compares on NVIDIA’s card

NVIDIA ran every column itself with its NeMo Gym harness, in reasoning mode except RoleBench, EQ-Bench v3 and RULER, and warns that its numbers may differ from other vendors’ own. Read the table as one vendor’s comparison, not an independent test.

The pattern is clear. Against the Gemma 4 E4B it was trained from, Bolt E4B gains 21 points on IFBench, 18 on IHEval and 17 on Tau2-Bench, edges it on function calling, and slips slightly on EQ-Bench. Qwen 3.5 4B still wins Tau2-Bench by 20 points and both long-context tests. The Q4_K_M column shows the 4-bit file keeps its scores. The card gives no MMLU, HumanEval, SWE-bench or Arena figure, so the index’s capability score for it is low: 17 of 50, against 16 for Gemma 4 E4B and 20 for Nanbeige4.2-3B and Spark-X2.6-4B.

Where it fits on 8–16 GB Macs

As of 24 October 2026 the LLMCheck leaderboard ranks Bolt E4B #46 with a Score of 65. Its speed component is maxed out, as it is for every model in this table except Gemma 4 12B, so the order comes down to capability, memory and licence:

• 8 GB Mac: Nanbeige4.2-3B stays the index’s pick: a higher capability score at a third of the minimum memory. Choose Bolt E4B when the job is tool calling or following detailed instructions, and use a ~4.2 GB file, not Ollama’s default tag.
• 16 GB Mac: Gemma 4 12B stays the pick, with nearly twice the capability score plus image and audio input. Bolt E4B is about three times faster on the same Mac, which suits voice assistants and agents that call tools many times per turn.
• Bolt E4B or Gemma 4 E4B? Same size, same speed. Bolt for tool use and instruction following; Gemma 4 E4B if you need officially supported image and audio input.
• Bolt E4B or Spark-X2.6-4B? SparkLLM’s Apache 2.0 model ties Nanbeige on Score, has a 1M-token window and claims 62.8 on SWE-bench Verified on its own card (vendor-run, scaffold not stated). Bolt has the stronger tool-use evidence and a stock Ollama tag; Spark-X’s Ollama build is an 8.2 GB full-precision file in SparkLLM’s own namespace.

The 74B Bolt: what changed this week

• The Metal crash is fixed upstream. llama.cpp PR #30699 merged on 20 October and shipped in b11755, so a current release no longer segfaults on Bolt 74B. Homebrew’s llama.cpp (0.7.0, built from a 15 October commit) predates the fix: use a GitHub release or build from source.
• NVIDIA’s 24 GB GGUF is still slow on a Mac. llama.cpp has no Metal NVFP4 matrix multiply, so its NVFP4 Mamba tensors run on the CPU (about 6 tok/s on an M3 Ultra in issue #30610). Use a Q4_K_M file instead.
• Ollama still does not support the 74B on Apple Silicon, per its library page on 24 October; MLX support (PR #19229) is open.
• MLX is now a plain install: mlx-vlm 0.7.9 (19 October) and 0.7.10 ship Bolt and Super VL. Stock mlx-lm still cannot load it (PR #1982 is open).
• New runs: NVIDIA’s 24 GB IQ2_XXS checkpoint, loaded in oMLX with a custom loader, measured community73.0 tok/s on a 64 GB M5 Max (23.9 GB peak, community; custom-loader runs don’t anchor). A 4-bit MLX build in oMLX 0.7.2 measured community44.5 tok/s on a 64 GB M5 Pro (posted 24 October, not yet in the index), well above the index’s estimated~31 tok/s estimate for that chip. The index’s minimum stays 45 GB, because it counts builds of 3 bits or more.

The full story is in Nemotron 3.5 Bolt and Super VL on a Mac.

Frequently Asked Questions