Nota AI's 1.58-bit 30B: shrink to 10 GB, then recover 89.4%

A 30B model on a 24 GB MacBook still sounds wrong. At 4 bits the weights alone are around 16 GB, and the KV cache and runtime buffers make 24 GB tight. Nota AI's 1.58-bit release changes the arithmetic. It shrinks Nemotron 3.5 Lightning 30B-A3B from about 60 GB in BF16 to 9.6 GB and recovers 89.4% of the original score on a seven-benchmark generative average.

The headline is not “1.58-bit makes it small.” As the Nota AI blog states, converting a BF16 model straight to ternary sends GSM8K and HumanEval to 0.00. Shrinking the file and preserving the model's ability are separate problems. This article focuses on the recovery procedure that closes the gap.

1.58-bit is information per weight, not file size

Ternary quantization represents each weight as one of -1, 0 or +1. The theoretical minimum information needed to distinguish three values is log₂3 ≈ 1.585 bits, hence the name. The file on disk is not exactly 1.58 bits per parameter. It also stores scales and container metadata.

Nota AI's procedure has three steps. Compute the group mean absolute value α, then set the threshold τ = α/2. Weights with |w| > τ become +1 or -1 by sign; weights with |w| ≤ τ become 0. Finally, compute the mean absolute value s of the non-zero codes and reconstruct ŵ = s × q.

α = mean(|w|)
τ = α / 2
q = +1   if w > τ
     0   if |w| ≤ τ
    -1   if w < −τ
s = mean(|w|)  over q ≠ 0
ŵ = s × q

In the blog's eight-weight example, α = 0.39, τ = 0.195 and s = 0.58. 0.90 becomes +1, -0.30 becomes -1 and 0.05 becomes 0. The reconstructed values are +0.58, -0.58 and 0. The share of weights that become zero matters. Too few and small values near zero grow to ±1, increasing error. Too many and the model loses information and expressiveness. τ sets that share.

Why direct conversion scores zero

Nota AI chose post-training quantization (PTQ) so it could reuse an already-trained BF16 checkpoint. This is a different path from quantization-aware training (QAT), which starts from random initialization and pretrains with ternary weights, as in BitNet b1.58. PTQ completed without error, but internal evaluation produced GSM8K 95.00 → 0.00, HumanEval 96.34 → 0.00 and IFEval-ko 89.46 → 10.36.

The cause is accumulated rounding error. Each weight picks up a small reconstruction error, and across dozens of transformer layers that error feeds into the next-token probabilities. On token-by-token generative benchmarks, it turns into a wrong answer almost every time. This is a common failure mode below 2 bits.

Recovery 1: on-policy distillation and STE

The first recovery step is distillation. Standard off-policy distillation has the student imitate answers the teacher generated in advance. But the ternary student's output distribution is badly shifted after quantization, and teacher-written sentences barely expose the errors the student actually makes. Nota AI added on-policy distillation. The student generates its own context; the teacher computes the next-token distribution in that same context; the student learns from the difference between the two distributions.

Quantized weights are not differentiable. The team uses a straight-through estimator (STE). The forward pass uses the ternary code as-is; the backward pass treats the quantization function as an identity and passes the gradient to the latent weights. Latent weights are the full-precision values behind the ternary code. As small updates accumulate, a weight crosses the threshold and its code flips.

About 5–7% of weights changed code in one epoch. That suggests recovery is less about adding new knowledge and more about fixing the small number of codes that quantization moved so the model matches the teacher's output again.

Recovery 2: boundary masking

Distillation and STE improved performance sharply in the first one or two epochs. GSM8K rose 82.56 → 87.57, HumanEval 72.56 → 80.49 and IFEval-ko 61.79 → 68.39. But training to three epochs sent GSM8K back to 85.90 and IFEval-ko to 64.82. The same pattern appeared with other models.

Nota AI's explanation is that STE sends a signal to every latent weight, so values that do not need to change keep updating and noise flips codes. The team narrowed the training range. It trained only weights whose PTQ initial value sat within ±10% of τ—about 5.9% of the total—and froze the rest. Only values near the threshold can flip their code with a small update; values far away rarely change code and mostly add noise. The method is called boundary masking.

Boundary masking at the same three epochs lifts GSM8K from 85.90 to 88.32 and IFEval-ko from 64.82 to 68.39. Where to train mattered more than training longer.

The final scorecard

The recovered model averages 83.82 across seven generative benchmarks, 89.4% of the teacher's 93.80. The per-benchmark numbers carry more information than the average.

The English four-benchmark average is 86.28 (90.6%); the Korean three-benchmark average is 80.55 (87.7%). Math (GSM8K) and code (HumanEval) recover to about 94%, but MATH500, which needs long solutions, recovers to 83.6%, and Korean instruction-following to 81.6%. Deploying this model into a Korean agent workflow on the strength of “89.4% average” can produce a different experience.

Read the evaluation setup too: temperature 0.7, full test sets, a 32K generation budget, and Nota AI's internal evaluation. There is no independent reproduction. The model card also states that this checkpoint is an intermediate release for public use, not the fully recovered model.

Fitting a 10 GB model onto a MacBook

The model card reports 60 GB in BF16 shrinking to 9.6 GB, 6.25× smaller. MLX has no 1.58-bit container, so the weights ship in a 2-bit affine container: bits=2, group_size=64, levels={0,1,2}, with level 3 unused. bias = -scale, so dequantization is scale × (q - 1) = {-a, 0, +a}. The stored values are ternary, but the file format is 2-bit affine.

The memory calculation is public. Weights are about 10 GB. At a 128K context the KV cache is about 0.8 GB, because only six attention layers need one. Mamba state and runtime buffers add at most about 2.4 GB, for a total of at most about 12.4 GB. A 24 GB MacBook's default GPU limit of about 16 GB fits it. A 16 GB MacBook's default limit of about 10.7 GB does not; raising the limit to about 13.5 GB makes it fit on paper, but leaves much less memory for the system and other apps. That is why Nota AI recommends 24 GB or more.

Where it fits, and where it does not

The appeal is the cost structure of “30B-class reasoning locally.” Start with code completion, local document summaries, classification and extraction, and repetitive agent subtasks where 89% quality is acceptable. Do not drop it into a Korean math or instruction-heavy workflow without a task-specific evaluation, because MATH500 and IFEval-ko are the weakest recoveries.

This result is also not “1.58-bit works for anything.” Teacher quality, distillation data, the boundary-masking ratio and MLX kernel support all affect the outcome. The model card notes that the architecture must be implemented in your mlx-lm or mlx-vlm build. A quantized file is not portable to any runtime.

Insight: extreme quantization is a recovery problem

The real lesson of 1.58-bit is not the bit count but the unit of recovery. Nota AI did not retrain every weight. It trained the 5.9% near the threshold where a code can flip. “Where to train” restored the score, not “train longer.”

The same lens applies to local model adoption. Do not decide from file size and memory math alone. Ask what a failure costs in your workflow, which benchmark is weak rather than which average is high, and whether the unrecovered capability is central to the task. A 10 GB model fitting on a 24 GB laptop is the starting point. An 89.4% average is not a promise that your workload will recover 89.4%.

For model selection and routing cost, read Muse Ember. For an engine that keeps a scene as inspectable data, read raea 0.1.0.