2021 – present · Independent · Local AI testing

Running a small language model on a laptop CPU — and measuring everything

An 8-billion-parameter AI model, no cloud, no GPU. What it took to make it fast enough to read, and cool enough to trust.

What I'm doing now

Since retiring in 2021 I have kept working the way I always did: take a system, find its real limits, and make it behave predictably. The system these days is a local large-language-model stack — an 8-billion-parameter AI model running entirely on a four-core laptop, with no cloud service, no GPU, and no data leaving the machine.

The interesting part was never “can it run.” It was whether it could run well and within limits — the same question I spent fifty years answering about geodetic instruments, avionics, RF power supplies, and superconducting magnet systems.

The short version: it runs at about 7 tokens per second — roughly reading speed — while holding 80 °C, drawing under 22 watts, and using 6.2 of the machine's 11.2 GB of memory. Every one of those numbers was measured, not estimated, and each represents a deliberate trade.

The model

The model is an 8.03-billion-parameter Llama-3 derivative, compressed to 4-bit precision with a technique called an importance matrix: the quantiser measures how much each weight actually affects output quality, then spends its limited bits where they matter most. The result is a 4.58 GB file — about a quarter the size of the same weights held at 16-bit precision — that behaves close to its full-precision original.

It is also an unaligned model — one trained without refusal behaviour. That was a deliberate choice for a technical test, not a preference: safety training adds a layer of behaviour between the weights and the output, and I wanted a clean measurement of the underlying inference performance without it. Treating it as a benchmark subject rather than an authority is the whole point.

Recognition matters in this work. The model file carries a published SHA-256 hash and mine matches it exactly — a simple check that the 4.58 GB on disk is the 4.58 GB the publisher shipped, unmodified.

Why no GPU

The laptop has an integrated Intel Iris Xe graphics processor. Benchmarked against this model, it looked like this:

The GPU is four times faster at the first task and two and a half times slower at the second — and the second is the one a person actually waits on.

The reason is architectural. Reading a prompt is one long, dense, parallel calculation: ideal for a GPU. Generating an answer is hundreds of sequential steps, each of which cannot start until the last finishes — a latency problem, not a throughput problem. It comes down to repeatedly pulling 4.58 GB of weights through memory, and an integrated GPU shares the same memory as the CPU with no bandwidth advantage and a per-step dispatch cost.

There was a second, less obvious finding. The GPU-enabled build of the inference engine reserved about 1.6 GB of system memory even when instructed to use zero GPU layers — memory the driver never actually got asked to use. Switching to a CPU-only build returned that 1.58 GB and cost roughly 0.3 tokens/sec. On an 11.2 GB machine, that is not a difficult decision.

The tradeoffs, and what each one bought

Two of these were counter-intuitive enough to be worth stating plainly.

More threads is not better. On a four-core, eight-thread processor the instinct is to use all eight. But this workload is not compute-bound — it is bound by memory bandwidth, and two threads sharing one core's cache do not double bandwidth. Four threads won.

Context size is nearly free. The intuition that a larger context window costs real memory is correct in general, but wrong at this scale: 4.58 GB of weights dwarf an 8K context. Doubling it changed peak memory by three megabytes. There was no trade to make, so I took the larger window.

Thermals and power — the part that mattered most

Left uncapped, the machine would spike to 93–98 °C during the first fifteen to twenty seconds of a session, then settle near 80 °C. The spike was a transient, not sustained heat, and transient or not, sitting a few degrees below a 100 °C throttle point is poor engineering.

The fix was not an inference setting at all. It was the processor's own power limit: the long-term constraint was reduced from 28 W to 20 W, and the short-term burst from 51 W to 22 W. The long-term limit averages over a 28-second window, which happens to cover the entire spike.

Twenty-nine percent less power for four and a half percent less speed. The workload was never fully power-bound, which is exactly why the trade was available: the machine now settles at 80 °C and stays there for the length of a session, with eleven degrees of headroom below the throttle point instead of two.

What 7 tokens per second actually means

At about 7 tokens per second and roughly three-quarters of a word per token, the model produces around 315 words per minute — slightly faster than most people read. (The capped benchmark figure is 6.3; seven is what it sustains in ordinary interactive use.) There is no waiting for the thought to arrive; the text appears at reading pace.

That is a genuinely useful result for a machine with no dedicated graphics hardware. It is also the ceiling for this class of hardware, and knowing where the ceiling is — and which knobs move it — is more valuable than a bigger number that only holds for a benchmark.

What this has to do with the rest of my career

The subject is new. The method is not.

Calibrating a geodetic instrument, bringing up an MRI magnet system, or verifying an RF power supply all start the same way: establish what the system is actually doing before changing anything, and know your measurement uncertainty before you trust a result. This project was that discipline applied to software — benchmark both paths rather than assume, check that the power limit was really in force rather than that it was configured, verify the model file against its published hash, and record the numbers so the next session starts from evidence instead of memory.

A fair amount of it was also just instrument work: watching package temperature climb under load, correlating it with wall-clock time, and noticing that the number everyone quotes was a twenty-second transient mislabelled as steady state.

The tools are different. The habits are the same.