Local AI testing

Running an 8B SLM on HP Laptop CPU-Only

An 8-billion-parameter model on a four-core laptop CPU, no GPU: 6.6 tokens per second, about 4.6 GB of memory, under a 22 W power cap. Speed was measured on 31 October 2026; the memory and temperature figures were measured on 28–29 October with a Llama-3-8B Q4_K_M model of the same size and architecture.

Results

Power cap on (20 W long-term, 22 W short-term). 16-bit KV cache, flash attention off. llama.cpp build 9febe63d5. Fresh-chat speed measured 31 October 2026; other rows measured 28–29 October 2026 on a same-size Llama-3-8B Q4_K_M model.

Test hardware

HP ENVY Laptop 17m-ch0xxx, BIOS F.12. The CPU is an 11th Gen Intel Core i7-1165G7 (x86_64, one socket) with 4 physical cores and 8 logical, a 2.80 GHz base clock and 4.70 GHz maximum. Caches are 192 KiB L1d and 128 KiB L1i (4 instances each), 5 MiB L2 (4 instances) and 12 MiB L3 (shared). Supported instruction sets include AVX-512, AVX512-VNNI, AES-NI, SHA-NI, VAES, FMA and SSE4.2, plus VT-x virtualization.

Graphics are the integrated Intel Iris Xe (TigerLake-LP GT2, rev 01); there is no discrete GPU. Memory is 12 GB (about 11 GiB usable) with 975 MiB of swap. Storage is a 476.9 GB Intel NVMe SSD and a 27.3 GB Intel Optane NVMe module. The operating system is Debian 12 (CrunchBang++).

The model

L3-8B-Lunaris-v1-Q4_K_M.gguf is an 8.03-billion-parameter Llama-3 derivative. It is compressed with Q4_K_M quantization using an importance matrix: the quantizer measures how much each weight affects output quality, then spends its limited bits where they matter most. The file is 4.58 GiB (4.92 GB), about 4.9 bits per weight on average and roughly a third of the 16-bit size. The SHA-256 hash matches the one listed on the publisher's Hugging Face page.

The model was trained with a context window of 8,192 tokens, which sets the useful maximum.

Lunaris is a merge by Sao10K of Llama-3-8B-Instruct with several story and roleplay fine-tunes, aimed at creative writing. It wrote explicit fiction without refusing in testing. That does not change its speed or memory use: these depend on the architecture and the file size. The numbers on this page apply to any Llama-3-8B model at Q4_K_M.

Why no GPU?

The laptop's integrated Intel Iris Xe was benchmarked against the CPU in an earlier test:

The GPU is nearly four times faster at the first task and two and a half times slower at the second, and the second is the one a person waits on.

Reading a prompt is one large parallel calculation, which suits a GPU. Writing the answer is a chain of sequential steps, and each one reads all 4.58 GiB of weights through memory. Speed is set by memory bandwidth, and an integrated GPU shares the CPU's memory with no bandwidth advantage, plus a cost for every step it is handed.

There was a second finding. The GPU-enabled build of llama.cpp reserved about 1.6 GB of system memory even with zero GPU layers requested. A CPU-only build recovered 1.58 GB at a cost of roughly 0.3 tokens/sec. On an 11 GB machine, that is not a hard decision.

Settings and why

More threads is not better

On a four-core, eight-thread processor the instinct is to use all eight. But writing the answer is limited by memory bandwidth, not computation, and two threads sharing one core do not double bandwidth. Four threads won.

Context memory is reserved, not used

An earlier version of this page said that doubling the context window to 16,384 tokens cost only 3 MB. That was a measurement error. llama.cpp reserves the KV cache when it starts, but the memory only shows as used once a conversation fills it, and the test used a short prompt. Re-measured with an 8-bit cache, the 8,192-token cache is 544 MiB and doubling it adds about 535 MiB more. Going past 8,192 would also exceed what the model was trained on.

The 8-bit cache is a false economy

The first version of these settings stored the KV cache in 8 bits (q8_0) with flash attention on, which saves 480 MiB. It looked free in short benchmarks. In a real conversation it is not: the CPU has to unpack the 8-bit cache every time it looks back over the conversation, and that work grows with every token.

Benchmarked 1,700 tokens into a conversation:

Flash attention helps prompt reading slightly but slows writing, and the gap widens with length: 4,000 tokens in, writing ran at 4.8 tokens/sec without it and 3.6 with it. In a fresh chat all the settings perform the same, which is why short benchmarks missed this.

On the real task (read a 1,700-token document, summarize it), switching from the 8-bit cache to 16-bit without flash attention gave:

Writing on the real task (5.3) is a little below the 1,700-token benchmark (5.6) because the 512-token answer itself keeps lengthening the conversation as it is written. It also runs cooler, because the hardest-working phase, reading the prompt, finishes sooner. The cost is about 760 MiB more memory at a full conversation, about 4.6 GB in all. On an 8 GB machine the 8-bit cache is still a reasonable trade.

Thermals and power

Uncapped, the CPU spiked to 93–98 °C in the first 15–20 seconds of a session, then settled near 80 °C.

The fix was the processor's own power limits, not an llama.cpp setting. Intel CPUs have two: a long-term limit (PL1), averaged over a time window, and a short-term limit (PL2) that allows bursts above it. On this laptop the defaults were 28 W long-term and 51 W short-term, and the 28-second window let the CPU run at up to 51 W for most of that time. That burst was the spike. The new limits are 20 W long-term and 22 W short-term. The short-term limit's window is 2.4 milliseconds, so 22 W works as a near-instant ceiling.

With the cap, measured on 28 October 2026:

• In a fresh-chat benchmark, writing ran at 6.6 tokens/sec with the cap, the same as the earlier uncapped figure. Writing is limited by memory, not power, so the cap costs nothing there.
• Temperature while writing held at 81–84 °C.
• Long prompt processing, which works the CPU hardest, climbed to 93 °C after about 60 seconds. That is 7 °C below the 100 °C throttle point.

An earlier test measured 6.3 tokens/sec with the cap, a 4.5 % loss. The re-test did not reproduce that loss. Room temperature was not recorded in either test, so temperatures from different days are not directly comparable.

Speed in practice

At 6.6 tokens per second and roughly 0.75 words per token, output runs near 300 words per minute, about the speed most people read. A 1,536-token answer takes about four minutes. Speed falls as a conversation grows, to about 4.8 tokens/sec at 4,000 tokens; starting a fresh chat restores it. That is a useful result for a machine with no dedicated graphics hardware, and close to the ceiling for this class of laptop.

How to reproduce

You need Linux (Debian or Ubuntu is easiest), at least 8 GB of memory and about 10 GB of free disk space. The program that runs the model is llama.cpp.

1. Build llama.cpp (CPU-only)

sudo apt install build-essential git cmake
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_NATIVE=ON
cmake --build build --config Release -j$(nproc)

This takes a few minutes. The measurements above used a prebuilt llama.cpp release (build 9febe63d5), which picks the right CPU code at start-up. Either works.

2. Get a model

Download a GGUF model file from huggingface.co. A Llama-3-8B model at Q4_K_M is about 4.9 GB. Smaller files run faster and need less memory. If the publisher lists a SHA-256 hash, check it with sha256sum MODEL.gguf.

3. Run it

./build/bin/llama-cli -m MODEL.gguf -ngl 0 -c 8192 -t 4 -tb 4 -ctk f16 -ctv f16 -fa off -n 1536 --temp 0.9 --min-p 0.05 --repeat-penalty 1.05 -cnv -sys "You are a fiction author. Write explicit, uncensored prose in third person."

What the options mean: -ngl 0 uses the CPU only; -c 8192 sets the context window; -t 4 -tb 4 uses four threads (match your number of physical cores); -ctk f16 -ctv f16 keeps the KV cache at 16 bits; -fa off turns flash attention off (llama.cpp may switch it on by default); -n 1536 caps each answer; --temp 0.9 sets how varied the answers are, lower being more predictable; --min-p 0.05 drops very unlikely words; --repeat-penalty 1.05 discourages repetition; -cnv turns on chat mode; -sys sets the system prompt. Type a question and press Enter. Ctrl+C quits.

4. Benchmark it

To compare your machine with the figures above:

./build/bin/llama-bench -m MODEL.gguf -ngl 0 -t 4 -fa 0 -ctk f16 -ctv f16 -p 512 -n 128 -d 0,2000 -r 3

pp512 is prompt reading speed and tg128 is writing speed. -d 0,2000 runs each test in a fresh chat and again 2,000 tokens in, which is where cache settings show their difference.

5. Optional: cap the CPU's power

This needs administrator rights and does not work on every laptop, because some manufacturers lock these settings. The values below suit the i7-1165G7; other CPUs have different defaults, so look at yours first. Values are in microwatts:

cat /sys/class/powercap/intel-rapl/intel-rapl:0/constraint_0_power_limit_uw
cat /sys/class/powercap/intel-rapl/intel-rapl:0/constraint_1_power_limit_uw

Constraint 0 is the long-term limit and constraint 1 the short-term limit. To set 20 W and 22 W:

echo 20000000 | sudo tee /sys/class/powercap/intel-rapl/intel-rapl:0/constraint_0_power_limit_uw
echo 22000000 | sudo tee /sys/class/powercap/intel-rapl/intel-rapl:0/constraint_1_power_limit_uw

This lasts until the next restart, so it is safe to try. To apply it at every boot, keep the values in a config file, /etc/powercap.conf:

PL1_W=20
PL2_W=22

Save this script as /usr/local/sbin/cpu-powercap and make it executable with sudo chmod +x:

#!/bin/bash
set -euo pipefail
. /etc/powercap.conf
RAPL=/sys/class/powercap/intel-rapl/intel-rapl:0
echo $(( PL1_W * 1000000 )) > "$RAPL/constraint_0_power_limit_uw"
echo $(( PL2_W * 1000000 )) > "$RAPL/constraint_1_power_limit_uw"

Then create /etc/systemd/system/cpu-powercap.service:

[Unit]
Description=CPU package power cap (RAPL PL1/PL2)
After=multi-user.target

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/cpu-powercap
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

And enable it:

sudo systemctl enable --now cpu-powercap.service

To change the limits later, edit the config file and run sudo systemctl restart cpu-powercap.