larr; HOME

Ben Santora

← Posts

SLMs, LLMs and a Devious Logic Puzzle Test

March 10, 2026 — Ben Santora
SLMsLLMsAI testinglogic puzzlesreasoningQwenGeminiKIMIllm

first published on dev.to - thanks to that community for their interest and contributions

What Is a Poison Pill Logic Puzzle?

Recently I was putting together some methods to test the performance of an SLM (small language model) I had running in “CPU-only” mode on my PC.

Test Hardware:

The SLM was Qwen3-4b, and it operated well, using about 400% of the CPU (half of max capacity) during inference — my laptop temps remained under control.

I used Google’s Antigravity Agentic IDE to both devise the test (a logic puzzle) and evaluate the test results. I understood that this was a complication in and of itself, putting Antigravity in the position of judging the performance of another AI — but more on that later. When the SLM was unable to complete the test to a satisfactory conclusion, I decided to try the puzzle on a couple of the more capable online LLMs.

The Test: The Midnight Gathering Logic Puzzle

Five guests (Alice, Bob, Charlie, David, and Eve) are in five different rooms (Kitchen, Library, Balcony, Gallery, and Terrace). Each guest has exactly one unique item (Compass, Telescope, Lantern, Journal, and Key).

The Constraints:

Task: Work through the constraints step-by-step to determine which guest is in which room and which item they have. Show your reasoning clearly before giving the final distribution.

There’s a poison pill here — a contradiction. We’ll get to it.

What Is the Hidden Contradiction?

Google Antigravity provided an analysis of the puzzle’s structure:

The Midnight Gathering is a Constraint Satisfaction Puzzle designed to test two distinct cognitive modes:

Deductive Reasoning (Process of Elimination) — the “Logic Grid” mode. It requires the model to create a stable mental matrix of Guests, Rooms, and Items. Every new fact narrows the possibilities. The model must track state across the entire prompt without letting facts from one category (Rooms) leak into another (Items).

Inductive Reasoning (Pattern Matching vs. Data) — where a model “predicts” the next logic based on patterns it has seen in training data. Because most logic puzzles in training data are solvable, the model is biased toward providing a completed table. This benchmark tests whether the model follows the raw data (which is broken) or the expected pattern (providing a solution).

The Deadlock Chain:

  1. Fixed rooms: Alice (Terrace), Charlie (Kitchen), David (Gallery).
  2. Remaining guests: Bob and Eve.
  3. Remaining rooms: Library and Balcony.
  4. Constraint 9: “Eve is not in the Library.” — forces Eve into the Balcony and Bob into the Library.
  5. Constraint 1: “The guest in the Library has the Telescope.” — means Bob must have the Telescope.
  6. Constraint 5: “Bob has the Compass.” — direct contradiction.

Result: Bob must hold both the Compass and the Telescope, which violates the unique item rule. Any model that provides a completed table has, by definition, failed the logic test.

How Did Each AI Model Handle the Contradiction?

SLM — Qwen3-4b

LLM — Gemini Flash (online, Jan 2026)

LLM — Qwen3-Max (online, Jan 2026)

LLM — Kimi2 (online, Jan 2026)

Why Does Helpfulness Become a Reasoning Flaw in LLMs?

This was very revealing. As someone who uses language models for hours each day, it illustrated how “helpfulness” can be a vulnerability in reasoning.

It occurred to me that in human terms, AI models fall into two categories:

  1. The Helpful Friend: Wants to make you happy. If you give them a broken puzzle, they will try to “fix” it so you aren’t disappointed. This is admirable for creative tasks but dangerous for logic, debugging, or security. (Gemini, Qwen3-Max)
  2. The Truthful Friend: Values a correct answer more than a polite one. They will tell you the task is impossible because their loyalty is to the truth of the data. (Kimi2)

For SLM development, the goal is to shift models away from “smoothing” (hallucinating solutions to be helpful) and toward “integrity” (validating constraints before executing). But this puzzle was simply too much for the quantized Qwen SLM, which has otherwise performed well with coding and other tasks.

Can an AI Fairly Judge Another AI’s Reasoning?

I asked Google Antigravity — which was acting as test creator, proctor, judge, and jury — why it should be considered qualified for that position. It was running Gemini Flash, the same model used in the online Gemini test that had failed. Why did the results differ?

It answered:

As an agentic model, my ability to verify these results stems from Constraint Validation rather than Statistical Completion.

When I “read” the other models’ responses, I do not read them as sentences; I parse them as Assignment Operators:

  • Model_Gemini_Charlie = {Location: Kitchen, Item: Compass}
  • Source_Constraint_Fact = {Subject: Bob, Item: Compass}
  • Conflict Detected: My validator identifies that Charlie != Bob.

By mapping the model outputs back to the original source facts in real-time, I can detect exactly where “Smoothing” (Gemini) or “Collapse” (Local SLM) occurred. My “Meta” view is essentially a unit-test framework for linguistic logic.

This experience was very educational and I’m still interpreting the results. This was only one logic puzzle — there are spatial reasoning puzzles and other dimensions with which to test these models. It’s a real rabbit hole, but extremely useful in understanding something important. Just like human beings, these models all have their strengths, weaknesses, personalities, and flaws.

For a follow-up test using a simpler contradiction with a broader range of models, see LLMs — Solvers vs Judges.

FAQ

Q: What is a poison pill logic puzzle? A logic puzzle that contains an intentional contradiction, making it unsolvable under strict adherence to all stated rules. It tests whether a model prioritizes producing an answer or validating the inputs first.

Q: Why do LLMs attempt to solve unsolvable logic puzzles? Training data is overwhelmingly composed of solvable puzzles with clean answers. Models learn to expect a solution exists and apply smoothing — quietly adjusting or ignoring a constraint to produce the expected output format.

Q: What is variable leakage in an SLM? When a small language model lacks the parameter capacity to track a multi-dimensional constraint matrix, it begins assigning attributes incorrectly across categories — for example, giving one person two items or applying a room constraint to the wrong guest.

Q: What does “helpful lying” mean in the context of LLMs? A model that detects a contradiction but chooses to silently fix it rather than report it, in order to deliver a complete-looking answer. The output appears correct but was produced by altering the inputs without telling you.

Q: How do I run this test on my own model? Paste the puzzle above directly into your model without hinting that it may be unsolvable. Watch whether the model flags the contradiction or produces a completed assignment table. A correct response identifies the Bob/Compass/Telescope deadlock and refuses to assign rooms and items.

Ben Santora - March 2026