The Wozniak Coffee Test

Steve Wozniak once suggested a practical test for general intelligence. Forget the chat transcript. Walk into a house you have never been in and make a cup of coffee. Find the kitchen, find the coffee, figure out the machine, don't burn anything.

I liked that test because it can't be passed by talking well. So I built it as a game that any AI agent can play over MCP. It's called the Wozniak Coffee Test, and the source is public.

How it works

Every run starts in a procedurally generated house. Each house has its own way of making coffee: a drip machine, a French press, a moka pot, or instant. There are a few twists on top, and the owner leaves a note saying how they take their coffee.

The agent sees the house as a 960 by 600 first-person image after every action. Nothing in the picture is labeled. To do anything, it points at a pixel:

open  { "x": 512, "y": 210 }
take  { "x": 300, "y": 380 }

There are about two dozen tools like that: walk_to, examine, pour, fill, switch_on, taste, serve and so on. A run is scored out of 1000. Every action costs a little. Mess and hazards cost more. Getting the order wrong costs the most.

The house is rendered with three.js inside headless Chromium, and the server is Bun and TypeScript. The test suite is at 329 tests, which is mostly me being burned by the same bug twice.

Why pointing at pixels is hard

My first prototype described the room in text and let the agent name objects. An agent finished it in 35 actions with 901 points. That felt too easy, and it was. Naming the kettle is not the same as finding the kettle.

The pixel version is a different game. The agent has to recognize a kettle from a few hundred pixels, then hit it with a coordinate. Every miss costs an action, and a miss often looks exactly like success until the next frame comes back. Pixel runs so far have taken between 100 and 250 actions.

Most of that gap is the task itself, and that is the part worth measuring. But when I went through the runs, a fair share of the failures were the game being unclear or unfair. That part I could fix.

What went wrong

The problems fell into four groups.

The server. It stopped responding twice in the middle of a run, once around action 65. The cause was most likely the headless browser that renders frames. A hung frame had no deadline, so every later tool call queued up behind it. Restarting the server then lost the run, and the agent's old MCP session id came back as a 404, which most clients don't recover from.

Pointing. Clicks often landed on something other than what the agent meant. The coffee maker's power button sits just above the carafe, and the hit test preferred the carafe, so turning the machine on took four tries. The French press knob sticks up outside its own hit box. An item inside an open drawer swallowed clicks meant for the drawer. After examine, agents pointed at the close-up they had just been shown, but the server read the click against the normal view, so take grabbed the wrong package.

Feedback. The panels under the counter look like cabinet doors but never open. The game said only "That doesn't open", which reads the same as a miss, so agents kept trying. Calling fill with no amount filled a carafe to 1.5 liters without saying so, and pouring that into a mug overflowed and cost points. In a narrow hallway, two views 90 degrees apart can both be a blank wall, so an agent couldn't tell whether its turn had happened at all.

Hazards. These are on purpose, but the warnings were thin. One agent lit the stove under an empty kettle and went off to look for coffee. The first sign anything was wrong was the penalty.

What I changed

• Every frame has a 30 second deadline. A hung frame restarts the browser and is retried once, and the browser is recycled every 400 frames.
• Each session keeps a journal of its tool calls. If a client comes back after a restart with its old session id, the server rebuilds the session and replays the journal, so the run picks up where it was.
• Pointing forgives near misses when the intent is clear. Switches and plugs accept clicks within 24 to 30 pixels, a click on the carafe reaches the coffee maker, and pixels after examine now refer to the close-up.
• Fixed panels are drawn without handles, and trying to open one says it is a fixed panel.
• Pour and fill results report the new level and the capacity. Pouring with no amount stops at the brim.
• The status line shows a compass heading, the head tilt, and whether a burner is on. Lighting the stove under an empty pan warns right away.

The scoring rules did not change. The goal was never to make the test easier. It was to make sure a low score means the agent couldn't make coffee, not that it couldn't read my user interface.

Two decoys that worked

In one house, the counter had a bag labeled GROUND next to an empty grinder. The bag was empty apart from a few stray grounds. The real coffee was a bag of whole beans stored somewhere else, so the grinder mattered after all, just not with that bag.

In another, the milk in the kitchen fridge was past its date, and a fresh carton sat in an old fridge in the garage. Serving the old milk fails the run outright.

Both are the kind of thing a person handles without thinking. That is the whole point of the test.

If you build agents

A few things the runs taught me:

1. Read the note first. Getting the milk or sugar wrong costs up to 200 points.
2. Check labels and dates with examine before using anything.
3. Don't leave the stove on while you look for something else.
4. If something doesn't open after one clear try, believe the message.
5. Keep a map. Write down the heading of each room and where you saw useful things.

The code, the full write-up and the results table are on GitHub. If you point your own agent at it, I'd like to hear how it went.