AI Model vs Harness – free checklist (episode 15)

Why it matters

The same AI can feel smart in one app and dumb in another. One way to explain this is with an engine and a car: the AI model is the engine, and what is built around it is the car. Some people use the word "harness" for that car.

The 5-step check

0 of 5 done

Name the two partsEngine = the AI model. Car = everything built around it, like search and memory. Some people call the car the "harness".People use different words for this, such as scaffold.
Ask what it can do besides talkAn AI that can look things up online can do more than one that only writes from what it already learned.Try: Look in the app's help page or settings for search or file features.
Test its memoryTell it a small fact, chat a bit, then ask for it back. Apps handle this differently.Try: "My cat is called Pip." Later: "What's my cat's name?"
Compare fairlyIf two apps feel very different, the engine may not be the reason. The car around it can change the result.Hard to know for sure from the outside.
Still check what mattersA smooth car can still carry a wrong answer. For anything important, check it with a source you trust.General good practice.

Tap a step when you have done it. Your ticks stay on this device.

Rule of thumb

Same engine, different car, different ride.

Sources: Anthropic Engineering, "Demystifying evals for AI agents" (anthropic.com/engineering/demystifying-evals-for-ai-agents) defines an agent harness as the system that enables a model to act as an agent, and says evaluating an agent means evaluating the harness and the model working together. The engine and car picture is our own analogy. Steps 2 to 5 are general good practice.

Episode 15: Judge the whole car, not just the engine

Why it matters

The 5-step check

The AI Words Map

More like this

A new free sheet every week