Stop asking the model to remember arithmetic

Most of what’s in my skill files shouldn’t be there.

About half of what I’d written into them was arithmetic dressed up as guidance, and I was paying a model to remember it on every single run.

Here’s the line that made me look. My watch scraper writes two numbers into every record:

"vph": 21600,
"frequency": 3

One is derived from the other. Frequency in Hertz is beats per hour over 7200, because a beat is a half swing of the balance wheel. 21600 over 7200 is 3, so the record is right.

Nothing in my pipeline knows that. A model wrote both numbers, a second model read them and agreed, and that is two models and roughly 8000 tokens spent on a division I could have written in JavaScript in 1994.

The day it writes 4, both models will nod and the record goes into the database.

That isn’t an edge case in my setup. That’s most of my setup.

Two jobs, and only one of them is a skill

A skill is knowledge the model needs while it’s working. Technique, order of operations, where to look when the obvious place is empty. It fires at generation time, and the model decides whether to use it.

A check is a fact about whether the output came out right. It fires afterwards, and it has no interest in what the model was thinking.

Advice gets ignored, misremembered, or pushed out of a long context. A gate doesn’t.

Almost everything I’d written into skill files was the second kind wearing the clothes of the first.

The test

Here’s the whole idea, and you can poke at it. Change a field and watch the checks decide. No model is involved in anything below.

Load “plausible enum” and look at what happens. Automatic is a perfectly sensible thing to write in a movement field. It’s the word on the dial. It is also not one of the four values my database will accept, so the row fails on insert at 2am with a constraint violation and no context.

A model reading that record sees a reasonable answer. A membership test sees a string that isn’t in a list.

So the rule I use now is a single question, and it sorts every line in a skill file:

Can you write a function that returns true or false, without asking a model?

Frequency equals beats over 7200, yes. Reference matches the brand’s pattern, yes, and the patterns already live in brand-prefixes.json so the check reads the file rather than knowing about Panerai.

“Try the brand’s novelty page before the product page”, no. That’s technique, and it stays where it is.

What it actually looks like

Not much. The whole thing is a list of predicates and a loop.

type Check = { id: string; run: (s: Spec) => string | null };

const checks: Check[] = [
  {
    id: 'frequency',
    run: (s) =>
      s.frequency === s.vph / 7200
        ? null
        : `frequency ${s.frequency} does not match vph ${s.vph}, expected ${s.vph / 7200}`,
  },
  {
    id: 'movement-enum',
    run: (s) =>
      MOVEMENT_ENUM.includes(s.movementType)
        ? null
        : `movementType "${s.movementType}" is not one of ${MOVEMENT_ENUM.join(', ')}`,
  },
  {
    id: 'reference-prefix',
    run: (s) =>
      prefixes[s.brand]?.test(s.reference)
        ? null
        : `reference ${s.reference} does not match the pattern for ${s.brand}`,
  },
];

export function verify(specs: Spec[]) {
  const failures = specs.flatMap((s) =>
    checks.map((c) => ({ ref: s.reference, id: c.id, message: c.run(s) }))
         .filter((r) => r.message)
  );
  return failures;
}

Returning the expected value in the message matters more than it looks. The agent that reads the failure gets told what the number should have been, so it can fix a field instead of re-deriving the rule.

Point it at a batch:

That photo URL is real. It has a ./ in the middle and product_en_file/file/ appears twice, and I would have found out about it when a customer opened a page with a missing image. A HEAD request settles it in 40 milliseconds and no model has to have an opinion.

Then close the door

A checker you have to remember to run is a suggestion, and I have written plenty of scripts I forgot about.

So the import command doesn’t take a results file. It takes a verified results file, and the only thing that produces one is the checker.

const report = verify(specs);
if (report.length > 0) {
  console.error(`${report.length} failing checks, refusing to import`);
  process.exit(1);
}

Four lines, and the unverified file now has no route into the database at all.

The same shape works for anything that can hurt you. Rather than telling an agent to be careful with a destructive operation, give it a command that cannot perform the destructive part, and put that part behind a second command with its own preconditions. A promote step that refuses to run unless a backup already exists on disk beats any amount of careful phrasing.

Keep the numbers out of the checks

There’s a trap here that will make you abandon the whole approach within a month.

If a check hardcodes the value it’s checking, every routine change edits the check. Bump Node from 22 to 24 and you’re editing the rule plus every fixture that mentions 22. Do that twice and you’ll stop writing checks.

So current state lives in data, and only the durable relationship lives in code. The check doesn’t know which Node version is right. It parses the toolchain file, parses the CI workflows, and asserts they agree.

Mine works the same way. The reference check has never heard of Panerai. It reads brand-prefixes.json, which the scraper already updates when it meets a brand it doesn’t recognise. Adding a brand is one line of data and no code at all.

Where it bites

Once the check exists, the interesting question is when it runs. Same code, four possible homes, and the difference is what you throw away when it fires.

I’d assumed CI. It’s the obvious home and it catches humans too. But for this pipeline the better spot is a hook on file write, because it fires the second the results file lands, while the agent still has the product page in its context. Finding out at import time means re-scraping from source. Finding out four seconds later means editing a number.

That’s the whole lifecycle argument. Session start, on-demand skills, hooks, CI, cron. The further down you go, the harder it is to dodge, and the more work is already in flight when it fires. Those two pull against each other, and where you land is the only real decision in any of this.

Don’t go and write forty of these

The tempting move now is a weekend of writing checks. Don’t.

Every check is production code. It needs tests, it breaks, and one day it will be wrong in a way that blocks legitimate work at 11pm while you’re trying to ship. You’re taking on a second codebase whose only job is policing the first one.

The path is comment, then doc, then check, and most things should stop at doc. Write the check the second time a bad record reaches your database, not in anticipation of the first.

Mine is a dozen checks and I expect it to stay that size. Scraping is still prose, because scraping is a genuine judgement call and 40 brands each hide their specs somewhere different. The arithmetic was never a judgement call.

The part I didn’t expect is how much came out of the prompt. Every rule that turned into a check left the skill file, and what’s left is genuinely about judgement: which page to try first, what to do when a brand buries its specs in a PDF, when to give up and read the calibre database instead.

That’s the version of a skill file I’d defend. The rest of what was in mine was a check I hadn’t written yet.