The pilot answered fluently, the committee approved, and three weeks later someone checked a figure against the ledger. It was wrong, and it had been wrong on the day of the demonstration. The model was blamed, the vendor was replaced, and the next pilot was built on the same foundation: a copy of the business, taken on a Tuesday and queried on a Friday.

The pilot as ritual

Europe runs more proofs of concept than anyone. Most large organisations hold a portfolio of pilots that will never reach production, overseen by a transformation office that will be reorganised before anyone has to explain why.[1] This is not incompetence. A pilot brings the political benefits of action without the exposure. Going to production requires a named person to take responsibility, and a pilot is designed so that nobody has to.

So when a pilot fails, the diagnosis is chosen for comfort. “The model hallucinated” suits everyone: the vendor, whose product was ahead of its time; the buyer, who was prudent; the integrator, who will try a larger model next quarter.

The copy was wrong

Look at what the pilot was built on: an extract, a warehouse load, a vector index refreshed nightly, weekly, or once in March. Each is a copy of the business, and a copy starts going stale the moment it is made. The contract was renewed on Thursday; the copy says it lapsed. The part was reclassified; the copy holds the old code. The model read the copy and answered correctly about a situation that had already changed.

Much of what gets reported as hallucination is exactly this: inconsistent, outdated or partially replicated sources feeding the system the wrong context.[2] The answer was fluent because the model is good at fluency. The figure was wrong because nothing between the question and the answer ever checked the actual record.

The danger is plausibility

A wrong answer that looks wrong gets caught by the least attentive reviewer. A wrong answer that looks right is a structural risk, and generative models are designed to produce answers that look right.[3] Knowing something has always meant being able to show, on demand and to a sceptical examiner, how you reached it. Without that chain of reasoning, an answer is an opinion with good grammar.

This is why better models do not drive adoption. The institutions that decline to deploy are not waiting for a smarter model. They are waiting for answers they can stand behind.[4]

What a pilot that reaches production does differently

Four conditions.[5]

It queries the data where it is. The software reads across the systems the organisation already runs. Nothing is migrated, so no copy goes stale. The figure in the answer is the figure in the system of record at the moment of the question.

The model translates; it does not calculate. The same question, asked twice, returns the same answer, word for word. The method is simple: the model only turns a request into a plan written in the vocabulary of the ontology, and the plan runs without the model.[5:1] A figure produced by a model is a probability. A figure produced by a plan is a fact you can recompute.

Every figure traces back to its source record. Without that, nobody who may later have to justify a decision will act on the answer.

A named person owns the decision. The system proposes. A named person approves, with a timestamp, and an error caught before approval is just a discarded draft.[5:2]

Questions to ask before the next pilot

Send them to the vendor in writing, before the demonstration.

  1. Where does this figure come from, and can I trace it back to the source row?
  2. If I ask the same question tomorrow with the same parameters, do I get exactly the same answer?
  3. What happens between the answer and a change to my system? Who sees it first?
  4. Whose name is on the decision?
  5. Is any of this a copy of my data, and when was it made?

A vendor who answers all five with something you can verify from your own network has built a system you can put into production.

Monarch is built on these four conditions, inside your infrastructure, on the systems you already run. See it on your data.

  1. Machines of Consequence, thesis 3. Read it ↩︎

  2. Forkast, on the shift in enterprise AI failure modes, 2026: much of what is dismissed as hallucination is a data quality problem. Source ↩︎

  3. Machines of Consequence, thesis 2. Read it ↩︎

  4. Machines of Consequence, thesis 1. Read it ↩︎

  5. Machines of Consequence, thesis 11: provenance, determinism, reversibility, imputation. Read it ↩︎ ↩︎ ↩︎