The Verification Age

Imagine you're responsible for a factory, and an AI tells you a machine is about to fail.

It recommends shutting down the line.

If it's right, you could prevent damage and keep somebody from getting hurt. If it's wrong, you're stopping work that people are counting on.

There's a very convincing explanation on the screen.

What would you need to know before you trusted it?

This is The Age of Imagineering. I'm Justin Thompson.

In our first episode, we talked about what happens when more people can turn an idea into something real. Last time, we looked at the infrastructure that would make that possible at a much larger scale.

Today, I want to stay with the person looking at that screen.

Because somewhere between an extraordinary demonstration and a system we can depend on, somebody has to answer a difficult question.

How do we know this works?

That's the question behind The Verification Age.

Let's keep our factory imaginary for a moment. Picture a maintenance team that's been looking after the same equipment for years.

They know the sound of a healthy machine. They know which readings tend to wander and which changes deserve attention. They've learned some of that from manuals, and some of it from being there when things went wrong.

Now they're given a system that can watch far more signals than any person could follow at once. Temperature. Vibration. Pressure. The small changes that might come before a failure.

There's real promise in that.

A problem caught early could mean a repair during a quiet shift instead of an emergency. Fewer people called away from dinner. Less pressure to make a difficult decision with a machine already coming apart.

But the team has questions.

Has this system seen equipment like ours? Did anybody test it under the conditions we work in? How often does it raise an alarm when nothing is wrong?

And how often does it stay quiet when something is?

Those are sensible questions. We should be building systems that can answer them.

We often talk about whether an AI is good. Whether it's smart. Whether it did better than another model on a test.

For the person making a decision, that description leaves a lot out.

A system might be very useful with one kind of machine and unreliable with another. It might work well in ordinary conditions and struggle when the load changes or a sensor starts giving bad readings.

Trust needs somewhere specific to stand.

There are two questions worth separating here.

First, has this system demonstrated that it can do the kind of work we're asking it to do?

Second, what evidence supports the particular answer it's giving us now?

A good record gives us a reason to pay attention. We still have to check the decision in front of us.

Think about an experienced mechanic. You value their judgment partly because of what they've handled before. When they tell you a part needs replacing, you can also ask them to show you what they've found.

We need something like that for AI. A record of demonstrated ability, and a way to examine the evidence behind an individual recommendation.

One place to start is with work that has already happened.

Our imaginary factory might have years of equipment readings, maintenance reports, and records of breakdowns.

Suppose we choose a point in those records before a particular failure. We gather the information that was available at that moment. We ask the model what it thinks will happen next.

Then we save its prediction before looking at the outcome.

Did it see the problem coming? How much warning did it give? Was its explanation useful? Was the warning early enough to be useful to the maintenance team?

We can compare the prediction with the record of what actually happened.

That's the idea of historical replay. Give a system a carefully reconstructed decision from the past, and find out how well it handles it.

There is an important catch, though.

An AI may already have encountered that outcome while it was being trained. Hiding the final page when we ask the question doesn't make the answer unknown to the model.

It could appear to predict something it has effectively seen before.

So the test needs more care than simply removing a date or withholding a report. We need cases kept separate from training wherever possible, and a clear account of what we can and can't establish about earlier exposure.

If we can't rule out that exposure, we have to say so. The result gives us weaker evidence than a genuinely unseen case.

And the collection of cases matters just as much.

If we only test on famous breakdowns, a system that always shouts "something's wrong" could look surprisingly useful.

We need the ordinary days, too. The machines that kept running. The odd reading that turned out to mean nothing. The conditions where a cautious person would keep watching rather than stop production.

Then we need a comparison. How does the AI perform against the existing alarm system? Against a simple rule? Against the way the team already makes decisions?

An impressive prediction has to be useful in the work people actually do.

It also needs room for uncertainty. A system should be able to say that the evidence is weak, that a sensor needs checking, or that this situation falls outside what it's been tested on.

That admission could be one of the most valuable things it tells us.

Historical testing can help us decide what deserves a closer look. After that, we still need to watch what happens on genuinely new cases.

For our factory, that might mean running the system alongside the maintenance team without giving it control of the equipment. Record the predictions as they arrive. Compare them with later inspections and outcomes. Learn where the system helps, and where it needs supervision.

Any eventual use would still need the plant's safety checks and the judgment of qualified people.

That takes patience. It also produces something a demonstration on its own can't give us: evidence gathered while the answer was still unknown.

Now think about the work involved in creating those records.

Someone has to establish where the information came from. Which readings were reliable. What was known at the time. Whether a repair changed the outcome. Where the records are incomplete.

A missing failure report doesn't prove that nothing failed.

The value is in the care taken with the evidence.

I think that work could become enormously important as AI improves. A model can change. A carefully maintained record of reality can help us test the next model, and the one after that.

It's easy to get excited about a machine producing an answer. The people doing the quieter work of checking it may end up making the whole thing possible.

And we need a useful way to communicate what they've learned.

Imagine opening a system's record and seeing a clear description of its limits.

Tested on this equipment. Evaluated under these operating conditions. Useful for these kinds of warning. Not enough evidence for that other situation.

You could see how often it missed a problem, how often it raised a false alarm, and how much uncertainty remained in those estimates.

You'd also want to know when it was last checked.

Equipment changes. Work changes. A model gets updated. A test from the past doesn't settle the question forever.

This is why I think verification becomes infrastructure.

We need ways to preserve evidence, repeat tests, inspect decisions, and notice when performance changes. We need those things to keep working after the excitement of launching something has passed.

The need gets sharper when a system can act.

There's a difference in responsibility between a recommendation somebody can consider and a command that shuts down a production line.

As we give AI more ability to do things, we have to be clear about which decisions it can make, what checks come first, and when a person needs to take over.

We should be able to trace what happened afterward, too.

What did the system see? What did it decide? What did somebody approve? What changed in the real world?

Without that record, even a failure can leave us with very little to learn from.

The same questions reach far beyond our imaginary factory. They matter wherever a generated design or a prediction could affect somebody's life.

A simulation can help us explore an idea. Its usefulness depends on how well we've checked it against reality, and how honestly we describe what remains unknown.

As our ability to create grows, the work of earning trust grows with it.

I find that hopeful. There is meaningful work here for people who know a field deeply. People who understand the awkward cases. People who can look at a beautiful result and notice the part that doesn't fit.

That knowledge belongs in what we build.

Let's come back to the person at the screen.

The AI has made its recommendation. This time, there's a record behind it. There's a clear explanation of its limits, a way to check the evidence, and a process for deciding what happens next.

The person still has responsibility. They have more help carrying it.

That's a future worth working toward.

We can imagine more possibilities than ever. Now we have to do the patient work of finding out which ones deserve a place in our lives.

If this gave you something to think about, follow The Age of Imagineering for the next episode.

And for the essays behind the show, subscribe at age of imagineering dot com.

I'll see you next time.