Blog
Human in the loop: why every AI automation needs a checkpoint
· 4 min read
An AI agent reads incoming requests, fills in the right form and immediately drafts a reply. A request for a quote, a job application, a refund request. At first it saves a noticeable amount of time. Six months later it turns out that for a certain type of request the agent systematically made the same judgement, and nobody noticed, because nobody was really looking any more. The agent worked, just not as intended.
That is the problem with "it works": an AI system that runs smoothly does not prove that it makes the right decision. Without a checkpoint you only notice a systematic error once the damage has been done. With a checkpoint you notice it at the first deviation.
Confirming is not oversight
Many businesses do build in a human step, but that step is often a button labelled "approve", while the human never sees the underlying data. This is called automation bias: the tendency to accept AI output because the system is usually right. The EU AI Act explicitly names this as a risk that people overseeing AI systems must be able to resist. Article 14 of the act states that people who oversee an AI system must be able to understand the system's capabilities and limitations, monitor its operation for anomalies, and disregard, reverse or override the output, or stop the system, when necessary. That is a fundamentally different role from clicking a button the AI puts in front of you, as you can read in the official text of Article 14.
That article is a legal obligation for AI systems classified as high risk. Think of systems that select or assess job applicants, that make decisions about promotion or dismissal, that assess someone's creditworthiness, or that set the price of life or health insurance. The exact descriptions are in Annex III of the AI Act. If you recognise one of those applications in your own use of AI, a checkpoint is no longer optional advice but a legal requirement. If you use AI for something else, such as customer communication or internal process automation, the obligation does not apply automatically, but the underlying problem is identical: a system that makes mistakes without anyone noticing.
A checkpoint is more than an approve button
A checkpoint that genuinely adds something does three things that an approve button does not.
First, it asks for a measured outcome, not a confirmation. AI tools readily confirm that something works without actually having tested it: an agent that says a request has been "processed correctly" has judged that itself, using the same assumptions that led to the error if there was one. Instead, ask for a concrete outcome you can check yourself: which amount, which date, which reason for rejection, and test it against the source data on a sample basis.
Second, it gives the reviewer real insight, not just the conclusion. Someone who only sees "approve or reject" cannot assess more than the agent already did. Someone who sees the underlying data on which the decision was based can recognise a deviation that the agent itself does not recognise as one.
Third, an irreversible action always gets a separate confirmation step, apart from the rest of the process. Refunding money, deleting an account, issuing a certificate definitively: actions like these deserve an explicit stop, precisely because a misclick or a wrong judgement cannot be undone there. That principle applies to people just as much as to AI agents, but with an AI agent that handles hundreds of cases in seconds, the scale of an error is immediately greater.
Anyone who builds this in practice also discovers that assessing the result has to be a separate step, carried out by someone other than the person who set up the system. A second pair of eyes often catches what the builder no longer sees, precisely because the builder now knows what was intended and unconsciously looks for confirmation rather than deviation.
Where to put the checkpoint
Not every step in an automated process needs the same amount of oversight. An agent that categorises an email as "invoice" or "complaint" has little impact if it gets it wrong: someone will spot it at the next step. An agent that then independently carries out a refund or rejects an applicant is a different matter. The question that counts is not "does the AI do this well enough", but "what happens if this goes wrong once, and who will notice". Put the checkpoint where the answer to that second question is "nobody, until it is too late".
What does this mean for your business?
Four steps to get started.
First, map out which steps in your AI-automated processes are irreversible or have an effect outside your organisation, such as a rejection, a payment or a certificate. Those need an explicit checkpoint; the rest can be lighter.
Always show the person doing the checking the source data, not just the AI's conclusion. An "approve" button without the underlying information is not a check, it is a signature under someone else's judgement.
Consistently ask for a measurable outcome rather than a confirmation, and test it on a sample basis. That way you discover a systematic deviation at the tenth request, not the thousandth.
Check whether any of your AI applications fall into the high-risk categories of the AI Act, such as recruitment, staff assessment or credit assessment. If so, a checkpoint under Article 14 is an obligation, not a choice.
Would you like to know where the checkpoint belongs in your own processes, or how to set up AI agents so that people and system strengthen each other instead of one blindly trusting the other? Get in touch with us; we are happy to think along based on what we have built ourselves.