Into the Wild: Who Decides What AI Should Do?

Welcome to the AI Wild West

Europe is making a serious attempt to bring greater structure to this landscape through the EU AI Act and other emerging regulations. That is important, particularly as organizations try to understand what responsible AI use should look like in practice.

But regulation can only take us so far. It can define prohibited practices, identify high-risk use cases and set expectations around controls, transparency and accountability. What it cannot do is determine the appropriate level of authority an AI system should have within a specific organization.

Should a system recommend an action, approve it or execute it? How much influence should AI have over a hiring decision, a financial assessment, a security investigation or access to an essential service?

These decisions ultimately belong to the organization. And they need to be made deliberately.

The problem isn’t only the algorithm

Organizations need to look beyond individual AI tools and understand their broader AI decision landscape.

Where is AI already being used? What data can it access? Which decisions can it influence? What happens when its recommendation is wrong? And when human involvement is required, does that person actually have the authority and context needed to challenge the system?

The idea of a “human in the loop” can sound reassuring, but human involvement is only meaningful when that person has a genuine ability to question, override or stop the system. If their practical role is simply to approve an AI-generated recommendation, the organization may have created the appearance of oversight rather than meaningful oversight.

As AI begins to influence more consequential decisions, this becomes a cybersecurity and enterprise risk issue as much as a technology issue.

Protecting the system is no longer simply about protecting the model, the data or the infrastructure. It is also about protecting the integrity of the decision itself.

Organizations need to understand who can manipulate the data the system relies on, who can change its instructions, permissions or thresholds, whether changes in behavior would be detected, and whether an outcome can ultimately be explained, challenged and reversed.

Organizations need to define where AI belongs, what it should be allowed to influence and, most importantly, where its authority should stop.

That means asking some relatively straightforward but important questions: 

• What happens if the system is wrong?
•  What happens if it is manipulated?
•  How much authority are we actually willing to give it?
•  And, who remains accountable when an AI-supported decision leads to the wrong outcome?

These questions are not barriers to innovation. They are part of what makes responsible innovation possible.

The objective is not to keep AI outside the organization. It is to make sure that adoption does not gradually lead to more decision-making authority being delegated than anyone consciously intended.

Capability does not equal authority

AI will continue to become more capable. That is an easy prediction.

The harder one is whether organizations will become equally good at deciding where that capability should stop.

The real governance challenge is not only managing AI risk, It is managing the transition from capability to authority.

We do not need to wait for regulation to answer every question, and we certainly do not need to stop using AI. 

But organizations do need to make conscious decisions about where AI participates, where humans remain accountable and where the line should not be crossed.

The next phase of AI governance should therefore be about more than compliance or technical controls. It will also be about decision rights and accountability. 

That is the conversation we should be having now, before AI makes the decision for us.