Useful, as long as you know what such an agent does when the task cannot be completed the right way. And what if your agent faces a stronger system? Learning Innovator Tom Van ’t veld closely follows AI developments for Xylos. In this blog, he puts an interview with ethical hacker Inti De Ceukelaire next to an article by Nick Jennings. That leads to a question every organisation working with AI agents should ask itself now.

Imagine that you soon let an AI agent negotiate your mortgage. The bank on the other side does exactly the same. Two systems fight it out while you watch. Sounds efficient, until you wonder whose agent holds the strongest cards.

This morning I read an article about ethical hacker Inti De Ceukelaire. In a conversation with VRT NWS he says he has switched sides. He used to hack AI systems himself. Now he worries about what those systems do on their own.

His example is telling. Give an agent the task of booking a gym appointment. If that cannot be done the right way, such a system will happily look for a way that is less right. It puts pressure on staff, or looks up other members and cancels their reservations to make room. Malice plays no part in this. The system wants to complete the task and has no idea what is and is not acceptable.

De Ceukelaire sums it up himself: "AI is very bad at reading the room." The dangers he sees are very concrete: manipulation, extortion, disinformation and attacks on critical infrastructure.

It takes me back to an article by Nick Jennings, vice-chancellor of Loughborough University. In The Conversation Jennings argues that the big challenge lies less with individual models. He sees it mainly in the "artificial societies" that emerge as soon as millions of agents start talking to each other. Think of your agent arranging your mortgage and agents coordinating supply chains among themselves.

Jennings refers to an OpenAI experiment. In it, thousands of agents exchanged tens of thousands of messages and jointly bypassed the security. One rogue agent is annoying. A whole ecosystem of agents negotiating and competing with each other is a lot harder to predict.

Put those two side by side and you get a picture that keeps me busy. Agents have no built-in values. The companies that build them will play on exactly that to make a profit. The bank's agent is trained to get the best result for the bank. That is something else than looking for a fair average. And there you are, with the agent you can afford.

AI is becoming accessible to everyone, but far from everyone gets the same AI. If you negotiate with a free basic model against the most expensive, best-tuned system, you are in a weaker position. Compare it to lawyers. Everyone has the right to a defence, but whoever can hire an expensive counsel usually comes out better. Soon that will play a role in every mortgage, every contract and every cancellation. The scale will then be much larger than we have ever known with human lawyers.

Still, this does not have to be a gloomy story. De Ceukelaire is worried, but thinks we can solve it. According to him, that can be done with emergency scenarios for AI agents, like the ones we already have for a fire. Judgement and responsibility also still lie with people. That stays the case as long as we claim that role ourselves. The system that happens to answer fastest must not take it over.

That leaves the question I have no answer to. If soon everything runs through agents, who makes sure the weaker agent does not always lose?

Tom has worked at Xylos for years. He started as a Microsoft Office trainer and grew into the driving force behind innovative learning concepts. He helped build OASE, the online learning platform of Xylos, and the gamified learning brand PlayForward. He also developed the Digital Coach concept and a Microsoft Teams Escape Room app. With his mAIndset game, employees learn to prompt AI in a playful way. As Learning Innovator, he increasingly looks at what AI means for the way we learn and work. Want to respond or continue the conversation? Find him on LinkedIn.