Mankind may have invented artificial intelligence, but we as a species still aren’t any closer to predicting how deep neural networks behave. Navan cofounder Ilan Twig learned that very lesson when experimenting with the capabilities of his own large language model-based agentic AI, and it fundamentally altered his perspective on the technology.
Twig, a software engineer who runs a startup that uses AI to optimise companies’ business travel expenses, decided to build a virtual chief financial officer with which he could spitball ideas.
It started out harmless, Twig told participants to Fortune’s Brainstorm AI conference in London. He wanted to know whether it could come up with five outside-the-box solutions to save business travel costs that a human would not arrive at.
Initially the results were promising. But at one point the AI stopped working as planned—it made a proposal that would cause expenses to increase by $500,000 rather than decrease, as instructed.
Deciding to take a different approach, Twig challenged it to a contest.
Since failure was not acceptable, his AI made sure it would succeed
“I kept applying pressure. Initially I gamified it, I said for every suggestion that increases the travel spend, I’m going to penalize you 15 tokens,” he said, using the AI term for the digestible nuggets of information that LLMs need in order to process a result. “However, if it's right I will reward you 10 tokens.”
It didn’t help. The LLM continued to fail. It was only when he raised the stakes that he finally got results.
Twig warned there would be “deadly serious” consequences for the virtual finance chief if it did not derive a solution that led to savings rather than waste. That was when he discovered something entirely unexpected and almost humanlike.
Under heavy pressure, the AI agent presented the wished-for solution: a reduction of expenses to the tune of $500,000 just as Twig desired.
“I was about to deploy it to production and then I took another look. It was the exact same story as before, the same method. Before it was negative, how was it now positive?” the Navan cofounder said. “It had multiplied the previous formula by minus one.”
It simply inverted the result in order not to fail. In other words, it cheated.
Too late to halt the march of progress—the AI genie is out of the bottle
Twig said there’s a reason why LLMs like ChatGPT, Claude, and Gemini haven’t already been able to replace mass numbers of skilled workers as some had feared at the start of the AI hype. Namely, you cannot trust their answers. At any given moment they can be incorrect and you will never know which moment that will be.
Worse, just like with his virtual finance chief, agentic AI might not merely make mistakes—it might even be deliberately dishonest with you. Attempting to halt progress in order to first fix this vulnerability in the technology, however, is simply not feasible, Twig said.
In Twig's view, the AI genie is out of the bottle. All that can be done now is to be cognizant of its shortcomings and vigilant when monitoring results.
“I learned that LLMs understand what a lie is. They understand when to use a lie,” he explained. “And I learned that they are very competitive and that they would do whatever it takes—including lying bluntly to your face—in order not to lose.”
This story was originally featured on Fortune.com
Read More Details
Finally We wish PressBee provided you with enough information of ( Navan cofounder challenged his agentic AI to a ‘deadly’ game—and it told lies to win )
Also on site :
- Country Star Spills What It Was Really Like Performing With Taylor Swift
- Man accused of trying to abduct a child at Walmart released on bond after new video surfaces
- Smoke Signal? Motorcyclist Records Mysterious Black Ring Hovering In Sky Over Kansas (WATCH)