Don’t let AI automate the wrong things
AI is a force multiplier. If your strategy is unclear or your processes are inconsistent, AI will only make it worse. Here's how to prevent that.
👋 Hi! It’s Jennifer. Welcome to my fortnightly newsletter where I challenge conventional business wisdom to help you become a different kind of thinker, leader and operator.
Did a friend forward this to you? Subscribe below to get my emails directly in your inbox.
Read time: 9 min.
In 2022, Klarna CEO, Sebastian Siemiatkowski was proud of his decision.
He replaced 700 customer service workers with AI chatbots, cutting his workforce nearly in half. The Swedish CEO even bragged about it in interviews. He believed these tools could match, even exceed, human performance.
The media ate it up.
The move looked smart. It was bold, efficient, and forward-thinking.
By 2024, Klarna was hiring humans back.
Customer complaints had surged. User satisfaction dropped. The AI systems that were supposed to solve everything had created new problems no one anticipated.
What went wrong wasn't the technology.
It was the same thing that goes wrong whenever leaders rush to implement powerful tools without first establishing the foundations to make them work.
AI amplifies what already exists
I use AI almost every day.
It helps me dig up research, reshape outlines, and brainstorm ideas. It's excellent at summarizing long documents and translating text.
But here’s the thing.
AI is only as useful as the thinking I do upfront. Whatever I put in such as my assumptions, research, clarity of thought, is what I get back, just in better packaging.
Tobin Leslie, Cofounder and President of Kinetic Solutions Group puts it well:
"AI isn't a silver bullet. It's a force multiplier. It amplifies your output. It doesn't improve your thinking. It doesn't clarify your goals. It doesn't fix your systems. That speed doesn't come with a built-in strategy. That part is still yours. And that's what most people skip.”
Here’s how that might play out in your company:
Marketing teams use AI to create content without a clear positioning and messaging strategy. The content ends up generic and doesn’t resonate. It fails to attract the right type of customers.
Customer service teams deploy chatbots without first streamlining their processes or updating knowledge bases. Customers get frustrated with unhelpful responses, and satisfaction scores plummet.
Sales teams use AI to personalize outreach emails based on customer data. But if those emails don't address real customer pain points, conversion rates stay flat.
It’s classic garbage in, garbage out.
If your processes are unclear —> AI makes them more unclear.
If your strategy is confused —> AI makes it more confused.
If your systems are inconsistent —> AI makes them more inconsistent.
Understanding why this amplification happens requires looking at what AI can and cannot do at this time.
AI struggles with complex reasoning
Apple recently ran a study called The Illusion of Thinking to test AI’s reasoning ability.
Instead of focusing on whether the final answer was right (like most benchmarks do), Apple wanted to see how the model got there. Was the thinking sound? Did more steps lead to better answers, or just more confusion?
To find out, they designed puzzle-style environments like Tower of Hanoi and River Crossing. These let them control the difficulty and watch how the AI handled each step along the way.
The results were mixed.
At moderate complexity, the models performed well, better than traditional language models.
But as the problems got harder, the systems broke down. They wasted resources on inefficient reasoning and often followed incoherent logic.
The takeaway?
Even the most advanced models can’t handle truly difficult reasoning. And more “thinking” doesn’t always mean better thinking.
So if AI can’t reason well on its own, how well can it operate without guidance?
AI still needs supervising
In April 2025, Carnegie Mellon ran an AI workplace study to explore a key question: Can AI agents manage real-world business tasks in team settings without constant human oversight?
While most AI today supports individual tasks like writing or summarizing, the researchers wanted to see if agents could act more like employees.
To do this, they created a realistic office simulation where ambiguity, miscommunication, and coordination challenges naturally arise.
Human “managers” were placed over these AI teams to understand what kind of supervision was still needed.
Ironically titled “Your next assignment at work: babysitting AI,” the study showed just how far we are from autonomous AI teamwork.
The agents frequently failed at basic tasks, made inconsistent decisions, waited indefinitely for clarification, and miscommunicated, ultimately becoming more of a burden than a help.
The experiment exposed the current limitations of agent-based AI and showed that AI still needs supervising.
Beyond supervision challenges, there's another fundamental limitation that affects everyday AI use.
AI confidently makes things up
These fabrications are called hallucinations, which are made-up responses that sound confident and plausible but are factually wrong.
You may have already noticed this in your day-to-day use.
A recent Wall Street Journal test illustrated this perfectly.
A reporter asked several AI systems who he was married to, which was a question with no public answer. Each system gave a different, incorrect response, including names of real people the reporter had never met.
None hesitated.
They confidently guessed.
How does this happen?
As Futurism explains, language models are trained to predict the most likely next word or phrase. They’re not rewarded for saying “I don’t know.” So instead, they guess because statistically, filling in a blank is better than leaving it empty.
These hallucinations aren’t rare edge cases. They can crop up in everyday use. Especially when AI is asked about niche, outdated, or ambiguous topics. That’s especially a problem when models are used in contexts like medicine, law, or journalism.
AI speaks with confidence. It doesn’t even know when it’s wrong.
And to the unaware user, there’s no reason to question it.
What happens when the foundation isn’t there
In February 2025, Tromsø municipality in northern Norway published a proposal to restructure its schools and kindergartens.
The report included a knowledge base with research references to support the recommendations. Shortly after publication, people discovered that some of the references were fake.
Although they looked legitimate, they didn't actually exist. They had been generated by an AI tool and added to the report without being verified.
When it was discovered, the municipality admitted the mistake, stopped the consultation process, and asked PwC to investigate what went wrong.
PWC found that the municipality had no strategy for using AI, no clear guidelines, no employee training, and no system for quality control. Some of the staff had even used personal ChatGPT accounts for work tasks, meaning they had no control over what sensitive information was being shared with external AI systems.
The investigation revealed what many organizations discover too late - AI doesn't fix poor foundations. It exposes them.
In Tromsø's case, the core conclusion of their proposal remained valid. But the way they used AI, including their failure to catch obvious errors, could have raised questions about their competence and judgment.
For a government organization, that's not a risk you can afford to take. And for private companies, perhaps this lesson is something that we can take with us too.
Trust is hard to come by.
Once you lose it, it’s a long road to getting it back.
So what can we do?
Get our house in order first
Using AI for personal productivity is different from deploying it across an organization.
When there are no guidelines or strategy in place, you could risk everything going “Helt Texas” (go completely haywire), as the Norwegians like to say.
According to a global survey by GoTo and Workplace Intelligence, only 45% of corporate IT managers have formal AI policies. Of those, 56% cited security concerns and integration challenges as main barriers.
The PWC investigation into what went wrong revealed key takeaways that any organization could consider before implementing AI at scale.
Basically, it’s a governance model:
Develop a clear AI strategy. You need policies defining when and how AI should be used, which tools are approved, and what the limitations are. Without this, employees make it up as they go along.
Train employees on both capabilities and limitations. People need to understand what AI can and cannot do, how to spot hallucinations and errors, and when human judgment is still required.
Establish quality control processes. Every AI output needs human verification, especially for high-stakes content. You need people who understand your standards well enough to catch when AI gets things wrong.
Define clear roles and accountability. When something goes wrong - and it will - someone needs to own the fix. There must be clear responsibility for AI decisions and outcomes.
Build security and privacy controls. Using public AI tools with sensitive information creates risks. You need approved tools and protocols for data handling.
Document what good work actually looks like. If you can't clearly define your standards for quality research, customer service, or content, AI will just automate inconsistency.
The goal isn't to make this a massive strategic process that takes ages to implement. But putting some thinking in ahead of time can prevent different departments from doing different things, all for the sake of speed.
Once the baseline for the company’s guidelines are created, each department could then create a subset of guidelines for how AI should be used.
This helps to reduce silo thinking.
The real question isn’t about replacing people
Being "AI first" doesn't have to mean replacing people with automation.
It’s a choice.
And it’s one that too many executives are making for the wrong reason. .
The question shouldn't be: "How many people can we replace with AI?"
That kind of thinking shows that people are expendable. It erodes trust. And it sends a clear message that cost savings matter more than people.
As leaders, we should be asking instead:
What do we want to be known for, and how can AI help us deliver more of that?
Where does speed actually matter, and where does quality matter more?
What parts of our work build trust, and need a human to do them well?
AI should enhance human work, not replace human judgment and care.
Our customers and employees can tell when we've replaced something meaningful with automation. They notice when responses feel generic, when service lacks empathy, or when important decisions lack human consideration.
The real value comes when we use it to free up humans to do what only humans can. That’s to think deeply, connect meaningfully, and lead with judgment.
The best companies won’t be defined by how much AI they use.
They’ll be defined by how wisely they use it, and what they refuse to give up in the process.
