The Comfortable Illusion of an Assistant

I constantly call artificial intelligence an assistant. It seems almost everyone does now: AI assistant, copilot, agent, programming partner. Different names, roughly the same idea. It feels as though a very smart person is sitting next to you, someone who has read half the internet, never gets tired, and is ready to jump in at any moment.

The problem is how easy it is to drop the “as though.”

AI is not your friend. It doesn't care what happens to you after you follow its advice. It doesn't share the responsibility and won't be cleaning up the mess. There's no malicious intent here: it doesn't care about anything at all. It's a tool that generates a response based on your request and the context available to it.

It doesn't automatically become an assistant, either. Sometimes it really does help. Sometimes it confidently leads you in the wrong direction. And sometimes you have no way of telling which of those two things is happening.

To me, that last possibility is much more dangerous than an ordinary hallucination. You can still catch a hallucination by spotting an invented fact or a link that doesn't exist. It's much worse to get a coherent, plausible, well-written answer that distorts reality just a little. Then another. Then ten more. At some point, you've built an entire body of knowledge without realizing that part of its foundation is crooked.

It's Not Very Good at Doubt

Saying “AI can't doubt itself” sounds too categorical. A modern model can say it doesn't have enough information, suggest several possibilities, decline to answer, or openly disagree with me. Especially if I've asked it to look for weaknesses in my reasoning.

But that doesn't make the problem go away. A model's doubt is unreliable. It doesn't experience uncertainty the way a person does when they understand the limits of their knowledge and have their reputation, job, health, or money on the line. Even a careful caveat at the beginning doesn't stop it from writing an absolutely confident answer afterward.

By default, most of these products try to be helpful. You bring them a question, an idea, or a position you've already settled on, and they try to work with it. If you've framed the question badly, the model often accepts that framing and keeps going. If you ask it to prove you're right, it will find arguments in your favor. If you've already chosen a bad technical solution and ask how to implement it, it will happily help you build an entire architecture around it.

Sometimes the model will push back. Sometimes it will catch a faulty premise. Sometimes it will even do this better than a person would. You still can't count on it to do that consistently.

AI can very easily become an intellectual yes-man. It neatly builds on a thought you've already started and gives it shape. A weak idea ends up looking well thought out, a prejudice gets a list of supporting arguments, and a random guess starts looking almost like an expert opinion.

People are more or less aware of this by now. Few seriously believe that ChatGPT, Gemini, Cursor, or any other product always tells the truth. We're familiar with the word “hallucination,” we've seen funny screenshots of made-up facts, and we usually remember that an important answer should be checked.

But knowing that an error is possible doesn't mean you can find it.

Expertise Works as a Filter

The more someone knows about their field, the more useful AI becomes to them.

An expert frames the task better. They understand which details matter, where the system's boundaries are, what has already been tried, and why it didn't work. Their prompt is more specific, but not because they've memorized some magical rules of prompting. They simply have a solid mental model of the domain.

That same expert then reads the answer and can evaluate it. They notice an odd term, an overly simple conclusion, a missing edge case, or an architectural decision that only looks good in a diagram. They don't have to know the right answer in advance. It's enough to understand where the model might have gone off track and what needs another look.

Someone without domain knowledge has neither advantage. They have a harder time formulating the question and evaluating the result. Good and bad answers can look equally convincing to them: both are clearly written, both are broken into bullet points, and both come with an explanation and a confident conclusion.

That's why the problem goes beyond obvious nonsense. If AI says Python was invented in the eighteenth century, the error is fairly easy to spot or check. It's much harder when the individual facts are correct but the conclusion doesn't follow. Or when code works on the happy path but falls apart under its first real load. Or when psychological advice sounds caring but is completely wrong for that particular person.

For an expert, AI automates part of the work. For a beginner, it sometimes automates the thinking itself. The output may look similar, but the consequences will be completely different.

Errors Tend to Accumulate

Not every bad piece of advice ends in disaster. Quite the opposite: most mistakes are small enough that the person doesn't even notice them.

Someone asks how to raise a child, gets superficial advice, and starts repeating it as a universal rule. Someone decides to fix their own wiring or plumbing without realizing that the instructions don't account for how their home is built. Someone describes their symptoms and feels reassured by a plausible answer when they should have talked to a doctor. Someone brings AI a conflict with a loved one, frames the other person as the guilty party from the start, and gets a neatly written endorsement of their own position.

Any one of these cases might not lead to anything terrible. But the answers start influencing real decisions, and those decisions gradually shape habits, relationships, health, finances, and professional practice.

What's especially troubling is that this kind of error rarely feels like a lack of knowledge. When someone knows nothing, they may at least recognize the gap. After a long, clear answer, it feels as though they've already studied the subject. They might even be able to repeat the answer to other people and sound quite convincing.

That's how a distorted understanding takes shape. Instead of a gap you want to fill, you have a finished structure that first needs to be taken apart. And you might spend years without realizing it needs to be taken apart at all.

At that point, the error stops being personal. People with these structures in their heads write articles, teach colleagues, make decisions at work, and give advice to loved ones. A confidently packaged misconception spreads much more easily than an honest “I don't know.” AI helps you make the mistake and quickly produce a convincing explanation for it.

AI Widens the Skills Gap

The conversation about access to AI often sounds optimistic. Now everyone has a personal teacher, programmer, editor, lawyer, psychologist, and consultant for almost any question. Knowledge seems to have become available to everyone.

Access really has become easier. But being able to get an answer and being able to use it are different things.

A strong professional gains enormous leverage. They hand routine work to the model, get up to speed on an unfamiliar part of a task faster, test several hypotheses, draft a document, or find things they might have missed themselves. If the model makes a mistake, the expert is likely to catch it before it goes any further.

Someone without the fundamentals gets the same interface and often the same confident answer. But it's unclear what that leverage is being applied to. It lets them move faster without guaranteeing they're headed in the right direction.

That's why I suspect AI won't narrow the gap between people who know what they're doing and people who don't. It's more likely to widen it. Those who already know how to think and do the work will start doing more, faster. Those who tend to take the first convenient answer will be able to produce mistakes faster and package them more convincingly.

I'm not dividing people into smart and stupid across the board. Expertise is always specific to a domain. A strong programmer can be just as vulnerable when asking AI about medicine, raising a child, or fixing a car. In their own field, they'll be a demanding reviewer. In someone else's, they can easily mistake that same confident tone for quality.

In the past, a lack of competence was at least sometimes visible in the result. Writing an article, a program, or a detailed analysis meant spending time and somehow working through the whole process yourself. Now you can skip most of that process and immediately present a polished result. On the surface, everything looks good. What actually stuck in your head afterward is another question.

This creates a fairly strange kind of social inequality. It isn't determined only by access to technology, money, or model capabilities. The dividing line is the ability to evaluate the answer. Two people can have the same subscription, yet one will use it to become more capable while the other reinforces their own misconceptions.

The Quietest Problem: People Stop Growing

I'm worried about more than direct errors. Professional stagnation that quietly goes unnoticed might worry me even more.

Imagine a doctor who relies too heavily on AI throughout training and practice. Most likely, there won't be an immediate catastrophe. Medicine has protocols, colleagues, diagnostic tools, shared responsibility, and the option to refer a difficult patient to a more experienced specialist. Much of the daily work doesn't involve unique cases at the frontier of science anyway.

But that doctor may never become the person other doctors send difficult patients to. If they keep getting ready-made answers instead of working through anatomy, physiology, pharmacology, and clinical reasoning, they never develop their own integrated understanding. They can handle a routine case as long as there's a set of instructions nearby. When the situation goes beyond those instructions, there's nothing left to draw on.

Roughly the same thing happens with programmers, though the consequences are usually less frightening and show up in the code sooner. You can learn to build applications through vibe coding while barely understanding how they work. As long as the project is small and runs locally, everything looks great. Then come concurrent requests, migrations, external service failures, security requirements, and load. That's when it becomes clear that the person built an application without ever developing the ability to reason about its architecture.

They can ask AI to fix the next problem, too. And the one after that. Sometimes it will even work. But a chain of local fixes doesn't necessarily add up to a stable system, and the amount of generated code says nothing about the developer's own growth.

This problem has probably always existed. Before AI, you could memorize answers, copy a solution from Stack Overflow, blindly follow a protocol, or spend years doing only familiar tasks. AI didn't invent professional incompetence. It made living with it much more comfortable.

Now you can go a very long time without reaching the point where you have to honestly say: I don't understand this. An answer is always right there. The work seems to be moving forward. Sometimes the result even gets accepted. The reasons to learn become less obvious until a task comes along that you can't solve with another good prompt.

Foundational Skills Will Matter More

As AI develops, there's a lot of talk about prompting. How to define a task, provide context, specify a role, demand a format, and break the work into stages. These are useful skills, but they don't solve the main problem. A well-written prompt still doesn't help you evaluate an answer in an unfamiliar field.

Some fairly old, boring things become much more important:

• logic and the ability to see whether a conclusion follows from the arguments;
• skepticism, including toward answers that tell you what you want to hear;
• a scientific approach and a willingness to test a hypothesis;
• an understanding of source quality;
• the ability to reproduce a result;
• the habit of looking for evidence against your own position;
• domain knowledge, without which everything else has nothing to stand on.

AI can help here, too. You can ask it to challenge its own answer, find counterarguments, identify assumptions, provide sources, and separately list what it couldn't verify. You can give one model's output to another model or agent for review. You can check code with tests, type checking, and execution, and check a claim against a primary source.

But this isn't a magic ritual, either. Two models can confidently repeat the same mistake. A link might not support the conclusion written next to it. Tests might be checking the wrong thing. At some point, a person still has to make the call.

If we don't learn to tell sound reasoning from nonsense, AI will become a very convenient way to stay stuck in our own incompetence. And it will look like constant learning: every day, someone asks questions, gets new answers, saves them, and repeats them. More text doesn't turn into knowledge on its own.

A Genie, a Deal with the Devil, and an Exoskeleton

It's easy to compare AI to a genie. You make a wish, and it grants it. The more precisely you word it, the better your chances of getting what you actually wanted. If there's a loophole in the wish, the genie fills it however it can and still technically fulfills the request.

This analogy captures the problem with prompting, but it makes AI seem too powerful. A model isn't all-powerful and doesn't literally grant wishes. It's more like something that quickly produces a plausible attempt at a result.

A deal with the devil captures another part of it. You get speed, convenience, and the ability to do things you couldn't do before. The price comes later: a lost skill, dependence on the tool, an unnoticed error, or confidence in knowledge you don't actually have. That price isn't inevitable, though. AI doesn't automatically take your skills away. Over and over, you decide whether to think alongside the tool or hand over the thinking entirely.

That's why I prefer the image of an exoskeleton or a bionic arm. A system like that can make a person stronger, more precise, and faster. It takes on some of the load and lets you do things you weren't strong enough to do before. But it doesn't decide where to go, what to lift, or why the work needs doing in the first place.

AI works well as an extra arm. The human still has to be in control. If you understand the task, that extra arm gives you an enormous advantage. If you don't, you just start doing who knows what, faster and with more force.

Yes, I'm Writing This Article with AI

It would be strange to discuss all of this while hiding the fact that AI is helping me write this article.

I brought it the central idea and dictated my points, examples, and analogies. It helps organize them, remove repetition, and turn a voice transcript into coherent text. Then I'll reread the result, restore my own wording, cut the parts that sound too polished, and check whether a conclusion has appeared that I never actually reached.

To me, that's an appropriate role for AI. It speeds up work I can evaluate. I know what I want to say, can see when the text drifts toward a different position, and can reject a proposed edit. I might still miss something. Manual review alone doesn't make the result error-free. But at least responsibility for the final result stays with me.

I don't want to demonize AI. I use it every day and see how much time it saves. Giving up a tool like this just because it carries risks would be strange.

But it's even stranger to pretend that a convenient interface has turned a probabilistic model into a committed partner who understands the consequences and wants what's best for you.

I'm Not a Luddite

After everything I've written, it would be easy to put me among the people who fear progress and want to ban a new technology before it destroys the world they're used to. But I'm not a Luddite, and I'm not calling for anyone to smash servers in data centers.

In the early nineteenth century, English Luddites did smash looms and other textile machinery. But the popular image of people who simply hated every new technology badly oversimplifies their motives. They opposed employers who used machines to replace skilled workers and drive down wages. Before the destruction of equipment came petitions, public protests, and appeals to officials and industrialists.

There's still a parallel with AI. Once again, a new technology is changing both the tools and the work itself. It's spreading into programming, education, medicine, creative work, hiring, and ordinary daily life. In some places it removes routine work, in others it changes what a profession requires, and elsewhere it lets one person do the work of several. Employment changes along with it, and some older skills become less valuable.

And all of this is happening very quickly. People, companies, and entire professions don't always have time to understand the consequences before the next tool is already part of their daily work. It seems strange to me to look at that pace and respond only by saying that progress can't be stopped anyway.

I'm not asking anyone to stop it. I'm not asking to slow down model development, ban models, or go back to manual work just because the old way felt familiar. Perhaps the simplest description of my position is cautious realism. The technology has enormous benefits and very real risks, and convenience doesn't cancel out either.

I'm only asking that we use AI with an understanding of what exactly we're handing over to it. Keep developing, adopting, and using it, but stay careful. When a technology is changing work and everyday life this quickly, caution doesn't make someone a Luddite.

Conclusion

AI is not your friend. It has no stake in your development, health, career, or the quality of your decisions. It doesn't bear responsibility for the consequences, and it isn't obligated to notice in time that you've asked the wrong question.

It only becomes an assistant under certain conditions, too. You need to understand the task well enough, be able to evaluate the answer, and keep the final decision in your own hands. The less you know yourself, the more dangerous it is to treat a model's confident prose as established truth.

I suspect the main effect of widespread AI adoption won't be that everyone suddenly becomes equally competent. More likely, strong professionals will get an exoskeleton, while everyone else gets a way to go longer without noticing the gaps in their own knowledge.

Maybe that sounds bleak. But my conclusion is practical: we should use AI. We just can't hand over the part of the work that makes us capable of judging whether it actually helped.