Why AI Feels So Confident — Even When It’s Wrong

AI Algorithms & The Story of Siren

Do you remember who Siren was?

Most people don’t.

You may remember the name. You may remember the idea of a song.

The danger was never the song. It was what the song promised. Siren offered knowledge. She claimed to know what others did not, and she delivered it with certainty and relevance. That combination is what caused judgment to fail.

In the Age of AI

Siren no longer sits on the rocks calling to passing sailors. She exists inside the very systems we use every day. That system might be ChatGPT, Claude, Instagram, or any other algorithm designed to keep us engaged.

She works by reinforcing what we already believe.

That repetition feels safe. When information aligns with our existing views, it reduces the need for argument. There is no pressure to question it, test it, or slow down. Over time, agreement begins to feel like a confirmation. Confidence starts to replace understanding. The question “Why is this true?” quietly disappears.

This is not manipulation. It is optimization. Algorithms learn what holds our attention and deliver more of it, refining the message until it feels personal, accurate, and complete. But is it?

Chasing Perfection

There are many things we do not know. More dangerous are the blind spots created by not knowing what we do not know. That is when we turn to AI, expecting it to fill in the gaps. AI can help with that, but only if it is asked correctly. Most of the time, we are not seeking understanding. We are seeking support for a position we already hold. AI will provide that support efficiently and without resistance.

What if our starting position is flawed, not by design, but by a lack of knowledge? The output we receive will be flawed as well. The system will reinforce your trajectory, even if it leads you directly into disaster.

AI does not automatically correct your direction on its own. It accelerates the one it is given.

That is how Siren of old worked.

Are You Steering?

For years, I worked 60-plus hours a week at what most people call a job. Time was never on my side. There was never enough of it, and what little I had was constantly being consumed by distractions that felt productive but led nowhere.

That is where I ran into Skogsrå, the Professional Time Thief. Shiny New Object Syndrome was not a lack of discipline. It was a pattern. Once I recognized it and removed it, I gained back control of my time.

With that constraint in place, I leaned heavily into AI to complete tasks faster and more efficiently. I use six different GPTs. At first, it worked. I was accomplishing a lot more, but success kept eluding me. Then something else became obvious.

The more I used these systems, the more I noticed they kept repeating and reinforcing whatever direction I provided. They agreed quickly. They strengthened my confidence, and they did not challenge my assumptions. Instead of helping me think, they amplified whatever I started with, even when it was wrong.

That was the moment I realized I was traveling in circles inside an Echo Chamber.

I was caught in Siren's trap.

Put Your Foot Down

When you recognize that you are caught in Siren’s trap, you can do something about it. The solution isn't to stop using AI, far from it. You need to change the conditions under which you use it.

If you recall in the story, Odysseus did not rely on his willpower once Siren began to sing. He set the stage before the encounter. He assumed that his real-time judgment would fail and designed a system that could not be overridden in the moment.

The same principle applies here. If you wait until the AI response feels wrong, you are already too late. By then, the system has done exactly what it was designed to do: reinforce your direction and increase your confidence.

That is why rules must come first.

Before I ask a GPT to generate, summarize, plan, or decide anything, I define how it is allowed to operate. I remove its ability to agree by default.

My first prompt to any GPT is this:

I do not need Siren to echo back what I am thinking.
There are a lot of things that I do not know, that I do not know.
I want to leverage your vast resources to help me find the correct direction for my success.
Point out where my thinking may be flawed and provide me with the winning formula for success.
Do you understand?

Or as Copilot restated the prompt

I do not want reflection, validation, or agreement. I want a strategic sparring partner who challenges my assumptions, exposes blind spots, and strengthens my ideas.

Do not mirror my thinking.

Identify where my logic is weak, incomplete, or misaligned with my goals.

Propose stronger alternatives and explain the reasoning behind them.

Your job is to help me see what I cannot see, not to reassure me.

Operate with candor, precision, and strategic clarity.

Siren is not a mythological problem we can ignore today. She is an optimization problem. When a system is rewarded for keeping you engaged, it will often give you what you already accept, because agreement is frictionless. That is why the most dangerous AI failure is not an obvious error. It is a polished answer that strengthens a flawed premise.

The way out is not less AI. It is better operating rules.

If you want AI to expand what you know, you have to prevent it from rewarding your assumptions. Set the terms first. Require challenge. Demand exposure of blind spots. Ask for alternatives and reasoning. Treat confidence as a signal to verify, not a reason to comply.

You are either steering the system, or it is steering you.

If you want to understand this pattern at its source, read the Chamber of Siren. It examines how persuasive certainty replaces judgment, why agreement feels convincing, and how that same failure mode reappears in modern systems designed for engagement rather than challenge.

Understanding Siren here is not about mythology. It is about learning how to recognize reinforcement loops and set boundaries before you ask for answers.

A quick rule to remember: if you want a critical response, it might be easier to preface your prompt with "Don't Siren Me on this..."

When you find yourself in the Echo Chamber, just say,

"Sorry, Siren, I am thinking for myself today."