July 2026
A few weeks ago, a client sent me a business idea that ChatGPT had helped them develop. It was neatly organized, well-written, and sounded surprisingly convincing. There was only one problem: it was a terrible business idea.
Not because ChatGPT had "lied." Not because my client had asked a bad question. The problem was much more subtle than that, and it's something I think every business owner needs to understand before AI becomes as common in your office as Google is today.
The AI had done exactly what it was designed to do. My client had simply assumed it was doing something else. If you're using ChatGPT, Claude, Gemini, Grok, or any of the other modern AI models, this distinction matters more than almost anything else you'll learn about them. Because it changes how you should interpret every answer they give you.
The biggest misconception about AI
Most people imagine AI works something like a really, really smart employee. And the normal process a (quality) human employee goes through is typically:
- You ask them a question.
- They think about it.
- They weigh the evidence.
- They reach a conclusion.
- Then they explain their reasoning and form a plan of action if you agree with their logic.
But with AI, that isn't what's happening.
Modern AI is extraordinarily good at recognizing patterns in language. That's a simplification, but it's close enough for this discussion. It's been trained on an almost incomprehensible amount of writing produced by humans: books, research papers, technical documentation, news articles, websites, programming languages, public conversations, and countless other sources.
When you ask it a question, it isn't opening a filing cabinet to retrieve the answer. And it isn't sitting quietly in thought like your accountant figuring out your taxes, either. It's predicting what sequence of words is most likely to answer your question well, based on everything it has learned about how humans write and communicate.
That sounds like a small distinction, but I promise you it isn't. It's the entire game, and if you're not well informed about the process by which these data centers come to their conclusions in the context of your question, your profile, your phrasing, and its unique weighting based on the model you're currently using, you could wind up chasing a whole colony of white rabbits.
The difference between facts and judgment
If you ask AI what the capital of France is, you can be remarkably confident in the answer. It will get that correct with a near 100% rate of accuracy.
If you ask it to explain a piece of code, summarize a legal document, or help you write a database query, it's usually operating in territory where objective answers exist. That's one of the reasons AI has become such an incredible productivity tool for traditionally white collar professions where the certainty of the answer you give is worth quite a lot.
Now change the question, though.
- "Should I hire another employee?"
- "Is my business idea any good?"
- "What's the best marketing strategy for my company?"
- "Should I homeschool my kids?"
These aren't questions with objective answers waiting to be retrieved. They're questions involving uncertainty, judgment, values, tradeoffs, economics, psychology, and experience. The AI still gives you an answer. That answer may even seem authoritative. And depending on what personal memory it has of your past conversations, the way you ask questions, the answers you tend to accept, it could wind up pretty far afield of the "best" answer, because subjective questions don't really have a "best" answer. Best is nothing more than a composite of your unique perspective earned through lived experience that an AI can only pretend to understand.
Ultimately, the AI can't tell you, at least not clearly enough, how much confidence you should actually place in the conclusion. That's where people get into trouble. Including and especially when using these bots for business purposes.
The confidence problem
One of the most fascinating conversations I've had recently wasn't about coding or websites. It was about philosophy. Specifically, why AI tends to sound equally confident whether it's explaining how to write a database query or offering an opinion on education policy.
The answer isn't that it's trying to deceive you. At least not on purpose. It's that today's AI has no good way of communicating the difference between those two kinds of questions.
Think about it from the AI company's perspective for a minute.
You're OpenAI or Anthropic, and you're competing with every other AI company on Earth. Trillions (not a typo) of dollars are being invested. You believe, perhaps sincerely, that whoever builds the most useful AI first may shape the next century of technology and that kind of power could have either glorious or disastrous outcomes for humanity depending on who "wins".
In that reality, if you're OpenAI (ChatGPT), would you rather your product answer questions like this?
"I'm moderately confident, although reasonable experts disagree for these reasons..."
Or this?
"Here's what I recommend."
The second experience feels dramatically better to the average user. Because most people don't like indefinite answers, especially from subordinates. They feel they're paying you for an answer, and thus you should give it to them. Human employees are pressured every day to set milestones and deliverables they don't actually feel confident in because there is no way to be confident in them when there are too many variables. The same is true of AI workers. The problem is that feeling confident and being well-supported by evidence are not always the same thing.
This is where people misunderstand me
Clients occasionally ask whether they should trust AI. Other times I just come out and warn them if I catch them mentioning something they tell me Claude did for them that was "so cool", but I actually see a bunch of red flags. My answer surprises and likely annoys them. But it's necessary to unpack this honestly.
I trust AI to help me write code every day.
I trust it to summarize technical documentation.
I trust it to spot patterns in data.
I trust it to help me brainstorm ideas I might not have considered.
I do not trust it to come up with its own tasks autonomously and deploy code without first personally testing it.
I do not trust it to decide whether a client's business idea is viable.
I do not trust it to come up with a full marketing strategy that accounts for a client's hyper-specific circumstances.
I do not trust it to tell me who to vote for.
I do not trust it to resolve philosophical questions that humanity has argued about for two thousand years.
And I certainly don't trust it to replace my own judgment. That's not because AI is "bad." It's because those aren't objective questions.
The feature AI desperately needs
During that same conversation, we landed on an idea that I wish every AI company would seriously consider.
Imagine every AI response came with something like this.
Answer Type: Objective fact
Confidence: Very High
Expected Consistency: Nearly identical regardless of who asks.
Now imagine another answer.
Answer Type: Interpretive synthesis
Confidence: Moderate
Expected Consistency: Different well-informed models may reasonably emphasize different evidence.
Suddenly, the user understands something incredibly important.
The AI isn't saying, "This is the truth." It's saying, "Given the available evidence, this is a reasonable synthesis." Those are completely different statements.
Why doesn't AI already do this?
Part of the answer is technical. AI doesn't think the way humans think. It doesn't have a tidy chain of reasoning sitting behind every sentence that it can simply print for you. Researchers are actively working on making AI systems more interpretable, but we're not there yet.
Part of the answer is human nature. People like certainty. We like decisive answers. We like experts who sound like they know what they're talking about. If you've ever noticed that asking the same AI essentially the same question twice can produce noticeably different answers, you've already seen this limitation in action. Small changes in wording change the context, and changing the context changes the answer. You can even see deviations (sometimes pronounced deviations) with prompts that are literally copied and pasted from one window to another. There is absolutely no difference between the prompts but the answers are still different, usually when the question is subjective.
That isn't necessarily a bug. It's a reminder that you're interacting with a language model, not an oracle.
The advice I give every client
AI is already changing my industry, and it's going to change yours too. The businesses that benefit most won't be the ones who blindly trust every answer. They'll be the ones who learn to ask a second question.
Not:
"Is this answer correct?"
But:
"How confident should anyone reasonably be that this answer is correct?"
Those are different questions. The first is about the answer. The second is about the evidence. And if there's one habit I'd encourage every business owner to develop over the next decade, it's that one. Because AI is only going to become more persuasive. Learning when not to be persuaded may turn out to be the most valuable AI skill of all.
AI isn't going anywhere. It is definitely a bubble, and I'll write all about that in a future post, because that's important. But the internet was once a bubble, too. That's what capitalism does, quite naturally and predictably so. When a brand new, novel, exciting technology suddenly emerges, everybody rushes to see if they can get in on the action. Then they realize that the people who had the capital from the previous gold rush are the ones who wind up with most of the benefits of the latest one, save for a few, and they dip out, causing a crash. But ultimately, the new technology survives, just as the internet before it, because it's far too good to ignore or shelve.
"Adapt or die." ~ (Allegedly) Billy Beane, among others.