Before you build anything, it helps to know what you're actually talking to. Not a search engine, not a database — something stranger and, once you get it, much more useful.
An AI chatbot is a prediction machine, not a filing cabinet. It doesn't "look things up" — it generates the most plausible next word, over and over, based on patterns it absorbed during training.
It only knows what's currently sitting in front of it — its context window — which is why every new chat starts blank, and why it can state a wrong answer as confidently as a right one.
None of that makes it untrustworthy. It makes it a specific kind of tool, with specific strengths, that works best when you understand the shape of it.
When you type a question into a database, it goes and retrieves a record. When you type a question into an AI chatbot, nothing gets "retrieved" in that sense. The model reads everything in front of it and asks, in effect: given all of this, what's the most likely next word? Then it picks one, adds it to the text, and asks the same question again. And again. A whole reply is just that process running thousands of times in a row, fast enough to look like thought.
This sounds like it should produce nonsense, and for a long time, versions of it did. What changed is scale: the pattern-matching got so good, over so much human writing, that "the most plausible next word" now tracks real reasoning an enormous share of the time. But the mechanism never changed — there is no drawer it opens to check a fact, no fact-checker running underneath. It is prediction, all the way down, which explains almost everything else in this article.
A useful comparison: drafting a reply on autopilot, letting your hands type whatever sounds right based on a thousand similar replies you've written before. That fluent autopilot is close to what's happening — except its "thousand similar replies" is a meaningful fraction of everything ever written down, and it never gets distracted.
Here's the idea that makes everything else click. Picture the AI working at a desk. Whatever's on that desk — your current message, the messages before it in this conversation, any documents you've attached, any instructions you've given it — is the only thing it can see while it's working. That desk is called the context window, and it has a fixed size.
Everything it says is shaped entirely by what's on the desk right now. Mention a deadline early in a long conversation, and if the desk gets crowded enough that the note slides off the edge, it stops factoring in — not because it "forgot," but because that paper is no longer in front of it. This is also why pasting background information at the start of a conversation changes everything that follows: you're loading the desk with what it needs to do good work.
Think of planning a trip with someone who's brilliant but has never met you: hand them your itinerary, your budget, and your travel preferences before asking for advice, and you get a plan built around your actual trip. Ask cold, and you get generic advice built around an imaginary average traveler. Same brilliance, different desk.
Close a conversation and open a new one, and the desk gets cleared completely. The AI you're talking to in the new chat has no idea the previous one happened — not because it's been instructed to forget, but because nothing from that old desk ever made it onto this new one. There's no memory sitting quietly in the background connecting your Tuesday conversation to your Thursday one, unless you (or a system you've built) explicitly carries something across.
This catches people off guard constantly. You explain your situation carefully on Monday, get great help, and come back Wednesday expecting it to remember — and it can't. It's more like meeting a brilliant temp handed a fresh, empty desk every morning. The fix isn't "get a better AI" — it's giving that desk a way to carry information forward on purpose, which is the entire idea behind turning a chatbot into a system, covered later in this series.
Because the model is predicting plausible text rather than retrieving verified facts, it can produce a wrong answer with exactly the same fluent, assured tone as a right one. This is usually called "hallucinating," which makes it sound like a malfunction — it's actually the mechanism working normally, just pointed at a gap. When the true answer isn't well represented in what it learned, it has no built-in "I don't actually know" reflex; it fills the gap with plausible-sounding text instead. Ask it to summarize a document you didn't attach, or name a specific source from memory, and this is exactly where it tends to go wrong.
The practical fix isn't distrust — it's a habit of checking the kinds of claims that are actually risky, and letting the rest go. A few tells worth watching for:
The single best defense is the one from the section above: give it the real source material to work from — the actual document, the actual numbers — instead of asking it to recall something from memory. A model working from material on the desk in front of it is dramatically more reliable than one working from what it "remembers" learning.
Once the mechanism clicks, its strengths and weaknesses stop looking random and start looking predictable.
None of this is a flaw to work around apologetically. It's the actual shape of the tool. Once you know the shape, you stop being surprised by it — and you start handing it the kinds of problems it's built to be extraordinary at.