There’s a version of this article that focuses on practical tips for optimizing output of large language models (LLMs): “make your prompts more explicit,” “provide more context,” “don’t assume the model will connect the dots.” That version is fine (and useful, even), but if you’ve spent any time trying to build functional, production-grade systems with LLMs then you’ve probably already written it yourself in the margins of your frustration. This isn’t that article.

The more philosophically interesting, and ultimately more practically important, questions are: why do common LLM failures happen, why is “prompt engineering” necessary at all, and what are the fundamental principles upon which this evolving symbiolinguistics between humans and AI are based? Put more simply: why is ChatGPT’s brilliance peppered with episodes of derpiness that would embarrass Jerry Lewis?

To understand how LLMs think, one must start with how they read, because for an LLM reading isn’t just one input channel among many, it’s the only input channel. In transformer models, the mechanism that processes input is the same mechanism that generates “reasoning.” The two are not separate systems; they’re the same system run in sequence, recursively. Chain-of-thought reasoning, scratchpad reasoning, and agentic planning are all processes by which the model reads its own output and feeds that output back in as new tokens. This means that every structural limitation in how an LLM reads propagates directly and multiplicatively into how it reasons.

LLMs and humans are not just quantitatively different readers; they are qualitatively different kinds of cognizers, and the differences trace back to at least three distinct architectural and epistemological gaps that no amount of fine-tuning or prompting is going to fully close. Understanding these gaps matters not just for writing better prompts, but for making sound architectural decisions about where and how to deploy LLM-based systems.

Let’s work through each gap in turn, and then explore the common root.

Gap One: Premise Inference, or The Problem of Implicit Reasoning Chains

Suppose I make the following two simple statements in succession: “I checked my phone. We should bring the plants in.” Upon hearing these, another human immediately reconstructs the implied chain of thought: He looked at a weather forecast, the forecast predicted something bad (frost, a storm, high winds), he values the life of the plants, he assumes I also value the life of the plants, and bringing the plants inside will protect them from impending doom. No human stops to ask, “what does checking a phone have to do with plants? Let me spin up a subagent to reason about that…” Humans insert the missing links automatically, fluidly, and without conscious effort.

On the other hand, an LLM may fail here in a way that is non-obvious if you’re not carefully scrutinizing the output. The model is unlikely to say, “I don’t understand.” It will generate something (it’s constitutionally incapable of not generating something, which is why in prompt engineering it’s important to give LLMs an “out” or “none of the above” option), and that something will be syntactically coherent. It may even sound reasonable, especially to those who have grown comfortable accruing cognitive debt. But in order to produce its output, the LLM may have invented intermediate premise(s) to bridge any logical gaps. Those premises may be spot-on. Or they may be completely wrong, partly wrong, or just ever-so-slightly off in a way that poisons downstream reasoning. Any such problems compound when the implied chain is longer than one link. A two-step implication might be resolved correctly most of the time. But start adding steps, where each step is left to inference, and the chances that model outputs begin casually stravaging into foppery grow.

This is not because LLMs are stupid. It is because they have no mechanism that is functionally equivalent to what cognitive scientists call mental simulation: the human ability to run a little movie in your head of what is being described and check whether the intermediate steps make sense. When a human hears “I checked my phone,” she briefly and unconsciously simulates the act of checking a phone and asks: what would I learn from that, and what would that information imply about plants? Her simulation is grounded in her prior experience of the world. But the LLM is doing something different: it is computing a probability distribution over tokens given the context, heavily weighted by the statistical structure of its training corpus. It doesn’t run a simulation; it computes a continuation. And continuations that bridge implicit reasoning gaps are underrepresented in training data, because (thankfully) most human writing doesn’t bother to explain every step of its own logic (yawn). Instead, we assume a shared experiential context that makes many premises not only too obvious to state, but counterproductive to include.

This has a direct implication for both content design and more complex reasoning systems. The more implicit your reasoning chain, the more likely an LLM is to insert a statistically plausible but contextually incorrect bridge. For simple cases like a few implied steps in a prosaic context (let’s bring the plants in), modern frontier models do fine. The implications are easy enough to infer from surrounding context and the model’s broad training. But when the model is working through a multi-step problem, whether in a chain-of-thought prompt or an agentic task, it is reading its own prior outputs as new input at each step. A missed or incorrectly inferred premise in step one becomes a stated premise in step two. There’s no external reality check between steps; the only check is the statistical plausibility of the next continuation given everything written so far. Errors compound. Drift accelerates. And the model, having no experiential ground to detect that something has gone wrong, proceeds with full confidence through conclusions that have quietly become untethered from the original intent. It’s a pernicious failure. Derp.

Gap Two: Context Selection, or Floating Abstractions All the Way Down

This is a deeper problem, and explaining it properly requires a brief detour through Ayn Rand: a philosopher who gets almost no serious attention in AI discourse, possibly because she’s known for her controversial, unpopular, and radical political and moral arguments. But her less known epistemological work is more directly relevant to LLM cognition than most of what currently passes for AI philosophy.

Rand’s core epistemological observation was that valid concepts are formed through a specific inductive process. Humans begin with direct sense perception: I see this red thing, and this red thing, and this other red thing. Through a process of observing similarities and differences, I form a new concept: “red”, a mental unit (with “measurements dropped”) that refers to a specific attribute perceived in reality across multiple instances. “Red” is a low-level concept because it has direct referents: actual red things in the world that I have experienced. Higher-level concepts, such as “justice,” or “property,” or “momentum,” are formed hierarchically by a process of non-contradictory integration of lower-level concepts that themselves trace back, through a chain of abstractions, to direct sense perception. A concept that cannot be traced back through this hierarchy to some perceptual base is what Rand called a floating abstraction: a word that has the grammatical form of a concept, but lacks a genuine cognitive referent. It’s ersatz: it floats in the mind unmoored to concrete reality.

Rand believed that floating abstractions are a primary source of flawed human thinking and philosophical or political disagreement. When people use terms like “freedom,” or “the public good” without being able to articulate how they relate to observable reality through a chain of concepts, those terms are “floating,” which empowers people to use them in contradictory ways without noticing, to equivocate between different meanings, and to feel certain about something they can’t actually specify with any certainty. Philosophical charlatanism, in Rand’s view, depends almost entirely on the exploitation of floating abstractions.

Now back to LLMs. Here’s the uncomfortable truth: for an LLM, every concept is a floating abstraction.

I want to be precise about this, because it’s easy to overstate. An LLM “knows” that red is a color, that colors are perceptual properties, that perceptual properties are attributed by beings with sense organs, and so on. It has learned the relationships between concepts through the statistical structure of human language. In a narrow sense, this is something like a hierarchical conceptual structure, without the process of non-contradictory integration. But it has never seen red. It has no sense organs. The concept “red” in the LLM’s weights is not anchored in any actual percept; it is anchored by a vector representation of the word “red” and its N-dimensional distance to other vectors. An LLM’s entire conceptual edifice is built on language rather than on reality, which means it “floats,” not in the sense of being undefinable (the LLM can define red quite fluently), but in the more fundamental sense of having no non-linguistic ground; there are precisely zero perceptual referents in liminal space.

The consequence for context selection is direct and serious. It’s one of the reasons that an otherwise competent LLM may trip over an apparent logical contradiction that humans easily resolve by importing the right experiential context (distinguishing process from outcome, short-term from long-term). The failure emerges when that distinction isn’t statistically obvious from the surrounding tokens, which is precisely when the absence of perceptual ground matters most.

When a human hears “they’re serving sushi at the picnic,” she automatically imports a cluster of causally-relevant real-world knowledge: sushi is raw fish, raw fish spoils in heat, picnics are typically outdoor events, it’s currently summer, outdoor summer temperatures in this region are probably high, high temperatures accelerate bacterial growth in raw fish, ergo someone might get sick. This inference chain feels effortless because it isn’t really inference at all; it is associative activation grounded in direct experience. She has felt heat. She has smelled spoiled food. She has spent an unpleasant night in the bathroom after eating bad food. She has a visceral understanding of what happens when food sits in the sun, and that understanding is not mediated entirely by language. Her conceptual hierarchy bottoms out in percepts, and those percepts carry implicit information that is automatically activated by the right contextual triggers.

Frontier models are impressively good at surface-level examples like my sushi picnic (I checked), but an LLM performing the “same” conceptual task is doing something structurally different. It activates context through attention over the token sequence. The context that gets imported is the context that is statistically proximate to the input in the training distribution, not the context that is causally relevant. In easy cases, statistical proximity and causal relevance coincide; the model gets it right. It recommends against sushi at picnics in the summer. In hard cases, such as novel situations, specialized domains, complex tasks, and low-frequency events, statistical proximity and causal relevance diverge. The causally necessary context is not always the statistically common context for a given phrase, and the model has no mechanism to detect this divergence. It has never eaten bad sushi. Now run this recursively, and the downward spiral ensues. Derp.

Gap Three: Attention and the Illusion of Large Working Memory

The third gap is more narrowly architectural, and it requires some care to characterize correctly because the popular framing of it is almost exactly backwards.

The common assumption is that LLMs have an advantage over humans in working memory because their context windows are enormous. The latest GPT and Gemini models have context windows of 1M+ tokens. Grok’s flagship has a context window of 2M tokens. The human cognitive “context window” (short-term working memory), on the other hand, can hold a paltry four chunks of information at once, with each chunk being at most a few words or concepts. So by that measure the common assumption is correct: an LLM context window is not merely larger than human working memory, it’s orders of magnitude larger.

But this comparison is almost entirely misleading, because it conflates the size of the context window with the quality of attention over that window.

Transformer attention is, at its core, a learned mapping from positions in the token sequence to weighted combinations of values at other positions. The attention weights are typically computed via the softmax of scaled dot products between query and key vectors. In practice, what this means is that the model’s ability to “notice” that a piece of information in position N is relevant to a decision at position M depends on the learned attention patterns. And those patterns are heavily biased toward local context, toward frequently co-occurring patterns in training data, and toward the beginning and end of the context window. The middle of a long context is famously underattended; several papers have documented the “lost in the middle” phenomenon, wherein retrieval accuracy for facts placed in the middle of long contexts drops significantly compared to facts placed at the beginning or end.

Human attention, by contrast, is not positional. It is both semantic and motivational. When a human is thinking through a problem, she doesn’t retrieve information by its position in a sequence of things she has read; she retrieves it by its relevance to her current goal, triggered by associative cues and modulated by emotional salience, novelty, and need. Working memory in humans is better understood not as a fixed-size token buffer, but as an attentional spotlight: it can be pointed at any part of the vast long-term memory store, and it dynamically selects for information that is relevant to the current task, updated in real time as the task evolves. The “window” may be small, but the search space it can address is vast, and the selection mechanism is remarkably good at finding what is relevant.

What an LLM has, in contrast, is a large but relatively low-resolution attention mechanism. It can see many tokens at once, but its ability to dynamically identify and foreground the right tokens, the ones with high relevance to the current decision, is limited by the fixed, trained structure of its attention heads. Grok marketing notwithstanding, you cannot tell a transformer to “think harder” about position 47,392 because that’s where the key constraint was specified. The model’s attention weights are determined by the forward pass, not by explicit executive control.

This matters materially in agentic contexts, where an LLM must make a series of decisions over time, integrating information from multiple sources, tools, memory stores, and prior reasoning steps. The human analogy would be a detective working a complex case: she doesn’t sit there and re-read every case file before each decision. Instead, she maintains a sparse, high-level model of the case in her head, knows roughly where to look for specific information when she needs it, and is triggered by new evidence to retrieve and re-examine specific old evidence with fresh eyes. The detective’s working memory is shallow; her long-term memory is deep. Her attention system is evolutionarily optimized for connecting the two in a goal-directed way. A “reasoning model” is an attempt to mimic this process, but it’s stuck with an attention mechanism that was not designed for executive goal-directed retrieval. It can hold a lot in its context window, but it cannot reliably focus on the right part of it or recognize that the “right” part is somewhere else. Again, run this recursively. In a multi-step reasoning chain, the model’s outputs at each step become new, trusted, context for the next step’s attention computation. Errors compound. Derp.

The Common Root: No Ground, No Sanity Check

Premise inference, context selection, and attention quality are not independent problems. They are expressions of a single underlying condition: LLMs have no non-linguistic ground truth against which they can judge their output, and no mechanism to use it if they did.

A human who infers a missing premise is doing so against the backdrop of a lifetime of embodied experience (think Kahneman System 1). If the implied premise leads to a conclusion that feels wrong, that feeling is a signal that something in the inference chain is off. It’s a chemical admonition to go back and look. Sorry, Star Trek Vulcan fans: it turns out that to be a clear, rational thinker and problem solver, emotional awareness matters. It matters not because emotions are magical epistemological divination rods, but because they are biological cheat codes that help focus your attention amidst a cacophony of possibilities. This is not mysticism; it’s an optimization and error-correction mechanism that has been trained by millions of actual interactions with reality, where wrong conclusions can have dire consequences.

LLMs have neither the perceptual faculty through which to anchor concepts, nor the efficient signaling mechanism to direct attention to relevant data or to raise the alarm over dubious results. Their “experience” is statistical exposure to human language, which is (indirectly) a derivative of human experience and lacks the feedback mechanism that keeps human representations correlated with reality. A language model trained on descriptions of gravity has not felt the sensation of falling; it has learned the statistical associations between the word “fall” and words like “impact,” “speed,” “weight,” and “damage.” In most circumstances, this is enough to produce useful outputs. In edge cases, this absence of ground truth is the gremlin that undermines your goal.

This is also why the limits of LLM reasoning are not, fundamentally, a data problem. Adding more training data doesn’t fix it; it just pushes the edge cases farther out. The fundamental issue is that statistical proximity in language space is not the same thing as causal relevance in reality. Human cognition, imperfect as it is, has a mechanism for closing this gap. Current LLM designs are destined to remain untethered to reality.

The Problem of Writing Asymmetry

There’s an implication worth calling out here, because it’s particularly relevant for anyone thinking about content design at any scale (which, for what it’s worth, is a problem we think about a lot at Sprite.)

Writing optimized for LLM consumption and writing optimized for human engagement are not just different in degree; they are in some respects structurally opposed. Writing that works for humans exploits the very mechanisms that cause LLMs to stumble. Powerful human prose relies on implication, on the reader’s automatic importation of the right context, on the emotional activation triggered by a well-placed image or a rhetorically charged phrase. Take one of my favorite Bashō haiku:

natsukusa ya
tsuwamono domo ga
yume no ato

summer grass
all that remains
of brave soldiers’ dreams

That’s horrible content for getting yourself indexed by ChatGPT. But it’s beautiful. And chilling.

By contrast, you know what kind of content is great for exploiting ChatGPT? Verbose, laboriously exhaustive technical documentation, like old school software specs with tables of contents from back when waterfall charts were how pointy-haired bosses managed engineering teams, and when humans still wrote code. Or your own corporate bylaws (have you ever actually read them?). Legal and technical documents like these are often packed with fully explicit premises, carefully specified context (including a glossary of jargon), and no reliance on automatic inference. They make great LLM input. And kindling.

The practical implication is that content optimized for a world in which LLMs gatekeep discoverability cannot simply be “good writing” in the traditional sense. It has to be structured in a way that makes it LLM-legible without becoming human-illegible. This is a real design challenge, and that tension is not going away. If anything, as LLMs become more embedded in search, recommendation, and retrieval pipelines, the tension is going to get worse. The structure has to carry the weight that rhetoric used to carry; the logic has to be explicit enough that a system with no experiential ground can follow it; but the prose has to remain engaging enough that a human who stumbles upon it doesn’t immediately close the tab.

Anyone who tells you otherwise is floating an abstraction.

Pay Attention (query, key, value)

I want to preempt a predictable misreading. None of the above is an argument that LLMs are useless, or that their limitations are fatal, or that human cognition is without its own serious failure modes. Humans are catastrophically bad at certain things: consistency, memory across time, resistance to emotional bias, scale, and remembering whether Oceana is at war with Eastasia or Euroasia. LLMs are extraordinarily useful precisely in areas where human cognition bottlenecks: generating first drafts at scale, retrieving and synthesizing large bodies of text, pattern-matching across domains, maintaining consistency of style across thousands of outputs.

The argument is narrower and more defensible: that the specific failure modes of LLMs (poor premise inference, unreliable context selection, and coarse attentional focus) are not accidental or fixable at the level of training or minor model iteration alone. They are structural consequences of what it means to learn solely from language without the benefit of embodied experience and real-world feedback. Understanding this will help you be a better prompt engineer and agentic system architect, because (and I say this with genuine admiration for Vaswani et al.) as it turns out, attention is not all you need.