Modern AI systems often appear intelligent because they produce coherent answers. They summarize documents, generate code, and solve complex problems in ways that feel increasingly human. With larger context windows and stronger reasoning models, it is tempting to assume that the main limitation of AI systems has already been solved.
In practice, a different problem emerges as systems grow. AI systems struggle not with understanding content, but with understanding structure.
The Illusion of Awareness
Large language models operate within context. At any given moment, the model only sees the information that has been provided in its input window. Within that space, it can reason, plan, and generate convincing outputs.
This creates an illusion of awareness. The system appears to know the environment it operates in, but that awareness disappears as soon as the context changes. Unlike traditional software, AI systems do not inherently maintain a persistent understanding of:
- What exists in a project
- Which version is current
- What has already been completed
- What should be ignored
Every interaction is effectively a fresh interpretation. For small tasks this works well. For larger systems, it introduces instability.
Comparison Models: How Modern AI Tools Handle Structure
Modern AI development tools expose two dominant approaches to handling structure inside large projects. These approaches are best understood as navigation models rather than product features.
On-Demand Traversal and Agentic Search
One approach relies on dynamic exploration. Systems such as Claude Code operate primarily through agentic search, where the model explores the repository in real time using tools like:
grepandripgrepfor text search- File listing for directory discovery
- Targeted file reads for content inspection
Instead of building an upfront index, the agent incrementally discovers structure while solving the task.
This model behaves similarly to a developer opening a codebase for the first time.
Strengths:
- High flexibility
- Avoids stale structural assumptions
- Information is always read directly from source files
Weaknesses:
- No persistent structural awareness
- Agent must repeatedly rediscover relevance
- Increased token usage in large repositories
- Can lead to inconsistent reasoning paths
Claude Code can be extended with semantic retrieval systems through external integrations, but its default behavior prioritizes exploration over precomputed structure.
Explicit Indexing and Semantic Search
A second approach builds structural awareness before reasoning begins. Cursor, for example, constructs a searchable index of the codebase that powers semantic search across files. Files are chunked, embedded, and indexed so the system can retrieve relevant information without repeated traversal.
Cursor's engineering documentation describes how this index is built incrementally using:
- Merkle trees to detect file changes
- Embedding caches to avoid recomputation
- Similarity hashing to reuse indexes across near-identical repositories
The result is faster navigation and improved response accuracy because the system begins with structural awareness rather than discovery.
The tradeoff is synchronization. Indexed representations must remain consistent with the underlying filesystem, otherwise the system risks acting on outdated structure.
Indexing in AI SystemsHybrid Semantic Indexing and Project Graphs
Systems such as Codex-style coding agents use a hybrid approach. Instead of relying purely on traversal or purely on indexing, they build semantic indexes alongside structural graphs of the project:
- Code is chunked into logical units
- Symbols and relationships are mapped through language tooling
- Semantic search retrieves relevant regions when needed
This combines embedding-based retrieval with project-aware structure, allowing the agent to maintain efficiency while still adapting dynamically as the codebase evolves.
Summary of Approaches
Agentic Traversal:
- Discovery: Real-time exploration
- Structural awareness: None (rediscovered each time)
- Tradeoff: Flexible but token-heavy
Semantic Indexing:
- Discovery: Pre-built index lookup
- Structural awareness: Persistent across queries
- Tradeoff: Fast but requires synchronization
Hybrid:
- Discovery: Index + dynamic graphs
- Structural awareness: Adaptive and persistent
- Tradeoff: Balanced but more complex
Comparison of AI Structural ApproachesAcross these approaches, the underlying problem remains the same. Reasoning improves once the system understands where things exist before attempting to interpret them.
The Problem of Traversal: The Fog of Discovery
Without indexing, AI systems rely on traversal — discovering structure dynamically by listing directories, opening files, and interpreting relevance repeatedly.
This leads to three predictable failure modes:
- State blindness — The system cannot determine whether something is relevant or current without reopening it
- Token inefficiency — Repeated discovery consumes context and computation without adding new knowledge
- Non-deterministic resume — When execution stops or context resets, the system must rediscover structure again and may follow a different reasoning path
These are architectural problems rather than model limitations.
Information, Context, and State
Understanding indexing requires separating three concepts that are often treated as identical.
Information is raw content such as files, documents, or assets. This is permanent and stored on disk.
Context is the temporary subset of information provided to a model during interaction. This is ephemeral and exists only per session.
State is the persistent understanding of what exists and what matters at a system level. This requires explicit management.
Language models are strong at processing information and context. They are weak at maintaining state across time. When state remains implicit, systems repeatedly rediscover structure before acting.
Indexing exists to make state explicit.
Navigating Without Rediscovery
Reliable AI systems separate two responsibilities:
- Knowing what exists — handled by indexing
- Understanding what it means — handled by reasoning
An index acts as a structural layer containing pointers, relationships, versions, and state information. It does not attempt to interpret content. Instead, it allows the system to navigate precisely before interpretation begins. By reducing ambiguity at the structural level, reasoning becomes more consistent and efficient.
Why Larger Context Windows Don't Replace Indexing
As context windows grow, it may seem possible to simply provide the entire project to the model. The limitation is persistence.
Context is temporary. Once the interaction ends, the model retains nothing about the structure it previously observed. Indexing provides continuity across interactions and allows systems to resume deterministically without reconstructing their environment from scratch.
Context Windows:
- Solve capacity — how much can be seen at once
- Lifetime: Single interaction
- Scale by adding more tokens
Indexing:
- Solves organization — what exists and where
- Lifetime: Persistent across sessions
- Scales by structuring existing information
These solve different problems.
Why Indexing Becomes Essential With Agents
As AI systems move from answering questions to performing multi-step tasks, indexing becomes infrastructure rather than optimization. Agents operate over time. They revisit environments, modify artifacts, and resume work after interruptions.
Effective indexing typically follows a few principles:
- The index should be derivable from source data so it never becomes the single source of truth
- It should contain structure and state rather than content itself
- When discrepancies occur, the underlying files or data remain authoritative
These constraints allow agents to behave consistently while preserving flexibility.
Key Takeaways
- Indexing solves structural awareness rather than reasoning itself
- Traversal-based systems maximize flexibility but struggle with scale and consistency
- Semantic indexing improves navigation efficiency by separating discovery from interpretation
- Context provides temporary visibility, while indexing provides persistence across interactions
- Reliable agentic systems make structure explicit before reasoning begins
- As AI systems grow in complexity, indexing moves from optimization toward core infrastructure