Agent Builder is available now GA. Get started with an Elastic Cloud Trial, and check out the documentation for Agent Builder here.
For the last few years, prompt engineering has dominated the conversation in AI. The focus was on finding the perfect words and phrasing to coax the right response from an LLM. But as we build more sophisticated AI agents designed to tackle multi-step, complex tasks, the focus is shifting, the topic moves towards challenges with managing context.
An LLM's context is like a human's short-term memory. Trying to stuff it with every potentially relevant piece of information is challenging. It leads to context rot, a concept where the model's 'attention budget' gets exhausted, causing it to lose focus and derail its own reasoning. For example, one recent "needle in a haystack" (NIAH) benchmark in NOLIMA: Long-Context Evaluation Beyond Literal Matching noted that performance "degrades significantly as context length increases," observing that at 32K tokens, "11 models drop below 50% of their strong short-length baselines." Inefficiency within the context window directly affects the agent's reliability.
The solution isn't just better prompts, but an emerging discipline: context engineering. It’s the art of managing the model's limited attention.
Beyond RAG: the shift to "just-in-time" context for AI agents
Previously, the standard approach was RAG, which involved a form of retrieval before inference, where all potentially relevant data was processed and fed into the system prompt upfront for it to reason over.