Elasticsearch retrievers architecture and use-cases

Elasticsearch is packed with new features to help you build the best search solutions for your use case. Learn how to put them into action in our hands-on webinar on building a modern Search AI experience. You can also start a free cloud trial or try Elastic on your local machine now.

In this blog post we'll take another deep dive with retrievers. We've already talked about them in previous blogs from their very introduction to semantic reranking using retrievers. Now, we're happy to announce that retrievers are becoming generally available with Elasticsearch 8.16.0, and in this blog post we'll take a technical tour on how we implemented them, as well as we'll get the chance to discuss the newly available capabilities!

Elasticsearch retriever

The main concept of a retriever remains the same as with their initial release; retrievers is a framework that provides the basic building blocks that can be stacked hierarchically to build multi-stage complex retrieval and ranking pipelines. E.g. of a simple standard retriever, which just bring backs all documents:

Pretty straightforward, right? In addition to the standard retriever, which is essentially just a wrapper around the standard query search API element, we also support the following types:

• knn - return the top documents from a kNN (k Nearest Neighbor) search
• rrf - combine results from different retrievers based on the RRF (Reciprocal Rank Fusion) ranking formula
• text_similarity_reranker - rerank the top results of a nested retriever using a rerank type inference endpoint

More detailed information along with the specific parameters for each retriever can also be found in the Elasticsearch documentation.

Let's briefly go through some of the technical details first, which will help us understand the architecture and what has changed and why all these previous limitations have now been lifted!

Technical drill down of retrievers

One of the most important (and requested) things that we wanted to address was the ability to use any retriever, at any nesting level. Whether this means having 2 or more text_similarity_reranker stacked together, or an rrf retriever operating on top of another rrf along with a text_similarity_reranker, or any combination and nesting you can think of, we wanted to make sure that this would be something one could express with retrievers!

To account for this, we have introduced some significant changes to the retriever execution plan. Up until now, retrievers were evaluated as part of the standard search execution flow, where (in a simplified scenario for illustration purposes) we reach out to the shards twice:

• once for querying the shards and bringing back from + size documents from each shard, and
• once for fetching all field data and perform any additional operations (e.g. highlighting) for the true top [from, from+size] results.

This is a nice linear execution flow that is (relatively) easy to follow, but introduces some significant limitations if we want to execute multiple queries, operate on different results sets, etc. In order to work around this, we have moved to an eager evaluation of all sub-retrievers of a retriever pipeline at the very early stages of query execution. This means that, if needed, we are recursively rewriting any retriever query to a simpler form, the specifics of which depend on the retriever type.

• For non-compound retrievers we rewrite similar to how we do in a standard query, as they could still follow the linear execution plan.
• For compound retrievers, i.e. for retrievers that operate on top of other retriever(s), we flatten them to a single rank_window_size result set, which is essentially a <doc, shard> tuple list that represents the top ranked documents for this retriever.

Let's see what this actually looks like, by working through the following (rather complex) retriever request:

The rrf retriever above is a compound one, as it operates on the results of some other retrievers, so we'll try to rewrite it to a simpler, flattened, list of <doc, shard> tuples, where each tuple specifies a document and the shard that it was found on. This rewrite will also enforce a strict ranking, so no different sort options are currently supported.

Let's proceed now to identify all components and describe the process of how this will be evaluated:

[1] top level rrf retriever; this is the parent of all sub-retrievers which will be rewritten and evaluated last, as we'd first need to know the top 10 (based on rank_window_size) results from each of its sub-retrievers.

[2] This knn retriever is the first child of the top level rrf retriever and uses an embedding service (my-text-embedding-model) to compute the actual query vector that will be used. This will be rewritten as the usual knn query by making an async request to the embedding service to compute the vector for the given model_text.

[3] A standard retriever that is also part of the top-level's rrf retriever's children, which returns all documents matching topic: science query.

[4] Last child of the top-level rrf retriever which is also an rrf retrievers that needs to be flattened.

[5] [6] similar to [2] and [3], these are retrievers that are direct children of an rrf retriever, for which we will fetch the top 100 results (based on the rrf retriever's rank_window_size [4]) for each one, combine them using the rrf formula, and then rewrite to a flattened <doc, shard> list of the true top 100 results.

The updated execution flow for retrievers is now as follows:

• We'll start by rewriting all leaves that we can. This means that we'll rewrite the knn retrievers [2] and [6] to compute the query vector, and once we have that we can move up one level in the tree.
• At the next rewrite step, we are now ready to evaluate the nested rrf retriever [4], which we will eventually rewrite to a flattened RankDocsQuery query (i.e. a list of <doc, shard> tuples).
• Finally, all inner rewritten steps for the top-level rrf retriever [1] will have taken place, so we should be ready to combine and rank the true top 10 results as requested. Even this top-level rrf retriever will rewrite itself to a flattened RankDocsQuery which will be later used to proceed with the standard linear search execution flow.

Visualizing all the above, we have: