Blog

How to use hybrid search for an e-commerce product catalog

Learn how to use hybrid search to build an e-commerce product catalog, using faceting, promotions, personalization and behavioral analytics.

Vector Database
AL
By: Andre Luiz

Try out vector search for yourself using this self-paced hands-on learning for Search AI. You can start a free cloud trial or try Elastic on your local machine now.

In this article, we will demonstrate how to implement a hybrid search that combines the results of full-text search with vector search. By unifying these two approaches, hybrid search improves the breadth of results, leveraging the best of both search strategies.

In addition to integrating hybrid search, we’ll demonstrate how to add features that make your search solution even more robust. These include faceting and personalized product promotions. Additionally, we’ll show you how to capture user interactions and generate valuable insights using Elastic’s Behavioral Analytics tool.

In this implementation, you will see how to build both the interface that allows users to view and interact with the search results and the API responsible for returning the information. To access the repositories with the source code, the links are provided below:

We have divided this guide into several steps, from creating the index to implementing advanced features such as faceting and result personalization. By the end, you will have a robust search solution ready to be used in an e-commerce scenario.

Environment setup for e-commerce hybrid search

Before we begin the implementation, we need to set up the environment. You can choose to use a service on Elastic Cloud or a containerized solution to manage Elasticsearch. If you choose containerization, a configuration via Docker Compose can be found in this repository: docker-compose.yml.

Index creation and product catalog ingestion

The index will be created based on a catalog of cosmetic products, which includes fields such as name, description, photo, category, and tags. Fields used for full-text search, like "name" and "description," will be mapped as text, while fields used for aggregations, such as "category" and "brand," will be mapped as keyword to enable faceting.

The "description" field will be used for vector search, as it provides more context about the products. This field will be defined as dense_vector, storing the vector representation of the description.

The index mapping will be as follows:

The script for creating the index can be found here.

Embedding generation

To vectorize the product descriptions, we use the model all-MiniLM-L6-v2. In this case, the application is responsible for generating the embeddings before indexing. Another option would be to import the model into the Elasticsearch cluster, but for this local environment, we chose to perform the vectorization directly within the application.

We used the cosmetics dataset available on Kaggle to populate the index, and to improve the efficiency of data ingestion, we used batch processing. During this same ingestion stage, we will generate the embeddings for the "description" field and index them into the new field "description_embeddings".

The complete data ingestion process can be followed and executed directly through the Jupyter Notebook available in the repository. The notebook provides a step-by-step guide on how the data is read, processed, and indexed into Elasticsearch, allowing for easy replication and experimentation.

You can access the notebook at the following link: Ingestion Notebook.

Hybrid search implementation

Now, let's implement hybrid search. For keyword-based search, we use the multi_match query, targeting the fields "name," "category," and "description." This ensures that documents containing the search term in any of these fields are retrieved.

For vector search, we use the KNN query. The search term needs to be vectorized before executing the query, and this is done using the method that vectorizes the input term. Note that the same model used during ingestion is also used for the search term.

The combination of both searches is accomplished using the Reciprocal Rank Fusion (RRF) algorithm, which merges the results of the two queries and increases search accuracy by reducing noise. RRF allows both keyword-based and vector searches to work together, enhancing the understanding of the user's query.

Now, let's compare the results of a traditional keyword search with hybrid search. When searching for "foundation for dry skin" using keyword search, we get the following results: