.png)
Imagine you're working at a small and midsized business (SMB) with around 100 employees. As the business continues to grow, both new and current employees constantly need to stay up to date and have access to resources related to company information.
These data sources may include customer service tools (e.g., Intercom, Salesforce), work documents (e.g., Google Drive, Microsoft Office 365), databases (e.g., PostgreSQL, MySQL), messaging apps (e.g., Gmail, Slack, Zoom), and other unstructured files (e.g., PDF, TXT).
This creates a need for a method to streamline enterprise information retrieval from various data source platforms.
Your SMB company needs:
This article will explore some of the most common retrieval solutions and evaluate how each can address the needs of a growing business. We will provide a comparative overview of the storage and search features of five passage retrieval options and one document retrieval option, along with a short description of each.
Before comparing the methods, it’s important to clarify what is meant by document and passage retrieval. Document retrieval refers to the process of retrieving entire documents by searching through them as a whole. While this can be useful, it may not always be practical, as it can be difficult to quickly locate specific information within a large document. On the other hand, passage retrieval focuses on retrieving smaller, specific sections of a document that match the query, making it easier to quickly find the information you need.
Document retrieval can still be relevant because it can be adapted to function similarly to passage retrieval by segmenting documents into smaller chunks during preprocessing. While this approach can simulate passage retrieval, it may require more effort than necessary and could introduce additional complexity. For this reason, we have decided to include a document retrieval method, as it can still be an option in certain scenarios where preprocessing and chunking are feasible.
Below is a brief side-by-side comparison of the data ingestion and processing features offered by each service. For more detailed information, please refer to the specific section.
[^1] Amazon Kendra provides example code and setup tutorial on how to build an image search engine with the help of Amazon Rekognition, however this involves training a model. The resulting image search engine allows for images to be found given text descriptions of what you’re looking for. Click here for more details.
[^2] Azure AI Search has multiple plans. The Free and Basic plans support 16 MB / blob file; S1 supports 128 MB / blob file; S2, S3, L1, L2 supports 256 MB / blob file. Click here to learn more about indexer limits for the plans.
[^3] Coveo file containers are temporary, private, and encrypted Amazon S3 data structures you can use to safely upload content, and is typically used for pushing batch items or larger items.
[^4] Elasticsearch provides example code to setup image similarity searches using image embeddings. Click here for more details.
[^5] Elasticsearch provides example code to setup image similarity searches using image embeddings. Click here for more details.
[^7] Algolia records are object-like collection of attributes with a name and a value. Depending on the Algolia plan, a single record may be limited to 10KB (in the Free Build plan) or 100KB (in the Grow, Premium, and Elevate plans).
It’s important to consider whether or not the service supports structured, unstructured, or both types of data.
This feature indicates the maximum amount of data that can be ingested in a single operation or document. The data size limit can vary depending on the connector, API method, or file type used. It's important to consider these limits when planning data ingestion, as large items may need to be split into smaller chunks before they can be processed.
You might be looking to ingest more complicated data and be able to search on them — like PDFs, images with text or audio files, which is when OCR and ASR support could be helpful. Typically, services supporting these file types would process these by applying OCR or ASR and store the extracted text as metadata or separate fields that can be searchable.
This feature refers to the ability of the service to analyze and extract insights from images, often through techniques such as image recognition, visual search, and similarity search. Services that support image analysis can identify objects, text, and patterns within images, allowing for more advanced search capabilities beyond traditional text-based queries.
Some of the products don’t provide native support, but there are tutorials on how to implement them with help with additional code and attaching additional services.
This feature describes the ability to enforce control over data access, restricting it based on the user's role. For example, certain documents or content may only be accessible to higher-level managers, ensuring that sensitive information is not readily available to all employees.
If your company operates in a multilingual environment or deals with multilingual documents, consider services that support indexing and searching across multiple languages.
As your data grows, it's important to consider solutions that can scale effectively, allowing the system to handle larger volumes of documents and information.
All of these services/products support both keyword and semantic search methods, all have autocomplete and query suggestions, and all have relevance tuning.
Algolia can support semantic search, but through the NeuralSearch service, which is available only with their Elevate plan.
Ideally, it would be best to have a service that offers both keyword and semantic search for more powerful searching.
Having the feature to autocomplete and query suggestions allows users to see suggested queries as they type, based on previous searches, popular terms, or context, helping them find accurate results more quickly.
Relevance tuning refers to the ability to adjust the weight of specific fields and prioritize certain types of documents based on their importance.
This feature indicates whether the service includes a full user interface (UI) application or provides UI components that can be integrated into existing systems. A full application would be a ready-to-use platform with all the necessary features, while UI components are smaller, customizable elements like search bars, filters, and result displays that can be embedded into a larger application. This is important to consider if you're looking to build a solution from scratch or if you need a complete out-of-the-box solution.
In addition to common file types like PDF, HTML, XML, CSV, PPT, DOC, DOCX, and TXT (except Algolia, which only supports JSON and CSV), the following data sources are also supported. (For a full list, see the links attached to each service name).
Below are five passage retrieval solutions to consider for enterprise search.
Amazon Kendra is a cloud-based AI-powered enterprise search service from Amazon Web Services (AWS) designed to help organizations quickly search, discover, and retrieve relevant information from a variety of data sources.
Amazon Kendra leverages machine learning to process queries, enabling natural language querying rather than relying on exact keyword matches. It uses models for reading comprehension and semantic matching to improve search results.
Kendra includes an Experience Builder, allowing users to create applications using UI components. Its search index can also be integrated into custom retrieval-augmented generation (RAG) applications.
As highlighted in the features comparison table, Amazon Kendra supports ASR and OCR indexing capabilities. This is achieved through integration with Amazon Textract and Amazon Transcribe, which process files, extract text, and index it to make the content searchable.
Designed for easy integration into developer RAG systems, Amazon Kendra also provides APIs for developers. For more information, check out the Amazon Kendra API documentation.
Amazon Kendra pricing is based on storage units, query units, and connectors. The minimum cost includes a Base Index with one Storage Unit and one Query Unit, with additional charges for extra units or connectors.
Kendra offers three types of indexes, ranked from least to most advanced: Basic Developer Edition (for proof-of-concept solutions), Basic Enterprise (which includes semantic search capabilities for production workloads), and Gen AI Enterprise Edition (for the highest accuracy).
Based on the example case, the Basic Enterprise Edition would be a suitable option, given its storage and querying capabilities. The estimated annual cost for this plan is $79k+.
For more information on pricing and features, visit this link.
Azure AI Search (formally called Azure Cognitive Search) is a cloud-based enterprise information retrieval system which can be used for RAG-based applications or document search and data exploration.
Azure AI Search offers hybrid searching, combining both keyword and semantic search capabilities. Its ingestion process includes an integrated vectorization extension, which provides end-to-end data processing tailored for RAG usage, including chunking, vectorizing, and indexing.
While Azure AI Search does not provide a built-in UI for interaction, it offers APIs for searching and accessing search results. For more details, check out the Azure AI Search REST API documentation.
Azure AI Search offers four tier options: Free, Basic, Standard, and Storage Optimized, each with fixed pricing. To determine the most suitable plan, businesses need to consider their storage, number of indices, and scale-out limits, as pricing is based on storage rather than querying. For more details about the pricing plans, visit this link.
For the example case, the Standard S2 plan would be the most appropriate choice, as it supports up to 512 GB of storage, meeting the data requirements. The estimated annual cost for this plan is $11k+.
Coveo provides enterprise search software specifically designed for commerce, websites, workplace (enterprise search), and service (customer self-service).
Coveo’s Enterprise AI-Search is a hybrid search system that combines keyword search with semantic search capabilities.
Access to Coveo’s index searching is available through the Search API or by using Coveo’s provided UI visual search components.
For more details, check out the Coveo API documentation.
Coveo offers both a Pros and an Enterprise Plan. The Enterprise Plan’s pricing is not publicly disclosed and must be discussed on a case-by-case basis. However, it has been reported on forums that Coveo can be a very expensive service, potentially more costly than Algolia.
Pricing may ultimately depend on factors specific to the organization’s needs. For more details, visit the Coveo pricing page.
Elasticsearch, developed by Elastic, is a popular open-source search and analytics engine built on Apache Lucene. It is used for storing, searching, and analyzing both structured and unstructured data, including logs, metrics, and security events.
Elasticsearch offers full-text search in near real-time and supports semantic searching using natural language processing (NLP) and vector searching. It also includes vector databases for storing and querying vectorized data using embeddings from third-party NLP models.
Elasticsearch provides a REST API for interacting with the search engine. For more details, check out the Elasticsearch REST API documentation.
Elasticsearch, when used independently from Elastic Cloud, is free to use.
Pricing for Elasticsearch largely depends on the chosen deployment method, self-management, and hosting options. In some cases, self-hosting may or may not be more cost-effective than using the Elastic Cloud alternative—Elastic Enterprise Search.
Not to be confused with Elasticsearch, Elastic Enterprise Search is an additional service offered through Elastic Cloud that uses Elasticsearch under the hood and provides additional features.
One additional feature of Elastic Enterprise Search is its built-in Search UI, which allows for easy integration of pre-made search bars into applications or websites.
Regarding data sources and ingestion, Elastic Enterprise Search offers an Elastic Web Crawler for data ingestion from websites. Its connectors are managed by the Elastic Cloud service, rather than being self-managed, making setup and maintenance simpler.
For more details, check out the Elastic Enterprise Search API documentation.
Since Elastic Enterprise Search is an additional service within Elastic Cloud, pricing depends on the chosen Elastic Cloud plan. The minimum costs could range from $95, $109, $125, or $175 per month, depending on the plan.
Given the conditions of our example business, the Platinum Plan would be the most suitable option due to its support for semantic searching. To estimate costs, visit the Elastic Cloud pricing page.
Algolia is a hosted search engine commonly used for e-commerce, SaaS applications, and knowledge bases.
It’s important to note that Algolia supports indexing and querying over structured data only. This means it may not be ideal for unstructured data retrieval, as unstructured data would need to be converted into structured formats for use with Algolia.
Algolia provides lexical searching and can also support semantic searching, but only through its NeuralSearch service, which is available exclusively with the Elevate plan. Additionally, Algolia offers UI widgets and API clients for easy integration.
For more details, check out the Algolia REST API documentation.
Algolia is known for being a very expensive search engine.
Algolia’s pricing plans include: Free, Grow, Premium and Elevate. Since we would like the ability to search semantically, the NeuralSearch service is required, which is only available with the Elevate plan, which is not disclosed and must be discussed. To learn more about the plans, check out this link.
The features table provides an overview of some general features to consider when choosing a service. These are not the only features of the services but highlight key aspects that might influence your decision. We also include the estimated pricing overview.
Most retrieval services offer similar capabilities, but depending on your business needs, some may be better suited than others:
When selecting an enterprise search solution, factors like storage, data type, connectors, search capabilities, and query frequency are critical. This article provides an overview of popular options to help guide your decision.
%20(2).svg)
.png)
