Deep|A discussion about RAG, Vector Database , and Their Monetization Potential, as well as an Analysis of Elastic as a Company-Part-1
A deep analysis of RAG, Vector Database and Their Monetization Potential
The title emphasize that these two concepts, RAG and vector databases, cannot be equated simply.
1. What’s RAG?
RAG is a broader concept that refers not just to a specific product or tech stack, but rather a methodology. It strengthens that when answering questions, it can adopt an open-book approach rather than relying solely on memory. This can also be seen as providing AI and large models with means to interact with external information, enabling them to access external data at any time to enhance generated results.
Many technologies can fall under the RAG concept. For instance, using a search API to allow GitHub to search the web when answering questions is essentially a form of RAG. Similarly, traditional databases can implement RAG without relying on vector databases; for example, in an order management scenario, reading business order data from a Snowflake data warehouse and generating order information descriptions through SQL queries can also be considered RAG.
RAG does not necessarily require the use of vector databases, but given that current AI capabilities are primarily focused on large language models, breakthroughs are often centered around language processing. Therefore, one of the best ways to retrieve information related to language is through vector databases. Vector search and traditional keyword search are two primary methods for information retrieval, and as a result, there is a close relationship between vector databases and RAG.
2. RAG’s Potential
The potential of RAG technology warrants higher-level discussion, particularly regarding its significance in the AI ecosystem. This can be analyzed through two perspectives: the value of technology and investment.
Technically, RAG operates alongside other essential technologies such as prompt engineering and post-training, enhancing the capabilities of LLMs. These technologies complement rather than replace one another. RAG stands out due to its relatively lower cost; it efficiently converts data into vectors, reducing the computational expenses associated with processing large data volumes through LLMs. Moreover, RAG is easier to implement, as it can integrate with vector databases, traditional relational databases, and search APIs. Its accessibility means that many programmers without deep learning expertise can adopt it, adding significant value to LLM applications.
From the perspective of investment, while RAG technology holds promise, it may not be the optimal time for investment due to its early lifecycle stage. Current commercial players are in a highly competitive environment, making it challenging to identify leading firms. Investing now may require long-term commitment and patience, as the timeline for RAG maturity is unpredictable. The low adoption rate primarily results from implementation challenges, as the methodology is relatively new and lacks widely accepted best practices. As clients face various parameters to adjust, the need for trial-and-error approaches can hinder progress.
Addressing these implementation frictions could signal a turning point for RAG technology. Vendors that can develop generalized configuration strategies to alleviate these user challenges may establish early competitive advantages in the market.
3. RAG vs. Fine-Tuning or Post-Training: Are They Substitutable?
RAG (Retrieval-Augmented Generation) and post-training are two distinct methods for enhancing language model capabilities, each with its own focus. RAG is relatively easier to implement and can be attempted by anyone, making it accessible even for those who may not achieve success. In contrast, post-training generally suits well-funded AI-centric companies, as it may pose challenges for traditional large enterprises.
From the perspective of information freshness, RAG allows models to accept new data immediately through databases or APIs, while post-training relies on training batches, leading to a cutoff point for the most recent information. This means RAG can modularize knowledge bases, allowing easier updates and management of information, such as for e-commerce product details, without the resource waste of retraining.
RAG’s simplicity in data preparation makes it user-friendly, whereas post-training requires a complex process involving the creation of query and answer pairs, with stringent data quality and scale requirements. While RAG does not change the model's original behavior, post-training is geared towards refining responses for more human-like interactions and improving task-specific performances, like translation.
Cost-wise, RAG incurs low to medium costs, primarily related to data storage and governance akin to traditional databases, while post-training demands higher investment due to its specialized knowledge and technology needs. However, post-training offers stronger interpretability and immediate response capabilities compared to the potential latency introduced by RAG when retrieving data.
Both methods can mitigate hallucinations, but neither completely eradicates the issue. RAG may still produce deviations from accurate language materials, whereas fine-tuning provides better control over responses, optimizing the model's output to signal when it lacks knowledge. RAG also enhances data privacy, while fine-tuning can risk data exposure.
In summary, both RAG and post-training serve to improve model capabilities but address different needs. RAG supports dynamic knowledge acquisition, allowing for real-time data incorporation, while fine-tuning trains models for specific queries, boosting adaptability based on defined objectives.
4. Will RAG Accompany the Evolution of AI to Its End?
I believe it will. A metaphor can help explain this: RAG is like an open-book exam, related to information, while the data-plus-AI stack is implemented through RAG. Post-training, on the other hand, is specifically aimed at mock exams and practice questions, focusing on training for specific question types, while pre-training corresponds to a student's foundational level. Therefore, I think that the current pre-training, post-training, and RAG are all essential. Just like a student uses different methods to solve problems during their learning process, ultimately, these methods need to be integrated.
From my perspective, large models are more like tools for mastering problem-solving and applying patterns, rather than merely compressing potentially infinite and diverse data through limited parameters. This approach is unnecessary; it primarily aims to discover and describe the patterns that humans have identified, whether they are linguistic patterns or workflow patterns, which are suitable for fitting with model parameters. RAG serves as an external library or the internet, acting as a source of information. Thus, my understanding is that RAG is not a temporary solution but an indispensable component. In summary, I believe RAG is worthy of in-depth study; this is the core conclusion.
By the way, Anthropic recently released a blog specifically focusing on how to effectively implement RAG. They are developers of long-context windows, initially emphasizing the need to achieve a million-token context window while also researching RAG. In their blog and the source code of their experimental projects, they use Elastic Search for the search component. In Gartner's Gen AI Hype Cycle, it indicates how many years away the technology is from maturity. RAG may mature in less than two years, while the maturity of vector databases could take a longer time. The difference between the two mainly lies in the fact that RAG is a broader concept and, as an application paradigm of Gen AI, may face higher acceptance.
Currently, vector databases are indeed a relatively immature type of database compared to traditional relational databases like MySQL, which have been extensively optimized for query performance, security, and other aspects, accumulating rich experience. In contrast, vector databases are still in their infancy in terms of algorithm implementation, with varying progress among different companies. For example, Zilliz may be ahead, while others are still catching up. Gartner predicts that vector database maturity will take 5 to 10 years, partly because the current penetration rate of vector databases is below 1%. This low penetration rate has many reasons, but the biggest barrier is that the technology is too novel, and many people have yet to master how to use it.
As a new technology, RAG currently lacks widely recognized best practices in the industry, and practical use involves many techniques. For example, NVIDIA conducted a promotional experiment with RAG, revealing that without RAG, the accuracy of answering questions was only 30%. However, by employing various detailed techniques and data processing during RAG usage, the accuracy could be improved to 50%. Yet, 50% accuracy is still inadequate, necessitating the combination of RAG with post-training so that the model first learns key terms and then integrates with RAG to enhance effectiveness. Additionally, SFT (Supervised Fine-Tuning), a post-training technique, can make the model's responses align more closely with proper citation behavior, increasing accuracy. Ultimately, RAG, SFT, and ranking must be combined to optimize the references available in RAG search results. Experiments have shown that using traditional keyword searches like BM25 together with contextual embeddings and other methods, by integrating various techniques, only marginally improved the success rate.
What is the best approach? Currently, no one knows for sure; this is the biggest problem, and everyone is exploring, even leading companies are still in a state of discovery. This explains why the adoption of RAG and vector databases faces certain barriers.
5. What’s Vector Databases?
In this section, I place "vector databases" alongside "vector search," as I believe the term "vector database" is inaccurate and does not accurately describe its functionality. I think "vector search" is a more appropriate term because, at its core, a vector database is not a true database but rather a search solution.
In the workflow of RAG, the role of a vector database is information indexing. Indexing refers to generating a directory for the data, allowing relevant documents to be found through retrieval. In this process, vector databases share some similarities with traditional databases. For instance, traditional databases involve data storage, querying, and computation, and it may seem that vector databases do the same, but there is a fundamental difference between them.
The most significant distinction is that vector databases store vectors, which can be viewed as labels for the raw data. The vector search solution projects data into a high-dimensional space, positioning synonyms as close to each other as possible. However, in this space, the coordinates of vectors have no specific meaning; only relative coordinates are meaningful, while absolute coordinates cannot be reverted to the original data. Thus, what is stored in a vector database is not the raw data itself but labels that point to the raw data. We can think of the embedding process as a labeling process, albeit done through a large model. From this perspective, a vector database stores labels rather than data, making it an indexing technology. This is similar to the core concept of inverted indexing mentioned earlier, where the index stores references rather than the original data. Additionally, vector databases must be used in conjunction with another storage system for raw data; for example, raw data might be stored in a database, while the vector database operates alongside it, or the raw data may be stored in object storage.
In terms of storage, there is a fundamental difference between vector databases and traditional databases. Traditional databases emphasize data structure and efficient querying, while vector databases do not adhere to row or column concepts. Currently, the storage solutions for vector databases are not complex, primarily involving the individual storage of vectors. There is no differentiation in storage, but in computation, many algorithms such as FLAT, LSH, and HNSW (with HNSW being the most popular) are widely used. However, all these algorithms are open-source implementations and are textbook-level, leading to rapid adoption by different vendors for vector search, making it difficult for the algorithms themselves to achieve differentiation.
Moreover, there is a detail in the query computation of current vector databases that may cause friction during their implementation. Mainstream vector databases typically use in-memory computing for queries, which requires vector caching in memory, resulting in hardware costs for queries that exceed those of traditional databases. Additionally, the integration of vector databases with large models presents a challenge; when responding to queries, the wait time for the vector database cannot be too long, as it impacts user experience. In the future, if asynchronous agents are designed—such as Anthropic's use—where users can delegate tasks to AI agents, then the application opportunities for vector databases in hard disk solutions will increase.
Finally, I want to reiterate the relationship between vector databases and RAG. RAG is a methodology, while vector databases are a tool for implementing these methods. In the workflow of RAG, many steps are completed outside of the vector database. For instance, when establishing the index for raw files, the initial step involves the embedding process, which segments the raw file into multiple vector chunks (chunking). This process can be facilitated by tools like LangChain, but chunking is not central to the core functionality of vector databases; the critical aspect is how to segment the raw data.
Embedding is even more vital, as it transforms raw data into vectors that are projected into vector space, relying on the embedding model (provided by large language model vendors), with the quality of this model directly affecting the effectiveness of vector retrieval. Therefore, the concept of a vector database as an independent product has logical flaws; it cannot operate in isolation.
6. The Calculation of TAM(Total Addressable Market) of Vector Databases
Let’s discuss the estimation of Total Addressable Market (TAM) from an investment perspective. Here are some calculations from three sources for reference
6.1 Morgan Stanley's Estimate: Morgan Stanley's analysis begins with data from 2022, estimating revenues from some top vendors, totaling around $100 million. By assuming these vendors capture 80% of the market and considering a growth anchor from the cloud database sector, the projected market size for RAG is estimated at $2.4 billion by 2026. However, the estimation process involves complex assumptions that may not be very reliable.
6.2 Snowflake's Report on Databases: The report discusses the characteristic challenges faced by vector databases, emphasizing their storage of high-dimensional data and the necessity for optimized solutions. It suggests that advancements, such as decreased memory costs and asynchronous AI operations, could enhance the utility of vector databases, but uncertainties regarding their maturity remain.
6.3 IDC's Forecast: IDC provides a top-down estimate for global enterprise Gen AI expenditure, projecting substantial growth by 2027. It estimates that vector databases could account for around $4.3 billion of the software spending within Gen AI. However, this projection contains inherent uncertainties, particularly regarding the assumptions made. It has an estimated earnings elasticity of 20%, suggesting potential earnings of $500 million by 2026.
7. Competitive Landscape of Vector Databases
The competitive landscape in the database industry has intensified since 2022, marked by an influx of players. Competitors can be categorized into two main groups: established pure play vendors, such as Zilliz, and new entrants adopting open-source models.
Pure play vendors have built a robust technological foundation over time and emphasize performance advantages, while newcomers often leverage open-source frameworks. Zilliz has garnered significant recognition within the open-source community, particularly for its Milvus vector database, alongside LlamaIndex, a basic tool for developing test projects that should not be viewed as a full-fledged vector database.
Key players include Zilliz and Pinecone, with Pinecone receiving attention due to its collaboration with OpenAI. Zilliz reported considerable revenue growth, while Pinecone's earnings were reported at $20 million, significantly lower than expectations. In addition to pure play vendors, traditional data platforms and public cloud service providers are also entering the database space, using their existing ecosystems to cross-sell products. Microsoft and Amazon have launched relevant products since 2023, such as Elastic's Relevance Engine and new offerings by Microsoft and Amazon.
Elastic positions itself between traditional data platform players and pure play vendors, focusing on document search, stemming from its robust existing market presence based on search technology. While Elastic maintains market leadership in this new search domain, it faces competition from pure play vendors that emphasize speed and performance. Comparatively, Elastic offers advantages in speed and scalability against traditional databases, including PostgreSQL.
Examining Milvus reveals that it highlights Elastic as a key competitor and claims to have achieved a tenfold performance increase based on various optimizations. The performance enhancements among these products often rely on fundamental improvements to existing technologies rather than revolutionary advancements. Elucidating on operational efficiencies reflects a manufacturing perspective where incremental enhancements lead to significant overall improvements.
Another notable open-source project, Qdrant, showcases its performance gains in comparison to Milvus and Elastic, illustrating how benchmark scores can be more of a marketing strategy than an objective measure of performance suitability. The competitive dynamics also include major public cloud players, particularly AWS and Azure. AWS’s vector search capabilities incorporate four modules that integrate vector search functionality into various database products like OpenSearch, DocumentDB, PostgreSQL, and Redis.
OpenSearch is tailored to function independently while enabling the addition of vector search capabilities, reflecting a strategy similar to Elastic's. Historically, OpenSearch has sought to emulate Elastic Search’s capabilities closely. While Elastic enjoys brand recognition and community support, AWS customers may favor OpenSearch due to its seamless integration into the AWS ecosystem. Similarly, Azure treats vector search as a functional module within multiple products but has not adopted a competitive strategy reflecting direct engagement or forking of Elastic's project.
Azure AI Search shares functional similarities with Elastic's vector search modules but faces criticism for lack of investment and maturity in its offerings. Users have reported dissatisfaction regarding costs and functional deficiencies, leading some to migrate back to Elastic solutions.
Overall, since 2023, the database field has expanded with diverse competing offerings, although Elastic still retains a competitive advantage primarily through brand image and ecosystem integration. Microsoft’s recommendations to customers often highlight Elastic’s relevance, demonstrating its significant influence despite the crowded market dynamics, indicating a lasting impact on the competitive landscape for database technologies and vector search functionalities. The interplay between established players and evolving technologies will shape future developments in this rapidly growing sector.
