Creating a RAG Pipeline
Last updated
Last updated
The process of creating a RAG pipeline in Vectorize is simple and divided into three main steps:
Configuring the vector database and vectorization strategy.
Configuring one or more source connectors to ingest data from.
Configuring when the pipeline should update the vector indexes – whether it should run in real time, 24/7, or on a scheduled basis.
Below are the detailed instructions on how to configure a RAG pipeline, using the provided images as a guide.
From the Vectorize dashboard, select New RAG Pipeline from the RAG Pipelines section on the left sidebar.
This will take you to the pipeline configuration screen, where you'll be asked to name your pipeline and configure its components. Here you must provide a name for your pipeline.
First, select a vector database where the vector embeddings will be stored. You can either select an existing vector database or create a new one. Click New Vector DB to add a new database integration.
This will bring up a list of available vector databases you can integrate with. You can refer to the relevant documentation on each connector for more details about how to configure the integration.
If you have existing vector database integrations, they will appear here. Choose the one you want to use for this pipeline.
Once you configure a vector database integration for a RAG Pipeline, it will become available to re-use in future pipelines.
Once you have configured a new vector database integration or selected an existing integration, you must specify the vector index/collection/table (depending on the vector database) that you want to populate.
Additionally, you can configure metadata filtering on the vector database configuration as well.
Next, select an AI platform that will be used to generate the text embeddings. You can either select an existing AI platform or create a new one by clicking New AI Platform.
If you already have existing AI platform integrations, they will be displayed here. Choose the appropriate platform for your pipeline.
Once you have selected your AI Platform integration, you can specify the vectorization strategy you want to use in your RAG pipeline. You will generally identify the optimal settings for your RAG Pipeline by using the RAG Evaluation feature to determine the embedding model and vectorization strategy that works bets for your data.
After configuring the vector database integration and the AI platform integration, the RAG Pipeline builder will take you to the source connector configuration screen.
When you click on the box to add a source connector, you will be presented with a list of available source connectors.
Once you select the source connector you want to use, you will be presented with a dialog box to configure the connection to the source system. You can find documentation for how to configure each individual connector in the connector's documentation.
The final step before creating your RAG Pipeline is to configure how often you want it to run. RAG Pipelines can run continuously or according to a schedule. For more information, see the documentation for Scheduling RAG Pipelines
After clicking Create RAG Pipeline, you will see the pipeline creation progress. The stages include:
Creating pipeline
Deploying pipeline
Starting backfilling process
Regardless of what schedule you configured, your pipeline will always run immediately to backfill the vector index.
