Creating a RAG Pipeline

The process of creating a RAG pipeline in Vectorize is simple and divided into three main steps:

1. Configuring the vector database and vectorization strategy.

2. Configuring one or more source connectors to ingest data from.

3. Configuring when the pipeline should update the vector indexes – whether it should run in real time, 24/7, or on a scheduled basis.

Below are the detailed instructions on how to configure a RAG pipeline, using the provided images as a guide.

Create a New RAG Pipeline

1. From the Vectorize dashboard, select New RAG Pipeline from the RAG Pipelines section on the left sidebar.

   

   New RAG Pipeline
2. This will take you to the pipeline configuration screen, where you'll be asked to name your pipeline and configure its components. Here you must provide a name for your pipeline.

   

   Create New RAG Pipeline

Configure Vector Database

3. First, select a vector database where the vector embeddings will be stored. You can either select an existing vector database or create a new one. Click New Vector DB to add a new database integration.

   

   Select Vector Database

This will bring up a list of available vector databases you can integrate with. You can refer to the relevant documentation on each connector for more details about how to configure the integration.

Configure Source Connectors

4. If you have existing vector database integrations, they will appear here. Choose the one you want to use for this pipeline.

   

   Existing Vector DB

Once you configure a vector database integration for a RAG Pipeline, it will become available to re-use in future pipelines.

5. Once you have configured a new vector database integration or selected an existing integration, you must specify the vector index/collection/table (depending on the vector database) that you want to populate.

   

   Specify Index Name
6. Additionally, you can configure metadata filtering on the vector database configuration as well.

   

   Metadata Filtering

   

   Metadata Filters

   

   Metadata Filters

Configure AI Platform

7. Next, select an AI platform that will be used to generate the text embeddings. You can either select an existing AI platform or create a new one by clicking New AI Platform.

   

   Select AI Platform
8. If you already have existing AI platform integrations, they will be displayed here. Choose the appropriate platform for your pipeline.

   

   Existing AI Platform

Configure Vectorization Strategy

9. Once you have selected your AI Platform integration, you can specify the vectorization strategy you want to use in your RAG pipeline. You will generally identify the optimal settings for your RAG Pipeline by using the RAG Evaluation feature to determine the embedding model and vectorization strategy that works bets for your data.

   

   Vectorization Strategy

Configure Source Connectors

10. After configuring the vector database integration and the AI platform integration, the RAG Pipeline builder will take you to the source connector configuration screen.

Source Connector

11. When you click on the box to add a source connector, you will be presented with a list of available source connectors.

Source Connector List

Once you select the source connector you want to use, you will be presented with a dialog box to configure the connection to the source system. You can find documentation for how to configure each individual connector in the connector's documentation.

Schedule the Pipeline

The final step before creating your RAG Pipeline is to configure how often you want it to run. RAG Pipelines can run continuously or according to a schedule. For more information, see the documentation for Scheduling RAG Pipelines

Pipeline Scheudler

Create the Pipeline

After clicking Create RAG Pipeline, you will see the pipeline creation progress. The stages include:

• Creating pipeline

• Deploying pipeline

• Starting backfilling process

Creating Pipeline

Backfilling

Regardless of what schedule you configured, your pipeline will always run immediately to backfill the vector index.

Backfilling Progress