AI Platforms

AI Platforms are essential components in RAG (Retrieval-Augmented Generation) pipelines within Vectorize. They are responsible for generating text embeddings, which are crucial for transforming unstructured text data into vector representations that can be efficiently stored and searched in vector databases.

What are AI Platform Integrations?

AI Platform integrations in Vectorize allow you to connect to various AI services that provide advanced natural language processing capabilities, particularly for generating text embeddings. These embeddings capture the semantic meaning of text, enabling more accurate and context-aware information retrieval in RAG pipelines.

Available AI Platform Integrations

Vectorize currently supports the following AI Platform integrations:

• OpenAI: Leverage OpenAI's powerful language models for generating high-quality text embeddings.

• Voyage AI: Utilize Voyage AI's specialized embedding models for various natural language processing tasks.

Configuring AI Platform Integrations

You can configure AI Platform integrations in two ways:

1. From the AI Platforms Section

1. Navigate to the Vectorize dashboard.

2. In the left sidebar, under "Integrations," click on "AI Platforms."

AI Platform Menu

1. You'll see a list of currently configured AI Platform integrations in your workspace.

2. To add a new integration, click the "New AI Platform" button.

AI Platforms

1. Choose from the list of available AI Platform options.

AI Platform Options

1. Follow the prompts to configure the selected AI Platform integration.

2. While Creating a RAG Pipeline

1. During the RAG Pipeline creation process, you'll reach a step to configure the AI Platform.

2. You can either select an existing AI Platform integration or create a new one.

RAG Pipeline Option

1. If creating a new integration, choose from the available options and follow the configuration steps.

RAG Pipeline AI Platforms

1. Select the specific embedding model you want to use for this pipeline.

Note: AI Platform integrations configured as part of a RAG Pipeline will automatically appear in the AI Platforms list for your organization and can be reused in future RAG pipelines.

Configuration Details

When setting up an AI Platform integration, you'll typically need to provide:

• API keys or access tokens for authentication

• Model selection (if the platform offers multiple embedding models)

• Any additional configuration options specific to the chosen AI Platform

Using AI Platforms in RAG Pipelines

In a RAG Pipeline, the integrated AI Platform is used to:

1. Generate embeddings for the documents ingested through source connectors

2. Create embeddings for user queries during the retrieval process

These embeddings enable semantic similarity searches in the vector database, allowing for more accurate and context-aware information retrieval.

For detailed information on configuring specific AI Platform integrations, please refer to their individual documentation pages linked above.