RAG Pipeline Quick Start

Approximate time to complete: 5-10 minutes, excluding prerequisites

This quickstart will walk you through creating and scheduling a pipeline that uses a web crawler to ingest data from the Vectorize documentation, creates vector embeddings using an OpenAI embedding model, and writes the vectors to a Pinecone vector database.

Step 1: Sign Up for Pinecone

1. Go to the Pinecone homepage (https://www.pinecone.io).

2. Click the Sign Up button located at the top right corner of the page.

3. Choose your preferred signup method, either:

   • Continue with Google

   • Continue with GitHub

   • Continue with Microsoft Or enter your email address to create an account manually.

Step 2: Set Up Pinecone

Access Your Pinecone Dashboard

1. After successfully signing up, log into your Pinecone dashboard.

2. In the dashboard, click on API keys in the sidebar under the Manage section.

Generate and Copy API Key

1. Click the Create API Key button if you don't have an existing key.

2. After your key is generated, click the copy icon to save your API key. You will need it in the upcoming steps.

Make sure to store your API key securely as it is necessary to interact with Pinecone from your application.

Step 3: Create a RAG Pipeline on Vectorize

Create a New RAG Pipeline

1. After logging into the platform at https://platform.vectorize.io, navigate to RAG Pipelines on the left-hand sidebar.

2. Click on New RAG Pipeline.

3. On the next screen, name your pipeline. For example, "quickstart-pipeline".

Configure the Pipeline

1. Under Select Vector Database, click on New Vector DB.

2. From the list of vector databases, select Pinecone.

3. Name your Pinecone integration (e.g., "quickstart-pinecone") and paste your Pinecone API key that you copied in earlier steps.

4. Click Create Pinecone Integration.

5. Provide an Index Name (e.g., "vectorize-quickstart"). If the index does not exist, it will be created automatically.

Select AI Platform

1. Click on New AI Platform to select the platform for generating text embeddings.

2. Select OpenAI from the AI platform options.

3. In the OpenAI configuration screen:

   • Enter a descriptive name for your OpenAI integration.

   • Enter your OpenAI API Key.

4. Accept the default values for embedding model, chunking strategy, and chunk size.

Add Source Connectors

1. Click on Add Source Connector.

2. Choose the type of source connector you'd like to use. In this example, select Web Crawler.

Configure Web Crawler Integration

1. Name your web crawler source connector, e.g., vectorize-docs.

2. Set both Seed URL(s) and Allowed URL(s) or prefix(es) to https://docs.vectorize.io.

3. Click Create Web Crawler Integration to proceed.

Configure Web Crawler Pipeline

1. Accept all the default values for the web crawler pipeline configuration:

   • Throttle Wait Between Requests: 500 ms

   • Maximum Error Count: 5

   • Maximum URLs: 1000

   • Maximum Depth: 50

   • Reindex Interval: 3600 seconds

2. Click Save Configuration.

Verify Source Connector and Schedule Pipeline

1. Verify that your web crawler connector is visible under Source Connectors.

2. Click Next: Schedule RAG Pipeline to continue.

Schedule RAG Pipeline

1. Accept the default schedule configuration

2. Click Create RAG Pipeline.

Step 4: Monitor and Test Your Pipeline

Monitor Pipeline Creation and Backfilling

1. The system will now create, deploy, and backfill the pipeline.

2. You can monitor the status changes from Creating Pipeline to Deploying Pipeline and Starting Backfilling Process.

3. Once the initial population is complete, the RAG pipeline will begin crawling the Vectorize docs and writing vectors to your Pinecone index.

View RAG Pipeline Status

1. Once the website crawling is complete, your RAG pipeline will switch to the Listening state, where it will stay until more updates are available.

2. Your vector index is now populated and we can try it out using the RAG Sandbox, to do so click on RAG Pipelines from the left hand menu.

Test Your Pipeline in the RAG Sandbox

1. After your pipeline is running, open the RAG Sandbox for the pipeline by clicking the magnifying glass icon on the RAG Pipelines page.

2. In the RAG Sandbox, you can ask questions about the data ingested by the web crawler.

3. Type a question into the input field (e.g., "What are the key features of Vectorize?"), and click Submit.

4. The system will return the most relevant chunks of information from your indexed data, along with an LLM response.

This completes the RAG pipeline quickstart. Your RAG pipeline is now set up and ready for use with Pinecone and Vectorize.