Create a RAG Pipeline with Elastic and a Web Crawler
Last updated
Last updated
Approximate time to complete: 5-10 minutes, excluding prerequisites
This quickstart will walk you through creating and scheduling a pipeline that uses a web crawler to ingest data from the Vectorize documentation, creates vector embeddings using an OpenAI embedding model, and writes the vectors to an Elasticsearch vector database.
Before starting, ensure you have access to the credentials, connection parameters, and API keys as appropriate for the following:
A Vectorize account (Create one free here ↗ )
An OpenAI API Key (How to article)
An Elastic account (Create one on Elastic ↗)
Navigate to the Elastic Cloud console and click Create project under the Serverless projects section.
Select Elasticsearch for building custom applications with your data, and click Next.
Name your project (e.g., vectorize-quickstart).
Under Configuration, choose Optimized for Vectors.
Click Create project to initialize.
Once initialization completes, click Continue.
Scroll down to the API Key section and click New to create a key.
Enter a name for your key (e.g., vectorize-quickstart) and optionally set an expiration date.
Click Create API key.
Copy the generated API key and save it securely—you won't be able to retrieve it later.
Copy your Elasticsearch endpoint URL as well. You'll need this to connect to your deployment.
Open the Vectorize Application Console ↗
From the dashboard, click on + New RAG Pipeline under the "RAG Pipelines" section.
Enter a name for your pipeline. For example, you can name it quickstart-pipeline.
Click on + New Vector DB to create a new vector database.
Select Elastic Cloud from the list of vector databases.
In the Elastic Cloud configuration screen:
Enter a descriptive name for your Elastic Cloud integration.
Enter the Host, Port, and your Elastic API Key.
Provide the index name you want to use in Elastic
The Index Name can be the same as your pipeline name.
Click on + New AI Platform.
Select OpenAI from the AI platform options.
In the OpenAI configuration screen:
Enter a descriptive name for your OpenAI integration.
Enter your OpenAI API Key.
Leave the default values for embedding model, chunk size, and chunk overlap for the quickstart.
Click on Add Source Connector.
Choose the type of source connector you'd like to use. In this example, select Web Crawler.
Name your web crawler source connector, e.g., vectorize-docs.
Set Seed URL(s) to https://docs.vectorize.io.
Click Create Web Crawler Integration to proceed.
Accept all the default values for the web crawler pipeline configuration:
Throttle Wait Between Requests: 500 ms
Maximum Error Count: 5
Maximum URLs: 1000
Maximum Depth: 50
Reindex Interval: 3600 seconds
Click Save Configuration.
Verify that your web crawler connector is visible under Source Connectors.
Click Next: Schedule RAG Pipeline to continue.
Accept the default schedule configuration
Click Create RAG Pipeline.
The system will now create, deploy, and backfill the pipeline.
You can monitor the status changes from Creating Pipeline to Deploying Pipeline and Starting Backfilling Process.
Once the initial population is complete, the RAG pipeline will begin crawling the Vectorize docs and writing vectors to your Pinecone index.
Once the website crawling is complete, your RAG pipeline will switch to the Listening state, where it will stay until more updates are available.
Your vector index is now populated and we can try it out using the RAG Sandbox, to do so click on RAG Pipelines from the left hand menu.
After your pipeline is running, open the RAG Sandbox for the pipeline by clicking the magnifying glass icon on the RAG Pipelines page.
In the RAG Sandbox, you can ask questions about the data ingested by the web crawler.
Type a question into the input field (e.g., "What are the key features of Vectorize?"), and click Submit.
The system will return the most relevant chunks of information from your indexed data, along with an LLM response.
This completes the RAG pipeline quickstart. Your RAG pipeline is now set up and ready for use with Elastic and Vectorize.
