Creating a RAG Evaluation

The Create RAG Evaluation page allows you to set up and configure an evaluation of various vectorization strategies and embedding models for your data. This tool helps assess the accuracy and relevancy of different strategies and provides insights into optimizing your Retrieval-Augmented Generation (RAG) applications.

Create New RAG Evaluation

Available Options

1. RAG Evaluation Name

• Description: Provide a name for your RAG evaluation. This will help you identify and track different evaluations.

• Input Type: Text Field

Create RAG Evaluation Overview

2. Upload Documents

• Description: You can upload up to 5 files to be used for your RAG Evaluation. These files will be chunked and embedded as part of the evaluation process.

• Supported File Types:

  • PDFs *.pdf

  • Markdown *.md

  • Text *.txt

  • HTML *.html, *.htm

  • Microsoft Word Docs *.doc, *.docx

• File Size Limits:

  • PDF, HTML, and Doc/Docx: Maximum file size of 5MB

  • Text and Markdown: Maximum file size of 500KB

• Additional Note: Empty files or files that contain only images will abort the RAG evaluation.

Upload Docs

3. Select Vectorization Strategy

You can choose between two types of vectorization strategies:

• Default:

  • This option evaluates 4 vectorization strategies using the following embedding models:

    • OpenAI Ada v2 (1536 dimensions)

    • OpenAI v3 Large (3072 dimensions)

    • OpenAI v3 Small (1536 dimensions)

    • Voyage AI Instruct v2 (1024 dimensions)

  • The chunking strategy used is paragraph-based.

• Custom:

  • Allows you to select up to 4 custom embedding models and chunking strategies to test. This option provides more flexibility in determining the best-performing strategy for your data.

Select Vectorization Strategy

4. Select Vector Database

The vector database stores and retrieves the vector embeddings. Vectorize provides a database engine for each of the available database options. You do not need to supply your own database or bring your own API key. You can select from the following cloud-based databases:

• Couchbase Capella: A modern cloud database for real-time applications.

• DataStax Astra: A cloud database based on Apache Cassandra.

• Elastic Cloud: Elastic's serverless cloud instance with vector support.

• Pinecone: A top-rated cloud vector database optimized for vector search.

Configure Vector Database Engine

5. Configure RAG Evaluation (Custom Strategy Only)

For each of the 4 strategies, you can customize the following options:

• Embedding Model:

  • Select from available models such as OpenAI Ada v2, OpenAI v3 Small, OpenAI v3 Large, VoyageAI 2, etc.

• Chunking Strategy:

  • Choose a chunking strategy to segment your document, with the option to chunk by paragraphs.

• Top K:

  • Select the number of top similar vectors (k) to retrieve from the database.

• Chunk Size (Tokens):

  • Specify the maximum number of tokens in each chunk (e.g., 500 tokens).

Configure RAG Evaluation