Skip to main content

Understanding Metadata in RAG

Metadata is the additional information that describes and categorizes your documents, like authors, dates, or topics. When stored alongside vector embeddings in your database, metadata enables powerful filtering and improves search relevance. Vectorize supports both system metadata (automatically provided) and user-defined metadata, which we'll explore in detail below.

For example, if you have a technical document, its user-defined metadata might include:

  • Document type: Technical specification
  • Author: Engineering team
  • Last updated: January 2025
  • Product area: Authentication

Why Use Metadata?

When processing documents in your RAG pipeline, important contextual details like authors, dates, or categories can help narrow down search results and provide crucial context.

For example, imagine you're building an AI assistant that needs to access:

  • Product documentation
  • Support tickets
  • Release notes
  • API documentation
  • Discord conversations
  • Intercom chat history

Without metadata, a search for "how to reset API key" would return semantically similar results across all these sources. With metadata, your assistant can filter for:

  • Content type (documentation vs. support conversation)
  • Status (resolved vs unresolved issues)
  • User role (admin vs. regular user questions)
  • Product feature (authentication, billing, API)
  • Issue priority (high, medium, low)
  • Response rating (high vs. low customer satisfaction)

This means you can build precise queries like "find recent, resolved tickets about API key resets" or "show me highly-rated support responses about authentication for admin users."

Metadata types

Vectorize supports two types of metadata:

  • System metadata: fields which are always provided in each document record in your vector index
  • User-defined metadata: user-provided metadata associated with documents

System metadata

The following metadata fields are provided in each document record in your vector index.

FieldDescription
textThe actual content being embedded
chunk_idIdentifier for this specific chunk of text (e.g. "20" out of 181 total chunks).
total_chunksTotal number of chunks this document was split into
unique_sourceUnique identifier combining origin_id and a hash of the content
origin_idUUID for the original document uploaded.
filenameOriginal filename of the document
sourceFull path/identifier of the document in storage
source_display_nameHuman-readable source path
originMethod of document ingestion. This corresponds to the source connector used to extract data in your RAG pipeline. Examples: google-drive, file-upload, web-crawler, etc.
metadataUser-defined metadata attributes

User-defined metadata

For each document you process, you can optionally provide an associated metadata file containing content to store in your vector database. When querying, you can combine semantic search with metadata filters to get exactly the results you need.

Let's look at a simple example using a collection documents containing Shakespeare's plays.

For each document, we'll create a metadata file containing JSON that contains information about the play. For example:

    {
"metadata" : {
"title": "A Midsummer Night's Dream",
"genre": "comedy",
"year_written": "1595",
"setting": ["Athens", "Nearby Forest"],
"characters": ["Puck", "Oberon", "Titania", "Lysander", "Hermia", "Demetrius", "Helena", "Bottom"],
"number_of_acts": 5,
"themes": ["love", "magic", "nature", "dreams", "mischief"],
"famous_quotes": [
"Lord, what fools these mortals be!",
"The course of true love never did run smooth"
],
"length": {
"words": 16087,
"lines": 2349
}
}
}

In this example, the metadata allows you to build more precise queries for your AI application, like:

  • Find all tragedies written before 1600
  • Search for plays with themes of revenge
  • Look for specific characters or settings
  • Filter by genre while doing semantic search

What's next?

Was this page helpful?