Scheduling Pipelines

This page relates to page-based plans.

🔍 Looking for our legacy usage-based plan documentation? Click here.

The pipeline scheduler keeps your vector indexes up to date with fresh data. You can configure your pipeline to check for new data at regular intervals, ensuring that your indexes are always current.

Pipeline Sync Settings

The pipeline sync settings determine how and when your pipeline checks for updates. There are three main components to configure:

1. Sync Schedule

   • Daily: Pipeline syncs every day at specified times
   • Weekdays: Pipelines sync Monday through Friday
   • Weekly: Pipelines sync on a single day of the week
   • Custom Days: Pipelines sync on specific days of the week
   • Manual: Pipeline syncs only when manually triggered

2. Frequency

   • 1 sync per day: Pipeline will check for updates once daily
   • Multiple syncs per day: Pipeline will check for updates multiple times daily at evenly spaced intervals
     • The number of syncs per day is determined by your plan level

3. Timezone

   • Select your preferred timezone to ensure syncs occur at the appropriate local time
   • The pipeline uses this timezone for all scheduling calculations

Data Sync Behavior

When your pipeline runs:

• It performs an initial load to populate your vector database
• Each subsequent sync will run until all data is fully updated, ensuring your vector database stays in sync with your source data
• Once the pipeline is fully synced it will return to it's idle state until the next scheduled sync or manual trigger