Deep Research (Beta)

📢 Note: Deep Research is currently in Beta.

The Deep Research feature allows you to generate comprehensive research reports based on your organization's data. This powerful tool combines AI with structured templates to produce in-depth analysis automatically.

Deep Research operates on the vectorized data associated with a pipeline. Deep Research will query the vector database configured on the pipeline multiple times to gather the required information to complete the report. It is also able to augment your private data with data searched from the web.

Deep Research is available in the Vectorize UI on the Pipeline Details page. It is also availabe as an endpoint on the Vectorize API. There is no additional charge for Deep Research, however it does use the retrieval endpoint when doing its research. Those retrievals will count against your plan's included retrievals and may trigger charges for additional retrievals.

Getting Started

To access the Deep Research feature, navigate to the Pipeline Details page and select the Deep Research tab. The interface is divided into three main sections:

1. Research Topic - Define what you want to research
2. Report Schema - Customize the structure of your report
3. Research Results - View the generated report

Step 1: Define Your Research Topic

1. Navigate to the Research Topic tab
2. Enter your research query in the text area (e.g., "Generate a research report on the financial status of the company"). For best results, be a specific as possible.
3. Toggle the Web Search option if you want to include internet data in your research
4. Click Next: Configure Schema to proceed

Step 2: Configure Your Report Structure

The Report Structure defines the parts of your research report. This is helpful if your organization expects a certain structure in the report. Defining the structure is optional. If you do not provide it, Deep Research will automatically generate a suitable structure for your report based on the instructions you provided in the Research Topic section.

When configuring your report structure, you can:

Use a Template

1. Click the dropdown menu labeled "start from a template"
2. Select a pre-defined template that suits your needs

Create a Custom Schema

1. Use the Schema Builder to create your own report structure
2. Add sections using the Add Section button at the bottom
3. For each section:
   • Provide a title and description
   • Add subparts using the Add Subpart button
   • Configure each subpart with its own title and description

Organize Your Schema

• Rearrange sections and subparts: Drag and drop items to reorder them
• Expand/collapse sections: Click the chevron icon next to a section title
• Edit content: Click on any field to edit its content
• Remove items: Click the trash icon to delete sections or subparts

Advanced Schema Options

• View JSON: See the raw JSON structure of your schema
• Download: Save your schema as a JSON file for future use
• Import: Upload a previously saved schema
• Clear All: Remove all sections and start fresh

Step 3: Generate Your Research

1. Review your research topic and schema
2. Click the Start Research button to begin the generation process
3. The system will automatically switch to the Research Results tab
4. You'll see a progress tracker on the left showing each phase of the research process

Viewing and Using Research Results

Once the research is complete:

1. The full report appears in the right panel
2. Review the content, which follows the structure defined in your schema (if provided)
3. Download the report as a PDF using the PDF button
4. To make changes and regenerate:
   • Navigate back to the Research Topic or Report Schema tab
   • Make your adjustments
   • Click Run Again

Tips for Effective Research

• Be specific: Clearly define what you want to research
• Structure matters: Organize your schema logically with appropriate sections
• Start small: Begin with a simple schema and expand as needed
• Save templates: Create and save schemas for research types you perform regularly
• Iterate: Use the "Run Again" button to refine results after reviewing initial output

Troubleshooting

• Deep research can take several minutes to complete, so be patient. If research seems to be taking too long, check your internet connection.
• If results don't match expectations, try refining your research question, adding more relevant source documents to you pipeline, or enabling web search to gather additional information from the internet
• If a section is missing in the results, ensure your schema is properly structured

Using the Deep Research API

In addition to the user interface, the Deep Research feature is also available through a REST API. This enables you to integrate research capabilities into your own applications and workflows.

API Endpoints

Start Deep Research

Initiates a new research process.

• URL: /v1/org/[organizationId]/pipelines/[pipelineId]/deep-research
• Method: POST
• Headers:
  • Content-Type: application/json
  • Authorization: Bearer [your-token]

Request Body:

{
  "query": "Your research question",
  "webSearch": true,
  "schema": "JSON schema for report structure",
  "n8n": {
    "account": "account-id",
    "webhookPath": "webhook-path",
    "headers": {
      "custom-header": "value"
    }
  }
}

Parameters:

• query (required): The research question or topic
• webSearch (optional): Set to true to include internet data in research
• schema (optional): JSON string representing the report structure
• n8n (optional): Configuration for n8n workflow integration

Response:

{
  "researchId": "unique-research-id"
}

Get Research Results

Retrieves the results of a research process.

• URL: /v1/org/[organizationId]/pipelines/[pipelineId]/deep-research/[researchId]
• Method: GET
• Headers:
  • Authorization: Bearer [your-token]

Response:

{
  "ready": true,
  "data": {
    "success": true,
    "events": ["Research started", "Processing data", "Generating report"],
    "markdown": "# Research Report\n\n## Introduction\n\nThis report explores..."
  }
}

• If ready is false, the research is still in progress
• If success is false, the error field will contain error details
• events contains progress updates
• markdown contains the full report in markdown format

Schema Format for API Usage

When using the API, you'll need to provide a schema in JSON format that defines the structure of your research report.

Schema Structure

{
  "report_schema": [
    {
      "type": "Section Name",
      "description": "Description of this section",
      "subparts": [
        {
          "type": "Subpart Name",
          "description": "Description of this subpart"
        }
      ]
    }
  ]
}

Key Components

• report_schema: The root array containing all sections
• type: The title/heading of a section or subpart
• description: Detailed instructions for what should be included
• subparts: An optional array of subsections within a main section

Example Schema

Here's an example schema for a financial report:

{
  "report_schema": [
    {
      "type": "Executive Summary",
      "description": "A high-level overview of the company's financial status and key findings.",
      "subparts": []
    },
    {
      "type": "Financial Highlights",
      "description": "A summary of key financial performance indicators.",
      "subparts": [
        {
          "type": "Revenue Summary",
          "description": "Total revenue generated, including a comparison with previous periods."
        },
        {
          "type": "Profitability Metrics",
          "description": "Gross profit, operating profit, and net profit margins."
        }
      ]
    }
  ]
}

Comprehensive Schema Example

Below is a more detailed schema example for a financial report:

{
  "report_schema": [
    {
      "type": "Title",
      "description": "The official title of the financial report, including the name of the company and the reporting period.",
      "subparts": []
    },
    {
      "type": "Executive Summary",
      "description": "A high-level overview of the company's financial status and key findings.",
      "subparts": []
    },
    {
      "type": "Company Overview",
      "description": "A summary of the company's business model, industry, and markets.",
      "subparts": [
        {
          "type": "Company Background",
          "description": "Brief history, mission, and vision of the company."
        },
        {
          "type": "Market Position",
          "description": "Competitive positioning and industry trends."
        }
      ]
    },
    {
      "type": "Financial Highlights",
      "description": "A summary of key financial performance indicators.",
      "subparts": [
        {
          "type": "Revenue Summary",
          "description": "Total revenue generated, including comparisons with previous periods."
        },
        {
          "type": "Profitability Metrics",
          "description": "Gross profit, operating profit, and net profit margins."
        }
      ]
    }
  ]
}

Best Practices for Schema Design

1. Be specific in descriptions: Provide clear guidance on what information should be included
2. Use logical organization: Structure the report with a natural flow from high-level to detailed information
3. Include context: Add details about relevant metrics, comparisons, or time periods
4. Balance depth and breadth: Include enough sections to be comprehensive without making the schema unwieldy
5. Consider your audience: Design the schema with the intended readers' needs in mind

API Usage Example

The following example demonstrates how to start a research task and retrieve the results using cURL:

# Start a new research task
curl -X POST \
  https://api.vectorize.io/v1/org/your-org-id/pipelines/your-pipeline-id/deep-research \
  -H 'Authorization: Bearer your-token' \
  -H 'Content-Type: application/json' \
  -d '{
    "query": "Analyze the financial performance of our company",
    "webSearch": false,
    "schema": "{\"report_schema\":[{\"type\":\"Executive Summary\",\"description\":\"Summary of findings\",\"subparts\":[]}]}"
  }'

# Response: {"researchId":"abc123"}

# Check research status
curl -X GET \
    https://api.vectorize.io/v1/org/your-org-id/pipelines/your-pipeline-id/deep-research/abc123 \
  -H 'Authorization: Bearer your-token'

Integration Tips

• Poll the results endpoint periodically to check if research is complete
• Consider implementing webhooks for asynchronous notifications
• Store the research ID to access results later
• Use proper error handling for cases where research fails