How to Create an GCP Cloud Storage Bucket and Service Account

Approximate time to complete: 5-7 minutes, excluding prerequisites

This how to article will give you step by step instructions for creating a GCS bucket and service account with credentials in Google Cloud Platform. It will also show you how to configure a source connector from Vectorize to your GCS bucket so you can use your bucket in a RAG pipeline.

Before you begin

Before starting, ensure you have access to the credentials, connection parameters, and API keys as appropriate for the following:

• A Vectorize account (Create one free here ↗)

• A GCP Cloud Account (Create one free here ↗)

Step 1: Make sure GCS is enabled

1. Open Google Cloud Console

   • Navigate to the Google Cloud Console.

   • Make sure you are working in the correct project.

2. Go to APIs & Services

   • From the dashboard, locate the "APIs & Services" option in the quick access section.

   • Click on APIs & Services.

   

   Go to APIs & Services
3. Enable Cloud Storage API

   • Click on ENABLE APIS AND SERVICES.

   

   Enable APIs & Services

   • In the API library, use the search box and type Cloud Storage to locate the API.

   

   Search for Cloud Storage

   • From the search results, click on the Cloud Storage option from Google Enterprise API.

   

   Select Cloud Storage

   • On the Cloud Storage API details page, click ENABLE.

   • If the API is already enabled, you can go to the next step.

   

   Enable Cloud Storage API

Step 2: Create The GCS Bucket in GCP Cloud Console

1. Open GCP Menu

   • Open the Google Cloud Platform (GCP) menu and navigate to Cloud Storage.

   • Under Cloud Storage, select Buckets.

   

   Open GCP Menu
2. Create a New Bucket

   • Click on the CREATE button at the top to start creating a new bucket.

   

   Create New Bucket
3. Configure Your Bucket

   • Bucket Name: Enter a globally unique name for your bucket (e.g., vectorize-gcs-aef220-dee21).

   • Storage Location: Choose a region for your data (you can accept the default or select a specific region).

   • Storage Class: Choose the default storage class for your data (e.g., Standard).

   • Access Control: Set Public Access Prevention to on and use Uniform access control.

   • Data Protection: Configure options such as soft delete, versioning, retention policies, etc.

   

   Configure Bucket

   • Once done, click CREATE to proceed.

4. Confirm Public Access Settings

   • A prompt will appear notifying you that public access will be prevented.

   • Click CONFIRM to proceed unless you have a specific use case that requires public access.

   

   Confirm Public Access
5. Save the Bucket Name

   • After the bucket is created, you'll be taken to the bucket details page.

   • Copy and save your bucket name, as you'll need it later (e.g., vectorize-gcs-aef220-dee21).

   

   Save Bucket Name

Step 3: Set Up the Service Account and IAM Permissions

1. Navigate to Service Accounts in GCP IAM

Navigate to Service Accounts

2. Click the button labeled CREATE SERVICE ACCOUNT

Create a Service Accounts

3. Fill in your service details

• Provide a Service account name, this will automatically populate the Dervice account ID field as well

• Click on CREATE AND CONTINUE, making sure you're clicking the button shown and NOT the more prominent DONE button.

3. Assign roles to your service account

Create a Service Accounts

• In the role selection section, grant the Project > Viewer role for the service account.

Grant Project Viewer Role

• Click on the ADD ANOTHER ROLE button

Add Another Role

4. Add Roles and IAM Conditions

   • After adding the Viewer role, select Storage Admin under the Cloud Storage section.

   

   Select Storage Admin Role

   • Now let's limit the service account to just this one bucket. To do so, add an IAM condition by clicking on ADD IAM CONDITION.

   

   Add IAM Condition
5. Create IAM Condition

   • Name your IAM condition and paste the bucket name into the value field. Set the condition as shown:

     • Condition Type: Name

     • Operator: is

   

   Create IAM Condition

   • Click SAVE to apply the condition.

6. Fixing Errors and Editing IAM Conditions (Skip if you don't get this error)

   • If you receive an error such as "Failed to add project roles", you can edit the IAM condition by clicking the pencil icon next to the condition name.

   

   Edit IAM Condition

   • In the Edit Condition screen, you can either modify it in the Condition Editor if needed.

   

   Go to Condition Editor

   • Once you're done, run the linter by clicking RUN LINTER and then click SAVE.

   

   Run Linter

   • After saving, click CONTINUE again to proceed.

   

   Try Continue Again
7. Finalizing the Service Account

   • To finalize the setup, click DONE to complete the creation of the service account.

   

   Click Done

Step 4: Create Keys for the Service Account

1. Select the Service Account

   • After creating the service account, you should see this list with the new service account listed.

   • If you left that page, you can always get to it by going to IAM & Admin > Service Accounts using upper left "hamburger" menu.

   • Click on the service account email to manage it.

   

   Select Service Account
2. Go to the Keys Tab

   • Once in the service account details, navigate to the KEYS tab.

   

   Go to Keys Tab
3. Add a New Key

   • Open the ADD KEY dropdown

   

   Add Key
4. Select Create new key to generate a new key for this service account.

Create New Key

5. Create a JSON Key

• When prompted, select the JSON key type and click CREATE.

Save the Key

1. Save the Key

   • A JSON file will be downloaded. Save this file securely as it contains the private key for your service account.

   • Choose a location on your hard drive to store the key file.

Key File Download

• Make sure to keep this key file secure, as it cannot be recovered if lost.

Step 5: Configure a Google Cloud Storage (GCS) Connector in Vectorize

1. Open the Vectorize Dashboard

   • Navigate to the Vectorize dashboard after logging in.

2. Select "Source Connectors"

   • From the left-hand menu, find the Source Connectors option under Integrations.

   

   Source Connectors
3. Click "New Source Connector"

   • On the Source Connectors page, click the New Source Connector button in the top-right corner.

   

   New Source Connector
4. Select "Google Cloud Storage"

   • In the list of connector options, find and select Google Cloud Storage.

   

   Google Cloud Storage
5. Configure the Google Cloud Storage Source

   • Name: Enter a name for the GCS integration (this can be your bucket name, but it's not mandatory).

   • Bucket: Enter the exact name of the GCS bucket you created in Google Cloud.

   • Service Account JSON: Copy and paste the service account JSON key you downloaded during the GCP setup process.

   

   Configure GCS Source
6. Create the GCS Integration

   • Once the details are filled in, click Create Google Cloud Storage Integration to finalize the connector setup.

   After the integration is successfully created, your GCS connector can now be used as part of a RAG Pipeline.