Storage Configs - Avala Documentation

Manage cloud storage configurations that connect your AWS S3 or Google Cloud Storage buckets to Avala. Storage configs allow datasets to read and write data directly from your own buckets.

Credentials (access keys, service account JSON) are validated on creation and never returned in API responses. Only bucket metadata and verification status are exposed.

List Storage Configs

GET /api/v1/storage-configs/

Returns all storage configurations for the authenticated user’s organization, ordered by most recent first.

Parameters

Name	Type	Required	Description
`cursor`	string	No	Cursor for pagination (query parameter)
`limit`	integer	No	Number of results per page (query parameter)
`organization`	string	No	Organization slug. Required if you belong to multiple organizations.

Request

curl "https://api.avala.ai/api/v1/storage-configs/" \
  -H "X-Avala-Api-Key: $AVALA_API_KEY"

import requests

response = requests.get(
    "https://api.avala.ai/api/v1/storage-configs/",
    headers={"X-Avala-Api-Key": "YOUR_API_KEY"}
)
configs = response.json()["results"]

const response = await fetch(
  "https://api.avala.ai/api/v1/storage-configs/",
  {
    headers: { "X-Avala-Api-Key": "YOUR_API_KEY" },
  }
);
const { results } = await response.json();

req, _ := http.NewRequest("GET", "https://api.avala.ai/api/v1/storage-configs/", nil)
req.Header.Set("X-Avala-Api-Key", "YOUR_API_KEY")

resp, err := http.DefaultClient.Do(req)

Response

{
  "next": null,
  "previous": null,
  "results": [
    {
      "uid": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
      "name": "Production S3 Bucket",
      "provider": "aws_s3",
      "s3_bucket_name": "acme-datasets",
      "s3_bucket_region": "us-west-2",
      "s3_bucket_prefix": "annotations/",
      "s3_is_accelerated": false,
      "gc_storage_bucket_name": null,
      "gc_storage_prefix": null,
      "is_verified": true,
      "last_verified_at": "2025-01-15T09:30:00Z",
      "created_at": "2025-01-15T09:30:00Z",
      "updated_at": "2025-01-15T09:30:00Z"
    }
  ]
}

Fields

Field	Type	Description
`uid`	string (UUID)	Unique identifier for the storage config
`name`	string	Display name of the storage configuration
`provider`	string	Storage provider: `aws_s3`, `cloudflare_r2` or `gc_storage`
`s3_bucket_name`	string \| null	S3 bucket name (AWS S3 only)
`s3_bucket_region`	string \| null	AWS region of the bucket (AWS S3 only)
`s3_bucket_prefix`	string \| null	Key prefix within the bucket (AWS S3 only)
`s3_is_accelerated`	boolean	Whether S3 Transfer Acceleration is enabled (AWS S3 only)
`gc_storage_bucket_name`	string \| null	GCS bucket name (Google Cloud Storage only)
`gc_storage_prefix`	string \| null	Object prefix within the bucket (Google Cloud Storage only)
`r2_account_id`	string \| null	Cloudflare account ID (Cloudflare R2 only)
`r2_public_base_url`	string \| null	Public read base URL for a public R2 bucket (Cloudflare R2 only)
`is_verified`	boolean	Whether the last connection test succeeded
`last_verified_at`	string (datetime) \| null	When the connection was last tested
`created_at`	string (datetime)	ISO 8601 timestamp of creation
`updated_at`	string (datetime)	ISO 8601 timestamp of the last update

Create Storage Config

POST /api/v1/storage-configs/

Creates a new storage configuration. Cloud credentials are validated during creation — the request will fail if Avala cannot connect to the bucket. Credentials are stored securely and are never returned in API responses.

Parameters (AWS S3)

Name	Type	Required	Description
`organization`	string	No	Organization slug (query parameter). Required if you belong to multiple organizations.
`name`	string	Yes	Display name for this storage config
`provider`	string	Yes	Must be `aws_s3`
`s3_bucket_name`	string	Yes	S3 bucket name
`s3_bucket_region`	string	Yes	AWS region (e.g., `us-west-2`)
`s3_bucket_prefix`	string	No	Key prefix for objects in the bucket
`s3_access_key_id`	string	Yes	AWS access key ID (write-only)
`s3_secret_access_key`	string	Yes	AWS secret access key (write-only)
`s3_is_accelerated`	boolean	No	Enable S3 Transfer Acceleration (default: `false`)
`s3_cloudfront_domain`	string	No	CloudFront distribution domain
`s3_cloudfront_public_key_id`	string	No	CloudFront public key ID for signed URLs

Parameters (Cloudflare R2)

R2 reuses the s3_bucket_name and s3_bucket_prefix fields — it speaks the S3 API — but takes its own account and credentials. There is no region (R2 is a single global namespace), and s3_is_accelerated, s3_cloudfront_domain and s3_auth_method: iam_role are rejected: they are AWS features with no Cloudflare equivalent, and accepting them would store a configuration whose URLs point at a host that does not serve the bucket.

Name	Type	Required	Description
`organization`	string	No	Organization slug (query parameter). Required if you belong to multiple organizations.
`name`	string	Yes	Display name for this storage config
`provider`	string	Yes	Must be `cloudflare_r2`
`r2_account_id`	string	Yes	Cloudflare account ID — exactly 32 hexadecimal characters
`s3_bucket_name`	string	Yes	R2 bucket name
`s3_bucket_prefix`	string	No	Object prefix within the bucket
`r2_access_key_id`	string	Yes	R2 access key ID (write-only)
`r2_secret_access_key`	string	Yes	R2 secret access key (write-only)
`r2_public_base_url`	string	No	Public read base URL (`r2.dev` subdomain or custom domain). Leave unset for private buckets, which are served through presigned URLs.

Parameters (Google Cloud Storage)

Name	Type	Required	Description
`organization`	string	No	Organization slug (query parameter). Required if you belong to multiple organizations.
`name`	string	Yes	Display name for this storage config
`provider`	string	Yes	Must be `gc_storage`
`gc_storage_bucket_name`	string	Yes	GCS bucket name
`gc_storage_prefix`	string	No	Object prefix within the bucket
`gc_storage_auth_json_content`	string	Yes	Service account JSON key content (write-only)

Request (AWS S3)

curl -X POST "https://api.avala.ai/api/v1/storage-configs/" \
  -H "X-Avala-Api-Key: $AVALA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Production S3 Bucket",
    "provider": "aws_s3",
    "s3_bucket_name": "acme-datasets",
    "s3_bucket_region": "us-west-2",
    "s3_bucket_prefix": "annotations/",
    "s3_access_key_id": "YOUR_AWS_ACCESS_KEY_ID",
    "s3_secret_access_key": "YOUR_AWS_SECRET_ACCESS_KEY"
  }'

import requests

response = requests.post(
    "https://api.avala.ai/api/v1/storage-configs/",
    headers={
        "X-Avala-Api-Key": "YOUR_API_KEY",
        "Content-Type": "application/json"
    },
    json={
        "name": "Production S3 Bucket",
        "provider": "aws_s3",
        "s3_bucket_name": "acme-datasets",
        "s3_bucket_region": "us-west-2",
        "s3_bucket_prefix": "annotations/",
        "s3_access_key_id": "YOUR_AWS_ACCESS_KEY_ID",
        "s3_secret_access_key": "YOUR_AWS_SECRET_ACCESS_KEY"
    }
)
config = response.json()

const response = await fetch(
  "https://api.avala.ai/api/v1/storage-configs/",
  {
    method: "POST",
    headers: {
      "X-Avala-Api-Key": "YOUR_API_KEY",
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      name: "Production S3 Bucket",
      provider: "aws_s3",
      s3_bucket_name: "acme-datasets",
      s3_bucket_region: "us-west-2",
      s3_bucket_prefix: "annotations/",
      s3_access_key_id: "YOUR_AWS_ACCESS_KEY_ID",
      s3_secret_access_key: "YOUR_AWS_SECRET_ACCESS_KEY",
    }),
  }
);
const config = await response.json();

body := strings.NewReader(`{
    "name": "Production S3 Bucket",
    "provider": "aws_s3",
    "s3_bucket_name": "acme-datasets",
    "s3_bucket_region": "us-west-2",
    "s3_bucket_prefix": "annotations/",
    "s3_access_key_id": "YOUR_AWS_ACCESS_KEY_ID",
    "s3_secret_access_key": "YOUR_AWS_SECRET_ACCESS_KEY"
}`)

req, _ := http.NewRequest("POST", "https://api.avala.ai/api/v1/storage-configs/", body)
req.Header.Set("X-Avala-Api-Key", "YOUR_API_KEY")
req.Header.Set("Content-Type", "application/json")

resp, err := http.DefaultClient.Do(req)

Response

{
  "uid": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "name": "Production S3 Bucket",
  "provider": "aws_s3",
  "s3_bucket_name": "acme-datasets",
  "s3_bucket_region": "us-west-2",
  "s3_bucket_prefix": "annotations/",
  "s3_is_accelerated": false,
  "gc_storage_bucket_name": null,
  "gc_storage_prefix": null,
  "is_verified": true,
  "last_verified_at": "2025-01-15T09:30:00Z",
  "created_at": "2025-01-15T09:30:00Z",
  "updated_at": "2025-01-15T09:30:00Z"
}

Credentials (s3_access_key_id, s3_secret_access_key, gc_storage_auth_json_content) are write-only. They are never included in API responses.

Test Connection

POST /api/v1/storage-configs/{uid}/test/

Re-runs credential validation and connectivity checks on an existing storage configuration. Updates the is_verified and last_verified_at fields.

Parameters

Name	Type	Required	Description
`uid`	string (UUID)	Yes	Storage config UID (path parameter)

Request

curl -X POST "https://api.avala.ai/api/v1/storage-configs/a1b2c3d4-e5f6-7890-abcd-ef1234567890/test/" \
  -H "X-Avala-Api-Key: $AVALA_API_KEY"

import requests

response = requests.post(
    "https://api.avala.ai/api/v1/storage-configs/a1b2c3d4-e5f6-7890-abcd-ef1234567890/test/",
    headers={"X-Avala-Api-Key": "YOUR_API_KEY"}
)
result = response.json()

const response = await fetch(
  "https://api.avala.ai/api/v1/storage-configs/a1b2c3d4-e5f6-7890-abcd-ef1234567890/test/",
  {
    method: "POST",
    headers: { "X-Avala-Api-Key": "YOUR_API_KEY" },
  }
);
const result = await response.json();

req, _ := http.NewRequest("POST", "https://api.avala.ai/api/v1/storage-configs/a1b2c3d4-e5f6-7890-abcd-ef1234567890/test/", nil)
req.Header.Set("X-Avala-Api-Key", "YOUR_API_KEY")

resp, err := http.DefaultClient.Do(req)

Response (Success)

{
  "verified": true
}

Response (Failure)

{
  "verified": false,
  "errors": {
    "s3_bucket_name": ["Bucket not found or access denied."]
  }
}

Get Setup Info

GET /api/v1/storage-configs/setup-info/

Returns the values you need to let Avala reach your S3 bucket through an IAM role in your account.

google_federation — for new connections. Your role trusts Google (accounts.google.com) pinned to Avala’s Google service-account IDs (subjects) and to your organization’s audience. trust_policy is the complete trust policy, ready to paste. No Avala AWS account appears in it. available is true once Avala reads new connections this way.
avala_aws_account_id and external_id — for existing connections whose trust policy names an Avala AWS account with sts:AssumeRole. Unchanged, kept for compatibility.

Parameters

Name	Type	Required	Description
`organization`	string	No	Organization slug. Required if you belong to multiple organizations.

Request

curl "https://api.avala.ai/api/v1/storage-configs/setup-info/?organization=my-org" \
  -H "X-Avala-Api-Key: $AVALA_API_KEY"

import avala
client = avala.Client()
info = client.storage_configs.setup_info(organization="my-org")

Response

{
  "avala_aws_account_id": "014424401899",
  "external_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "google_federation": {
    "available": true,
    "provider": "accounts.google.com",
    "subjects": ["108304072919389817770", "114542767323571216519"],
    "audience": "https://avala.ai/customer-s3-access/a1b2c3d4-e5f6-7890-abcd-ef1234567890",
    "trust_policy": {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Sid": "AvalaReadsViaGoogleFederation",
          "Effect": "Allow",
          "Principal": { "Federated": "accounts.google.com" },
          "Action": "sts:AssumeRoleWithWebIdentity",
          "Condition": {
            "StringEquals": {
              "accounts.google.com:aud": ["108304072919389817770", "114542767323571216519"],
              "accounts.google.com:sub": ["108304072919389817770", "114542767323571216519"],
              "accounts.google.com:oaud": "https://avala.ai/customer-s3-access/a1b2c3d4-e5f6-7890-abcd-ef1234567890"
            }
          }
        }
      ]
    }
  }
}

Delete Storage Config

DELETE /api/v1/storage-configs/{uid}/

Permanently deletes a storage configuration. Datasets that reference this configuration will no longer be able to access the bucket.

Parameters

Name	Type	Required	Description
`uid`	string (UUID)	Yes	Storage config UID (path parameter)

Request

curl -X DELETE "https://api.avala.ai/api/v1/storage-configs/a1b2c3d4-e5f6-7890-abcd-ef1234567890/" \
  -H "X-Avala-Api-Key: $AVALA_API_KEY"

import requests

response = requests.delete(
    "https://api.avala.ai/api/v1/storage-configs/a1b2c3d4-e5f6-7890-abcd-ef1234567890/",
    headers={"X-Avala-Api-Key": "YOUR_API_KEY"}
)

const response = await fetch(
  "https://api.avala.ai/api/v1/storage-configs/a1b2c3d4-e5f6-7890-abcd-ef1234567890/",
  {
    method: "DELETE",
    headers: { "X-Avala-Api-Key": "YOUR_API_KEY" },
  }
);

req, _ := http.NewRequest("DELETE", "https://api.avala.ai/api/v1/storage-configs/a1b2c3d4-e5f6-7890-abcd-ef1234567890/", nil)
req.Header.Set("X-Avala-Api-Key", "YOUR_API_KEY")

resp, err := http.DefaultClient.Do(req)

Response

Returns 204 No Content on success.

Supported Providers

Provider	Value	Description
AWS S3	`aws_s3`	Amazon Simple Storage Service
Google Cloud Storage	`gc_storage`	Google Cloud Storage

Error Responses

Validation Error (400)

{
  "s3_bucket_name": ["Bucket not found or access denied."],
  "provider": ["Unsupported provider: azure_blob"]
}

Returned when credentials are invalid, the bucket is unreachable, or the provider is not supported.

Unauthorized (401)

{
  "detail": "Invalid API key."
}

Returned when the X-Avala-Api-Key header is missing or contains an invalid key.

Permission Denied (403)

{
  "detail": "You do not have permission to perform this action."
}

Returned when the authenticated user does not have access to the requested storage config.

Not Found (404)

{
  "detail": "Not found."
}

Returned when the specified storage config UID does not exist.

​List Storage Configs

​Parameters

​Request

​Response

​Fields

​Create Storage Config

​Parameters (AWS S3)

​Parameters (Cloudflare R2)

​Parameters (Google Cloud Storage)

​Request (AWS S3)

​Response

​Test Connection

​Parameters

​Request

​Response (Success)

​Response (Failure)

​Get Setup Info

​Parameters

​Request

​Response

​Delete Storage Config

​Parameters

​Request

​Response

​Supported Providers

​Error Responses

​Validation Error (400)

​Unauthorized (401)

​Permission Denied (403)

​Not Found (404)

List Storage Configs

Parameters

Request

Response

Fields

Create Storage Config

Parameters (AWS S3)

Parameters (Cloudflare R2)

Parameters (Google Cloud Storage)

Request (AWS S3)

Response

Test Connection

Parameters

Request

Response (Success)

Response (Failure)

Get Setup Info

Parameters

Request

Response

Delete Storage Config

Parameters

Request

Response

Supported Providers

Error Responses

Validation Error (400)

Unauthorized (401)

Permission Denied (403)

Not Found (404)