> ## Documentation Index
> Fetch the complete documentation index at: https://avala.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Cloud Storage

> Connect S3 and GCS buckets for dataset storage

Avala supports bring-your-own-storage so your data never has to leave your infrastructure. Connect an Amazon S3 or Google Cloud Storage bucket, and Avala will read assets directly from it — no copies, no transfers. You control access, retention, and encryption while Avala handles the annotation layer on top.

## Amazon S3

### IAM Role (Recommended)

Cross-account IAM roles let Avala access your bucket using temporary credentials — no long-lived keys to rotate or leak. Avala assumes a role in your AWS account using [STS AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html) with an external ID for [confused deputy protection](https://docs.aws.amazon.com/IAM/latest/UserGuide/confused-deputy.html).

#### 1. Get Your Setup Info

Go to **Mission Control → Settings → Storage → Add Storage → Amazon S3**. Select **IAM Role** as the authentication method. You will see two values:

* **Avala AWS Account ID** — the account that will assume your role
* **External ID** — a unique identifier for your organization (used in the trust policy)

Copy both values — you will need them in the next step.

<Tip>
  You can also retrieve these values programmatically via the SDK:

  ```python theme={null}
  import avala
  client = avala.Client()
  info = client.storage_configs.setup_info(organization="my-org")
  print(info.avala_aws_account_id)
  print(info.external_id)
  ```
</Tip>

<Info>
  If your account belongs to multiple organizations, you must pass the `organization` parameter (your organization's slug) to all storage config API calls. Single-organization accounts can omit it.
</Info>

#### 2. Create an IAM Role

In your AWS account, create an IAM role with the following trust policy. Replace `AVALA_ACCOUNT_ID` and `EXTERNAL_ID` with the values from Step 1.

```json theme={null}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::AVALA_ACCOUNT_ID:root"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "EXTERNAL_ID"
        }
      }
    }
  ]
}
```

#### 3. Attach a Permissions Policy

Attach an inline or managed policy that grants Avala access to your bucket:

```json theme={null}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::your-bucket-name",
        "arn:aws:s3:::your-bucket-name/*"
      ]
    }
  ]
}
```

If you want Avala to write exports back to the bucket, also include `s3:PutObject`.

Replace `your-bucket-name` with your actual bucket name.

#### 4. Connect in Mission Control

1. Go to **Mission Control → Settings → Storage**.
2. Click **Add Storage** and select **Amazon S3**.
3. Select **IAM Role** as the authentication method.
4. Enter the **Role ARN** (e.g., `arn:aws:iam::123456789012:role/AvalaStorageAccess`).
5. Enter your **Bucket Name** and **Region**.
6. Click **Test Connection** to verify access.
7. Save the configuration.

### Access Key (Legacy)

You can also authenticate with long-lived AWS access keys. This method is still supported but **not recommended** — IAM roles are more secure because credentials are temporary and automatically rotated.

1. Create an IAM user with programmatic access.
2. Attach the same permissions policy shown above.
3. In Mission Control, select **Access Key** as the authentication method.
4. Enter the **Access Key ID** and **Secret Access Key**.
5. Click **Test Connection** and save.

<Warning>
  Access keys are long-lived credentials. If compromised, an attacker has access until you revoke them. Prefer IAM roles whenever possible.
</Warning>

### CORS Configuration

The browser-based editor loads your objects directly from your bucket, so the bucket must allow cross-origin requests from `https://avala.ai`. Add this CORS rule:

```json theme={null}
[
  {
    "AllowedHeaders": ["*"],
    "AllowedMethods": ["GET", "HEAD"],
    "AllowedOrigins": ["https://avala.ai"],
    "ExposeHeaders": [
      "ETag",
      "Content-Length",
      "Content-Range",
      "Accept-Ranges"
    ],
    "MaxAgeSeconds": 3600
  }
]
```

<Note>
  **Why the extra `ExposeHeaders`?** Large files — MCAP recordings especially — are streamed with HTTP **range requests** rather than downloaded whole. The browser can only read a response header that is explicitly exposed, and the range machinery needs `Content-Range`, `Content-Length` and `Accept-Ranges`. Exposing only `ETag` is enough for images, but MCAP episodes will fail to open.
</Note>

## Google Cloud Storage

### Service Account

Create a service account that Avala can use to access your bucket:

1. In the Google Cloud Console, navigate to **IAM & Admin > Service Accounts**.
2. Create a new service account (e.g., `avala-storage-reader`).
3. Grant the following roles on the bucket:
   * `roles/storage.objectViewer` — read access to objects
   * `roles/storage.legacyBucketReader` — list objects in the bucket
4. If Avala should write exports to the bucket, also grant `roles/storage.objectCreator`.
5. Download the JSON key file for the service account.

<Warning>
  The JSON key file grants access to your GCS bucket. Never commit it to version control. Store it securely, restrict file permissions, and rotate keys regularly.
</Warning>

### CORS Configuration

The browser-based editor loads your objects directly from your bucket, so the bucket must allow cross-origin requests from `https://avala.ai`. Save the following as `cors.json`:

```json theme={null}
[
  {
    "origin": ["https://avala.ai"],
    "method": ["GET", "HEAD"],
    "responseHeader": [
      "Content-Type",
      "ETag",
      "Range",
      "Content-Length",
      "Content-Range",
      "Accept-Ranges"
    ],
    "maxAgeSeconds": 3600
  }
]
```

Apply it:

```bash theme={null}
gcloud storage buckets update gs://your-bucket-name --cors-file=cors.json
```

Replace `your-bucket-name` with your actual bucket name. (The older `gsutil cors set cors.json gs://your-bucket-name` still works.)

<Warning>
  **`Range` is required, and it is easy to miss.** Large files — MCAP recordings especially — are streamed with HTTP **range requests** rather than downloaded whole. A request carrying a `Range` header triggers a CORS preflight, and Google Cloud Storage only permits it if `Range` appears in `responseHeader`. Omit it and GCS rejects the preflight with *no* `Access-Control-Allow-Origin` header at all, so the browser reports a generic CORS failure and the episode never loads.

  In GCS, `responseHeader` populates **both** `Access-Control-Allow-Headers` (what the browser may *send*) and `Access-Control-Expose-Headers` (what it may *read*) — which is why `Range` and the `Content-Range` / `Content-Length` / `Accept-Ranges` trio all belong in that one list.
</Warning>

### Connect in Mission Control

1. Go to **Mission Control > Settings > Storage**.
2. Click **Add Storage** and select **Google Cloud Storage**.
3. Enter your **Bucket Name**.
4. Upload the **Service Account JSON key** file.
5. Click **Test Connection** to verify access.
6. Save the configuration.

## Storage Configuration Options

Once a bucket is connected, you can configure it in Mission Control:

| Option                 | Description                                                                                      |
| ---------------------- | ------------------------------------------------------------------------------------------------ |
| **Default storage**    | Set the bucket as the default destination for new datasets.                                      |
| **Prefix filter**      | Limit Avala's access to a specific path prefix within the bucket (e.g., `datasets/production/`). |
| **Export destination** | Enable writing completed exports back to this bucket.                                            |
| **Signed URL expiry**  | Control how long signed URLs remain valid when serving assets to annotators (default: 1 hour).   |

## Uploading Data from Connected Buckets

After connecting a bucket, you can create datasets from its contents:

1. Create a new dataset in Mission Control.
2. Select **Import from Cloud Storage** as the data source.
3. Browse or search the connected bucket for the files or folder you want.
4. Select the assets and confirm the import.

Avala will register the assets by reference — it reads them from your bucket on demand rather than copying them.

<Tip>
  Your data stays in your bucket at all times. Avala generates short-lived
  signed URLs to serve assets to the annotation editor and never persists
  copies of your files.
</Tip>