> ## Documentation Index
> Fetch the complete documentation index at: https://avala.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Runbook

# Avala Cloud local-upload rollout and reversal

This runbook covers opt-in Avala Cloud uploads on R2. Personal and organization allocations are independent; Arcade campaigns use separate organization-funded allocations. BYOB remains unchanged. The preferred serving domain is `data.avala.ai`; bucket provisioning does not attach that domain. Private reads use the recorded S3-compatible endpoint and existing authenticated URL-issuing APIs.

## Approval boundary

Code and local tests do not authorize production activation. Obtain explicit approval before Terraform apply, creating cloud credentials, any billable R2 test, enabling an organization, migrating bytes, changing source retention or deleting objects. Terraform proposal and offline tests live in `infra/avala-cloud`. No real R2/canary/5 GB/throughput/soak acceptance is claimed by this runbook.

Review the actual pending migration chain and immutable release before deployment. Migration `0248` alone is no longer sufficient: personal allocation, quota overrides, browser recovery and capture have later schema requirements. Finish deploying all compatible API and Celery tasks before enabling new admission. Follow [browser deployment ordering](https://github.com/avala-ai/avala/blob/develop/reports/infrastructure/r2-migration/browser-recovery-production-readiness.md) and retain schema-aware readers after accepting R2 data; rollback admission policy instead.

## Activation order

1. Refresh the [metadata preflight](https://github.com/avala-ai/avala/blob/develop/reports/infrastructure/r2-migration/activation-metadata-preflight.md), then review the exact environment's [development](https://github.com/avala-ai/avala/blob/develop/reports/infrastructure/r2-migration/development-provisioning.md) or [production](https://github.com/avala-ai/avala/blob/develop/reports/infrastructure/r2-migration/production-provisioning.md) plan. The initial proposal creates only two private `user-managed` buckets, global and EU. It does not provision derived buckets, runtime credentials or application allocations.
2. After the separately approved apply, verify bucket privacy, CORS, lifecycle and jurisdiction. Deliver separate API and worker credentials using [runtime credential injection](https://github.com/avala-ai/avala/blob/develop/reports/infrastructure/r2-migration/runtime-credentials.md), deploy the compatible release, and verify the actual running services. Complete the approved development driver checks below before production enrollment.
3. For a named personal canary, review the [100 GB quota override](https://github.com/avala-ai/avala/blob/develop/reports/infrastructure/r2-migration/personal-quota-policy.md), [personal allocation](https://github.com/avala-ai/avala/blob/develop/reports/infrastructure/r2-migration/personal-allocation.md) and [browser recovery enrollment](https://github.com/avala-ai/avala/blob/develop/reports/infrastructure/r2-migration/browser-recovery-enrollment.md) separately. The requested allowance is exactly `100000000000` bytes. Null overrides still mean 10 GiB personal or 100 GiB organization; enabling R2 does not change the allowance. Verify the server's recovery response before enabling browser journals. Apply only the reviewed owner, target, jurisdiction and policies.
4. For Arcade, first provision reviewed `user-managed` and `derived` targets in the same jurisdiction, then bootstrap the campaign's original and derived allocations with positive allowances and its exact private Avala dataset binding. The [multipart enrollment command](https://github.com/avala-ai/avala/blob/develop/reports/infrastructure/r2-migration/capture-multipart-enrollment.md) requires those rows to exist and be enabled; it does not create them. Review bootstrap as a separate operation. The initial bucket-only plan and personal allocation command do not supply it. Then review multipart policy, the individual collector enrollment and the separate campaign cohort gate using [collector enrollment](https://github.com/avala-ai/avala/blob/develop/reports/infrastructure/r2-migration/capture-collector-cohort.md). Deploy the compatible native consumer before testing the selected cohort.
5. Verify the named canaries end to end, including suspension/restart, completion, Range reads, derived outputs, annotation/export, quota attribution and configuration rollback. Expand only after the required evidence passes. Migration parity, the seven-day quota soak and approved S3 teardown remain separate acceptance gates; do not infer them from new uploads succeeding.

Quota, browser enrollment and managed allocation commands preview by default in the updated operator release. Managed allocation and capture enrollment require `--apply --expected-policy-digest` from their exact current preview. Use explicit `--preview` for allocation inspection: older releases reject that flag before their immediate mutator. No execution flag is read-only only in the updated release. Even the quota GET can reconcile accounting; use the documented preview commands for read-only policy review.

## Configure a reviewed target

Use the staff Django admin `ManagedStorageTarget` to register the reviewed Terraform output:

* `name`: stable operator name such as `local-uploads-eu`.
* `account_id`: exact 32-character lowercase Cloudflare account ID.
* `bucket`: reviewed private bucket name.
* `jurisdiction`: `global` or `eu`; EU allocations require an EU target.
* `secret_id`: approved AWS Secrets Manager ARN.
* `secrets_region`: AWS region containing that secret.

Use the current [runtime credential contract](https://github.com/avala-ai/avala/blob/develop/reports/infrastructure/r2-migration/runtime-credentials.md): `AVALA_R2_CREDENTIALS_SECRET_JSON` maps immutable secret references to separately scoped API and worker credentials. Populate payloads outside Git/Terraform and inject exact Secrets Manager versions through the existing ECS execution role. Do not add application task roles merely to enable R2, since that changes ambient AWS credential resolution. The legacy direct Secrets Manager resolver remains available when injection is absent, subject to its existing permissions. Retain old aliases, tokens and versions through outstanding URL and multipart lifetimes plus the rollback window.

Each batch stores a location snapshot containing the account, bucket, jurisdiction and secret reference, never the token value. Editing the target changes only future batches. Retain old secret references and matching bucket access while any old batch still needs them.

## Canary allocation

In the verified updated operator release, preview the exact organization and target. Replace the example UUID and target with reviewed values; preview does not change DB configuration or infrastructure.

```bash theme={null}
python manage.py set_managed_storage_allocation <organization-uuid> --target local-uploads-eu --jurisdiction eu --enable --preview
```

Review the current/proposed policy, possible DB-only quota-row creation and returned digest. After explicit approval for that environment and scope:

```bash theme={null}
python manage.py set_managed_storage_allocation <organization-uuid> --target local-uploads-eu --jurisdiction eu --enable \
  --apply --expected-policy-digest <reviewed-policy-digest>
```

Only protocol-v2 clients for the enabled owner receive R2 grants. Clients generate one `dataset_upload_uid` for the whole upload and include it with `upload_protocol_version=2` in every file presign request. Small files use conditional PUT. Files above 64 MiB use fixed 64 MiB multipart parts, at most 10,000. S3 retains its configurable 5 GiB per-file ceiling. Capable R2 uploads may exceed it subject to 10,000 fixed parts and the owner's configured quota. Managed private GET grants last five minutes; old BYOB expiry behavior is unchanged.

Multipart part signing and completion/abort use `/api/v1/datasets/manual-upload/uploads/<upload_uid>/{parts,complete,abort}/`. Completion checks server-listed part sizes and HEAD size/type/encoding. Final dataset creation includes `dataset_upload_uid` and requires every upload to be complete. Ingest dispatch is recorded on the batch and retried by daily reconciliation if queue publication fails.

Canary acceptance requires approved real R2 tests, a representative large MCAP upload, successful indexing and thumbnail generation, viewer Range reads, annotation/export checks, denied unauthorized reads, retry/resume, and quota behavior. Do not widen the flag while any required check is missing.

## Allocation rollback

```bash theme={null}
python manage.py set_managed_storage_allocation <organization-uuid> --disable --preview
```

Review the pause digest and obtain approval, then apply it:

```bash theme={null}
python manage.py set_managed_storage_allocation <organization-uuid> --disable \
  --apply --expected-policy-digest <reviewed-pause-digest>
```

This immediately disables new R2 admission. Global owners can create new S3 batches; a disabled personal EU allocation pauses new batches because no per-owner EU S3 fallback exists. Review the selected owner's fallback before applying. Existing batches, part signing, completion, reads and accounting continue using their pinned locations. Keep quota overrides and browser recovery history. This does not copy or delete objects. An S3 POST capability issued to a legacy client remains valid until its original expiry. R2 PUTs are conditional to prevent replay overwriting a completed object.

This is not a full R2-outage rollback: R2-only objects still need R2. Verified same-generation S3 replicas and client URL refresh or edge indirection are prerequisites for transparent provider-outage recovery. Existing direct R2 URLs cannot be redirected by an API flag. Replica migration, public release publication, and domain failover are later phases.

## Accounting and retention limits

The existing quota row and S3 floor/reconciliation safeguards remain authoritative for S3. R2 batch records add reserved/committed bytes to every owner usage and admission check, so S3 reconciliation cannot erase R2 usage after allocation reversal. Daily reconciliation also lists persisted R2 source and thumbnail prefixes, including idle batches. Local-upload thumbnails are charged to the same owner cap. Arcade's separate derived allowance is required independently and is not enabled by a local-upload allocation.

Pending and aborted reservations remain counted until the final signed capability has expired plus the existing 24-hour in-flight upload grace. Daily reconciliation fences the multipart upload, rechecks HEAD under the quota and upload locks, and releases the reservation only on authoritative object absence. Completed receipts require the additional [purge tombstone and write-fence checks](https://github.com/avala-ai/avala/blob/develop/reports/infrastructure/r2-migration/purge-write-fence.md) before release. Network, permission and unexpected errors preserve the reservation for retry. Reservation reconciliation does not delete object bytes. Do not manually zero reservations to bypass a cap.

This runbook authorizes no purge or teardown. Before migration, separately review source S3 lifecycle and purge workers so original bytes survive verification and soak. The managed purge implementation has its own write-fence and worker deployment prerequisites; it is not the rollback procedure. Do not remove a batch or secret reference while its data is retained. Archive remains available; retained bytes remain counted.

## Verification record

Record actual local test results in the PR. Real R2 integration tests are opt-in and must stay skipped without a separately approved cost budget. This implementation does not establish seven-day ≤ 0.1% quota drift, S3/R2 parity, full outage rollback, anonymous public download, or production readiness. Keep those acceptance items open until measured.

After explicit approval covering both billable requests and deletion of this test's temporary objects, use the current three-driver matrix. Each selected bucket must be explicitly allowlisted, disposable, unversioned and end in `-dev`, `-test`, or `-development`; production names are refused. Supply these variables through the approved environment, with location JSON containing secret references only:

| Variable | Required value |
| - | - |
| `AVALA_STORAGE_ALLOW_BILLABLE_TESTS` | `1`, only for the approved execution |
| `AVALA_STORAGE_LIVE_DRIVERS` | `s3,r2-global,r2-eu` for the complete matrix |
| `AVALA_STORAGE_DEV_BUCKET_ALLOWLIST` | JSON array of the exact three reviewed development bucket names |
| `AVALA_STORAGE_S3_LOCATION` | Reviewed S3 `StorageLocation` JSON |
| `AVALA_STORAGE_R2_GLOBAL_LOCATION` | Reviewed R2 global `StorageLocation` JSON |
| `AVALA_STORAGE_R2_EU_LOCATION` | Reviewed R2 EU `StorageLocation` JSON |

The old `AVALA_R2_ALLOW_BILLABLE_TESTS` and `AVALA_R2_DEV_LOCATION` variables are not accepted by this suite. Using them alone skips all three driver cases. With the reviewed environment supplied, run from `server/` using verified local test database settings:

```bash theme={null}
python -m pytest server/apps/dataset/tests/test_managed_storage_live.py -m integration -v -rs
```

The location JSON uses `provider`, `bucket`, `account_id`, `jurisdiction`, `secret_id`, and `secrets_region`; S3 can use its existing credential chain and region. The suite validates every selected location before resolving any credentials. It verifies conditional presigned PUT/GET, private response cache headers, Range and If-Match, HEAD, LIST, COPY, direct PUT, fixed-size multipart completion, destination collision refusal and abort, then attempts cleanup only for its random integration prefix. Its small multipart probe uses two equal 5 MiB parts and a final tail; it does not exercise the application's 64 MiB chunker or the 5 GB+ canary.

Worker conformance uses those same approved disposable upload buckets. An immutable derived-style PUT must reject an incorrect Content-MD5 without creating an object, preserve the expected private cache and provenance metadata, and refuse a competing write without changing the retained receipt or bytes. A second multipart probe invokes the actual worker `StorageProvider.upload_part` method, which supplies Content-MD5, for two equal 5 MiB parts and a four-byte tail. It compares listed part numbers, sizes and ETags, then validates the completed HEAD, an independently calculated full-body SHA-256 and an ETag-conditioned tail read. These are synthetic payloads, not parsed MCAPs or actual derived artifacts; the test does not exercise campaign reservations, checksum-job dispatch or separate derived-bucket policy/credentials.

Include the larger probe budget in each execution approval: per selected driver, the worker checks add 20 logical storage calls on a successful run, including three cleanup DELETEs, before SDK retries or failure recovery. They upload roughly 10 MiB plus three small PUT bodies and read roughly 10 MiB back. Two additional objects may exist temporarily; a third key belongs to the intentionally rejected checksum attempt and is also cleaned if a defective provider created it. Failures retain the exact known multipart handle for abort and still attempt cleanup of all owned keys. Cleanup failure fails the case; it is not proof that no test objects remain.

The [R2 S3 compatibility matrix](https://developers.cloudflare.com/r2/api/s3/api/) lists Content-MD5 support for PutObject and UploadPart, and [R2 multipart rules](https://developers.cloudflare.com/r2/objects/upload-objects/#multipart-upload-details) require equal non-final parts (checked September 14, 2026). The SDK request shapes follow [S3 PutObject](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html) and [UploadPart](https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPart.html). The suite measures this contract when approved; the documentation alone does not establish live compatibility.

Every selected disposable upload bucket must explicitly allow `http://localhost:3000`, including the S3 test bucket. The existing development R2 upload policy includes it. The suite sends three additional OPTIONS requests per driver for signed PUT, conditional Range GET and the first multipart part. It checks the actual responses with that Origin as well: PUT and every multipart part must expose ETag; private reads must expose ETag and Content-Encoding, plus Content-Range on a range response. HTTP 200 with missing or wrong CORS headers fails. These checks require the reviewed explicit private-bucket header policy and do not accept wildcard origin or header exposure. Content-Length remains signed but is browser-controlled, so it is not included in the preflight request headers. No additional object is created for CORS checks.

This tests HTTP behavior at the S3-compatible endpoint. Actual Mission Control/browser execution, canonical application origins, custom-domain delivery, cache behavior and expired-URL recovery remain separate canary evidence. Cloudflare documents that valid presigns still need CORS and expired presigns omit CORS error headers; see [R2 CORS](https://developers.cloudflare.com/r2/buckets/cors/) (checked September 12, 2026). Never infer an absent CORS policy solely from a browser's generic error on an expired URL.

Require three passing driver cases with no skips for complete driver conformance. A deliberately partial selection leaves the unselected cases skipped and the full acceptance criterion open. `--collect-only` can confirm the three node IDs without running object operations, but collection alone cannot prove that execution would avoid skips. Default-off execution, collection checks and offline synthetic routing checks are not live R2 evidence. API quota, the actual browser journey, native recovery and migration parity require separate verification.

### S3 fallback batch retention

Protocol-v2 fallback uploads retain their S3 destination snapshot even when the
organization allocation later changes. Successful finalization links the batch
and dataset in one transaction; retries return that dataset and ingest dispatch
runs after commit with reconciliation retries. S3 finalization uses the original
bucket, region and prefix and does not attach the R2 dataset client.

Abandoned fallback pins deliberately remain while their owner exists. Do not
expire or delete them on a timer: delayed retries could otherwise reuse the UUID
against R2 while S3 uploads remain in flight. Owner quota or dataset deletion
collects fallback pins through ordinary model cascades. R2 upload objects and
provider configurations retain their protective references, so stored R2 locators
cannot be collected this way. This change performs no object deletion. Migration
0249 changes model deletion behavior and is reversible to protective references;
no production migration was run.

### Managed quota counter rollout

Migration 0251 backfills the per-owner managed usage counter and installs database
triggers before applications read it. Follow-up migration 0252 adds persistent
database defaults to both counters, including databases that already applied
0251, so older workers can insert rows without knowing these fields. Apply both
migrations before rolling out the new workers. Keep these defaults during an
allocation rollback. The
triggers also account for writes from older workers during the rolling deployment. Ordinary allocation rollback still
uses the organization disable command and retains these counters and all stored
locations. No database reversal is needed for that rollback.

A separate schema reversal requires deploying the prior aggregate-reading code
first, then reversing migration 0251. Removing counter columns while the current
reader is deployed is unsafe. Production schema reversal requires explicit
approval; none was performed. The migration contains the reverse trigger and
function operations. Internal design choices are recorded in
`reports/infrastructure/r2-migration/decisions.md`.