Avala Cloud: S3 to R2 discovery audit
Date: 2026-09-08. Source:origin/develop at 6e2a2d99d5297bbd31c71edfa96e2088fb9f3508, with recursively initialized pinned submodules. Working branch: docs/r2-discovery-audit.
Status: source audit ready for review; live inventory and migration authorization pending. No infrastructure, application behavior, feature flag, object, credential or production configuration was changed. This report does not certify production parity or readiness to cut over.
The first implementation slice is now narrowed, per the user’s follow-up, to the default Avala Cloud local-upload destination. BYOB remains unchanged. The broader inventory below records later dependencies so that a small first PR cannot accidentally claim the entire storage migration is complete. A second requested surface is the reactivated avala.ai/storage marketing page. The user prefers data.avala.ai for data serving; earlier datasets.avala.ai references describe the initial proposal, not a reserved domain. A full update is authorized for the designated Slack thread after the implementation PRs are ready, not during this audit.
1. Findings that change the plan
- Local uploads use presigned POST, not PUT. Server, browser and Python SDK depend on the POST form response. R2 requires a new upload protocol; changing the bucket or endpoint alone is insufficient. Preserve already-issued S3 grants and old clients.
- The quota premise differs from the code. Defaults are 10 GiB per personal user and 100 GiB per organization. An existing transactional quota/reservation/prefix ledger should be extended, not replaced. The requested 100 GB per user is a policy change as well as a migration requirement.
- R2 support already exists as a connector. It is not managed-storage routing, does not establish Avala ownership, and lacks the requested EU endpoint and checksum controls. Do not change BYOB to accomplish the managed default.
- Annotation history requires S3 versions. Sessions and LiDAR restore/finalization read and persist VersionId. Current-key copy alone loses product history.
- Some inference handoffs require native S3. SageMaker async input/output paths cannot be migrated by replacing a boto endpoint. Choose an invocation redesign or an explicitly approved temporary staging exception.
- Source preservation must cover existing deletion. Purge workers, lifecycle expiry, noncurrent-version expiry and overwrite jobs can invalidate a soak window without migration tooling deleting anything.
- Public visibility does not mean downloadable public bytes. The anonymous MCAP viewer deliberately uses bounded, watermarked slices. Raw open releases require separate publication eligibility and licensing review.
- Rollback has two meanings. Returning new writes to S3 is a flag change. Serving R2-only objects during an R2 outage additionally requires verified S3 replicas and URL re-resolution or edge indirection. A flag alone cannot supply missing bytes.
notes.md (in this folder; not in the site navigation); choices and rejected alternatives are in decisions.md (reports/infrastructure/r2-migration/decisions.md, internal repository inventory, not published here). The route inventory and draft copy are in marketing-proposal.md (marketing-proposal.md, in this folder; not in the site navigation).
2. Scope and evidence quality
This is Avala’s Physical AI data loop: managed sensor ingest, visualization, labeling, derived artifacts and exports. Customer S3/GCS connectors and the dedicated read-only account068760014274 are excluded. A dataset provider type, public flag, bucket name or accessible credential is not ownership proof. Each migration source needs an owner, account, class, jurisdiction and retained-generation manifest.
The whole-tree scan included hidden/ignored source, IaC, CLI instructions, workflows, SDKs, generated files, tests, historical reports and pinned submodules. It excluded credentials/environment files, Git internals and installed/build caches. It covered boto3, aws-sdk, @aws-sdk/client-s3, S3Client, s3://, presigning, multipart, S3/CloudFront domains, bucket bindings, AWS region settings, IAM, Terraform/CDK/Pulumi/CloudFormation, event/SQS references, lifecycle, Inventory/Storage Lens, CORS and CLI copies/syncs.
The scan found 2,756 broadly matching files / 22,573 lines, including 1,177 files with storage-specific signals. These are search candidates, not a count of production calls. Generic lifecycle/CORS matches, non-storage boto clients, vendored examples and historical plans are not migration targets.
Detailed internal evidence accompanies this report:
- Server/API touchpoints (
reports/infrastructure/r2-migration/api-touchpoints.md, internal repository inventory, not published here) - Browser, SDK, MCP, Sessions and pipelines touchpoints (
reports/infrastructure/r2-migration/clients-touchpoints.md, internal repository inventory, not published here) - Infrastructure, bucket and compliance evidence (
reports/infrastructure/r2-migration/infra-touchpoints.md, internal repository inventory, not published here) - Per-file search index (
reports/infrastructure/r2-migration/storage-search-index.tsv, internal repository inventory, not published here) and search methodology (reports/infrastructure/r2-migration/search-method.md, internal repository inventory, not published here)
avala-prod profile targets account 014424401899. No bucket/object listing, billing query, object download, histogram scan, benchmark, Vanta tenant query or Terraform plan was run. A separate approval request for a read-only AWS inventory capped at $5 is pending. Root/management and development account ownership, deployed versions and actual bucket bindings still need validation.
Unknown is not zero. Current GB, object counts, histograms, active object versions, delete markers, multipart uploads, live lifecycle/CORS/policies, actual CloudFront mappings, EU commitments, exact Vanta tenant mappings and recent billing remain unverified. No absolute migration cost can responsibly be certified from source alone.
3. Avala-owned bucket inventory candidates
These are IaC declarations or explicit operational references. Verify ownership and existence before use.{env} and {stage} are real deployment substitutions, not literal bucket names. Region provider means var.region, not a verified live location. For every row, current GB, current object count and size histogram are unknown. U means no explicit control found in the reviewed declaration, not disabled in production.
Historical metrics are not current inventory: the March cost report records 179 GB for the LiDAR phase-1 bucket and 6.4 GB for server media (
reports/infrastructure/COST_SAVINGS_ANALYSIS_2026_03.md:248). Do not extrapolate today’s managed total from those values or from the 5.6M-object comment.
For each live bucket collect current/noncurrent bytes and counts separately; incomplete multipart bytes; size bins <64 KiB, 64 KiB-1 MiB, 1-5 MiB, 5-100 MiB, 100 MiB-1 GiB, 1 GiB-1 TB, >1 TB; storage class and restore status; lifecycle rule filters; versions/delete markers; replication; encryption/key custody; access policy/public blocks; CORS; inventory/Storage Lens configurations; event destinations; CloudFront origins/aliases; owner and residency. Prefer an existing Inventory export for exact histograms. Daily bucket metrics cannot reconstruct a histogram or tenant ownership. Full list/version scans or export reads require an approved request budget.
4. Upload, quota and conversion paths
Existing quota behavior
server/server/settings/common.py:1649,1652 defines 10 GiB per user and 100 GiB per org. OwnerUploadQuota, OwnerUploadReservation and OwnerUploadPrefix already track usage, reservations, legacy floors, owner prefixes and reconciliation state.
Presign uses row locks and reserves declared bytes. Excess returns HTTP 413 with usage/limit detail; inability to measure safely returns 503. S3 POST policy currently enforces MIME and content-length-range. Grants last one hour and reservations are retained 24 hours. R2 PUT must replace the POST size-policy guarantee deliberately: reservation admission, validated completion, quarantine of oversized/uncommitted objects, and a hard upload-size mechanism tested against actual signed headers or an enforcing gateway. CORS is not authorization or quota enforcement.
Finalize lists the prefix, rejects confirmed emptiness with 409, saves the Dataset, then settles reservations best effort. It is not yet the requested atomic verified object record + final quota check + enqueue transaction. Nightly reconciliation scans selected active/incomplete/stale owners, not all idle tenants. Archive is quota-neutral; seven-day delayed purge credits confirmed deletion. All of these race protections must survive provider/class expansion.
The browser already renders server usage and maps 413 to quota exceeded. Distinct 80/95 percent warnings are absent. Derived bytes written under the same prefix count implicitly; external-field, exports, capture and inference buckets do not participate in a unified managed allowance. Proposed policy: separately metered and bounded org-derived allowance, explicitly approved before activation. Do not silently change the current org cap when offering 100 GB/user.
Conversion trigger findings
Local finalize saves through the dataset serializer and schedules indexing. MCAP thumbnail/proxy generation reads/writes through the dataset client (dataset/mcap/tasks.py:574,657,698); ALF/extraction pipeline uploads and registration are separate worker paths. This is useful for explicit completion dispatch, without inventing an S3 event dependency.
No automatic ROS bag/db3 -> MCAP conversion-on-local-upload implementation was found in the reviewed server/local-upload paths. Pipelines contain ROS decoding, MCAP extraction and format-specific converters, plus source-specific bag inspection. Those do not establish the requested automatic conversion job. Record its actual deployed owner/trigger or implement it as a separate approved feature. A 5 GB+ end-to-end acceptance flow cannot be claimed from existing code or a mock test.
5. Read, serve, SDK and MCP paths
Private MCAP presigns commonly last 24 hours; generic dataset URLs default to seven days. Proxy grants are approximately 60 seconds. MCAP’s immutable cache header is lifetime-bound, with a 300-second fallback on failed HEAD. The proxy emits
private, no-store. Authenticated OPFS caches are encrypted and scoped to user plus content/ETag identity; anonymous viewer cache scope is null. Preserve Range, Content-Range, ETag, If-Range and conditional headers, and measure cache invalidation when multipart ETags change. Do not globally apply a year-long private edge cache policy.
Python _uploads.py:39,99 rejects R2 and arbitrary custom upload hosts today. The public resolver separately knows R2 API endpoints but rejects the proposed datasets.avala.ai. Extend exact trusted origins and test redirects/SSRF boundaries. Existing avala.load should gain a supported downloadable manifest origin, not a duplicate loader.
dataset/mcap/public_api.py:56,133,164,173 forbids raw URLs and requires bounded watermarked slices with no-store responses. Catalog seeding defaults to external links and does not copy bytes. Keep downloadable open releases separate from catalog visibility and public previews. Use an immutable revision manifest and an explicitly eligible release for the anonymous one-line load.
MCP already uses opaque handles and current authorization checks before releasing a URL. Preserve that contract and never introduce bucket credentials or persistent private URLs in tool responses.
6. Workers, events and external handoffs
Customer and vendor sources are not owned output buckets. The internal pipeline table records these boundaries without advertising customer relationships here. The existing direct-client guard must include CLI subprocesses and framework storage backends, not just constructors. Non-S3 boto use in Sessions SDK, local filesystem search indexes, and local reconstruction staging are not storage migration work.
No separate storage-backed Slices/AutoTag or LeRobot export path was confirmed by name in the reviewed source. Dataset item metadata and generic exports are present; obtain the active job definitions and deployed producer/consumer map before claiming complete pipeline coverage. EC2 runtime scripts, external vendor services and scheduled jobs can exist outside the checkout. Multi-GB throughput and NAT/proxy processing costs were not measured.
7. Infrastructure and compliance couplings
Root Terraform already configures Cloudflare, pinned to~> 4.0 in infra/versions.tf:17. R2 resources may require a separately reviewed provider upgrade. Existing state is remote; no init/plan/apply was run. Runtime bucket bindings are injected into server, scheduler, chunk-worker and worker templates, including a literal S3 external-field base URL (infra/ecs_task.tf:394,395).
No active Inventory or Storage Lens resource was identified in the reviewed root IaC. Live presence is unknown. Manual LiDAR event notification is explicitly documented outside active IaC and retained Lambda permission proves a dependency. Current CloudFront declarations cover download/static origins; public-dataset distribution identity remains unknown. Cloudflare CDN in front of S3 is not R2.
The infra/modules/customer_aws_bridge/ boundary and customer credential flows remain unchanged. The managed asset proxy has a deliberately narrow read-only IAM role and cannot simply be given broad bucket access to compensate for missing ownership metadata.
No live Vanta tenant test export or signed EU/DPA commitments were available in this audit. These are explicit blockers to declaring compliance complete, not invented test IDs. Existing Sessions AWS KMS and S3 backup Glacier controls require a reviewed replacement or scoped exception.
Proposed legal/evidence diff, not applied: update
reports/compliance/system-description.md AWS/S3 encryption, retention and vendor sections; add Cloudflare’s object-storage purpose and jurisdictions to reviewed subprocessor disclosures; retain AWS compute/database/backup scope where still used. marketing/content/legal/privacy-policy.mdx:95,143 references AWS and US hosting; the subprocessor page uses localized legal keys. The user handles customer notices. EU storage routing must cover derived/tmp and fallback, and contracts must separately permit the location of compute processing.
8. Proposed PR sequence and approval contract
Every implementation PR includes changed behavior, tests, rollback, andKnowledge encoded:. Flags are per org and class, default off, with Avala internal as canary. No production flip, paid integration test, Terraform application, migration batch or teardown is authorized by this report. Narrowing the first slice does not remove compatibility prerequisites.
Stack only dependent contract/client layers. Marketing and independent infrastructure proposals can be separate reviews. User requested a full Slack update after the infra/server/Mission Control implementation PRs are complete; include all links if delivered as a stack. Nothing is posted yet.
9. Proposed verification and rollback requirements by phase
These are proposed controls, not runnable commands or deployed flag names. Exact commands must be written against the implemented configuration service in the implementation PRs. No current runtime flag can safely perform the complete migration today.
ETag/size/count parity is a useful gate, not a universal content proof. Multipart layouts and KMS objects make ETags unsuitable as universal hashes. A sampled hash adds evidence, not certainty for every unsampled byte. Propose capture of a cryptographic digest during the first authorized transfer and destination verification where practical, with explicit sampling coverage and residual uncertainty in
06-verification.md. Budget source re-reads separately. Do not create a verification document that implies migration has occurred before it does.
10. Ranked risks
11. One-time egress estimate and recurring baseline
Actual migration total: unknown pending live bytes, versions, classes and payer rates. The source tree cannot supply a defensible aggregate estimate. Do not use catalog-declared external dataset sizes or multiply historical object counts by a guessed mean. For an approved batch, calculatetransfer_cost = sum(billable_GB_by_region_and_tier * marginal_rate). Include only required source generations, then separately budget source retrieval/restore, requests, destination operations, verification re-reads, retries, NAT/proxy processing, and retained S3 storage during soak. Existing DR replicas must not be counted as a second logical dataset unless their unique generations are needed. Account for payer-level tier usage and unused allowances; do not assume a new free tier per bucket.
The user’s 0.09/GB range is a planning sensitivity, not a verified account rate. AWS’s S3 pricing page documents regional/tier-dependent transfer charges and a $0.09/GB internet example. Obtain current applicable region/payer tiers and discounts before approving each batch.
This table assumes the stated rate throughout for sensitivity; actual tiered totals differ. Sippy is not an exactly-once bound, so its first-access rationale is not a hard cost ceiling. Migration and integration tests remain billable work requiring explicit approval.
The last three complete months for comparison are June, July and August 2026. All three S3 storage/request/egress amounts are currently unknown. R2 projection inputs are stored GB-month by class, A/B operations, IA retrieval GB, Workers/Queues/auth-serving costs and retained AWS costs. The reviewed Standard/IA unit prices are recorded in
notes.md (in this folder; not in the site navigation). No monthly savings claim or cold-derived transition age is approved from the available evidence.