Data Connectors & Import Pipelines

Avala provides multiple ways to ingest data depending on your dataset size, infrastructure, and automation needs. This page covers each import method, when to use it, and how to build automated data pipelines.

Import methods overview

Method	Best for	Storage boundary	Automation	Setup
Mission Control upload	Interactive local imports	Owner’s managed-upload allowance	Manual	Avala account
Presigned URL upload	Programmatic uploads from any language	API-reported allowance and transfer limits	Full	API key
Cloud storage (S3/GCS)	Existing data in your own bucket	Your bucket’s access and service limits	Full	Verified storage configuration
MCAP import	Multi-sensor robotics data	Limits of the selected upload or cloud source	Full	API key or bucket configuration
SDK bulk upload	Local directory trees with progress and recovery	Owner’s managed-upload allowance	Full	SDK installed

Managed uploads use the allowance of the owning personal account or organization. The configured default organization allowance is 100 GiB, but the quota API reports the actual limit for each owner. See Upload local files for usage, reservations, and file limits. A connected customer bucket is a separate storage path; it does not receive a managed-upload allowance.

Mission Control upload

Use the Add Dataset wizard to select an owner and data type, choose Local files, and upload files accepted for that type. Wait for every selected file to finish before submitting the dataset for indexing. For directory trees or scripted recovery, use the CLI or SDK. Keep original files available while a browser upload is pending and follow any recovery prompt shown by the wizard. See the upload guide.

Presigned URL upload

Presigned URLs let you transfer files directly to the storage selected by Avala without supplying your own storage credentials. The API admits each upload against the owner’s allowance and returns the transfer contract.

How it works

Request a grant from POST /datasets/manual-upload/file-upload-url/ with the dataset name, relative file path, and byte length. Protocol v2 requests also carry upload_protocol_version: 2 and a stable dataset_upload_uid. Keep the same owner and batch identity through finalization.
Follow the returned method. A legacy POST requires the returned form fields. A managed PUT requires its URL and headers. MULTIPART requires part grants, uploads using the returned geometry, and completion with the uploaded part evidence. Do not treat every response as an S3 POST form.
Complete managed objects before finalizing the dataset through POST /datasets/manual-upload/. File transfer, dataset creation, and indexing are separate steps.

The current Python SDK and CLI handle this negotiation and recovery. For a custom client, use the interactive API reference for request and response schemas. The server selects the available storage route; a new client does not itself enable an unavailable route or increase an allowance.

Example: Upload with the CLI

pip install "avala[cli]"

# Set AVALA_API_KEY before running the command.
avala datasets upload \
  --source ./data/images \
  --name "Camera run 001" \
  --slug camera-run-001 \
  --data-type image \
  --wait

Add --organization-uid "$AVALA_ORGANIZATION_UID" for an organization-owned dataset. Resume is enabled by default. Rerun the same command after an interruption with the original source files and saved upload state. See the CLI flags and quota behavior or use client.datasets.create_from_local().

Cloud storage integration

Bring your own bucket (BYOB) when source data should remain in your S3 or GCS bucket and region. Avala indexes the existing objects in place instead of uploading another copy into managed storage. Your bucket permissions, provider limits, and dataset indexing requirements still apply.

Setup

Configure access to your bucket using the Cloud Storage guide.
Save and verify the storage configuration under the organization that will own the dataset.
Create a dataset from that configuration and, optionally, a narrower prefix within it.

Example: Import an existing bucket

For this S3 example, set AVALA_STORAGE_CONFIG_UID to your verified storage configuration, AVALA_ORGANIZATION_UID to its owning organization, and AVALA_S3_ROLE_ARN to the IAM role configured for bucket access. The API key’s user must be a member of that organization. Saved configurations supply bucket settings, but do not return credentials or the role ARN; pass the role separately for this BYOB import.

import os
from avala import Client
from avala.importers import import_cloud

with Client() as client:
    dataset = import_cloud(
        client,
        storage_config_uid=os.environ["AVALA_STORAGE_CONFIG_UID"],
        organization_uid=os.environ["AVALA_ORGANIZATION_UID"],
        role_arn=os.environ["AVALA_S3_ROLE_ARN"],
        name="Camera archive",
        slug="camera-archive",
        data_type="image",
        wait=True,
    )
    print(dataset.uid)

For local managed uploads, use avala datasets upload without --storage-config or --aws-profile. Those options do not select a destination for the managed-upload command.

MCAP Import

MCAP files contain synchronized multi-sensor data (cameras, LiDAR, IMU). Avala parses MCAP files to extract and align sensor streams for annotation.

Supported Message Types

Message Type	Description
`sensor_msgs/Image`	Camera images
`sensor_msgs/CompressedImage`	Compressed camera images
`sensor_msgs/PointCloud2`	LiDAR point clouds
`sensor_msgs/Imu`	IMU readings
`geometry_msgs/TransformStamped`	Sensor transforms (TF)
`sensor_msgs/NavSatFix`	GPS coordinates

Import Workflow

Upload MCAP files via the SDK or presigned URLs
Avala processes the file, extracting camera frames and point cloud scans
Sensor streams are synchronized by timestamp
Camera images and projected LiDAR data appear together in the annotation editor

For detailed MCAP setup, see the MCAP / ROS integration guide.

Building Import Pipelines

For production workflows, automate data ingestion so new data flows into Avala as it is collected.

Pipeline Architecture

Data Source                  Avala
┌──────────────┐            ┌─────────────────┐
│ Collection   │            │ Dataset          │
│ System       │──upload──→ │ (items created)  │
│ (cameras,    │            │                  │
│  sensors)    │            │ Project          │
└──────────────┘            │ (tasks assigned) │
                            └────────┬────────┘
                                     │
                              webhook │
                                     ▼
                            ┌─────────────────┐
                            │ Your Pipeline    │
                            │ (export, train)  │
                            └─────────────────┘

Example: Automated Ingestion with Webhooks

Combine the CLI upload with webhooks to build a fully automated pipeline:

# upload_pipeline.py
import os
import subprocess
from datetime import datetime

INDUSTRY_ID = os.environ["AVALA_INDUSTRY_ID"]
LICENSE_ID = os.environ["AVALA_LICENSE_ID"]

def ingest_batch(data_directory: str) -> str:
    """Upload a directory snapshot and create a new Avala dataset."""
    batch_name = f"camera-batch-{datetime.utcnow():%Y%m%d-%H%M%S}"

    subprocess.run(
        [
            "avala", "datasets", "upload",
            "--source", data_directory,
            "--name", batch_name,
            "--slug", batch_name,
            "--data-type", "image",
            "--industry", INDUSTRY_ID,
            "--license", LICENSE_ID,
        ],
        check=True,
    )
    return batch_name

if __name__ == "__main__":
    dataset_name = ingest_batch("/data/incoming")
    print(f"Created dataset {dataset_name}")

Schedule this script with cron, Airflow, or any task scheduler to periodically ingest new data.

Example: Watch Directory and Upload

#!/bin/bash
# watch_and_upload.sh - Upload new files as they appear

WATCH_DIR="/data/incoming"
INDUSTRY_ID="123"
LICENSE_ID="456"

inotifywait -m -e create "$WATCH_DIR" --format '%f' | while read filename; do
    if [[ "$filename" == *.jpg || "$filename" == *.png ]]; then
        dataset_name="camera-file-$(date +%Y%m%d-%H%M%S)"
        avala datasets upload \
          --source "$WATCH_DIR/$filename" \
          --name "$dataset_name" \
          --slug "$dataset_name" \
          --data-type image \
          --industry "$INDUSTRY_ID" \
          --license "$LICENSE_ID"
        echo "Created dataset from: $filename"
    fi
done

Choosing an Import Method

Use this decision tree to select the right approach:

Question	If Yes	If No
Fewer than 100 files?	Mission Control upload	Continue
Data already in S3/GCS?	Cloud storage integration	Continue
MCAP or ROS bag files?	MCAP import	Continue
Need automation?	SDK bulk upload or presigned URLs	Mission Control upload
Using Python or TypeScript?	SDK bulk upload	Presigned URL (any language)

Next Steps

Cloud Storage

Detailed S3 and GCS configuration for bring-your-own-storage.

MCAP / ROS

Import multi-sensor recordings with camera, LiDAR, and IMU data.

Python SDK

Install the Python SDK and start uploading data programmatically.

Webhooks

Set up event notifications to trigger downstream pipelines.

​Import methods overview

​Mission Control upload

​Presigned URL upload

​How it works

​Example: Upload with the CLI

​Cloud storage integration

​Setup

​Example: Import an existing bucket

​MCAP Import

​Supported Message Types

​Import Workflow

​Building Import Pipelines

​Pipeline Architecture

​Example: Automated Ingestion with Webhooks

​Example: Watch Directory and Upload

​Choosing an Import Method

​Next Steps