> ## Documentation Index
> Fetch the complete documentation index at: https://avala.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Data Connectors & Import Pipelines

> Get data into Avala using uploads, cloud storage, APIs, and presigned URLs

Avala provides multiple ways to ingest data depending on your dataset size, infrastructure, and automation needs. This page covers each import method, when to use it, and how to build automated data pipelines.

## Import Methods Overview

| Method                     | Best For                               | Max Size       | Automation | Setup         |
| -------------------------- | -------------------------------------- | -------------- | ---------- | ------------- |
| **Mission Control upload** | Small datasets, one-off imports        | 10 GB per user | Manual     | None          |
| **Presigned URL upload**   | Programmatic uploads from any language | 10 GB per user | Full       | API key       |
| **Cloud storage (S3/GCS)** | Large datasets, zero-copy access       | Unlimited      | Full       | Bucket config |
| **MCAP import**            | Multi-sensor robotics data             | 10 GB per file | Full       | API key       |
| **SDK bulk upload**        | Medium datasets with progress tracking | 10 GB per user | Full       | SDK installed |

## Mission Control Upload

The simplest way to get data into Avala. Drag and drop files directly in the web interface.

### Steps

1. Go to **Mission Control** > **Datasets** > **Create Dataset**
2. Name your dataset and select the data type
3. Drag files into the upload area or click **Browse**
4. Wait for processing to complete

### Limitations

* Browser-based upload is limited by your connection speed and browser memory
* Not suitable for datasets with more than 1,000 files
* No resumable uploads — interrupted uploads must restart

<Tip>
  For datasets larger than a few hundred files, use the SDK or presigned URL approach instead.
</Tip>

## Presigned URL Upload

Presigned URLs let you upload files directly to Avala's storage from any HTTP client. This is the most flexible programmatic upload method and works from any language or tool that can make HTTP requests.

### How It Works

1. Request a presigned upload URL from the Avala API
2. Upload your file directly to the presigned POST URL
3. Create the dataset from the uploaded files

### Example: Upload with cURL

```bash theme={null}
# Step 1: Get a presigned upload URL
curl -X POST https://api.avala.ai/api/v1/datasets/manual-upload/file-upload-url/ \
  -H "X-Avala-Api-Key: your-api-key" \
  -H "Content-Type: application/json" \
  -d '{
    "dataset_name": "robot-run-001",
    "file_path_in_dataset": "frame_001.jpg",
    "content_length": 1024000
  }'

# Response:
# { "method": "POST", "url": "https://s3.amazonaws.com/...", "fields": { ... } }

# Step 2: Upload the file with the returned POST fields
curl -X POST "https://s3.amazonaws.com/..." \
  -F "key=..." \
  -F "policy=..." \
  -F "file=@frame_001.jpg"

# Step 3: Create the dataset from uploaded files
curl -X POST https://api.avala.ai/api/v1/datasets/manual-upload/ \
  -H "X-Avala-Api-Key: your-api-key" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "robot-run-001",
    "slug": "robot-run-001",
    "data_type": "image",
    "visibility": "private",
    "industry": 123,
    "license": 456
  }'
```

### Example: Upload with the CLI

```bash theme={null}
# Handles presigned URLs, direct storage upload, and dataset creation.
# Limit: 10 GiB total per local-upload dataset.
avala datasets upload \
  --source data/images \
  --name robot-run-001 \
  --slug robot-run-001 \
  --data-type image \
  --industry 123 \
  --license 456
```

## Cloud Storage Integration

For large-scale datasets, connect your own S3 or GCS bucket so Avala reads data directly from your storage — no file transfers, no copies.

### When to Use Cloud Storage

| Scenario                               | Use Cloud Storage?                     |
| -------------------------------------- | -------------------------------------- |
| Dataset > 10,000 items                 | Yes                                    |
| Dataset > 100 GB total                 | Yes                                    |
| Data must stay in your infrastructure  | Yes                                    |
| Quick prototype with \< 100 items      | No — direct upload is faster           |
| Data is spread across multiple buckets | Yes — connect multiple storage configs |

### Setup

1. Configure your bucket with the appropriate IAM policy (see [Cloud Storage guide](/docs/integrations/cloud-storage))
2. Add the storage configuration in **Mission Control** > **Settings** > **Storage**
3. Create a dataset and select your connected storage as the data source
4. Reference items by their storage paths

### Example: Create Dataset from S3

```python theme={null}
from avala import Client

client = Client()

# Create a dataset backed by cloud storage
dataset = client.datasets.create(
    name="driving-data-2026-02",
    data_type="image",
    storage_config_uid="stg_your_config_uid"
)

# Register items by their S3 paths
items = [
    {"path": "s3://your-bucket/captures/frame_001.jpg"},
    {"path": "s3://your-bucket/captures/frame_002.jpg"},
    {"path": "s3://your-bucket/captures/frame_003.jpg"},
]

for item in items:
    client.datasets.create_item(
        dataset_uid=dataset.uid,
        source_url=item["path"]
    )
```

<Tip>
  Cloud storage datasets load faster in the annotation editor because images are served directly from your bucket's region, avoiding cross-region transfers.
</Tip>

## MCAP Import

MCAP files contain synchronized multi-sensor data (cameras, LiDAR, IMU). Avala parses MCAP files to extract and align sensor streams for annotation.

### Supported Message Types

| Message Type                     | Description              |
| -------------------------------- | ------------------------ |
| `sensor_msgs/Image`              | Camera images            |
| `sensor_msgs/CompressedImage`    | Compressed camera images |
| `sensor_msgs/PointCloud2`        | LiDAR point clouds       |
| `sensor_msgs/Imu`                | IMU readings             |
| `geometry_msgs/TransformStamped` | Sensor transforms (TF)   |
| `sensor_msgs/NavSatFix`          | GPS coordinates          |

### Import Workflow

1. Upload MCAP files via the SDK or presigned URLs
2. Avala processes the file, extracting camera frames and point cloud scans
3. Sensor streams are synchronized by timestamp
4. Camera images and projected LiDAR data appear together in the annotation editor

For detailed MCAP setup, see the [MCAP / ROS integration guide](/docs/integrations/mcap-ros).

## Building Import Pipelines

For production workflows, automate data ingestion so new data flows into Avala as it is collected.

### Pipeline Architecture

```
Data Source                  Avala
┌──────────────┐            ┌─────────────────┐
│ Collection   │            │ Dataset          │
│ System       │──upload──→ │ (items created)  │
│ (cameras,    │            │                  │
│  sensors)    │            │ Project          │
└──────────────┘            │ (tasks assigned) │
                            └────────┬────────┘
                                     │
                              webhook │
                                     ▼
                            ┌─────────────────┐
                            │ Your Pipeline    │
                            │ (export, train)  │
                            └─────────────────┘
```

### Example: Automated Ingestion with Webhooks

Combine the CLI upload with [webhooks](/docs/integrations/webhooks) to build a fully automated pipeline:

```python theme={null}
# upload_pipeline.py
import os
import subprocess
from datetime import datetime

INDUSTRY_ID = os.environ["AVALA_INDUSTRY_ID"]
LICENSE_ID = os.environ["AVALA_LICENSE_ID"]

def ingest_batch(data_directory: str) -> str:
    """Upload a directory snapshot and create a new Avala dataset."""
    batch_name = f"camera-batch-{datetime.utcnow():%Y%m%d-%H%M%S}"

    subprocess.run(
        [
            "avala", "datasets", "upload",
            "--source", data_directory,
            "--name", batch_name,
            "--slug", batch_name,
            "--data-type", "image",
            "--industry", INDUSTRY_ID,
            "--license", LICENSE_ID,
        ],
        check=True,
    )
    return batch_name

if __name__ == "__main__":
    dataset_name = ingest_batch("/data/incoming")
    print(f"Created dataset {dataset_name}")
```

Schedule this script with cron, Airflow, or any task scheduler to periodically ingest new data.

### Example: Watch Directory and Upload

```bash theme={null}
#!/bin/bash
# watch_and_upload.sh - Upload new files as they appear

WATCH_DIR="/data/incoming"
INDUSTRY_ID="123"
LICENSE_ID="456"

inotifywait -m -e create "$WATCH_DIR" --format '%f' | while read filename; do
    if [[ "$filename" == *.jpg || "$filename" == *.png ]]; then
        dataset_name="camera-file-$(date +%Y%m%d-%H%M%S)"
        avala datasets upload \
          --source "$WATCH_DIR/$filename" \
          --name "$dataset_name" \
          --slug "$dataset_name" \
          --data-type image \
          --industry "$INDUSTRY_ID" \
          --license "$LICENSE_ID"
        echo "Created dataset from: $filename"
    fi
done
```

## Choosing an Import Method

Use this decision tree to select the right approach:

| Question                    | If Yes                            | If No                        |
| --------------------------- | --------------------------------- | ---------------------------- |
| Fewer than 100 files?       | Mission Control upload            | Continue                     |
| Data already in S3/GCS?     | Cloud storage integration         | Continue                     |
| MCAP or ROS bag files?      | MCAP import                       | Continue                     |
| Need automation?            | SDK bulk upload or presigned URLs | Mission Control upload       |
| Using Python or TypeScript? | SDK bulk upload                   | Presigned URL (any language) |

## Next Steps

<CardGroup cols={2}>
  <Card title="" icon="cloud" href="/docs/integrations/cloud-storage">
    <p style={{fontWeight: 600, fontSize: '18px', marginBottom: '4px', marginTop: '8px', color: 'inherit'}}>Cloud Storage</p>
    <p style={{fontSize: '14px', marginTop: '0px', opacity: 0.6}}>Detailed S3 and GCS configuration for bring-your-own-storage.</p>
  </Card>

  <Card title="" icon="robot" href="/docs/integrations/mcap-ros">
    <p style={{fontWeight: 600, fontSize: '18px', marginBottom: '4px', marginTop: '8px', color: 'inherit'}}>MCAP / ROS</p>
    <p style={{fontSize: '14px', marginTop: '0px', opacity: 0.6}}>Import multi-sensor recordings with camera, LiDAR, and IMU data.</p>
  </Card>

  <Card title="" icon="python" href="/docs/sdks/python">
    <p style={{fontWeight: 600, fontSize: '18px', marginBottom: '4px', marginTop: '8px', color: 'inherit'}}>Python SDK</p>
    <p style={{fontSize: '14px', marginTop: '0px', opacity: 0.6}}>Install the Python SDK and start uploading data programmatically.</p>
  </Card>

  <Card title="" icon="bell" href="/docs/integrations/webhooks">
    <p style={{fontWeight: 600, fontSize: '18px', marginBottom: '4px', marginTop: '8px', color: 'inherit'}}>Webhooks</p>
    <p style={{fontSize: '14px', marginTop: '0px', opacity: 0.6}}>Set up event notifications to trigger downstream pipelines.</p>
  </Card>
</CardGroup>