> ## Documentation Index
> Fetch the complete documentation index at: https://avala.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Model Inference

> Connect ML models for AI-assisted annotation

Avala can call your ML models to generate pre-annotations, turning a blank canvas into a head start for human annotators. Connect a SageMaker endpoint or any custom HTTP model server, and Avala will send assets to it, receive predictions, and render them as editable annotations in the labeling editor.

## Supported Providers

| Provider                 | Status    | Description                                                                                                       |
| ------------------------ | --------- | ----------------------------------------------------------------------------------------------------------------- |
| **Amazon SageMaker**     | Available | Managed inference endpoints with IAM-based authentication. Supports SAM (Segment Anything Model) and YOLO.        |
| **Custom HTTP Endpoint** | Available | Any HTTPS server that accepts a POST request and returns predictions in Avala's format. Includes SSRF protection. |

<Tip>
  **Batch auto-labeling** is now available — run inference across entire datasets to generate draft annotations. See [Batch Auto-Labeling](/docs/integrations/batch-auto-labeling) for details.
</Tip>

<Warning>
  **Current limitations:**

  * Supported models: **SAM (Segment Anything Model)** and **YOLO** only. Multi-model support (Florence-2, RADIO) is planned.
  * Custom HTTP endpoints must use **HTTPS** and must not resolve to private/internal network addresses (SSRF protection).
</Warning>

## Amazon SageMaker Setup

### IAM Role

Create an IAM role that allows Avala to invoke your SageMaker endpoint:

```json theme={null}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "sagemaker:InvokeEndpoint",
      "Resource": "arn:aws:sagemaker:us-east-1:YOUR_ACCOUNT_ID:endpoint/your-endpoint-name"
    }
  ]
}
```

Set up a trust relationship so Avala's AWS account can assume this role. The Avala account ID is provided in Mission Control during configuration.

### Endpoint Configuration

Your SageMaker endpoint must accept image or point cloud data and return predictions in Avala's annotation format (see [Prediction Response Format](#prediction-response-format) below).

### Connect in Mission Control

1. Go to **Mission Control > Settings > Inference**.
2. Click **Add Provider** and select **Amazon SageMaker**.
3. Enter the **Endpoint Name** and **Region**.
4. Provide the **IAM Role ARN** that Avala should assume.
5. Click **Test Connection** to verify Avala can invoke the endpoint.
6. Save the configuration.

## Custom HTTP Endpoint Setup

If you are running your own model server (PyTorch Serve, Triton, BentoML, a plain Flask app, etc.), you can connect it directly.

### Request Format

Avala sends a POST request to your endpoint with the following JSON body:

```json theme={null}
{
  "asset_id": "asset_abc123",
  "asset_url": "https://signed-url-to-the-image-or-pointcloud",
  "asset_type": "image",
  "width": 1920,
  "height": 1080,
  "labels": ["car", "pedestrian", "cyclist", "truck"]
}
```

| Field        | Type      | Description                                                                                        |
| ------------ | --------- | -------------------------------------------------------------------------------------------------- |
| `asset_id`   | string    | Unique identifier for the asset.                                                                   |
| `asset_url`  | string    | Signed URL to download the asset. Valid for 1 hour.                                                |
| `asset_type` | string    | One of `image`, `point_cloud`, or `video_frame`.                                                   |
| `width`      | integer   | Width in pixels (images and video frames only).                                                    |
| `height`     | integer   | Height in pixels (images and video frames only).                                                   |
| `labels`     | string\[] | The label set configured for the project. Your model should return predictions using these labels. |

### Prediction Response Format

Your endpoint must return a JSON response with an `annotations` array:

```json theme={null}
{
  "annotations": [
    {
      "type": "bounding_box",
      "label": "car",
      "confidence": 0.94,
      "coordinates": {
        "x": 120,
        "y": 340,
        "width": 200,
        "height": 150
      }
    },
    {
      "type": "polygon",
      "label": "pedestrian",
      "confidence": 0.87,
      "points": [
        [450, 200],
        [470, 200],
        [480, 350],
        [440, 350]
      ]
    }
  ]
}
```

### Connect in Mission Control

1. Go to **Mission Control > Settings > Inference**.
2. Click **Add Provider** and select **Custom HTTP Endpoint**.
3. Enter the **Endpoint URL** (must be HTTPS).
4. Optionally configure **Authentication** (Bearer token or custom header).
5. Set the **Timeout** (default: 30 seconds).
6. Click **Test Connection** to verify Avala can reach the endpoint.
7. Save the configuration.

## Supported Prediction Types

| Type                  | Key                 | Description                                                                                                        |
| --------------------- | ------------------- | ------------------------------------------------------------------------------------------------------------------ |
| **Bounding Box**      | `bounding_box`      | Axis-aligned rectangle defined by `x`, `y`, `width`, `height`.                                                     |
| **Polygon**           | `polygon`           | Closed polygon defined by an ordered array of `[x, y]` points.                                                     |
| **Segmentation Mask** | `segmentation_mask` | Pixel-level mask as a run-length encoded (RLE) string or a URL to a PNG mask image.                                |
| **Classification**    | `classification`    | Asset-level or frame-level label with a confidence score.                                                          |
| **3D Bounding Box**   | `bounding_box_3d`   | Cuboid in 3D space defined by `center` (`x`, `y`, `z`), `dimensions` (`l`, `w`, `h`), and `rotation` (quaternion). |
| **Polyline**          | `polyline`          | Open polyline defined by an ordered array of `[x, y]` points. Used for lanes, edges, etc.                          |

## Auto-Labeling Workflow

Once a provider is connected, you can use it to pre-annotate tasks:

1. **Select a project** in Mission Control and open **Settings > Auto-Label**.
2. Choose the **Inference Provider** to use.
3. Configure the **Confidence Threshold**. Predictions below this threshold are discarded (default: 0.5).
4. Click **Run Auto-Label** to send all unlabeled assets in the project to the model.
5. Avala displays the predictions as pre-annotations in the labeling editor.
6. Annotators **review** each prediction — they can accept it as-is, adjust it, or delete it.
7. Once reviewed, the task is submitted normally through the project workflow.

You can also trigger auto-labeling via the API:

```bash theme={null}
curl -X POST https://api.avala.ai/api/v1/projects/{project_id}/auto-label \
  -H "X-Avala-Api-Key: your-api-key" \
  -H "Content-Type: application/json" \
  -d '{"provider_id": "inf_abc123", "confidence_threshold": 0.6}'
```

<Tip>
  Model predictions are always treated as suggestions. Every prediction must be
  reviewed and either accepted or corrected by a human annotator before it
  becomes a final annotation. This ensures your labeled data meets quality
  standards even when using AI assistance.
</Tip>