> ## Documentation Index
> Fetch the complete documentation index at: https://avala.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Camera Projection
> Pinhole and double-sphere camera models for LiDAR-to-camera projection
Camera projection maps 3D LiDAR points onto 2D camera images. This is used for image projection visualization, camera-LiDAR overlay in the point cloud panel, and multi-camera verification of 3D annotations. Avala supports two camera models: pinhole (standard perspective) and double-sphere (wide-angle and fisheye lenses).
## Projection Workflow
Projecting a 3D point onto a camera image involves three steps:
The 3D point is transformed from the LiDAR frame into the camera's coordinate system using the extrinsic calibration (rotation and translation from TF messages or calibration data in the recording).
The transformed 3D point is projected onto the 2D image plane using the camera's intrinsic model -- either pinhole or double-sphere. This produces normalized image coordinates.
The normalized coordinates are scaled to pixel coordinates using the focal length and principal point. For pinhole cameras, distortion correction is not yet applied (see note below).
## Pinhole Camera Model
The pinhole model represents a standard perspective camera. It is the most common model for automotive and robotics cameras with moderate fields of view (up to approximately 120 degrees).
### Parameters
| Parameter | Symbol | Description |
| --------------------- | ---------------------- | --------------------------------------- |
| Focal length X | `fx` | Horizontal focal length in pixels |
| Focal length Y | `fy` | Vertical focal length in pixels |
| Principal point X | `cx` | Horizontal optical center in pixels |
| Principal point Y | `cy` | Vertical optical center in pixels |
| Radial distortion | `k1`, `k2`, `k3`, `k4` | Lens radial distortion coefficients |
| Tangential distortion | `p1`, `p2` | Lens tangential distortion coefficients |
### How It Works
For a 3D point `(X, Y, Z)` in camera coordinates:
1. **Normalize** -- Divide by depth: `x = X / Z`, `y = Y / Z`
2. **Map to pixels** -- `u = fx * x + cx`, `v = fy * y + cy`
Lens distortion correction using the radial (`k1`--`k4`) and tangential (`p1`, `p2`) coefficients is not yet applied in the pinhole projection path. The distortion parameters are stored and will be used in a future update. For most use cases with low-distortion lenses, the ideal pinhole projection (without distortion correction) provides accurate results.
The result `(u, v)` is the pixel location on the image where the 3D point appears.
Points behind the camera (`Z <= 0`) are rejected before projection. Points that project outside the image boundaries are also discarded.
## Double-Sphere Camera Model
The double-sphere model handles wide-angle and fisheye lenses with fields of view exceeding 180 degrees. It uses two additional parameters beyond the standard focal length and principal point to model the extreme distortion of fisheye optics.
### Parameters
| Parameter | Symbol | Description |
| ----------------- | ------- | -------------------------------------------------------------- |
| Focal length X | `fx` | Horizontal focal length in pixels |
| Focal length Y | `fy` | Vertical focal length in pixels |
| Principal point X | `cx` | Horizontal optical center in pixels |
| Principal point Y | `cy` | Vertical optical center in pixels |
| Xi | `xi` | First sphere parameter (controls fisheye distortion strength) |
| Alpha | `alpha` | Second sphere parameter (balances the two spheres, range 0--1) |
### How It Works
The double-sphere model projects points through two conceptual spheres, which naturally handles the extreme bending of light rays in fisheye lenses:
1. **First sphere** -- Compute the distance from the origin to the 3D point: `d1 = sqrt(X^2 + Y^2 + Z^2)`. This maps the point onto a unit sphere.
2. **Shift by xi** -- Offset along the optical axis: `xi * d1 + Z`. The `xi` parameter controls how strongly the fisheye effect warps the projection.
3. **Second sphere** -- Compute a second radial distance incorporating the shifted coordinate. This second sphere adds another layer of distortion modeling.
4. **Blend with alpha** -- The `alpha` parameter blends the contributions of the two spheres. When `alpha = 0`, the model reduces to a single sphere; when `alpha = 1`, the second sphere dominates.
5. **Map to pixels** -- The blended result is scaled by focal length and offset by the principal point, producing the final pixel coordinates `(u, v)`.
The double-sphere model is a strong choice for fisheye lenses because it handles fields of view beyond 180 degrees with just two extra parameters. It avoids the numerical instability of polynomial distortion models at extreme angles.
## Multi-Camera Setup
Recordings often include multiple cameras covering different directions (front, left, right, rear). Avala handles multi-camera projection automatically.
### Calibration Data
Camera calibration is extracted from the recording's transform (TF) messages:
| Source | Data Provided |
| ------------------------- | ------------------------------------------------------------------------------- |
| `tf2_msgs/TFMessage` | Extrinsics: position and orientation of each camera relative to the LiDAR frame |
| `foxglove.FrameTransform` | Extrinsics: same as above, Foxglove schema |
| Camera info messages | Intrinsics: focal length, principal point, distortion model and coefficients |
The system builds a transform tree from all TF messages in the recording, then resolves the transformation chain between each camera frame and the LiDAR frame.
### Camera Selection
When projecting a point onto an image (as in image projection visualization mode), the system must choose which camera to use. The selection process:
1. For each point, project it into every available camera's image plane
2. Discard cameras where the point falls outside the image bounds or depth range
3. Among the remaining cameras, select the one where the projected pixel is closest to the image's principal point (optical center)
4. Sample the pixel color at the projected location
This principal-point-proximity strategy favors cameras that see the point near the center of the image, where lens distortion is lowest and image quality is highest. It produces seamless color mapping across the full 360-degree field of view when enough cameras are present.
### Supported Configurations
| Configuration | Cameras | Typical Use |
| ------------------- | ------- | -------------------------------------- |
| Single front camera | 1 | Basic forward-facing setups |
| Stereo pair | 2 | Stereo vision with depth estimation |
| Surround view | 4--6 | Autonomous vehicles with full coverage |
| Surround + fisheye | 6--10 | Mixed pinhole and fisheye rigs |
Camera projection requires valid calibration data in the recording. If calibration is missing or incorrect, the projection will be misaligned. Verify calibration accuracy by checking that projected point cloud edges align with object boundaries in the camera images.
## Verifying Calibration Quality
The camera projection overlay is one of the most effective tools for checking calibration:
1. Enable **Image Projection** mode in the point cloud panel
2. Open the camera image panels alongside the 3D view
3. Check that point cloud silhouettes align with object edges in the images
4. Misalignment indicates calibration errors -- check extrinsics (rotation/translation) first, then intrinsics (focal length, distortion)
## Related
* [Visualization Modes](/docs/visualization/rendering/visualization-modes) -- Image projection and other coloring modes for point clouds
* [Image Panel](/docs/visualization/panels/image-panel) -- Camera image display with LiDAR overlay
* [Point Cloud Panel](/docs/visualization/panels/point-cloud-panel) -- 3D viewer with projection overlay controls