Support Camera Depth

[MichealReed]

1. Pixel Format Enum (MiniAVPixelFormat):

   • Add new formats to represent depth data, specifying the data type and units (e.g., MINIAV_PIXEL_FORMAT_DEPTH_U16_MM for unsigned 16-bit depth in millimeters, MINIAV_PIXEL_FORMAT_DEPTH_F32_M for 32-bit float depth in meters).
   • Consider formats for confidence maps if provided by the hardware.

2. Buffer Structure (MiniAVBuffer):

   • Option A (Preferred for Synchronization): Modify the video struct within the MiniAVBuffer union to potentially hold both color and depth data if captured simultaneously from the same device. This might involve:
     • Adding fields like depth_width, depth_height, depth_pixel_format, depth_stride_bytes, depth_planes.
     • Ensuring the single timestamp_us applies to both synchronized frames.
     • The internal_handle would need to manage the release of both buffers.
   • Option B (Simpler, Less Synchronized): Introduce a new MiniAVBufferType like MINIAV_BUFFER_TYPE_DEPTH and potentially a separate depth struct in the union. This might be simpler but makes explicit synchronization between color and depth streams (if needed) the responsibility of the application using timestamps.

3. Metadata:

   • Intrinsics: Depth cameras require specific intrinsics (focal length, principal point) which might differ from the color sensor's intrinsics, especially on multi-sensor devices. The API needs a way to expose both color and depth intrinsics, potentially linked to the buffer.
   • Extrinsics: If a device has separate color and depth sensors, the rigid body transformation (rotation + translation) between them (extrinsics) is needed for alignment. The API should provide a way to query this.
   • Depth Scale/Units: Explicitly define the units for depth formats (e.g., via the format enum name or a separate metadata field) to allow correct interpretation.

4. API Functions:

   • MiniAV_Camera_GetSupportedFormats: Needs to be updated to report supported depth formats and potentially combinations (e.g., Color 1080p@30 + Depth 480p@30). The MiniAVVideoFormatInfo struct might need expansion.
   • MiniAV_Camera_Configure: Needs to accept requests for depth streams, potentially alongside color streams.
   • (Optional) New functions might be needed to query depth-specific properties or intrinsics/extrinsics if not handled via the generic property API.

5. Platform Implementations (miniav_c/src/camera/<platform>):

   • Windows: Investigate Media Foundation extensions or vendor-specific SDKs (e.g., Intel RealSense SDK, Azure Kinect SDK).
   • macOS: AVFoundation might offer some depth capabilities on newer hardware (e.g., TrueDepth). Vendor SDKs might be needed otherwise.
   • Linux: V4L2 might expose some depth formats. Vendor SDKs are common (RealSense librealsense).
   • Android: Camera2 API (DEPTH16, DEPTH_POINT_CLOUD).
   • iOS: AVFoundation (AVCaptureDepthDataOutput).

Tasks

• Define depth-related MiniAVPixelFormat enums.
• Decide on and update the MiniAVBuffer structure (Option A or B).
• Design mechanism for exposing depth/color intrinsics and extrinsics.
• Update MiniAV_Camera_GetSupportedFormats and MiniAV_Camera_Configure APIs.
• Implement depth capture for at least one platform (e.g., Windows using RealSense SDK or iOS/Android using native APIs).
• Add platform implementations for other systems.
• Ensure MiniAV_ReleaseBuffer correctly handles releasing depth resources.
• Update documentation.
• Add examples/tests for depth capture.

Considerations

• Synchronization: Ensuring tight timestamp synchronization between color and depth frames from the same device is critical for many applications (favors Option A for MiniAVBuffer).
• Performance: Depth processing can be demanding; maintaining zero-copy is important.
• API Complexity: Adding depth increases API surface area and complexity.
• Vendor SDKs: Relying on vendor SDKs adds external dependencies and build complexity. Prioritize native OS APIs where available and sufficient.
• Format Variety: Depth cameras output various formats (raw disparity, point clouds, IR); focus initially on common depth map formats (e.g., 16-bit unsigned integer).