[DMP 2025]: Hand Gesture Occlusion - 3D Hand Landmark Extraction and Gesture Similarity Pipeline #724
Description
Ticket Contents
Description
This project aims to develop a scalable pipeline for extracting 3D hand landmarks from RGB images and comparing hand shapes or gestures using point cloud similarity metrics. By leveraging MediaPipe for hand landmark detection and PyTorch3D for Chamfer Distance-based comparisons, the system enables quantitative analysis of hand motion and gesture similarity. The pipeline supports batch processing, GPU acceleration, and scalable evaluation of gesture datasets, making it suitable for applications in human-computer interaction and gesture recognition.
Goals & Mid-Point Milestone
Goals
Extract 3D hand landmarks from RGB images using MediaPipe.
Convert extracted landmarks into 3D point clouds (.ply format) using Open3D.
Compute pairwise Chamfer Distances to quantify gesture similarity.
Enable parallel processing and GPU-accelerated computations.
Export results in CSV format for downstream analysis.
Create histograms to visualize Chamfer Distance distribution patterns for further analysis.
Iteratively explore and evaluate multiple approaches to develop an optimal and efficient pipeline for gesture similarity recognition.
Goals Achieved By Mid-point Milestone
Successful extraction and export of .ply point clouds from RGB image batches.
Basic implementation of Chamfer Distance comparison module between two sets of point clouds.
Initial test runs validating point cloud generation and distance metrics on sample data.
Setup/Installation
Python-based backend using modular scripts and Jupyter notebooks.
Install required libraries:
MediaPipe for hand landmark detection.
Open3D for point cloud manipulation and export.
PyTorch3D for Chamfer Distance computation.
Joblib and multiprocessing for parallel execution.
NumPy and OpenCV for data processing.
Expected Outcome
The final product will be a reusable, scalable pipeline for generating and comparing 3D hand shape data from RGB images. The pipeline will: Generate 3D hand landmark point clouds and export them as .ply files with accurate geometry.
Compute pairwise Chamfer Distances between point clouds to quantify gesture similarity.
Produce CSV files summarizing similarity metrics (e.g., trial1_ply, trial2_ply, chamfer_distance).
Provide visualizations of hand point clouds using Open3D’s draw_geometries() for qualitative inspection.
Support efficient batch-level processing for large image datasets.
Accurately classify gesture similarity based on robust Chamfer Distance metrics and histogram-based analysis of distance patterns.
Acceptance Criteria
Generation of 3D hand landmark point clouds from RGB images.
Export to .ply files with correct geometry.
Pairwise Chamfer Distance computation between two trials.
Parallelized execution for scalability.
CSV output summarizing similarity metrics.
Implementation Details
Languages/Frameworks: Python, MediaPipe, Open3D, PyTorch3D, Joblib, NumPy, OpenCV.
Data Sources: Local image folders (data collected using depth cameras).
Core Modules: Landmark extraction using MediaPipe.
Point cloud creation and visualization using Open3D.
Chamfer Distance computation via PyTorch3D using k-NN and ICP-based point matching.
Parallel execution using multiprocessing or DataParallel.
Outputs: 3D point clouds as .ply files (21 keypoints per hand image).
CSV file with columns: trial1_ply, trial2_ply, chamfer_distance.
Histograms and histogram analysis reports representing Chamfer Distance patterns across the dataset.
Mockups/Wireframes
Visualizations of 3D point clouds using Open3D’s draw_geometries() function.
Product Name
Hand Point Cloud Comparison Pipeline
Organisation Name
Bandhu
Domain
Healthcare
Tech Skills Needed
Python, Other
Mentor(s)
Kirti Lakra, Manshul Belani, Pushpendra Singh Category
Computer Vision
HCI
Gesture Recognition
Category
Other