Guidance request — hand landmark detection on black assistive rehabilitation gloves

[Paul Oladapo]

Dear MediaPipe team,

My name is Paul Oladapo, and I am a thesis-based Master's student in Mechatronics Engineering at Ontario Tech University (Canada), working under the supervision of Dr. Aaron Yurkewich.

My thesis focuses on soft robotic gloves used as assistive devices for hand and finger joint rehabilitation in people with reduced mobility. The goal is to use computer vision to estimate the 21 hand landmarks and joint angles while the glove is worn, and to feed those measurements into an interactive rehabilitation game: the game adapts its difficulty and the glove adjusts its assistance level based on the measured joint angles, providing partial mobility assistance during gameplay.

The challenge is that MediaPipe Hands, which works impressively on bare hands, does not detect the hand when our textureless black glove is worn — understandably, since the glove removes the skin texture and color cues the model relies on. To work around this, I have been building a custom 21-landmark CNN (EfficientNet backbone, heatmap regression with a hierarchical wrist → palm → per-finger cascade), trained on ~600 self-captured frames, using white fiducial rings on the glove as physical ground truth for labeling and inference-time refinement. Results are promising but still far from MediaPipe's robustness, largely due to limited pose and scene diversity in my dataset.

I would be very grateful for any advice you could offer, in particular:

1. Is there a supported path to fine-tune or retrain the MediaPipe Hand Landmarker on custom data (e.g., gloved hands)? If not, is the training pipeline or dataset specification documented anywhere so I could replicate the approach?
2. What dataset size and diversity (poses, viewpoints, lighting, backgrounds) would you consider a realistic minimum for robust gloved-hand landmark detection?
3. Are there domain-adaptation or data-augmentation strategies you found especially effective for appearance shifts like ours (bare hand → black glove)?
4. Do you have any recommendations on combining physical fiducial markers with a learned landmark model, or prior work in that direction you would point me to?
5. Is there any planned or existing MediaPipe support for hands with gloves or hand-worn devices?

Any pointers, documentation, or prior art you could share would be extremely valuable for this assistive-technology application. I would of course be happy to share our results with the community.

Thank you very much for your time and for making MediaPipe available to researchers.

Best regards, 

Paul Oladapo

Research Assistant, Mechatronics Engineering

HERO Gloves | KeyGloves

Master of Applied Science

Ontario Tech University

[José Daniel Sarmiento]

Hi Paul,

This is a fantastic project—using computer vision for adaptive assistive rehabilitation is a great application of human-computer interaction and robotics.

Regarding your questions on MediaPipe and tracking gloved hands:

1. Retraining / Fine-Tuning MediaPipe Hand Landmarker

• Model Maker Limitations: MediaPipe Tasks Model Maker supports fine-tuning for gesture classification, but the core 21-landmark regression pipeline (Palm Detector + Landmark Model) is generally not exposed for end-to-end retraining out of the box.

• Alternative Path: Since the open-source TFLite backbone architectures (BlazePalm / Hand Landmark) are available, you can re-implement the landmark head architecture in PyTorch/TensorFlow, load the feature extractor weights, and fine-tune or train the regressor on your domain-specific dataset.

2. Dataset Size & Diversity Requirements

• Volume: 600 frames is a solid start for proof-of-concept, but deep landmark models usually require at least 5,000–10,000 diverse annotated frames to generalize well across lighting, background, and minor marker occlusions.

• Synthetic Data: To bridge the gap without manually labeling thousands of images, consider generating synthetic gloved hands using 3D models (e.g., using Blender with the MANO hand model modified with glove textures and white rings).

3. Domain Adaptation & Data Augmentations

• Synthetic-to-Real Transfer: Blend synthetic gloved images with real-world data.

• Heavy Augmentation: Apply aggressive color/brightness jitter, random shadows, background swapping, and contrast adjustments to force the model to focus on geometric structure and white fiducial rings rather than skin contrast.

• Sim2Real / GANs: Image-to-image translation (like CycleGAN) to map bare-hand datasets (e.g., FreiHAND or InterHand2.6M) into the "black glove + white marker" domain can quickly scale your training set.

4. Combining Physical Markers with Learned Models

• Classical CV + Deep Learning: Since you already have white fiducial rings, consider using high-speed Classical CV (e.g., OpenCV blob detectors or color/contour tracking) to extract precise 2D marker centroids. You can use these detections to continuously refine or constrain the CNN’s heatmaps/predictions.

• Kinematic Constraints: Post-process landmark predictions using forward kinematics / inverse kinematics (IK) or a Kalman Filter constrained by the fixed bone lengths of the human hand to prevent unrealistic joint deformations.

• Automated Auto-Labeling: Use the fiducial markers + multi-camera setup (if available) to build an automated pseudo-labeling pipeline for unannotated video sequences.

5. Official MediaPipe Gloved Support

• There is currently no official or publicly announced MediaPipe support for gloved or heavily covered hands, as the core models are primarily optimized for RGB bare skin cues in consumer environments.

Best of luck with your research at Ontario Tech! Feel free to keep the community posted on your progress.

Best regards,

Jose Sarmiento

--
You received this message because you are subscribed to the Google Groups "MediaPipe" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mediapipe+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/mediapipe/CADT%2BigUASg-pAiGgCzu8u0a%3DnpagV2LB6QEyV3NvX6niKasbmw%40mail.gmail.com.

--

Cordialmente,

José Daniel Sarmiento Blanco

Estudiante de Ingenieria Mecanica UIS

Codigo:2192232

[Paul Oladapo]

Hi Jose, 

Thank you for the information. It was very helpful!

Have a great week.

Best regards, 

Paul Oladapo

Research Assistant, Mechatronics Engineering

HERO Gloves | KeyGloves

Master of Applied Science

Ontario Tech University