Jan 27, 2026

Build face & hand tracking effects with Bolt x MediaPipe

With Bolt and Google's MediaPipe, you can create interactive apps that incorporate human tracking using nothing but a webcam. Here are four unique tools and games you can build in minutes.

Projects

You’ve probably seen the viral hand and face tracking effects taking over X in recent months. Builders are creating endlessly fun interactive experiences online.

In this tutorial, we’ll show you how to build interactive apps yourself using MediaPipe and Bolt.

MediaPipe lets you track faces, hands, and bodies in real time using nothing but a camera. Bolt, of course, lets you turn that raw capability into real, usable products fast.

This walkthrough features a total of four interactive apps:

  • A MediaPipe interactive showcase
  • A fitness rep counter
  • An air piano controlled by your fingers
  • A posture guardian and meeting engagement tracker

Watch the walkthrough with Jakub Skrzypczak, designer and Bolt.new superuser:

First: What’s MediaPipe?

MediaPipe is Google’s open-source framework for real-time human tracking. It runs entirely in the browser and gives you structured landmark data you can build on immediately.

MediaPipe supports four core detection modes:

Face

  • Tracks 468 facial landmarks
  • Use for tracking expressions, gaze, and attention

Hands

  • Tracks 21 landmarks per hand
  • Use for tracking gestures, instruments, and controls

Pose

  • Tracks 33 body landmarks
  • Use it for tracking fitness, posture, and movement

Holistic

  • Tracks face + hands + body
  • Use it for full-body interactions

Instead of raw pixels, you get coordinates, angles, and visibility scores, which serve as perfect inputs for apps.

Explore MediaPipe with an interactive showcase

MediaPipe Showcase

This showcase lets you toggle between:

  • Face detection: expressions, head rotation, blinking
  • Hand tracking: open palms, fists, finger positions
  • Pose detection: arms, legs, squats
  • Holistic mode: all of the above at once

You’ll quickly notice which angles are reliable and which aren’t (for example, flat palms facing the camera can get glitchy). It’ll help you decide how to design interactions in your app.

App #1: Fitness rep counter (pose tracking)

You can have a lot of fun with interactive apps. But you can also build things that are genuinely useful. This Fitness Tracker is one such example.

Fitness Tracker

Prompt

Build a clean fitness rep counter for basic exercises using MediaPipe Pose detection.

=== CORE CONCEPT ===

Simple, focused fitness app. Pick an exercise, do it, watch your reps get counted automatically with real-time form feedback. Full-screen camera view with minimal UI.

=== TECH STACK ===

React + TypeScript + Vite MediaPipe Pose Landmarker Tailwind CSS

=== EXERCISES (Pick 3-4 Simple Ones) ===

Push-ups (easiest to detect) Track elbow angle Count: Straight arms → bent elbows → straight arms

Squats (clear movement) Track knee and hip angles Count: Standing → deep squat → standing

Jumping Jacks (fun to track) Track arm height and leg spread Count: Arms down/legs together → arms up/legs apart → back

Arm Raises (simple) Track shoulder to wrist height Count: Arms down → arms raised above shoulder → back down

=== USER FLOW ===

Setup Screen:

Title: "FITNESS TRACKER" Exercise selection (large buttons with icons) Target reps: Quick buttons (10 / 15 / 20) `