YOLO-people-projection

A camera watches people on a floor. The computer works out where each of them is standing, in metres, sends that out as OSC, and a projector draws on the floor exactly under their feet.

This guide shows how it works: the physical setup, the calibration that lines everything up, and how the files feed each other. The step-by-step commands are in the README.

1. How it works, in one picture

Each camera frame goes through the same chain. YOLO, an image recognition model, finds each person and their ankles. Their ankles are mapped from camera pixels to metres on the floor. A tracker follows each person from frame to frame and gives them an id. Their positions go out as OSC messages, which anything can receive. A Processing sketch draws in metres, and its drawing is warped onto the projector so it lands where people are.

Top row: track_osc.py. Bottom row: a Processing sketch (its People tab, your drawing, its Projection tab). Anything else that speaks OSC can take the bottom row's place.

2. The physical setup

You need a projector pointing at the floor, a camera that can see the whole projected area, and a rectangle of tape on the floor inside both. Nothing needs to be square on: calibration corrects for the camera's and projector's angles.

Side view. Mount the camera high and looking down: seen at a shallow angle, people's feet (and calibration markers) are squashed and hard to find.

Top view. The tape rectangle must sit inside both the projected area and the camera's view. Every position (OSC, sketches) is measured in metres from tape mark 1: x towards mark 2, y towards mark 4.

1Projector

Aim it at the floor and connect it as a second screen (extended, not mirrored). Note its resolution and its position on the desktop, e.g. 1280x800 at 1920,0.

2Camera

Mount it high, looking down, seeing all of the projected area. Any camera, size or frame rate works: a webcam, a better USB camera, or a depth camera's colour feed.

3Tape

Mark four corners of a rectangle inside both, measure its width and height in metres, and remember which corner is mark 1.

4Darkness

Calibrate with the room as dark as you can, so the camera can read the projected markers.

3. Three ways of measuring the floor

The same spot on the floor has three different addresses: a pixel in the camera's image, a pixel in the projector's image, and a position in metres from tape mark 1. Camera and projector see the floor at angles, so their pixels are stretched differently across it. Calibration measures how to convert between the three, as two homographies: maps that take any flat surface seen from one angle to the same surface seen from another.

A person found at a camera pixel becomes a position in metres (left arrow), which the tracker sends out. A sketch draws in metres, and its drawing becomes projector pixels (right arrow), so it lands under the same person.

4. Calibration

Calibration happens in two scripts, once you know which camera to use. Redo it whenever the camera or projector moves.

Choosing the camera

Cameras are chosen by number. cameras.py lists the ones the computer can see, and --snapshots saves a picture from each, to tell them apart:

$ python cameras.py --snapshots
  camera 0: 1280x720 (V4L2), average brightness 104, saved camera_0.jpg
  camera 2: 1920x1080 (V4L2), average brightness 97, saved camera_2.jpg

Then calibrate with that number. Ask for a size, frame rate or compressed frames if you want more than the camera gives by default:

python calibrate.py --source 2 --proj-pos 1920,0 --proj-size 1280x800 --floor-size 2.4x1.4

python calibrate.py --source 2 --width 1920 --height 1080 --fps 60 --fourcc MJPG \
    --proj-pos 1920,0 --proj-size 1280x800 --floor-size 2.4x1.4

The camera, its size, and any frame rate and format you asked for are saved with the calibration, so everything after that uses them without being told. A camera over HDMI (a DSLR or camcorder) works the same way, through an HDMI-to-USB capture device.

`calibrate.py`: camera, projector and floor

1Markers

The projector shows a grid of square ArUco markers, each with its own pattern. The camera finds them (press SPACE), and since every marker's projector position is known, this links camera pixels to projector pixels.

2Floor

The projector turns white. You click the four tape marks in the camera view, in order. With the rectangle's measured size, that links camera pixels to metres, and through step 1, metres to projector pixels.

3Verify

The projector draws a grid in metres. If its outline sits on your tape, press s to save.

`foot_calibrate.py`: where feet land

YOLO finds ankles, not the spot where a foot meets the floor, and from an angle the difference is several centimetres. The projector shows targets one at a time; you stand on each with the cross between your feet, and the script learns a small correction from where you were detected to where you really were. In testing, it took the error from about 30 cm to about 4 cm.

What gets saved

calibration.json
├── floor        width_m, height_m              the taped rectangle
├── cameras
│   └── cam0     source, resolution             which camera, at what size
│                fps, fourcc                    if you asked for them
│                H_cam_to_floor                 camera pixels → metres
│                foot_correction                from foot_calibrate.py
└── projectors
    └── proj0    resolution, position           which screen it is
                 H_floor_to_proj                metres → projector pixels

It's specific to your space, so it isn't part of the repository. The Processing sketches find it in their own folder or the nearest one above it.

5. Running a piece

Two programs run side by side: the tracker, and whatever turns positions into a piece.

Start them in either order. Without a camera, track_osc.py --simulate 3 sends three imaginary people wandering the floor; --simulate 2 --with-camera adds two to the real ones.

follow.py is the all-in-one test: it tracks and draws a circle under each person itself, straight onto the projector, with no sketch and no OSC needed. Use it to check calibration and tracking before running a piece.

6. The OSC messages

One message per tracked person, every time the tracker processes a frame: as often as the camera and computer allow, or as --max-fps says.

/person  id x y

Argument	Type	Meaning
`id`	int	the same while the person is tracked
`x`	float	metres from tape mark 1 along the width, 0 to W
`y`	float	metres from tape mark 1 along the height, 0 to H

Nothing is sent when someone leaves. Forget anyone you haven't heard about for most of a second. If the tracker loses someone and finds them again within a couple of seconds, close to where they were lost, they come back under the same id: keep a person that long if you want them to come back as themselves.

The README shows how to receive them in TouchDesigner, Max, Pure Data and Python.

7. How the files fit together

Arrows point from a file to the files it uses.

pipeline.py is the heart of the tracking, used by follow.py and track_osc.py; foot_calibrate.py uses its foot-finding and models.py directly. (p_track.py, a quick YOLO test, uses only cameras.py and models.py.)

File	What it does
`calibrate.py`	Markers, tape clicks and a check: makes `calibration.json`.
`foot_calibrate.py`	Targets to stand on: adds the foot correction.
`follow.py`	Circles under people, drawn straight onto the projector, to check tracking.
`track_osc.py`	The tracker for running a piece: sends OSC, no projector window. `--simulate` for imaginary people.
`pipeline.py`	One camera frame → feet (the midpoint of the ankles, or the bottom of the person's box if their ankles aren't clear) → floor metres, inside the tape → tracks.
`models.py`	Loads YOLO on this machine's best hardware: an NVIDIA card if there is one, else the CPU, with a thread limit so it leaves room for the sketch.
`tracking.py`	Matches each frame's feet to the people already known, smooths their movement, and keeps their ids when they're missed for a moment.
`cameras.py`	Frames from any camera, chosen by its number, or a video file, all the same to the rest. Run it to list your cameras.
`calib_utils.py`	The calibration maths (homographies), the marker pattern, and window helpers.
`osc.py`	Builds and sends the OSC messages, with no extra package.
`recording.py`	Keys c / p / f record the camera view, the projector image or the screen, to `recordings/`.
`processing/…/People.pde`	Receives the OSC, glides each person between messages, and fades them in and out.
`processing/…/Projection.pde`	Finds the calibration and the projector screen, and warps the sketch's metre-scaled canvas onto the floor.