Workflows

AI Mask

Set up the AI runtime, then mask anything in your shot by describing it in words.

AI Mask finds things by name. Type person, pick a frame where they're visible, and it tracks them through the rest of the shot on its own.

The usual reason to want this is a solve that keeps drifting. Anything moving on its own gives the solver features that contradict the camera motion, and masking it out fixes that in one step.

First-Time Setup

AI Mask is the only part of Dope Track that downloads anything. Two pieces, and the node asks for them in order.

1. Install AI Runtime (about 2 GB). This is PyTorch and the libraries around it. Connect an Image Sequence to the node and the button appears. It takes a while and prints progress to the console.

Restart Blender when it finishes. The runtime can't load into a session that already started without it.

2. Download AI Model (about 900 MB). The segmentation model itself. Same idea, a button on the node.

Both are one-time. After that the node is ready in every project.

Writing A Prompt

Type what you want to find. Separate multiple things with periods:

person
person. car
red bag. dog

Singular words work better than plurals. person finds people more reliably than people does. Commas get converted to periods for you, so either is fine.

Be specific when the shot is busy. red bag will pick out one bag where bag might grab three.

Preview Before You Commit

Press Preview Mask and you get one frame back with the mask tinted over it. Change the tint color and opacity next to the buttons if green doesn't read well against your footage.

Preview is fast. Generating the whole sequence is not. Get the prompt right on one frame first.

If nothing is found, lower Box Threshold a little. If it's grabbing things you don't want, raise it. Text Threshold works the same way but for how closely something has to match your words. Both start at 0.25, and moving them by 0.05 at a time is usually enough.

Max Boxes caps how many separate things get masked. Lower it when the prompt is catching extras in the background.

Anchor Frames

An anchor frame is where the AI actually looks for your subject. From there it propagates forward and backward through the shot.

Single frame uses one anchor. Leave Current Frame ticked and it follows your playhead, or untick it and type a frame number. This is the right choice most of the time.

Multi frame lets you give it several. Add them by typing numbers, or scrub to a frame and press Add Current Frame.

40, 90, 130

When To Use Multi

Single-frame detection assumes what it saw at the anchor stays recognizable. Reach for multi-frame when that stops being true:

• The subject enters late. Anchor on frame 1 and there's nothing to find. Put an anchor where they actually appear.
• Something passes in front of them. After a heavy occlusion the propagation can latch onto the wrong thing. An anchor on the far side re-establishes it.
• They change a lot. Someone who turns around, opens a coat, or walks from bright sun into shadow looks different enough that one anchor won't cover it.
• Long shots. Small errors accumulate. Anchors spread through the shot keep pulling it back on track.

Space anchors where the shot changes, not evenly. One at each entrance, one after each occlusion, one after any big lighting or pose change. You can use up to 32.

Dilate and Erode

Dilate/Erode grows or shrinks the finished mask. Positive numbers grow it, negative shrink it, and the value is in pixels.

Grow the mask when you're masking for a solve. A mask that hugs the subject exactly still leaves a fringe of edge pixels that belong to them, and those are precisely the high-contrast pixels a tracker likes. Something like 5 to 15 pixels of dilation cleans that up.

Shrink it when you're masking to keep a subject rather than remove one, and the edge is picking up background.

It updates the preview live, so you can watch the edge move while you drag. When you settle on a number, regenerate the sequence, because the value is baked into the output files.

Generating The Sequence

Press Generate Masks. The node reports how many masks and how many detections it produced, and the output goes into the Mask Frames socket.

If you already generated matching masks in another session, Load Existing Masks picks them up instead of running everything again.

Change any setting afterwards and the node tells you the output no longer matches, so you know to regenerate rather than shipping stale masks.

Wiring It In

Import as Footage -> AI Mask -> Auto Track: Mask Frames

Once a mask is connected, Auto Track shows a Mask Mode dropdown:

You don't need to invert anything to switch between them. The same mask does both jobs, which is what makes object tracking straightforward. See Object Tracking.

To mask several things and combine them, run more than one AI Mask node into Combine Masks, which can add, subtract or intersect them.