How AI Background Removal Actually Works

A background remover looks like magic the first time you use it: drop in a photo, and a few seconds later your subject is floating on a transparent checkerboard. It's not magic, it's a specific kind of neural network doing a job it was trained on thousands of times over. Understanding roughly how it works also explains why some photos come out perfect and others come out with a slightly rough edge.

The task has a name: segmentation

What you're actually asking the AI to do is called image segmentation, deciding, for every single pixel in the photo, whether it belongs to the subject or the background. The output isn't a simple yes/no per pixel either; it's usually a confidence value between fully foreground and fully background, called a mask or matte. That's what eventually becomes the alpha (transparency) channel of your finished PNG.

The model doing this was shown a very large number of photos where humans had already marked, pixel by pixel, what counted as foreground. Over that training process it built up a general sense of what subject edges tend to look like: the contrast, the shapes, the way hair, fabric, and skin behave differently at an edge than a wall or a sky does. It's pattern recognition at a massive scale, not an understanding of what a "person" or "chair" actually is.

Why some edges come out clean and others don't

The model is best at exactly the kind of edge it saw the most of during training: a solid, continuous outline with decent contrast against the background. That's why a person in a solid shirt against a plain wall comes back almost perfectly clean.

The cases that trip it up all share one thing: the "edge" isn't really one clean line.

• • Flyaway hair isn't a boundary, it's dozens of thin, semi-transparent strands, each one partially foreground and partially background at the same time. The model has to guess an average, and averages look slightly soft.
• • Glass and anything translucent genuinely shows background through the foreground object, which contradicts the whole foreground-or-background premise the model is working from.
• • Motion blur smears the subject's edge into the background over several pixels, so there's no crisp line for the model to find even in principle.
• • Low contrast, a subject wearing colours close to the background, removes the main clue the model relies on.

None of these are bugs to be patched out; they're the actual hard cases of the underlying problem. Every background remover, ours included, gets worse at the same set of edge cases, because they're all solving the same task with the same fundamental constraints.

Why this can now run in your browser instead of a server

A few years ago this kind of model needed a beefy GPU on a remote server, which is why most background removers worked by uploading your photo, waiting, and downloading the result. Model sizes have shrunk enormously since then without giving up much accuracy, to the point where a capable segmentation model can be shipped as a file your browser downloads once and then runs locally using WebAssembly, a technology that lets browsers execute near-native-speed code.

That's exactly how our Background Remover works: the model loads into your browser the first time you use it, gets cached, and every image after that is processed on your own device. Your photo never travels to a server, because the computation that used to require one now fits on the machine you're already using.

Getting a better result out of it

Since the model's biggest weakness is ambiguous edges, the biggest lever you have is giving it a clearer edge to work with in the first place:

1. Shoot against a background that contrasts with your subject. Dark hair against a dark wall gives the model almost nothing to work with.
2. Use even lighting. Hard shadows create fake edges the model can mistake for the actual subject boundary.
3. Start from a higher resolution photo. More pixels at the edge means more information to base the mask on, particularly for fine detail like hair.