How AI Photo Search Works
By Jay Harb · Published · 10 min read
Searching photos for "dog on a beach" without anyone ever tagging them is not magic and it is not a cloud service. Here is the actual mechanism, and why a model small enough to ship in an app can do it on your CPU.
The thing that needs explaining
You type "dog on a beach" and get back photos of a dog on a beach — from a library where nobody ever wrote the words "dog" or "beach" anywhere. No filename says it. No EXIF field says it. No tag says it.
People reasonably assume this means the photos went to a server somewhere. It does not have to, and understanding why is the difference between accepting a monthly bill for it and running it on a laptop on a plane.
Step one: turn a picture into a list of numbers
The core idea is an embedding: a model reads an image and produces a fixed-length list of numbers — a few hundred of them — that encodes what is in it. Two photos of dogs land close together in that number space. A photo of a dog and a photo of a filing cabinet land far apart.
Crucially, nothing about this is a set of labels. The model is not deciding "this is a dog" from a list of 1,000 categories. It is placing the image somewhere in a continuous space where similar things are near each other, which is a much more flexible thing to have.
This is why an embedding-based search can find "a person holding a red umbrella" without anyone having anticipated that query. It is not looking up a tag; it is measuring distance.
Step two: turn your words into numbers in the same space
The clever part, and the reason this technique took off, is that the same trick works on text. A companion model reads a phrase and produces a list of numbers in the same space, trained so that a caption lands near the image it describes.
The family of models that does this is generally called CLIP — Contrastive Language–Image Pre-training. It was trained on a very large set of images paired with the text that appeared near them online, with one objective: put matching image/text pairs close together and non-matching pairs far apart. That is the entire training signal, and it turns out to be enough to produce a shared space where "dog on a beach" sits near photographs of dogs on beaches.
Step three: the search is just arithmetic
Once both sides are numbers in one space, searching is not AI at all — it is geometry.
- When your library is indexed, each photo is run through the image model once and its embedding is stored.
- When you type a query, the phrase is run through the text model once.
- The app compares the query vector against every stored photo vector — usually with cosine similarity, which is the dot product of two normalised vectors and measures the angle between them.
- Sort by score, show the top results.
Step 3 is a few hundred multiply-and-adds per photo. For a 100,000-photo library that is tens of millions of floating-point operations, which a modern CPU does in a fraction of a second. The search itself is trivially cheap. All the cost is in step 1, and step 1 happens once per photo, ever.
That asymmetry is the whole reason this can be a local feature. If searching required running a model over your library every time you typed, it would need a data centre. It does not.
Why it can run on your machine
Three things make on-device viable:
The model is small. The commonly-used CLIP ViT-B/32 image encoder is a few hundred megabytes — the size of a big game texture pack, not a language model. It ships inside an app.
Inference runs fine on CPU. Runtimes like ONNX Runtime execute these models with heavily optimised CPU kernels. It is slower than a GPU, but "slower" here means the indexing pass takes a while in the background, not that the feature is unusable.
Indexing is a background job. Nothing has to be interactive. The app can work through your library while you do something else, and once a photo has an embedding it never needs the model again.
What it is genuinely good at, and what it is not
Being honest about this matters, because "AI search" gets oversold.
Good at: scenes, settings, objects, weather, composition, colours, general activities. "Snow", "at the beach", "food on a table", "someone in a red coat", "documents", "screenshots", "fireworks". Sorting through a decade of holiday photos is exactly the job it was born for.
Bad at: proper nouns and specifics. It does not know your dog's name, which of two similar beaches you were on, or that a particular building is your grandmother's house. It has no idea when anything happened.
Not a substitute for the boring metadata. Dates, GPS coordinates and face clusters are precise where embeddings are fuzzy. The right mental model is that plain-English search is one lens among several — you narrow by person, or by year, or by place, and then describe what you are looking for.
It will confidently return near-misses. Because it is ranking by similarity rather than matching a condition, there is no such thing as "no results". The bottom of any result list is noise. That is inherent to the technique, not a bug in a particular implementation.
Text in images is a separate problem
A common expectation is that searching for a word will find photos containing that word — a sign, a receipt, a whiteboard. That is OCR, not embedding search, and it is a genuinely different pipeline: detect text regions, recognise the characters, index the strings.
It is worth knowing that most bundled OCR models are trained on documents — scanned pages, clean type on white. Pointed at a photo of a shop sign at an angle in bad light, they produce gibberish rather than an honest failure. If reading text in photos matters to you, test it on your own worst case before believing any claim about it.
The privacy consequence
If embeddings are computed on your machine and stored on your machine, then the search feature never needs to see a network. Not "your photos are encrypted in transit" — the photos genuinely do not leave.
This is checkable rather than something you have to take on faith. Disconnect the machine from the network and search your library. If it works, the answer was local. If it does not, it was not.
In Pluto Photos
Pluto Photos ships CLIP with the app and computes embeddings locally during indexing, so AI photo search works with no account, no upload and no per-search fee. The same applies to face detection, which uses a separate detector-plus-recogniser pair, and to background and object removal — every model runs on your own CPU.
That is what the AI photo organizer page covers in more detail, and why the offline photo organizer claim is a literal one rather than a marketing one. Download Pluto Photos and try the aeroplane test yourself.
Part of the AI Photo Management Guides guide.
Download Pluto Photos for Windows, macOS or Linux, or read more from the blog.