Thoughts · · By Ben Paget
A secret message, hidden in plain sight. You can’t see it, but an AI model can.
Live demoHide a message in your own photoghostframe.electricsheep.id (opens in a new tab)Vector embeddings had been on my “I should really understand this” list for a while. So I recently went down a rabbit hole trying to understand them a bit better.
To keep it short, an image embedding is the list of numbers a model uses to describe a picture. Each number captures something the model learned to notice (a texture, an edge, a shape...), and similar-looking pictures end up with similar lists.
Then, I wondered: could you subtly shift the pixels in a picture in such a way that those numbers could spell out a message of your choosing? Turns out you can! Researchers have published versions of the idea since 2018.
It's not a new field; so I spent the rest of the rabbit hole on narrower questions: if I hide a message, can a stranger with the same public model and the passphrase reliably read it back? How little can you change the pixels and still get the message back every time? Does it still work on photos it has never seen, on a plain CPU?
Ghostframe is where I ended up.
How Ghostframe works
It hides a short encrypted message in a photo. It nudges the pixels, by at most 2 levels out of 255 per channel (8 in Long mode), until a frozen, off-the-shelf model (MobileNetV3-Small, with its ImageNet weights) describes each tile of the photo with numbers that spell out the message. Reading it back needs the PNG, the passphrase and the same public model (you don't need the original photo).
Each of the 80 tiles has fixed public directions in the model's 576-number embedding space, and the sign of each projection is one bit. The message is encrypted (scrypt, AES-256-GCM) with Reed-Solomon parity on top, and the encoder runs gradient descent through the model on the rounded 8-bit pixels.
(enlarge)Does it hold up?
The method isn't new: Fernandez et al. (2022) (opens in a new tab) and Kishore et al. (Fixed Neural Network Steganography, 2022) (opens in a new tab) are close to this concept. What I wanted to see was if it would hold up reliably outside of a research paper.
So I added a preregistered benchmark on an ordinary CPU, with the pass/fail rules committed before the test photos were encoded: 123/123 held-out trials decoded for both modes (Subtle and Long), on 41 photos that were never used for tuning.
(enlarge)Try it
You can hide a message in your own photo on the Ghostframe demo (opens in a new tab). It runs one encode at a time on a single CPU container, so "Hide" is capped at 50 a day for the whole site. To try lots of photos, you can run it locally: the code is on GitHub (opens in a new tab), under the MIT licence.
The downsides
A resize, a crop or saving the pic to JPEG destroys the message, which is what most messaging apps do. It also doesn't hide that a message is there: a simple detector separates every marked photo from its original.
(enlarge)Reminder: it's an experiment, not a privacy tool.







