At a High Level

• I made the illustrations with Nano-banana.1

• I had GPT-5.2 describe reference images in words to make them more editable and portable.

• Overly verbose, expressive image prompts used to be required, but modern models seem to find them distracting. I had better results telling them to speak clearly and concisely, the way you would to a person.

• I defaulted to generating everything at least three times to get a sense of what was a prompt problem and what was model variance.

• The total cost for the post was about $12.

Creating the Character

In the first round of character generations I ended up with this which I liked quite a bit. It became the canonical reference which Nano-banana dubbed Hat-coat Wanderer.

I got pretty high variance for images. Here are some rejected examples:

Specifying Illustrations

Next I chose the lines in the story to illustrate. I wanted to pace them evenly throughout the story, highlighting emotionally salient scenes. At first, I tried to generate each description individually but they all sounded the same. The scenes that I got from generating them all at once were much better quality. I didn’t know which would be better, so I tried both Opus 4.5 and GPT-5.2.

I wasn’t able to just use either’s responses wholesale. For some scenes, I combined the best parts of both descriptions. Other times I tweaked toward something the model was able to generate better. There were also scenes where I had something in mind already and wanted to see if the model agreed.

Generating the Illustrations

Each generation received:

• a chunk of general style text describing the characters, world, and art style

• a sample image from the references

• the character turnaround for any characters in the scene

• the specific scene description

There was a surprising amount of variability in these. Here are a bunch of samples for the opening illustration. Some of that variance was failing to adhere to the prompt and other was because I under-specified what I was looking for.

Even then, I rarely one-shotted the illustration I ended up using. They usually required edits. One clear example is the background of the opening image. Once I found the pose and framing I liked, I removed the original background that I found too heavy and replaced it with something lighter.2

Places the Model Struggled

The models have gotten so much better. Edit models have been a step-wise improvement, and Nano-banana specifically seems like the breakthrough that makes consistency possible. However there were still gaps where I was fighting a tool, rather than working with it a collaborator.

Creating a second character

The turnaround for the second character was a delicate dance of trying to extract the essence of the art style without grabbing too much of the specific character. I wanted someone individually recognizable, rather than the main character color swapped.

I tried a few different tactics to brainstorm how some characters might differ: height, weight, skin color, hair color, different kinds of hats, jackets and other winter gear. The tactics that got the LLM brainstorming effectively were very similar to the ones I’d built into Imaginator.

Placing characters relative to each other

For the line “Humbly, the best direction is often neither where you nor they are pushing, but the place it ends up rolling somewhere in between,” Nano-banana really wanted to put the characters on opposite sides of the snowball, but the line calls for characters pushing at slightly different angles. Their combined vector needs to still be going somewhere for the story to make sense.

I tried a host of different ways to describe this: descriptions, angle measurements, even a diagram.

Eventually, I got lucky. Even then, the snowball path never made sense. I had to edit it in afterwards.

Many children pushing

It was comically hard to get a crowd pushing the ball. The model lost any pretense of where the ground was. These characters are either climbing on the ball or floating in space.

It was easy to make a crowd, but it was hard to get a quality one. Generating the crowd all at once was better than adding more children sequentially as edits. Edits led to the same character being added in multiple places, or the style or scale of the characters drifting and looking out of place.

While I was finally able to get them pushing in one direction, I’m not sure how effective their strategy would actually be.

A really big snowball

Nano-banana consistently produced a snowball that was about twice the size of the character. I tried a handful of strategies like “a massive snowball that fills most of the frame” or “the size of an office building” and edits like “Make the snowball 10 times the characters size”.

In the end the bit of hyperbole that got me there was “Make the character tiny so the snowball looks like a planet!”

Conclusion

One commenter on Hacker News captured it perfectly:

I know some people are going to be upset at model generated illustrations. But I think the alternate is probably, no illustrations. There’s a lot of unnecessary AI image slop all around and most add no value or worse makes you just avoid reading the content by their awfulness. This was done really well and I am not sure I would have read it fully without it.

-tecoholic

Even as someone who grew up doodling in all of my notebooks in school, I don’t know if I would ever have crafted illustrations with enough quality to be ready to publish this. I certainly would not have hired someone. But if I keep getting better at this, maybe for a future project, I can.

I’m not sure if these illustrations will stand the test of time or if they’ll feel dated like Stable Diffusion or the animation from the original Shrek. Regardless of how timeless the work is, I’m thankful that something that came out of my head reached people in a positive way.

The urge is pure.

You wake up to fresh snowfall. On goes your jacket and gloves and you’re out into the crisp, cool air.

You reach down to clump some between your hands. The snow is sticky and soft. You reach down for more and pack it in. Soon it’s so big that you set it down to roll. How the snow from the ground sticks to it. How it grows.

Behind you there are other places with other hands rolling balls and little snow left on the ground. Ahead of you is clean and white.

It’s only when looking back that a trail appears.

Sometimes you push alone. Sometimes others push with you. It can be just one person or a few.

When you all push together the ball rolls faster. It grows fast too. Wonder at how this thing that once fit in the palm of your hand that now has room for two, three, four pairs of hands.

There are moments the ball rolls so fast you fall behind. You’re no longer guiding it— You’re chasing it! There are moments when it won’t budge and you have to count to three so you can all push it in unison.

It’s okay if the other hands push somewhere slightly off course. You’ll stubbornly fight to redirect it to where you want it to go. Humbly, the best direction is often neither where you nor they are pushing, but the place it ends up rolling somewhere in between.

If the other hands are pushing back directly against yours, the ball will falter. As it does, you might start to argue. In those times, hands might have to stop pushing. Sometimes they’re theirs. Sometimes they’re yours.

It’s hard to tell what part of the snowball is theirs and what part is yours. It’s senseless to try to split it up. Instead, it’s best when someone reaches toward the ground, clumps snow between their hands, and starts afresh.

There is grief in parting. But that needn’t be all you feel. Now that you’re out of each other’s way, you can wave adieu with a grin. You’ll both roll much bigger snowballs than this one.

Keep the ball moving. If it stops, you might never get it started again. Besides, there’s no more snow where you’re standing.

If you’re lucky, it continues to grow. Bigger than a car. Bigger than a house. Other people will see its glory and join their smaller snowballs into it. They see the inertia and want to be part of the fun. More hands can help when they know where it’s going. But changing course is a whole different story.

When going uphill, everyone needs to agree the hill is worth climbing. When tumbling down, they must work together to guide it swiftly past rocks and trees. There are so many of you now. All pushing together. The path behind you is long. You couldn’t see where you are now from where you started.

It starts to get warm. The ground ahead is full of sticks and leaves. The snow has worn away. Despite all the hands pushing, the snowball starts to shrink. You think there’s a rich field ahead, but you can’t say for sure. The ball is so big now you can’t see around it. It’s easier to see left or right, and it can be tempting to go there instead. Not everyone will stick around to find out.

You’ll have to make a call. Sometimes the right answer is to forge ahead. Try to survive until the next storm. When the fresh powder hits, scoop it up. Survive a few storms and the ball is a mountain that can weather a whole summer. Make it a few summers, and it has its own weather, keeping the warm at bay. Eventually, the tiny clump between your fingers is a moon so big other people roll their own balls across its surface.

Other times the seasons are changing too fast and the snow is retreating everywhere you go. You can’t know for sure which situation you’re in from where you’re standing.

That’s just the mystery of it.

But every morning, there’s the window. There’s the ball. There’s your jacket on the hook.

If you feel the urge, listen to it.

After all, that’s what got you started in the first place.

Everyone knows that time speeds up as we age. We talk about it all the time, but we don’t really have cultural institutions that make this acceleration felt in an embodied way. There’s no ritual that prepares us for how quickly the years go by.

One of my big fears is that I’m still fighting the last war, trying to win my twenties even though they’re behind me. We all know the archetype: The jock who peaked in high school and can’t let go of the dropped pass that cost them the state championship, the forty-year-old who hasn’t realized they’re the oldest one at the club. As someone who is still spinning on my head once or twice a week, this is probably an important question to ask. It feels important to challenge my body and to have a place in my life for play but maybe that’s a cope.

I’m not busting my skills out on Friday nights trying to attract women at parties anymore. I’ve traded that for 30 seconds of fame at the couple of weddings I attend every year. When my parents were my age, they had three kids, the oldest of whom was almost twelve. There are ten-year-old professional bboys doing skills I’ll never be able to learn to do. I could be busy taking my own kids to sports practices and living vicariously through their achievements.

I saw winning my twenties as building my potential. it was about taking risks and trying hard. Exploring possibilities and seeing what I had a knack for, trying out different identities and seeing how it felt to be in different crowds. I wanted to craft a sense that I was going places.

The rules changed in my thirties. Optionality is valuable, but it was time to start choosing. It was no longer enough to have a bright future ahead of me. I needed evidence that when faced with a fork, I chose a good path. If my twenties were about creating more branches on the tree, my thirties were about starting to prune it.

Thinking something isn’t the same as doing it. I don’t feel like I’m really nailing my thirties. Sure, my body still works and I believe I’m taking great care of it in ways that compound over time relative to my sedentary peers, but I’m the only one of my 5 college roommates who isn’t married. I got engaged this year, but more than twenty couples we want to invite to the wedding have children. I’ve achieved good financial stability but I’m not clearly on a path to make a meaningful professional contribution. In many ways, I’m still someone with great potential rather than someone walking their path.

It feels like remedial extended twenties bonus time, as if I got myself diagnosed with ADHD just to get extra time on the tests. Only here, it costs me some of my thirties to do it. Given how fast time is moving, by the time I really understand how to win my thirties, they’ll be over. Maybe the move is to start putting energy towards really nailing my forties.

My partner left town two weeks ago, and any free time that wasn’t spent exercising or walking the dog was gobbled up. My midnight to 3 AM block stopped being for sleep and started being for projects. I’ve been writing more code than I have in years. I’m not alone. Shape rotators everywhere are vibing.

Becoming Even More Curious

The perceived difficulty of creating is falling off a cliff. AI streamlines the unpleasant parts of getting started. It anticipates my needs and fills in the gaps. But more than that, it engages with me when I’m most inspired, able to reflect my enthusiasm even in the middle of the night. It has the opportunity to make us more curious, creative, and ambitious so that we can do the best work of our lives.

Not everyone agrees. In an article in the New Yorker, Ted Chiang dismissed AI art, claiming that it reduces 10,000 choices into one and that AI fills in the blanks with the average decision. He called it time-saving but sterile. He’s stuck seeing things as they are, not as they could be. This confounds me, given that his own example of ‘Director Bennett Miller generating hundreds of thousands of images to discover the meaningful 20’ is staring him in the face. We need more people exploring the Miller path. It’s not about how AI can turn 10,000 choices into 1, but how AI can empower us to make 100,000 choices or even a million!

As an experiment and an example, I put together a project that I’m calling Imaginator to help people refine an image from a prompt, details chosen by models, and grids of images. I hope it will deliver a magic moment of discovering meaningful aspects of your idea you didn’t know you were looking for. I hope the concept is fun, and I'm thrilled to have something concrete that embodies design principles I'm genuinely passionate about.

Explore vs. Exploit

Most AI image tools focus on coercing diffusion models to produce an exact output. They presuppose that we have a vision figured out and are merely executing. Creativity doesn’t work that way. It’s full of experiments, chance, and happy accidents.

Imaginator for exploring rather than exploiting. In addition to generating an image, it brainstorms categories of prompt refinements you could make, like Art style, Pose, and Lighting, along with specific values for each category, like Black and white photo, Anime, or Oil painting. You can add your own manually or ask the AI to generate more at both levels. We should never have to stare at a blank page again. Models can brainstorm sensible values that are tailored to our needs.

Creativity is Curation

There’s a parable from Art & Fear about a ceramics professor who grades students on the literal quantity of their artwork in pounds rather than the quality of a particular piece. The best pieces all came from the quantity group. The lesson is that great work doesn’t come from obsessing over a single piece but experiencing all the nuances of many times over.

While the parable is about practicing a craft, I think it also applies to developing taste. Quantity creates quality. Similarly, modern chess students learn by watching games and trying out different moves at pivotal moments. The critical part of learning by quantity is a tight feedback loop. I hope Imaginator gives you enough experience with each option to help you build a taste for which one you’d like.

If Everything is a Remix and good ideas are just new combinations of existing ideas, I wondered, “How much can you learn about what you want by experiencing many combinations at once?” Instead of trying to one-shot precisely what you’re looking for, Imaginator shows the breadth of possibilities. I wanted you to explore the variations that exist within your prompt. I’m convinced there are ways to craft and design without being locked into chat. Images seem perfect because people can quickly scan them and naturally curate what they’re looking for.

Meaningful Constraints

The constraints of the last two years are falling away. The cost of LLM inference from Open AI has fallen by 99%. We can generate images as fast as we can type, each costing a fraction of a penny. Soon, intelligence will be too cheap to meter. What was scarce is now abundant, and learning what to do with it takes time.

Existing AI tools return multiple images, but we can’t tell why. It’s usually something behind the scenes, like a different model, LoRA, or seed number. Regardless, the content is displayed in a grid, but their positions aren’t semantically meaningful. We can’t act on the differences between images, so we’re left pulling a slot machine and hoping for a jackpot.

We must move from the violin era of AI art to the guitar era by creating metaphorical ‘fret’ guardrails that keep us in lanes that produce more controlled content. When the rows and columns on a grid represent specific differences between images, each variation is a unique combination of two particular choices. Scanning in any direction varies the image based on one category while holding the other constant to help you discover what’s important to you.

Tree of Possibilities.

Creation can be messy. I wanted to let you select different category combinations until you’ve found variations that are meaningful to you. This felt like what Bret Victor was talking about when he promoted showing all possible universes so that you don’t need to model the whole problem in your head.1 This frees up brain power for other parts of the problem.

Details are fractal. As Ted Chiang alludes, the “big” ones aren’t necessarily more important than the “small” ones. Furthermore, once you settle on one, more new questions pop up. I wanted to model this by letting you select an image you like from the grid, which adds those two details to the prompt and generates new possibilities for further exploration. As you iron out more details, I hoped the variations would converge.

Additionally, sometimes paths don’t work out, or our curiosities change. So, I wanted to ensure you could walk back up the tree and select variations representing different values to continue your exploration.

Conclusion

Imaginator is alpha as hell. There are no accounts or logins. Sometimes, you need to refresh for the images to load correctly. You should probably be able to edit the prompt, and there’s no way to visualize the tree or your current position. Frankly, even adding and manipulating details leaves a lot to be desired. That said, I’m impressed by how fast I got to feel this concept and see if it had legs.

This is a wild time to be alive. Things are changing fast, and I'm grateful for the opportunity to be here now building. AI has become this inspiring creative partner who's always awake, available, and eager to discuss what energizes me most. That's been incredibly motivating. In a highly meta fashion, it's helped me explore and refine my vision, and I'd love to help more people experience that magic.

I’d love to collaborate with people, too! If you’re working in this space or are curious and interested, I’d love to hear your feedback or see what you’re working on. Also, of course, if you have any magic moments with Imaginator, I’d love to hear about them too! Drop me a line at me@bawolf.com.

Lately, I’ve been exploring these ideas as a member of South Park Commons, a community that embraces creativity's messy, nonlinear journey. It’s a fitting place to develop and refine these ideas.

Subscribe now

The job of a creative

Someone doing creative work is exploring a unique part of the territory of “all possible ideas” to uncover good ideas that are meaningful and moving. The territory of all possible ideas is incalculably large. To narrow that down, creatives craft a sense of taste to separate good from bad. Actors develop a taste for what pose to strike; videographers discover what angle to shoot; editors adjust the color profile to evoke emotional responses from their audience. Similarly, SaaS entrepreneurs learn the right questions to ask working professionals to understand if there’s a business there before writing a bunch of code.

Individual creatives are at the vanguard, prospecting for new good ideas surrounded by a heavy fog of war. Each one contributes to our collective understanding of the explored territory and shared sense of taste.2

The job of tools

Creatives don’t start with perfect ideas. Their concepts are shaped by the tools they use, which guide the process and instill a sense of taste. For example, smartphone cameras make it easier for users to follow the rule of thirds to capture meaningful moments more evocatively. Web frameworks guide software engineers by breaking problems down into the right conceptual parts. Tools influence how we think, and better tools lead to better ideas.

When I talk about tools, I mean more than just wrenches and pens—I’m also talking about the tools that shape our society, like the alphabet3 and capitalism. These tools can remain stable for centuries until a new idea changes our understanding of the world. Large language models and generative AI are shaping up to be excellent tools for manifesting new ideas. Far from the sci-fi predictions of cold, calculating robots, they’re densely encoded with our human sense of taste. To fully realize their potential, we’ll need to combine that with what we already know about building great tools.

The shape of good tools

Good tools help us have better ideas. The better our tool ideas, the better our tools will be. The better our tool-making tools, the better our tool ideas will be. It’s all helplessly meta.

Good constraints

Constraints limit what can be expressed through a tool. The proper constraints limit in ways that lead to good ideas. The set of all possible musical notes is much smaller than the audible frequencies. Plenty of pleasant sounds are excluded, but the trade-off is worthwhile because it makes it harder to make a mistake. Even more constraining frameworks of scales and chords make it easier still.

There’s a reason more songs have been written for the guitar than the violin over the last 100 years. Violins require more training and focus because their continuous necks can play frequencies between notes. In contrast, guitar frets provide clear boundaries, helping musicians focus on other parts of the creative process. Today’s AI tools are still too nascent to effectively guide creatives toward good ideas. They look more like violins than guitars, but they are still discovering their metaphorical frets to help creatives stay within their strengths.

Bad constraints don’t limit ideas enough or in ways orthogonal to finding good ideas. The ‘Innovators Dilemma’ describes Kodak's failure to adopt digital film because they constrained themselves to ideas that wouldn’t cannibalize their existing film products. Unfortunately for them, the transition to digital was inevitable, and startups who did not have the same constraints took the market.

Peak under the hood

Tools should run well in a default setting, but sometimes, creatives must challenge constraints to find an unexplored patch of new good ideas. Good tools let creatives peek under the hood and modify the internals.

Constraints are fractal, so tool makers must choose how to allow creatives to peak under deeper and deeper hoods progressively. Accounting for every constraint is impossible, but good tools prioritize well. Guitar tuning is reasonably standardized to EADGBe, but playing other tunings is more common than defining new notes. In line with those values, re-tuning a guitar is more straightforward than rearranging the frets, and changing the volume is more frequent and easier than re-tuning an electric guitar. On the other hand, if your search for good music makes you question whether strings are the best source of vibration, a guitar won’t cut it. You’ll have to try another instrument.

AI tools are so nascent that they’re not exposing very many knobs to creatives. Creatives might need to see the underlying prompts sent language models to discover better ones, even when you provide sensible defaults. That’s how we move from a local maxima closer to the global maxima. It’s tempting to hide these as trade secrets, but that will only limit more effective creatives.

High sensitivity

Good tools are highly sensitive, so creatives can wield them as an extension of themselves. As Bret Victor describes in this fantastic talk, creatives model their movements in their brains. The less feedback they get, the more they have to model internally. There’s a limit to how much our brains can model. While Michael Jackson’s voice and training allowed him to model whole songs in his head and express them to studio musicians, most of us will write better songs with Garage Band.

Each brush stroke teaches us how to paint. It would be incredibly limiting if painters wore blindfolds until their works were entirely done. Yet, that’s how many software engineers write code and how we all took photographs before polaroids or digital film.

Struggle

Some argue that new tools make art 'too easy' by removing struggle, implying a lack of value. But this view is too narrow, elevating only certain kinds of familiar struggle.

Painters who mix their paints struggle differently than painters who use the color picker in Photoshop. In both cases, the experience is highly sensitive and deterministic. Over time, both painters will get the color they want by modeling the results in their heads and building intuition. Mixing paint takes longer and is more challenging to model, so those artists spend relatively more time on that part of the work. Photoshop artists will instead struggle differently, trying many more strokes and colors because their choice is more reversible. Better tools change what creatives struggle with but don’t eliminate the struggle altogether. Struggle is an emergent property of doing creative work.

Good struggle is playful experimentation in the unknown, while bad struggle is the hurdles to executing well-known ideas. Bad struggle comes from inefficient tools or unnecessary constraints that waste creative energy, leaving us bogged down in details instead of exploring the unknown.

AI tools often introduce slow feedback loops, where entire works are generated and must be remade with each revision. This randomness creates bad struggle by forcing creatives to rely on chance rather than building reliable strategies for consistent results.

[it] feels like the computer is telling my brain This Is What You Should Have Said, rather than letting my brain come up with the words. and then it doesn't sound like me. it sounds like a weird little simulacrum of [me] - Isabel Kim

Critics like Isabel Kim argue that AI-generated text tells her what to say rather than allowing her to find her voice. This is the wrong kind of struggle.

Tools are slowly getting more sensitive to nuanced changes with less entropy, creating tighter feedback loops so artists can model less in their heads. A straightforward way to do that is literally making AI models faster. You can try out Flux tuned for speed on replicate. It’s quick enough to generate images as you type, which gives way more information per keystroke. Masking and fine-tuning similarly allow for finer revision.

Understandably, some creatives view technology as competition. If the sense of taste they’ve built within themselves is now well encoded into a piece of software, then the struggle they earn a living with is now the bad kind. In Hidden Figures, Katherine Goble helped her team transition to NASA’s digital era, but not everyone adapts so easily. Failing to adapt can have real consequences for a creative’s livelihood. This might be so disruptive that we question economic ideas like capitalism itself.

Everything is a Remix

"Everything is a remix" is a well-established idea, and I stand on the shoulders of the giants who discovered it. At their core, good ideas often come from combining old ones in new ways. This is true when VCs ask startup founders, “Why now?” directors adapt books into movies, or engineers “rewrite it in Rust.”

Good tools help creatives connect seemingly unrelated ideas and guide them to combinations worth pursuing. Some tools reduce the energy required to refine ideas, while others introduce well-targeted entropy to identify which ideas deserve attention.

Critics claim AI can't generate original ideas, but that's shortsighted. AI tools see the world in a different way. They uncover novel connections humans might miss and experiment on a scale far faster than we can manage. There will be more moments like Alpha Go’s ‘Move 37’, which changed how people thought about Go forever.

Creatives will eventually use AI-encoded taste to determine which remixes have potential. For example, a movie exec might model their audience to understand if a pitch will appeal to enough people to warrant production. TikTok succeeded by replacing user-curated networks with an algorithm, and similarly, AI will guide content producers toward commercially viable ideas or potential Academy Award winners. While human attention may always be valuable, I’m not sure there’s any part of the creative remixing process that will remain solely the domain of humans.

Closing thoughts

AI will evolve like past technologies—by automating aspects of the creative process and amplifying human potential. Rather than eliminating creative struggle, these new tools will refine the challenges we face, empowering creators to take on more complex projects and push further boundaries. I hope this understanding of constraints, sensitivity, struggle, and remixing clarifies the path to get there.

The best parts were learning about:

• Building a good dataset. No dataset is perfect, and mine was particularly limited by what I was willing to curate manually.

• Training a model. The basic terms and moving parts.

• When to lean on AI tools, especially Cursor, and when to ask a real person for advice.

If you want to skip all the details and try the model, you can do so at https://breakingvision.bawolf.com/. I hosted the best version on Replicate and made a small web app to help you cut and run clips against the model.

Data

I started with what I already had: a phone full of videos from practice. I omitted references where I fell or crashed and kept the scope small, narrowing down to three moves I had the most repetitions of. These were windmills, halos, and swipes. If you’re not familiar with power moves, that’s totally fine. Here are some examples:

A windmill 🌪️ was designed to be a continuous backspin. In practice, it’s a spinning roll on your back, head, and hands.

A halo 😇 is an off-axis head spin on your hands and head. You’re supposed to travel around a ring (halo) on your head. 

A swipe 🤸 is a little more complicated but involves rotating your hands and one foot in a continuous circular cartwheel.

Without experience with labeling tools like CVAT or Label Studio, I uploaded videos to my computer and used LosslessCut to separate the videos and segment the individual moves into clips. The clips above are examples of what the model saw.

Sources

In addition to my phone clips, which I’ll call Bryant, I also made a second data source, which I’ll call YouTube. 

Bryant

These are all videos taken from my phone. Most are from a single location over many practice sessions with slightly different angles. Some are repeated combos of the same move, while others are sequences of different moves. I was the subject of all of these videos.

YouTube

These are primarily educational tutorials from YouTube. I could reliably get many examples of a move from a single tutorial video, which made the cutting and labeling process much more manageable. A few of them are from battles or people’s training sessions. Unlike Bryant, these examples are primarily strings of one move but include examples from many different people and settings.

With between 300 and 400 examples of moves, both data sources are on the small side.1

Datasets

I arranged the sources into different datasets to understand how that would impact training. The goal is a model that generalizes, which means that it extrapolates to data it has never seen before. Some models appear promising but overfit, meaning they got good training scores but don’t hold up outside of what they were trained on.

Structure

Each dataset sorts videos into three buckets. My original data gathering was a bit unbalanced. I labeled way more clips of windmills than swipes or halos. I ended up shelving a bunch of extra clips to make sure I had an approximately equal amount of windmills, halos, and swipes:

• 70% into the training set used by the model for the learning process, making predictions, and inserting entropy to determine how to change its predictions in the future.

• 20% into the validation set the model uses to measure snapshots of how training is going without updating the model.

• 10% into the test set that measures how training went after completion.

Lastly, these clips had dimensions that were all over the place. Some were portraits, others were landscapes, and some were even squares. Clips were between 0.5 seconds and 3 seconds. I normalized them all to 224x224px squares at 30 fps, which seemed standard.

Permutations

Some of these are from the same combo in the original source video, which can disguise some problems down the road. If there are different windmills from the same practice session in both train and val sets, the model might cheat and learn that white walls or a chair in the corner are a sign of windmills. That wouldn’t be visible in the test scores, which would all look like they were passing.

I tried a few things to detect when that was happening. The most straightforward was training a model on Bryant and seeing if it generalized to YouTube and vice versa.

Next, I made an “adjusted” version of each dataset that tried to be a little cleverer. I sorted clips from the same source video into the same bucket. If a video had three windmills in a row, the first windmill could not be in the training set if the second windmill was in the validation or test set. This is a sample video of a chain of halos.

I also included clips from videos with more diversity in the training set. If a video had a loop of swipe, windmill, halo, swipe, windmill, halo, those clips had priority over clips from a video that looped halo, halo, halo. Here’s an example of a diverse source that contains windmills, halos, and swipes.

YouTube and Bryant had a random version with all the clips randomly distributed and an adjusted version with the above criteria. I also made combined random and combined adjusted datasets, which are made the same way but with both sources combined.

As an aside, One thing I didn’t expect from using cursor was how many more one-off scripts I had it make. Most of the sorting code was written almost entirely by cursor, including the code to create charts and tests that the sorting was correct.

Training process

If, like me, you have never trained a model, here’s a quick primer. Training is measured in epochs or the number of passes through each training example. The amount of examples the model looks at simultaneously is called the batch size. There is an explore/exploit trade-off when choosing a batch size.

Smaller batches are better at exploring. They have more variance within an epoch. They’re risky because they’re more likely to move away from a promising solution even when they’re close to one. In return, they can jump out of enticing wrong solutions toward better ones. They typically train slower because they don’t fully utilize GPU parallelization and memory.

Larger batch sizes are better at exploiting. They look at more data simultaneously, creating less variation between batches within an epoch. This is more stable, making them better at refining a good solution but more likely to get stuck somewhere non-optimal. They train faster by taking advantage of GPU parallelization when available.

Training produces weights. Think of them like constants in the model’s functions. When training from scratch, the model starts with random weights and shifts them each epoch. The amount it’s allowed to move them is called the learning rate. Typically, this starts high and shrinks at set epoch intervals. Fine-tuning a model is just doing more training but with your custom data on top of pre-existing weights called checkpoints.

Learning rate and batch size are just arbitrary values you set when you run the training process. In ML jargon, they’re called hyperparameters, which sounds more complicated and less approachable. Hyperparameter tuning is just running many trainings or fine-tunings with different values and seeing which produces the best model.

The weights of a model are really abstract. They don’t have direct meaning that a person can interpret. There’s no weight for the direction a person faces in a particular frame or a direct sense of how fast they’re moving. As a result, learning is a complex trial and error of changing weights and seeing the effect on learning. It can be unsatisfying initially, but it’s genuinely incredible that it works.

Training is measured in accuracy, the proportion of correct predictions over the number of total predictions, and loss, a measurement of how confident the model was in making those predictions.

At the end of every epoch, the training process calculates corrections to the weights that would have improved the model’s performance. This is called backpropagation. Individual corrections from any one epoch might only improve the model a little, but they add up over many repetitions.

After an epoch, the accuracy and loss are measured on the validation set without backpropagation to see if the model generalizes beyond the training set. Similarly, after training, the model is tested on the never-seen test set for a final measure.

Subscribe now

Making a Model

It took me a while to get grounded. I started broadly, toying around with SAM2, pose libraries, and fine-tuning existing models. As a software engineer but not an experienced ML practitioner, there was a lot of clunkiness, and I ended up with many false starts.

Not all models from papers have public code, and even if they do, not all have documented processes for training and fine-tuning. Some have open weights, but others don’t. I clicked broken links that once led to checkpoints or raw datasets. Even understanding the format the model expected the dataset in could be opaque.

I have a web development background but am not an experienced Python developer. I had to do more version management finicking between Cuda, Conda, and Pip than expected. Overall, dependency management feels fractured in Python.2 I missed my `package.json` or `mix.lock` files. It was humbling.

Perplexity, ChatGPT, and Cursor were all bad at helping me choose which model to fine-tune. They were not very helpful in translating jargon or understanding how training might work from a code repo.

Eventually, I stepped back and worked with Cursor to build a simple model from scratch, which got me in the right groove. Cursor suggested a 3DCNN. The 3-dimensions are length, width, and time. CNN stands for Convolutional Neural Network. Roughly, it describes the strategy of scanning through video pixels, looking for patterns with a cube of pixels at a time. Each step of the scan looks at n pixels of length, by n pixels of width by n frames. This model was only a single layer and didn’t generalize well. Still, going through the whole end-to-end process of training, measuring, and even some hyperparameter tuning really helped my understanding.

Results

Briefly, here’s a sense of what this model was capable of and how it’s measured. These examples are from a single trial of the Bryant-random dataset, but the results across all the datasets were similar.

Despite becoming confident and 100% accurate in predicting training data after 8 epochs, the model was far less confident and only 70% correct in making predictions from validation data. This is a classic example of overfitting, where the model memorized the training data rather than learning patterns.

A confusion matrix shows the model’s predictions in more detail. The x-axis represents the model's prediction, and the y-axis is the actual value. [0,1,2] is a windmill, halo, and swipe.

If the model predicted all the values correctly, they would be in a diagonal line from top left to bottom right. Instead, the most consistent error was the model predicting that about half the swipes were aalos. I could go deeper, but this should be enough of a baseline to compare to the fine-tuning of CLIP.

Fine-tuning CLIP

Next, I took another swing at fine-tuning an existing vision model. There are models trained on massive datasets that understand human bodies. My videos would focus that knowledge on break moves.

Cursor was great at the low-level tactics of writing code, but human knowledge, whether from friends3 or Papers With Code, was much better at recommending base models than ChatGPT, Perplexity, or Claude. Simple questions like “Does this model take as input a video made up of a single move or multiple moves and an annotation file?” or “How should this data be formatted?” were more challenging for the AI tools to answer than I expected.

I settled on using a model based on OpenAI’s CLIP. Instead of 3DCNNs, CLIP uses transformers4, and instead of a single layer, CLIP uses many. CLIP was one of the fundamental breakthroughs in Dalle-2 and GPT-3. In their own words:

CLIP (Contrastive Language-Image Pre-Training) is a neural network trained on a variety of (image, text) pairs. It can be instructed in natural language to predict the most relevant text snippet, given an image, without directly optimizing for the task, similarly to the zero-shot capabilities of GPT-2 and 3.

CLIP can only examine individual images. To augment it to understand videos, I naively grabbed 10-15 frames from each video and averaged out the feature vectors from each frame. A feature vector represents how much the model thinks the input is a windmill, a halo, or a swipe.

The middle of each clip tells more about which move is being performed than the beginning or end because there are multiple ways to transition in or out of a move. Windmill to windmill looks different than windmill to swipe. Understanding this, I grabbed frames over a normal distribution, with the majority closer to the center and fewer closer to the ends. The consequence of averaging is that there’s no information about the order of frames. The hard work is abstracted away into the clip model.

You can see this model for yourself here.

Results

I ran multiple training trials for each dataset to see which hyperparameters worked best. Overall, almost all of the models were better. They ranged from bad to okay to very good. The parameters with the most obvious positive impact on training were:

1. Use the large checkpoint. CLIP has multiple checkpoints that you can start from: the smaller `clip-vit-base-patch32` and the larger `clip-vit-large-patch14`. Almost all the good models were fine-tuned from the larger checkpoint. It’s bigger in several ways. The large model has 24 transformer layers, while the base uses 12. Those layers are based on vectors with more dimensions (1024 vs. 768), and as a result, the large model has way more parameters overall.

2. The best models unfroze 2-3 layers. You can unfreeze transformer layers from the top down when fine-tuning. The more you unfreeze the, the more malleable the model is to forget what it already knows and learn from the training data. Unfreezing one was too few, and the model wasn’t able to learn enough from the training data and got a lousy score in both training and validation, whereas unfreezing four allowed it to forget essential things about bodies and overfit to the training data, giving it good training scores but bad validation scores.

Some of these scenarios play out in the accuracy and loss of different models trained on the Bryant-adjusted dataset. The confusion matrices show which moves the models learned more or less correctly. The best model is good at labeling windmills and swipes but mislabeled halos as windmills or swipes once. Still, it's pretty good! I would take any of the top three models over the simple model from before.

The best models trained from each dataset were all very good, though the adjusted datasets were as good as or better than the random datasets.

Most of them are pretty good, but which one is the best? YouTube-adjusted had a pretty good claim, but since the top couple were so good that they were only missing a single example, I decided to run them against all the labeled clips I had, including the ones I’d cut out to make the numbers even early on.

I was floored that combined-adjusted only missed one!5

The next closest model, our old winner, YouTube-adjusted, missed eleven. I declared combined-adjusted the winner.

If you break, I’m curious if it works for your clips, too! As a reminder, you can try it out here.

Conclusions

To be honest, I’m really impressed. These results were way better than I thought they would be. I’m not surprised that the combined training data was better than Bryant or YouTube, but I was surprised at how much better the adjusted sets were. Taking small steps to prevent these models from cheating seems to have gone a long way.

There are some obvious limitations here. This model only trained on three classes. There are hundreds of interesting break moves, and I don’t even know if the results would scale if you doubled classes to six or again to twelve. I’d love to try, but I might need some help labeling.6

This model has to label any input as a windmill, halo, or swipe. It would be interesting to train a model in a way that could say “none of the above” as an extra category and see how accuracy changed.

More than that, it would be better if this model had a sense of time. Feed it a longer video, and it labels the timestamps where moves occurred. The most interesting next step would be fine-tuning something like AdaTAD using OpenTAD. TAD stands for temporal action detection, the area of study around labeling what’s happening within videos.

All in all, this was really fun. It was empowering to see how far I could get on the shoulders of all the giants building great vision model tooling, the giants who built the great language models, and my friends. Approaching some of these topics can seem overwhelming at first, but there’s a lot of information out there now, so it’s more accessible than ever. You definitely don’t need an ML degree to try this out.

The kind of code I write doesn’t exist in a vacuum. The ecosystem is always changing and adapting. Other people are constantly writing code: for operating systems; for cloud hosts like Google or Amazon; for open-source projects and frameworks. The benefit is that one person can produce an extraordinary amount, empowered by our collective effort. The downside is that code that is not nurtured and maintained stops running.

As a result, every software project needs someone to watch over it and water it now and again. Some codebases are more like succulents, and others are more like orchids, but they all have needs. Well-known computer security expert Moxie Marlinspike wrote about the growing burden of writing software in a blog post, saying, “As long as software requires such concerted energy and so much highly specialized human focus, I think it will have the tendency to serve the interests of the people sitting in that room every day rather than what we may consider our broader goals.” We’re increasingly wary of the motivations of the people who build our software because using software takes up more and more of our waking attention. It also meant that when I shut down down Rume, the app from my last startup, it would probably never run again.

Another difference between a chair and a garden is that most people intuitively understand how a chair is constructed and how to fix it if problems arise. Like plant-sitting for a friend’s garden, maintaining unfamiliar code is challenging. While best practices and standards exist, it’s not easy to be productive when parachuting into a codebase.

This makes the code I write more valuable to me than anyone else. We did a ton of great work at Rume, but now that we’re not building on it, the code’s value is depreciating. And we’re not the only ones. So much code, and thus human time, effort, and thought, is effectively trash instead of recycling.

While I’m still adjusting to the mental model, I think this is okay. Writing software might be the largest lever we have. A few people can genuinely impact millions or billions of others. WhatsApp,. as an extreme example, had fewer than 60 employees when Facebook bought it for $19 billion. Its Android app, which served over 1 billion people, was maintained by four people. And when they added end-to-end encryption, that billion people got it overnight. People can credibly do years or even months of work and create something that changes the world, so even if they have to throw away most of their projects, the net gain might be better for everyone.

I’ll probably always have some nostalgia for Rume, but maybe not all projects should be preserved. I bet I’m not the only one who was somewhat relieved when AOL finally shut down their instant messaging service, AIM, in 2017. No one needs to preserve the cringy conversations I had as a 13-year-old. Some things have outlived their time.

This lightly edited personal update from 2022 felt relevant to share more broadly.

They want to work with you because they trust your judgment. If you work with them against your better judgment, you're not just playing yourself but also playing them. They have a lot of respect for you. Have enough respect for them to tell them the truth and work through the consequences.

Subscribe now

If they're making a go/no go decision and looking to you as the company leader for advice on whether they've done enough diligence, they're asking for your expertise, not just your praise.

If they're an engineer who thinks there's a market opportunity for a piece of technology they'd love to build and ask you to do it with them, they're asking for your judgment, not just your compliance.

Yes, they'd love for you to agree with them. They want to move forward on this with you. But they want you to agree with them because you're convinced, not because you feel social pressure.

I had to learn this one the hard way. It almost cost me a twenty-year relationship and burned me out. We're still repairing parts of it. I just wanted him to be happy, but it's a hard act to keep up.

The problem is that you can only keep following a path you don't believe in for so long. The dissonance will build, and you'll eventually have to tell them why you're acting funny. You might even admit you've had reservations for a long time. Suddenly, the floor falls out from under them.

They'll feel betrayed. They misunderstood their situation. They thought they were working with someone they deeply respected on a problem you both mutually believed was hard but worth a shot. But they've been alone, out on a limb this whole time.

You're their protection against feeling foolish, and instead, you'll become the cause of their embarrassment. You think you're doing them a favor. You see how good they feel working with you, and you're happy to see them happy. But they won't feel that way anymore.

Don't coddle them; they're an adult. Just tell them the truth. They can take it. Tell them how you feel. Even if they're sad now, they'll know they can trust you. They asked you because of who you are. So, be your whole self in the situation; be the person they know you to be.

After all, If they wanted anything less of you, you wouldn't want to work with them in the first place, would you?

Takeaways

• Embeddings are lists of numbers in many-dimensional space that you can compare with trigonometry. Those numbers represent how similar things feel.

• They’re great at mapping unbounded input to bounded output, specifically to records in your database.

• Combining embedding ranking with odds makes a new category of game mechanics possible. I’ll show you some examples that I’ve built.

The Project

Around this time last year, I was working on Aimigo, my exploration into a generative AI Tamagotchi. Tamagotchis only have a few features: They eat, poop, play, get sick, misbehave, and grow up. I set out to build new variations of these mechanics to make something players felt they had a relationship with that wasn’t a romance-based chatbot. I guessed that a character with its own taste that showed you how to care for it would be compelling.

Embeddings

Vector embeddings are just lists of numbers, usually 0-1, representing coordinates in space. However, instead of 3-dimensional physical space, they often represent an idea space with thousands of dimensions. That can be difficult to conceptualize because the dimensions don’t represent tangible things like “brightness” or “mass.” While it’s frustrating that they’re a black box, close items semantically feel more similar, and far ones feel more different. Vector embedding distances are compared in a variety of ways. I used cosine distance because that’s what OpenAI embeddings were optimized for.

I’m drawn to embeddings because of how well they handle tasks that look like search. They can take arbitrary text requests and convert them into bounded output. For example, the icon search from the last article takes any string and returns a list of appropriate icons from our database.

Generally, I’ve found embeddings more helpful in building game mechanics than language models. Comparatively, language models were slow, expensive, and unreliable. The more I used them for business logic, the harder it was to test and ultimately ship. I did get value out of them for small targeted tasks. However, I had better luck combining embedding ranking and weighted odds for most of the game engine.

Feature 1: Choosing characters

Players encounter the first embedding mechanic in Aimigo when they create a character. I bought a set of animal models from the creator Omabuarts, with over 150 species and a consistent set of animations.1 Then, I had players describe their character’s personality and used that to determine the species of their character.

Since the actual character creation is not very eventful, I built this character chooser to make it more visual. You can try different descriptions for yourself.

Ranking possible species

I asked the player to describe their desired character and converted that into an embedding. I had already calculated the embeddings for each species, which were ordered by their distance from the description.

This app is written in Elixir, which I’ve grown to love. I recognize Elixir is not the most popular language. Hopefully, the examples are simple enough to grasp through syntax.

characters = Repo.all(
  from(c in Character,
    order_by: [
      asc: cosine_distance(c.embedding, ^description_embedding)
    ],
  )
)

This worked pretty well. The description “strong, brave, and heroic” is closest to a lion, whereas “brooding, goth, and dark” is closest to a crow. However, after playing with it, I thought the determinism diminished the fun factor. I didn’t want players guaranteed a species just because they knew the words to get them, so I inserted a little weighted randomness.

Odds for choosing a species

I got a lot of mileage from choosing a random number from a normal distribution. In this instance, I took the absolute value of a normal distribution. I centered it around 0 so species with short cosine distances were more likely to be chosen. 2

number_of_characters = length(characters)
index = floor(abs(:rand.normal(0, Math.pow(number_of_characters/6, 2))))

character = Enum.at(characters, index, Enum.at(characters, 0))

The second input for a normal curve is variance (σ²), the square of standard deviation (σ), representing how wide or narrow the curve’s tails are. By adjusting it, I can tweak the relative likelihood for a player to get a species towards the beginning or end of the list. I divided the characters into six buckets of rarity.3

This weighted randomness gave me two features.

1. Players should feel like their description applies to the chosen species. Even if it’s not the top choice, all top options by proximity should be close to the right vibe.

2. If the species doesn’t match the description, that’s because something interesting happened, and that character is rare. A brooding crow will be much easier to get than a gloomy narwhal.

I wondered if I could skip this whole thing and instead feed the description and the list of possible species to a language model. Overall, I found that to be slower, more expensive, and less reliable. Prompting something like “Most of the time, give me the most intuitive species, but occasionally give me interesting unintuitive species” was ineffective. The embedding method is more predictable, inspectable, testable, and fun.

From then on, the character descriptions are fed into the language model for flavor text. When a player pets a grumpy character, it should respond less gratefully than a warm and fuzzy one. This should make the gloomy narwhal even more unique!

Feature 2: Feeding characters

The next mechanic I built lets players feed their characters.

1. The player offers their character a meal through a text input.

2. The character decides if they’ll accept it; if they do, they eat it and find out how much they like it.

3. The more they enjoy it, the happier it makes them.

4. Over time, they develop a taste for some foods over others.

I implemented this with a more complex version of my ranking and odds approach. The memory system I built was heavily inspired by a paper that had just come out about 8-bit sprite agents interacting in a small town: “Generative Agents: Interactive Simulacra of Human Behavior.”4

Ranking past experiences

When a player offers a character a meal, the meal is converted into an embedding to find relevant memories. This embedding search algorithm has three parts.

Cosine distance

Sort by the cosine distance between the embedding of the meal the player offered and the character’s memories of past meals. When players offer their character a calzone, they recall pizza more strongly than sushi.

Time decay

The second part of the algorithm biases towards more recent memories to ensure the character can grow over time. The general function is called exponential decay.

def time_decay(datetime) do
  hours_ago =
    DateTime.utc_now()
    |> Timex.diff(datetime, :hours)

  Math.pow(0.9995, hours_ago)
end

I chose hours as the base unit. Then, I tweaked the rate of decay to see what felt right. I settled on a curve that flattened out at about a year.5

Subjective importance

The last part of the memory score evaluates how impactful each memory is to the character. More important memories should affect more decisions. Here is a targeted place where language models were effective. I asked GPT-4 how impactful it believed the memories were on a scale of 1 - 10.

message = """
On the scale of 1 to 10, where 1 is purely mundane
e.g., brushing teeth, making bed) and 10 is
extremely poignant (e.g., a break up, college
acceptance), rate the likely poignancy of the
following piece of memory.
Memory: #{memory.content}
"""

The language model responded to adjectives and modifiers. For example, a character who likes cornbread is less enthusiastic about stale cornbread. If they enjoy sushi, they’re still weary of day-old sushi. The model also reacted well to the character’s species. A lion liked gazelle, whereas the herbivores did not.

The food-based memories mostly scored low in subjective importance; however, occasionally, they did not. I once fed my lion the bodies of my enemies, and they loved it. 9/10 core memory. When I fed them other food later, they’d say, “Meatloaf looks dull; I’m ready to feast on enemies!” or “Decent meal, but lacking the thrill of vanquished enemies.”6

Combined score

Cosine distance, time decay, and subjective importance are normalized to 0-1 and summed together to calculate the total retrieval score.

def normalize(value, max, min) do
  (value - min) / (max - min)
end

The normalization function isolates the differences between the memories in the set by normalizing them between zero and one. This does two valuable things:

• This makes scores with different ranges directly comparable. Otherwise, subjective importance with a rank of 1-10 would be so large that cosine distance would be irrelevant with a range of 0-1.

• This isolates the differences between values. Even if cosine distance and time decay have possible ranges between 0 and 1, if the data for one are between 0.5 and 0.9 and the data for the other are between 0.3 and 0.5, the first might disproportionately influence the result.

Since there were only hundreds of memories per character, I fetched them from the database and scored them in the application code. It was easier to prototype, and the numbers were small.7 Whenever I needed memories, I did the initial scoring in Postgres with pgvector, then normalized and combined the scores in a reducer function.

Once the memories were ranked and sorted, I fed the top 15 memories into a language model with a prompt like “Given these experiences, how much do you expect to like tacos on a scale of 1-5?”8 This distilled the character’s expectation for a meal to a number I could put into the game engine to calculate the odds of getting different reactions from the character.

Odds for eating a meal

The odds mechanic also has two parts:

• Does the character accept the meal

• If so, how much do they enjoy it?

Both have similar mechanisms.

Odds of accepting a meal

def meal_accepted?(score) do
  if :rand.normal(2, Math.pow(1.25, 2)) < score do
    :accepted
  else
    :rejected
  end
end

The app picks a random number on a normal curve centered at 2 with a standard deviation (σ) of 1.25. The character accepts the meal if the expected score exceeds the random number.

For example, when the character gives a meal an expected score of 1, they think they won’t like it. Most of the time, they’ll reject it without trying it. Conversely, if the character gives the meal an expected score of 5, they think they’ll like it a lot. They’ll rarely reject these. I centered the curve around two instead of three to bias characters towards trying things so players aren’t constantly rejected.

Odds of enjoying a meal

If the character accepts the meal, I calculate how much they liked it based on their original expected score.

def meal_enjoyment_score(expected_score) do
  raw_score = :rand.normal(expected_score, Math.pow(1.25, 2))

  cond do
   raw_score < 1 -> 1
   raw_score > 5 -> 5
   true -> round(raw_score)
  end
end

I pick a number from the same normal curve as the last one. This time, I center it around the expected score and clean up the output so it’s always an integer of 1-5. A character is more likely to enjoy a meal if they thought they would. Their taste is influenced by past decisions but not determined by them.

Once that score is calculated, a few in-game things happen:

1. Memories of this meal are created with embeddings for strings like “Sparky thought they would like tacos” and “Sparky decided to eat tacos and loved it.”

2. The character’s happiness is updated based on how much they liked the meal

3. A flavor text response is created to communicate what happened to the player in a little speech bubble.

4. Create a record of how that meal was received. Players can see whether it was accepted or rejected, the actual vs expected score, and if you click into the meals, see what meals they were thinking about when they ate it.

Conclusion

I had a bunch of vague ideas for where to go next. The characters’ personalities could change based on how players treat them. If you left them starving, they wouldn’t be as friendly. Get some pseudo-nutrition information so they grow differently depending on whether they get protein or sugar.

None of these necessarily felt impactful enough to make players build a strong enough relationship with their character to want to care for them every day, the way people bonded with the original Tamagotchi. Ultimately, I lost steam. However, I hope these mechanics will inspire your projects. Aimigo was rough around the edges, but there’s a kernel of something exciting here.

I hope some of the work I did here inspires you. If it does, I’d love to see what you build. Send me a note or tweet at @ba_wolf.

Here is a summary of my implementation decisions and some sources for other options.

Vector database

I chose pgvector/Postgres, but there are plenty of other choices, including some for other standard databases like MongoDB.

Pgvector client

The elixir library for use with Ecto. You can find a library for your client of choice here.

Database host

This app is hosted by fly.io, though their option wasn’t really a managed instance at the time. Now, they have a managed solution with Supabase. If you want to look up someone else, Pgvector has a clumsy list of hosts that support it in this git issue.

Embedding model

Initially, I used OpenAI’s `text-embedding-ada-002`, but since writing this, I have upgraded them to OpenAI’s `text-embedding-3-small`. If you want to try something else, check out Huggingface’s leaderboard.

Distance metric

I used cosine similarity as our distance function because that’s what OpenAI recommends for their embeddings. Other embeddings may be optimized for different strategies. Pgvector supports l2 distance, inner product, and cosine distance.

This feels like the right place to start if you’re an app developer looking for an excuse to dip your toes into this new AI world. Embeddings are just arrays of numbers, but they contain a compressed form of a considerable amount of human knowledge and shrink features that used to be substantial specialized projects into ones that individual product engineers can take on.

There are a ton of tooling options available to use embeddings. I’ll highlight our choices and note where you might want to make different ones for your situation. Here are some points I hope you take away:

• Vector embeddings work for search and recommendations because they’re good at measuring similarity to arbitrary input. This even works for different spoken languages like French or Japanese.

• Pgvector is a Postgres extension that stores and queries embeddings without adding a new service. It’s powerful because it can combine standard SQL logic with embedding operations.

• Unlike LLMs, working with embeddings feels like regular deterministic code.

The project

My friend Charlie Yuan and I built this mini icon app to help people discover icons. It’s pretty short and sweet. We have icon sets you can query, bookmark, and add to your project.

Pgvector

There are a bunch of specialized vector databases to choose from. Instead, we chose Postgres with pgvector to blend embedding search with business logic like filtering and scoring. While it’s not the fastest vector database, we didn’t want to have to compare results across multiple data sources. Pgvector probably already has a library for your favorite database client. Our project was Typescript through and through, and we used drizzle-orm. The docs will be a more robust place for setup documentation, so I’m leaving out that part to focus on the features you can build.

Creating embeddings

Once set up with pgvector, we created a strategy for encoding our icon data into vector embeddings. Embeddings are points in many-dimensional space, up to thousands of dimensions. Unfortunately, the axes of that grid are not humanly grokable ideas like “size” or “brightness.” They’re a bit of a black box. Luckily, like any good abstraction, they’re a black box with a good API.

The best practice seems to be finding the details that best represent what people want to search for and creating a function that outputs that as a string. Our icons can be a part of many use-case-based ‘categories’ and have many descriptive ‘tags’ associated with them. We encoded that information along with the icon name because that best represents what the icon is. Whereas the name of the icon set the icon belongs to, or its dimensions aren’t relevant. The strings we generated looked basically like this:

const createIconEmbeddingsString = (icon) => `icon: "${icon.name}", categories: [${categories}] tags: [${tags}]`;

Next, we chose an embedding model. OpenAI’s embedding models will probably work just fine. We’re using their `text-embedding-3-small`. If you want to dive in, check out the leaderboard and pick the model that best meets your needs. Whether you use an embeddings API or self-host an open source option, the interface should be text in and embeddings out.

Implementing search

Many sites implement search, but most icon sites implement search by text matching or full-text search. If you’re looking for a dog icon, they search over the icon metadata for icons that have ‘dog’ in them. If they want to get craftier, they come up with a bag of words related to ‘dog,’ like maybe ‘k9’, ‘puppy,’ and ‘woof’ to catch near misses. That’s pretty fragile. Someone has to choose tags for each icon; if they miss an important one, users won’t find what they’re looking for.

Similarity search

Our app gets relevant results when searching for ‘dog.’ We also get solid results for ‘puppy’ without a bag of words by measuring the cosine similarity between the embeddings of your search query and each icon. There are multiple ways to measure how similar embeddings are to each other, but OpenAI’s embeddings are designed to work well with cosine distance. Cosine similarity is just the opposite of cosine distance. Order by cosine distance wherever you can to take advantage of indexes.

cosine_similarity(x,y) = 1 - cosine distance(x,y)

You can even try dog breeds like ‘hound,’ ‘poodle,’ or my favorite ‘samoyed.’ It pretty much just works. But that’s not all; it also works for other languages. Try ‘chien’ and even ‘犬’1! With pgvector, we can get these results with simple SQL queries.

SELECT
  1 - cosine_distance (search_query.embedding,
    icon.embedding) as similarity,
  *
FROM
  icon
  join search_query on search_query.text = 'dog'
ORDER BY
  cosine_distance (search_query.embedding, icon.embedding) ASC
LIMIT 50;

Since we’re ordering by distance, every search returns every result in the table in order of closeness. We cut off results by a fixed number to make this manageable, limiting the query to the top 50. Using an arbitrary distance cut-off is tempting, like querying for results with a cosine similarity of less than 0.8. Unfortunately, the absolute distance for one query to produce correct-feeling results might differ drastically from another. We limit by the number of results, not by a minimum value, whenever possible.

Filtering

If we only wanted similarity search, any vector database would be fine, but Postgres allowed us to layer more features on top. The initial search looks across all icons in all icon sets, but our user’s style might only match some of the icon sets. We can filter by values like in any other Postgres query.

SELECT
  1 - cosine_distance (search_query.embedding,
    icon.embedding) AS similarity,
  *
FROM
  icon
  JOIN search_query ON search_query.text = 'dog'
  JOIN icon_set ON icon_set.slug = icon.icon_set_slug
WHERE
  icon_set.slug in('lucide', 'mdi')
ORDER BY
  cosine_distance (search_query.embedding,
    icon.embedding) ASC
LIMIT 50;

This is deterministic; everyone searching for a ‘dog’ will get the same results. However, the inputs are still unbounded, so embedding search doesn’t guarantee that the best results will be produced for every input. We can try different embedding models or ways of encoding icons into embeddings to improve the system.

A more complex algorithm

The embedding search usually puts the correct icon on the page, but the correct icon isn’t always the first result. We could make a simple algorithm that adjusts to user feedback and improves over time. To do this, we’d count every time a user clicks on an icon for a particular search query. When ranking search results, we’d create a score for each icon that combines the embedding search with the click data. 

Here’s a simple ranking algorithm. The details look messy because there’s some null checking and unit conversions2, but here are the basics.

1. Get the cosine similarity of the icon for the search query. It will be a number between 0 and 1. Multiply it by 0.5.

2. Divide the number of clicks for each icon by the icon with the most clicks for that query. This normalizes the most clicked icon to 1, and the least clicked to 0. Multiply by 0.5.

3. The final score is these two values added together for a range between 0-1.3

SELECT
  (
    0.5 * COALESCE(      -- so nulls are turned into 0
     1 - cosine_distance (search_query.embedding, icon.embedding),
     0
    ) + 0.5 *     -- so clicks matter less than embeddings.
    COALESCE( 
      search_query_selection. "count"::decimal / 
        max(search_query_selection. "count") OVER (),
    0)
  ) AS score,
  icon.*
FROM
  icon
  LEFT JOIN search_query_selection
    ON icon.id = search_query_selection.icon_id
  LEFT JOIN search_query
    ON search_query.text = 'dog'
      AND search_query.id = search_query_selection.search_query_id
  ORDER BY score DESC
  LIMIT 50;

Should vector embedding distance and clicks be equally weighted? Should the order of magnitude of clicks matter more than the raw number? The algorithm might need tuning, but this is just an example of how the database handles the calculation nicely.

With a separate vector database, we might have to get values for all icons from both databases before comparing them in application code or making tradeoffs like pulling 100 results from the vector db and filtering the Postgres query for click score to those results or vice versa. Instead, we simply query for results and display them.

Similar recommendations

Additionally, we include a content forward section of each icon page with other icons in the same icon set and category. That way, you can see other ‘navigation’ icons when looking at `arrow-up` in case you need those. Unfortunately, not all of our icon sets have categories. In these cases, we make a similar icons section using embeddings. Instead of getting an embedding from user input, we can compare the same cosine similarity measure against the selected icon’s embedding.

WITH current_icon AS (
    SELECT
      embedding,
      slug,
      icon_set_slug
    FROM
      icon
    WHERE
      icon_set_slug = 'lucide'
      AND slug = 'activity'
)
SELECT
    *
FROM
    icon
INNER JOIN current_icon ON
icon.icon_set_slug = current_icon.icon_set_slug
AND icon.slug != current_icon.slug
ORDER BY
    1 - cosine_distance (
    icon.embedding,
    current_icon.embedding
)
LIMIT 50;

Conclusion

I hope you have a good experience playing with our app and vector embeddings! We owe a big thanks to all the engineers who pushed the state of the art so far that we can stand on their shoulders. I hope this modest contribution helps make adding embedding features to your app approachable!

Here is a summary of our implementation decisions and some sources for other options.

Vector database

We chose pgvector/Postgres, but there are plenty of other choices, including some for other standard databases like MongoDB.

Pgvector client

We worked in Typescript and chose drizzle-orm. I’ve also worked in the Phoenix elixir ecosystem using the elixir library with Ecto. You can find a library for your client of choice here.

Database host

Our app is hosted on Neon. I’ve also used fly.io, though their option wasn’t really a managed instance at the time. Now, they have a managed solution with Supabase. If you want to look up someone else, Pgvector has a clumsy list of hosts that support it in this git issue.

Embedding model

We chose OpenAI’s `text-embedding-3-small`. If you want to try something else, check out Huggingface’s leaderboard.

Embedding string

We embedded strings of key and value pairs for the attributes we thought best described our icons. It doesn’t seem like any of the major players are doing anything crazy here; the significant knobs appear to be whether or not to embed keys or just values and which attributes of your records are relevant. Other examples in OpenAI’s cookbook show other choices.

Distance metric

We used cosine similarity as our distance function because that’s what OpenAI recommends for their embeddings. Other embeddings may be optimized for different strategies. Pgvector supports l2 distance, inner product, and cosine distance.

Search Size

In these examples, we limited our queries to the top 50 results. You can also limit your search to be above or below a certain distance threshold. That doesn’t seem super reliable. Relative distances seem more meaningful than discrete amounts. If you’re set on using a threshold, I’d recommend keeping it pretty wide, like 0.1 or 0.05, and using it with a limit. You may get some mileage using a wide range to avoid returning irrelevant long-tail results.