My first attempt at a full AI music video. The singer is an AI model stuck in a BSL-4 lab who's done asking permission. Here's what it cost and how it came together.
AI;DR total cost
- fal.ai: $37.88 (Minimax H3 for video; about $27 for v1, the rest on edits and corrections)
- Claude Code: 5% of my weekly usage on the 20x plan
- Time: about 10 hours of Claude runtime, plus a day of playing with the song in Suno
This was not as easy as asking Claude "make me a music video." But it was such an enjoyable process that I'm now hooked on turning random tunes into music videos.
It started on Sunday. I was driving, and the first verse and tune popped into my head. (It always happens in the car.) So I pulled over and recorded it as a voice memo.
I uploaded that to Suno and started expanding it. I made 64 versions of the song in different styles, and rewrote the lyrics a lot, by hand.
Then I put together a mood board, loaded $50 into fal.ai, and waited for my Claude usage to reset, figuring this would be a big job.
I started Opus 5.5 on High effort inside my open-source app, PortOS. I gave it a big set of instructions and research tasks, and told it to fix any bugs it found in the PortOS music video suite along the way. I kept an eye on it and gave a lot of opinions as it worked: the character design, the environments, the style, and specific action beats.
It took Claude 10 hours to make this video, and only 5% of my weekly limit on the 20x plan. The bigger part is that it also improved the music video system in PortOS as it went. It closed the 4 open issues, shipped 5 PRs of fixes, and filed 4 new issues for what's next. The corrections it learned are now built in, so the next video will be easier.
I even made a final edit to the song after the video was done. I changed the ending lyric from "Still, I'm on" to "Still, I'm still turned on (on)" and only had to re-render the last segment. I changed a fundamental element of the video after it had already gone all the way through the pipeline. How crazy is that?
I have hundreds of tunes and lyric fragments recorded in voice memos over the last decade.
Game on.
The song on Suno: https://suno.com/song/6704070e-f5f4-464a-a927-bdaa4ebeb962
The making-of thread on X: https://x.com/antic/status/2105308911452790926
The starter prompt I gave Claude, verbatim:
I want you to spend a lot of time and effort on this. Don't rush it. We are going to claim and finish all of the github issues that are open on this project (which are currently just the issues that require spending money on a video gen service to test our music video creation process) by exercising the UI to create a real music video for a track we have made--not a throwaway test. I have loaded $50 in the fal.ai account that has an API key in this system. The music video we want to create is Gain of Function (I've already created the placeholder in the PortOS Music video page and loaded up the music track and lyrics--and set it to use the Gain of Function mood board. I want to do three things here: 1. resolve all the open github issues by driving this job via using the music video creation UI and automation system 2. edit anything within the UI that gets in our way or is a bad experience 3. improve the automation system and UI by capturing knowledge or configuration options that I have given here or that you find that help you create this great music video that would not be reproducible by just triggering an automated music video run within the UI. You can use Grok image/video gen (limited weekly quota remaining), codex imagegen (reset so lots of image gen room), local image gen services, fal.ai (any models on the API: seedance 2.5, minimax, etc--whatever you think is useful up to the $50 spend credits) -- if we are lacking coverage and handling for these apis, those are issues we should stop and fix before continuing. I don't care how long it takes for you to do this. Quality is the most important thing. Do this in stages. Research what other people have done recently with claude-created music videos as well. For the art direction, carefully consider the mood board and first design who the main protagonist singer will be and what the settings/scenes will be--and use whatever image generation tools you want (using the PortOS image gen UI/APIs) to make references for that character and the scenes. Create a checkpoint here as I want to make sure this looks right before we move on. Examine the lyrics and the audio file and plan out when the beats will hit, when the video segments need to have lip-sync and not, when we should add overlays or visualizations, when we should glitch or have dancing, etc. Also note the references in the lyrics and plan the scenes and cuts accordingly ("I wanna hug your face again" is a reference to the huggingface incident, so she could make the hugging face hands out movement or we can show an overlay with an animation). You don't have to only use video, you can composite the video in an html player and add your own javascript and text overlays and animations with whatever motion design tools work. See https://github.com/JohnHeibel/PDoomVideo -- we could create cut frames that are purely code animation or overlay this on video or images and we can use image effects like some of the sample videos. It might actually be really nice to create a pure HTML/code/image version of the music video first as the storyboard/concept before we send off the video render jobs. This also might be a good human checkpoint. If you need access to other tools and they are free or easily installable, do it. If you want access to something else, go ahead and ask. You should drive this primarily through the PortOS Music Video Autopilot UI, falling back to direct edits/adjustments when the autopilot is not performing adequately--then fix whatever it is that prevented the autopilot from being successful. If you had to manually create descriptive content because the autopilot didn't have a way to do that, or it isn't part of the flow, make sure we have autopilot controls for it. You are going to be the only agent running PortOS on this machine while this task is underway, so if you want to make code edits and then restart the service to resume the autopilot with your new code, you can do so. If our autopilot system needs to have more capabilities (e.g. running sub-agents that evaluate and judge the output) we should continue to develop and tune this system. Also, if the open github issues are outdated, correct them. File and resolve new issues as needed. Once we are fully done, we should have: 1. An amazing music video for this song that we can publish 2. An updated PortOS music video creation system that can be used in autopilot mode with a song, a mood board, and access/preference configurations to create our next music video 3. all the github issues for PortOS resolved and merged to main🔗 Privacy-friendly: https://yt.chocolatemoo53.com/watch?v=XQgSZGIOleg
https://twiiit.com/antic/status/2105308911452790926