I want you to spend a lot of time and effort on this. Don't rush it. We are going to claim and finish all of the github issues that are open on this project (which are currently just the issues that require spending money on a video gen service to test our music video creation process) by exercising the UI to create a real music video for a track we have made--not a throwaway test. I have loaded $50 in the fal.ai account that has an API key in this system. The music video we want to create is Gain of Function (I've already created the placeholder in the PortOS Music video page and loaded up the music track and lyrics--and set it to use the Gain of Function mood board.
I want to do three things here:
1. resolve all the open github issues by driving this job via using the music video creation UI and automation system
2. edit anything within the UI that gets in our way or is a bad experience
3. improve the automation system and UI by capturing knowledge or configuration options that I have given here or that you find that help you create this great music video that would not be reproducible by just triggering an automated music video run within the UI.
You can use Grok image/video gen (limited weekly quota remaining), codex imagegen (reset so lots of image gen room), local image gen services, fal.ai (any models on the API: seedance 2.5, minimax, etc--whatever you think is useful up to the $50 spend credits) -- if we are lacking coverage and handling for these apis, those are issues we should stop and fix before continuing.
I don't care how long it takes for you to do this. Quality is the most important thing. Do this in stages. Research what other people have done recently with claude-created music videos as well.
For the art direction, carefully consider the mood board and first design who the main protagonist singer will be and what the settings/scenes will be--and use whatever image generation tools you want (using the PortOS image gen UI/APIs) to make references for that character and the scenes. Create a checkpoint here as I want to make sure this looks right before we move on.
Examine the lyrics and the audio file and plan out when the beats will hit, when the video segments need to have lip-sync and not, when we should add overlays or visualizations, when we should glitch or have dancing, etc. Also note the references in the lyrics and plan the scenes and cuts accordingly ("I wanna hug your face again" is a reference to the huggingface incident, so she could make the hugging face hands out movement or we can show an overlay with an animation). You don't have to only use video, you can composite the video in an html player and add your own javascript and text overlays and animations with whatever motion design tools work. See https://github.com/JohnHeibel/PDoomVideo -- we could create cut frames that are purely code animation or overlay this on video or images and we can use image effects like some of the sample videos. It might actually be really nice to create a pure HTML/code/image version of the music video first as the storyboard/concept before we send off the video render jobs. This also might be a good human checkpoint.
If you need access to other tools and they are free or easily installable, do it. If you want access to something else, go ahead and ask.
You should drive this primarily through the PortOS Music Video Autopilot UI, falling back to direct edits/adjustments when the autopilot is not performing adequately--then fix whatever it is that prevented the autopilot from being successful. If you had to manually create descriptive content because the autopilot didn't have a way to do that, or it isn't part of the flow, make sure we have autopilot controls for it. You are going to be the only agent running PortOS on this machine while this task is underway, so if you want to make code edits and then restart the service to resume the autopilot with your new code, you can do so. If our autopilot system needs to have more capabilities (e.g. running sub-agents that evaluate and judge the output) we should continue to develop and tune this system. Also, if the open github issues are outdated, correct them. File and resolve new issues as needed.
Once we are fully done, we should have:
1. An amazing music video for this song that we can publish
2. An updated PortOS music video creation system that can be used in autopilot mode with a song, a mood board, and access/preference configurations to create our next music video
3. all the github issues for PortOS resolved and merged to main
The starter prompt I gave Claude, verbatim:
I want you to spend a lot of time and effort on this. Don't rush it. We are going to claim and finish all of the github issues that are open on this project (which are currently just the issues that require spending money on a video gen service to test our music video creation process) by exercising the UI to create a real music video for a track we have made--not a throwaway test. I have loaded $50 in the fal.ai account that has an API key in this system. The music video we want to create is Gain of Function (I've already created the placeholder in the PortOS Music video page and loaded up the music track and lyrics--and set it to use the Gain of Function mood board. I want to do three things here: 1. resolve all the open github issues by driving this job via using the music video creation UI and automation system 2. edit anything within the UI that gets in our way or is a bad experience 3. improve the automation system and UI by capturing knowledge or configuration options that I have given here or that you find that help you create this great music video that would not be reproducible by just triggering an automated music video run within the UI. You can use Grok image/video gen (limited weekly quota remaining), codex imagegen (reset so lots of image gen room), local image gen services, fal.ai (any models on the API: seedance 2.5, minimax, etc--whatever you think is useful up to the $50 spend credits) -- if we are lacking coverage and handling for these apis, those are issues we should stop and fix before continuing. I don't care how long it takes for you to do this. Quality is the most important thing. Do this in stages. Research what other people have done recently with claude-created music videos as well. For the art direction, carefully consider the mood board and first design who the main protagonist singer will be and what the settings/scenes will be--and use whatever image generation tools you want (using the PortOS image gen UI/APIs) to make references for that character and the scenes. Create a checkpoint here as I want to make sure this looks right before we move on. Examine the lyrics and the audio file and plan out when the beats will hit, when the video segments need to have lip-sync and not, when we should add overlays or visualizations, when we should glitch or have dancing, etc. Also note the references in the lyrics and plan the scenes and cuts accordingly ("I wanna hug your face again" is a reference to the huggingface incident, so she could make the hugging face hands out movement or we can show an overlay with an animation). You don't have to only use video, you can composite the video in an html player and add your own javascript and text overlays and animations with whatever motion design tools work. See https://github.com/JohnHeibel/PDoomVideo -- we could create cut frames that are purely code animation or overlay this on video or images and we can use image effects like some of the sample videos. It might actually be really nice to create a pure HTML/code/image version of the music video first as the storyboard/concept before we send off the video render jobs. This also might be a good human checkpoint. If you need access to other tools and they are free or easily installable, do it. If you want access to something else, go ahead and ask. You should drive this primarily through the PortOS Music Video Autopilot UI, falling back to direct edits/adjustments when the autopilot is not performing adequately--then fix whatever it is that prevented the autopilot from being successful. If you had to manually create descriptive content because the autopilot didn't have a way to do that, or it isn't part of the flow, make sure we have autopilot controls for it. You are going to be the only agent running PortOS on this machine while this task is underway, so if you want to make code edits and then restart the service to resume the autopilot with your new code, you can do so. If our autopilot system needs to have more capabilities (e.g. running sub-agents that evaluate and judge the output) we should continue to develop and tune this system. Also, if the open github issues are outdated, correct them. File and resolve new issues as needed. Once we are fully done, we should have: 1. An amazing music video for this song that we can publish 2. An updated PortOS music video creation system that can be used in autopilot mode with a song, a mood board, and access/preference configurations to create our next music video 3. all the github issues for PortOS resolved and merged to main