In 2010 I directed a music video for my friend Mark and his daughter. A fun little recording project they called “Catastrophic Neon Rainbow”. We created a simple set in his music studio and filmed away. Now, 15 years later, we are collaborating again, this time for a new track for his band “No Berlin“.
A few months ago we started talking about making a music video for one of his songs. The catch? He’s based in Berlin, Germany these days. So since I wasn’t about to fly over there to shoot a music video I thought I’d try having him send me photos of himself and I would see if I could try some generative AI tools. I also had him send me video of him singing the song so that I could try Runway’s “Act One” feature which is basically like motion capture of one’s face that you can apply to another photo or video.
The final video will be on this site soon but for now you can see some short clips and how I went about achieving them. I ended up doing a lot of prep in Photoshop and then fed those images into Runway with “image to video” and eventually using those resulting videos for my Act One shots for lip syncing.
Here are a few behind the scenes of the process I’ve been going through. After a lot of time spent figuring out what I was going to do with these shots I eventually landed on the idea that I could “inform” Runway with some simple visual cues. For example in this first sequence I added some small fires using Photoshop Generative AI. Runway took it to the next level.
This one demonstrates a use of Runway’s Act One feature. Sound on:
This kind of blew my mind. Using a few photos Runway was able to tween together a sequence. Although the lighting in the original photos wasn’t entirely consistent you could really do a lot with this.
This final clip was a bit of a process. I knew I wanted to have a drone style fly-in shot revealing Mark laying lifeless on a long road at night. It turned out that the most realistic way to achieve this was to go in reverse. This would make sure the final shot was my composed image. There is certainly some degradation the wider the shot went but I think the idea is there.
I’ve thought about making another music video. We’ll see!