An experiment in AI filmmaking to create the style and feel of a 1970’s gangster film, using off the shelf AI tooling.
In the style of a gangster film …
Summer Experiment
With summertime comes time to play, and a backlog of experiments on my whiteboard. During the year, I see updates across the AI space, and with downtime, I set aside time to play.
This experiment was based on the discover of Runway’s GEN-2 AI model that creates video. I wanted to know if I could make a movie!
I’ve been a long time fan of Runway, using it since it’s first version, but this upgrade of the platform is something else. You can prompt shots. Yes, shots. Like from the movies.
So for example, I can prompt:
“In the style of a gangster film, 1970’s filmmaking, establishing shot los angeles, action film. Morning, sun comes up.
I will create it’s best “guess.” I can change things on the prompt and get specific on the time of day or the camera changes. It looks bad most of the time, but some of the outputs are amazing.
Every shot, in the movie above, was generated with a formulae using
1. Leonardo.ai to prompt the image generation model for Art Direction,
2. and then a systemic approach to shot development in Runway’s Gen-2.
I left a balance of shots in the edit. Some are amazing, and some are clearly generated. I deliberately wanted it to be generative for good or bad. The point was to see if it is possible.
Ultimately the technology is still early. But, man, is it coming quick!
So, here are a couple of notes going through the process of making a completely generative short film with Gen-2 from Runway, Leonardo.ai, and some geeky love of action movies.
(You should try it out for yourself)
Coherent Story is Hard
Its hard to get a coherent story through random shots – you have to play this game with the AI, to figure out what the model can deliver. These image models can do individual medium close ups of humans. So, that’s the what you have to build.
This is why most of the AI filmmaking (to this point) has been single close up “trailer type” of content. You might have seen the Wes Anderson stuff, or the Harry Potter Balinciaga. Getting one shot to play cleanly to the next is a multi-faceted battle and the models struggle to do this.
I desperately wanted to do cars, action, shooting, doors opening… no one got shot in this film! So while close up of actors is cool, it’s limited and not “gangstery.”
I did try two things to try to pull something together.
- I used a track by The Herbalizer, a talented DJ Duo, to give the piece structure. I thought about generating the music, but this track gave it presence. This gave the “gangster tone” I was looking for, but also the key changes for scenes and shots.
- I used tropes that clearly the model knew how to do. That was “sunglasses,” “1970’s”, “filmmaking,” and it clearly understands “Los Angeles,” because that’s probably where most of the cinematic data has come from.
AI Reference Material is Huge
I had difficulty figuring out the shots I wanted, so I went through a sort of “Art Direction phase.” Once I had a series of images that worked for each piece, I knew how to prompt for it. I also could use the image as a prompting reference in Runway.
Opening Day: Los Angeles, Palm Trees, Day, Cadillac, Downtown, 1970’s
The Deal: Interior of Man with Sunglasses, 1970’s architecture, Gangster
The Night time: Night, Palm Trees, Rodeo Drive, Hollywood
The woman: Interior Hollywood Bar, 1970’s, Beautiful Woman, Martini
the club owner: Interior Los Angeles, Black Club Owner, 1970’s
The fight: Man with Sunglasses, dramatic cinematic, gun fight, action, 1970’s
You can’t do Action… yet
I have to reiterate the frustration I had trying to get any motion of a Cadillac driving down the 405. There just doesn’t seem to be shots of cars in motion, or the model can’t do it. I also found, most often, that the model can’t move the camera or change the framing. Great filmmaking has architected shots where the camera and subject move, and we can’t quite do that yet. It’s just actors sort of smooshing around.
Also, It’s PG Only – Words like “sexy,” “seductive,” “breasts” all trigger alarms. I’m sorry, I can’t make a gangster movie without naked women. So, I don’t what that means you, Gen-2 model people at Runway,
— I just want to call attention to it!
Runway Gen-1 didn’t hold attention like this model did. This upgrade is quite promising, and I eagerly want to see what the next one is like.
If I could do this exercise but actually be able to flip a van over, or actually stage a gun fight, or maintain a lead actor, maybe I could actually tell action driven gangster stories. Just by writing each shot.
That’s unbelievably amazing if that is true.
I am convinced it is coming. Thanks for reading. Let me know if you have any thoughts or feedback.
