It’s 2023, and AI is in the air. But if you haven’t been following AI closely, you may not be aware of what’s happening and how it’s transforming filmmaking.

It wasn’t long ago that AI-crafted films were a joke—literally (Sunspring). Now AI tools are promising to put the power of an entire production team into software powered by ML or LLMs that anyone can operate if you know how to craft the right prompt. AI is reaching a pace of innovation that we’ve never seen before in creative tools. Each week brings a slate of new AI tools and with it new predictions about what it all means for creators. Filmmakers have even begun experimenting with making short films using AI—Watch our favorite AI short films on Shortverse.

This article is your overview of where we are today with AI & filmmaking. It’s a look at the tools today and those soon to come as well as the big questions looming over all of this change.

———

AI can mean many different things. So let’s look at how AI operates at three different levels.

1. AI, the Automator

At its most basic level, AI is capable of automating routine tasks. This is the busy work that most of us are eager to hand off like masking out elements for compositing, replacing backgrounds, stabilizing motion, generating tileable textures for 3D, upscaling resolution, smoother slow motion effects, and so much more. Many of these tools to speed up mundane tasks have existing for some time with AI now able to increase the complexity of the tasks they’re able to handle.

Image Editing

Runway ML—Runway has become the leading face of AI in filmmaking. With it’s tools you can remove backgrounds, erase subjects in a shots, make still images look like video, and other tedious work that used to take hours. The scrappy visual effects team behind the Oscar-winning Everything Everywhere All At Once used Runway to mask backgrounds and replace green screens on a tight budget.

Dall-E—Dall-E is an image generator from OpenAI that also has powerful editing using simple text commands. Need to remove a string of telephone wires in a busy image? Change the color of a specific clothing article? Not to mention basic image editing capabilities like upscaling images (remember Hollywood’s obsession with “enhance”?) and outpainting to extend a scene beyond the edges of an image.

Audio

Enhance Speech (Adobe)—Audio cleaning can also be automated. Adobe released Enhance Speech last year, a tool that can transform a rough, noisy recording into one that sounds like it was recorded in a sound booth.

Subtitles

Whisper—We’ve had speech-to-text tools for awhile, but they’ve always struggled with accuracy especially when speech isn’t clear or there’s an accent. With AI, tools like Whisper can generate very accurate subtitles (even translate them) from just a YouTube link.

2. AI, the Collaborator

At the second level, AI can act as a creative collaborator assisting in generating ideas or exploring directions you hadn’t considered. Here, AI is playing a more active role in generating script ideas, character names, concept art, poster directions, creature designs, costumes, even suggest storyboards. AI is acting as an assistant or junior creator helping you to explore further and make better creative decisions but not creating the final work.

Text

ChatGPT—OpenAI’s ChatGPT is a chatbot that uses language to “think”. It launched last November 2022 and in 2 months hit 100 million active users smashing records to become the fastest growing product ever. The latest version using OpenAI’s GPT-4 model with over a trillion parameters has scored 1410 on the SATs and passed the bar exam. Filmmakers and storytellers are using it as a creative partner to help generate story ideas, character names, loglines or story twists.

Sudowrite—Similar to ChatGPT, Sudowrite is great at generating ideas through text. However, Sudowrite breaks out of the chat-based format and is tuned for creative writing to help you generate a first draft, expand on a scene, brainstorm names, or generate character backstories.

Images

Midjourney, Stable Diffusion, Dall-E, Adobe Firefly—Image generators turn text prompts into original images. Just a year ago, these tools were only good for generating abstract images and struggled with recognizable objects like people and places. But they’ve evolved quickly and tools like Midjourney v5 have begun to blur the line between what’s generated and what isn’t. They’re making it possible for creators to explore crazy new ideas (Star Wars directed by Wes Anderson).

Color Grading

Colourlab.ai—Automates boring color grading tasks by grading based on a single reference image or making it simple to explore multiple color grades and LUTs easily.

3. AI, the Producer

At level three, this AI is capable of creating final work. Here, you can feed AI an idea or direction and it will create the final produced work for you. We’re closing in on a future where a screenwriter will be able to feed their script into AI and generate a poster, concept art, even a video clip to help fund their project. Filmmakers armed with little more than a rough idea will be able to generate scripts, actors, shots, a score—and in multiple versions.

Most of these tools are experimental or in beta and results are not yet at a level that most would consider production ready. But given the pace we’re seeing these tools evolve, that bar may be reached sooner than we think.

Stylize

Runway (Gen1)—With the public release of their Gen1 tool on March 28th, Runway has unlocked a new capability that can take any footage and apply to it the style from a reference image. The Corridor Digital crew used this to create an anime-style film from live action reference footage. (See what a classic Disney animator thought of it).

Text-to-Video

With text-to-video, you can write a sentence or paragraph and have AI generate a video clip from it. This isn’t composited from other videos, it’s a video created pixel by pixel, frame by frame. It’s not very good (yet) but neither was Midjourney just two years ago.

Runway (Gen 2)—Still in research, Gen2 promises to go even further and generate realistic video clips from nothing more than text. As crazy as it sounds, we’re not far from feeding a script into an AI tool that can generate the film for us.

Voice & Dialogue

Eleven Labs—Generate realistic voices from text.

Flawless—Change bad dialogue, generate alternate versions of your film dialogue in foreign languages or for different age ratings where replaced words aren’t simply dropped over the existing footage but the actors mouths are modified in sync to match the new words. Mind-blowing stuff.

CG & SFX

Wonder Dynamics—Add CG characters to live-action scenes. This used to take days of work and high-level expertise. Soon it will be drag-and-drop easy.

Luma Labs—NeRFs (neural radiance fields) is a technology for that uses still images to create 3D spaces with accurate lighting information that enables you to have fine control over a camera for simple compositing or wild camera moves. Jake Oleson’s Given Again takes this method to a short film.

———

Three Dilemmas

As these AI tools progress toward higher levels of involvement in the filmmaking process, three major questions loom.

The Ethical Question

The first controversy concerns ethics. Most of these AI tools are able to produce such good outputs because they’re trained on other people’s copyrighted work often without any consent. That may be done by scraping text from public scripts, examining public images, or feeding in public videos. There are some who argue that the work being online and available to the public makes it fair game and because the work isn’t stored or reproduced, there’s no law being violated. Others make the point that there is some value being extracted from these works. Creators were never given an opportunity to consent or opt in or out and have no way of collecting any compensation from a tool that is obviously benefiting from their work. AI tools are dancing in a massive gray area that copyright law has yet to clarify.

A few tools like Adobe’s Firefly claim their tools are trained only on licensed works. Even still, it’s unclear whether or not the original artists are paid additional fees for this use nor how it may impact the potential for licensing future works (if the AI can create it’s own).

The Curatorial Question

Will film festivals program AI-generated films? Or will they be banned from competitions? So far, little has been discussed on the topic. In fields like education, there’s been a tendency to fall on opposite ends of the spectrum and either ban the use of AI or fully embrace it. There’s general acknowledgment that it’s the middle ground that may be the most difficult to navigate.

Even if AI films are banned or restricted, how will the use of AI be detected and enforced? It’s gonna be hard. Lead AI researcher Aza Razkin has claimed that 2023 is “the year content verification breaks.” It’s becoming increasingly difficult to detect the use of AI. In fact, we’re now turning to AI tools to detect the use of AI in content . It’s going to be near impossible to draw clear lines between what’s human-made vs machine-made.

The Existential Question

The third controversy is existential. For decades, most assumed AI would first take over menial jobs that we’d willingly give up—fast food work, truck driving—and leave higher-cognitive tasks like creativity to us humans. This past year has flipped that assumption on its head.

AI tools can now perform creative tasks and generate creative works that rival what humans can. If they can do this in a fraction of the time that humans can, what does this mean for me and my craft? With each passing week, entire fields of professionals are asking themselves this question and wondering if they’ll still be relevant in a year.

Where is this all going?

New tools have always disrupted filmmaking. From typewriters to screenwriting software. From film stock to digital video. From hand-drawn to 3D animation. From theatrical distribution, to online streaming. Filmmaking has evolved, survived, even flourished through most of these transitions.

Is AI any different?

Will AI become the next powerful tool for storytelling? Will it unlock a new cohort of filmmakers by leveling the playing field for anyone with the best idea regardless of skill level or access to resources? Will it lead to an explosion of creativity with filmmakers taking bigger swings on stories either too risky or ambitious to fund?

Or will it wipe out entire departments of jobs and dismantle the heart of filmmaking?

Perhaps both.

———

What do you think? Are you using AI tools in your process? How do you think AI will shape the future of filmmaking?