Generative AI, the technology behind popular programs like Midjourney, Stable Diffusion, and DALL-E 2, which allows anyone to turn words into inspiring images, has been gaining a lot of mainstream attention recently. And few groups seem more excited about it than writers. Using written language to build images is what they do. But often, to introduce fans to characters and worlds, they need to resort to borrowed photos found on Google. This new generation of AI tools presents a much better way.
Brazilian best-selling author PJ Caldas used this technology to launch a book trailer for his first novel in English. The 82-second meditative animation uses AI for all aspects of production—from design to movement and music—and can be found at http://www.pjcaldas.com/ and his Instagram channel @thegirlfromwudang.
The film, inspired by the upcoming novel THE GIRL FROM WUDANG, uses visual references ranging from ancient Chinese paintings and calligraphy to modern comics and graffiti. Combined, they create a unique look that pulls the audience into the story’s universe. Even the theme music was composed by AI and then performed by real people. Animator Nick Tengri from iamYork and sound designers and composers from Quiet City lent their vast experience with big studio films and Netflix series to the project.
“Letting the AI create each frame on its own, without any human interference, was spine-chilling, and that’s what made it so interesting,” said PJ Caldas. The only parts made by hand were the title cards because, ironically, generative AI can create amazing images but doesn’t know how to write. “We left any writing within the animation exactly as the AI created it, but if you look closely, it’s all gibberish.”
Artificial intelligence is one of the main inspirations for The Girl from Wudang, which even features AI characters in the plot. “Letting the AI create the book trailer was like allowing the characters to tell their own story,” the writer said.
The technology is developing so quickly that the novelist didn’t want to wait for the book’s release, a year from now, to launch the trailer. Instead, he is already looking for other opportunities to experiment with AI. While the novelist was finishing this animation, for example, companies like Meta and Google released research papers with promising results from their tools, suggesting that users will soon be able to generate realistic videos from prompts as well. “In the future, we might be able to send the text, and readers will choose between reading, listening, or watching,” predicts PJ Caldas.
Meanwhile, he and his digital characters will focus on creating the best books he can write and the most impactful trailers that AI can produce. PJ Caldas is an award-winning, international best-selling author with four fiction books published in Brazil. The Girl from Wudang is his first novel in English.