Adobe has officially entered the world of generative AI video with its Firefly Video Model, which is now integrated into Premiere Pro. This development brings exciting new tools that enable creators to enhance footage and generate videos from still images and text prompts.
Here’s a closer look at what the Firefly video model is offering.
One standout feature is the Generative Extend tool, which is currently in beta for Premiere Pro. This tool allows users to lengthen the start or end of video footage that may be too short or make slight adjustments mid-shot, like fixing, shifting eyelids or unexpected movements.
Read also: Zoom unveils ‘digital twin’ AI avatars to attend meetings on behalf of users
While clips can only be extended by two seconds, this feature could save time by eliminating the need for retakes when only minor adjustments are required. The extended clips can be generated in 720p or 1080p at 24 FPS.
The tool also offers audio capabilities, allowing users to extend sound effects and ambient “room tone” by up to 10 seconds. However, this function does not support spoken dialogue or music.
Text-to-Video and Image-to-Video tools: From words and images to video
In addition to the Generative Extend tool, Adobe has launched Text-to-Video and Image-to-Video tools, which are now available in a limited public beta on the Firefly web app.
Text-to-video enables people to generate videos from descriptions in text form, thus generating simple videos. The tool works great with all styles, such as traditional movie-making, 3D animated movies, and even stop-motion. It also has “camera controls,” whereby users can modify the camera angle, movement, or shooting range for a better video.
Image-to-Video extends this notion one step ahead by allowing users to upload a reference image along with a text description of a video. Users will be able to control the final output in a manner that might make sense for a b-roll from photos or visualising reshoots.
However, several malfunctions, such shifting backgrounds or shaky objects in the video, show that the technology is still not flawless.
Limitations of Adobe’s AI video tools
Although Adobe’s new tools offer powerful new possibilities, they have limitations.
Adobe’s new tools have restrictions even as they present exciting new possibilities. Text-to-video and image-to-video clip lengths are currently capped at five seconds, with a maximum resolution of 720p at 24 frames per second (FPS).
This is inferior to OpenAI’s Sora, albeit not publicly accessible, can produce video up to one minute in length.
Although Adobe is working on a faster “turbo mode” to speed things up, the current generation time for content is about 90 seconds.
Notwithstanding these limitations, the tools mark a substantial advancement in generative AI for video editing.
Read also: Teens World Foundation launches SkillXplore cohort to empower young innovators
Ethical considerations and commercial safety
One significant advantage of Adobe’s Firefly Video Model is that it is trained on legally acquired content, which the company calls “commercially safe.” This could be a key selling point for users concerned about AI tools’ ethical and legal implications. Other models, such as Runway, have faced scrutiny for allegedly being trained on scraped online videos, raising copyright concerns.
Adobe also offers Content Credentials for videos created using its AI tools. This feature helps disclose AI involvement and ownership rights, adding transparency when publishing content online.
What’s next for Adobe’s AI tools?
These AI video features were unveiled at Adobe’s MAX conference, where the company introduced several other AI-powered updates across its suite of creative apps. Though it’s unclear when these tools will be officially out of beta, they are already accessible to the public. This puts Adobe ahead of competitors like OpenAI’s Sora, Meta’s Movie Gen, and Google’s Veo, none of which are publicly available yet.
With Adobe’s Firefly Video Model and its suite of new tools, creatives have more ways to explore AI-powered video editing. Technology is still evolving but represents a promising future for video production and content creation.