Meta has introduced Movie Gen, an advanced AI that can produce and edit videos while incorporating music and sound effects based on text prompts. This AI stands out due to its exceptional video and audio generation abilities, offering features and realism that surpass those of any other AI available.
AI Specifications
Movie Gen is built on a 30-billion parameter AI model that can create 16-second HD clips from text prompts. It has been pre-trained with one billion images and 100 million videos, selected from a much larger dataset to enhance quality for training purposes. On the audio side, Movie Gen Audio utilizes a 13-billion parameter model designed to generate 48 kHz sound effects and music from text prompts, having been pre-trained on one million hours of audio. The AI has been improved through human feedback along with high-quality audio and video samples.
Realistic Video Generation
When provided with a photo of a person and a description of that person in a specific scene, the AI can produce a realistic video featuring an animated actor in that environment. It has been programmed with knowledge of 22 different camera motions and positions, such as wide angle, tilt up, and truck left, allowing filmmakers to determine virtual camera placement and movements similar to actual filming. For filmmakers who prefer traditional methods, high-end DSLRs like the Nikon Z6III, available on Amazon, can still be utilized. Interestingly, Movie Gen is capable of editing videos in a way that is both precise and realistic, which is something other AIs currently struggle to achieve.
Audio Integration and Limitations
Additionally, text prompts can be used to incorporate professional-quality audio into the video clips, featuring sound effects and music scores. While the AI can generate music lasting several minutes, it is restricted to 16-second video clips due to the significant computing power required. The audio is synchronized with the scene's beats and can produce off-screen sounds, like birds chirping in a forest, based on the scene's context.
Meta is actively working on implementing safeguards for Movie Gen and plans to launch the AI once it is assured of its safety.
