It is an AI model that creates videos at the state-of-the-art level called Lumiere.
Originally, Google had a video maker called VideoPoet already
VideoPoet – Google Research
A Large Language Model for Zero-Shot Video Generation. VideoPoet demonstrates simple modeling method that can convert any autoregressive language model into a high quality video generator.
with the name shape generator. ImageGen
Image Video
High-definition video Generation with Diffusion Models
But Lumiere came out. This is much cooler than the original, and it can create high-quality short-form videos.
Basics
- Lumiere is trained with a dataset of over 30 million videos. 80 frames long with text captions inserted to practice it.
- It can produce 16 frames of video per second.
- It is not said that the 30 million videos are taken from the same source.
This work is a joint research between the Google Research team and Israel's leading technology universities, namely:
Lumiere has 4 cool features:
- Video Editing - You can solve it by prompting the command (easy but very divine).
- Stylized Generation - You can create a video from the reference we provide (you can give the reference as an image or video). The model will learn the style and then create a video to use that style.
- Image & Text to Video - Don't just put a prompt as a message. For example, throw a picture of us with a still face and then tell it to create a picture of us laughing. It can be done, it's too godly.
- Cinemagraphs - Select only a part of the image and animate it into a video.
source
All of this information is Google research published on GitHub.
Lumiere - Google ResearchSpace-Time Text-to-Video diffusion model by Google Research.lumiere-video.github.io
Paper งานวิจัย
One of the researchers, named Hila Chefs From Tel Aviv University, it was posted on X.