Phenaki

Phenaki is a remarkable model for generating videos from text. It offers the ability to create videos based on a sequence of textual prompts that can change over time, allowing for the generation of videos of varying lengths, even up to multiple minutes. The model addresses the challenges of video generation from text by introducing a new causal model for learning video representation. This model compresses the video to a small representation of discrete tokens, using causal attention in time to work with variable-length videos. To generate video tokens from text, a bidirectional masked transformer conditioned on pre-computed text tokens is employed. The generated video tokens are then de-tokenized to create the actual video. Phenaki demonstrates how joint training on a large corpus of image-text pairs and a smaller number of video-text examples can lead to generalization beyond what is available in the video datasets. Compared to previous video generation methods, Phenaki can generate arbitrary long videos based on a sequence of prompts in the open domain. The proposed video encoder-decoder outperforms per-frame baselines in terms of spatio-temporal quality and the number of tokens per video. Overall, Phenaki represents a significant advancement in the field of video generation from text.