MagicAnimate: Temporally Consistent Human Image Animation

MagicAnimate is an exciting open source project that allows you to create animated videos from a single image and a motion video. It is a cutting-edge diffusion-based framework for human image animation, maintaining temporal consistency and enhancing animation fidelity.

Key Features:

• Preserves the reference image faithfully.
• Can animate reference images with motion sequences from various sources.
• Integrates seamlessly with T2I diffusion models like DALLE3.

Advantages:

• Offers the highest consistency among all dance video solutions.

Disadvantages:

• Some distortion in the face and hands.
• Default configuration may result in style shifts.

Getting Started:

• Download the pretrained base models for StableDiffusion V1.5 and MSE-finetuned VAE.
• Download MagicAnimate checkpoints.
• Installation prerequisites: python>=3.8, CUDA>=11.3, and ffmpeg. Install with conda: conda env create -f environment.yml and conda activate manimate.

Generating Motion Video:

• Use OpenPose, a real-time multi-person keypoint detection library, to convert a motion video to OpenPose.

API Usage:

• You can use the Replicate API to generate animated videos. Here's an example code:

import Replicate from "replicate";
const replicate = new Replicate({ auth: process.env.REPLICATE_API_TOKEN, });
const output = await replicate.run(
  "lucataco/magic-animate:e24ad72cc67dd2a365b5b909aca70371bba62b685019f4e96317e59d4ace6714",
  {
    input: {
      image: "https://example.com/image.png",
      video: "Input motion video",
      num_inference_steps: 25, // Number of denoising steps
      guidance_scale: 7.5, // Scale for classifier-free guidance
      seed: 349324 // Random seed. Leave blank to randomize the seed
    }
  }
);

More information about MagicAnimate can be found in the official introduction, paper, arXiv, GitHub code, and demos.