How AI Video Generation Works

The short answer

AI video generators like Gemini Omni are latent diffusion models. They are trained to add noise to videos, then learn to reverse that process generating video by progressively removing noise, guided by your text prompt. The result is new video that never existed, generated frame-by-frame with physics-plausible motion.

Step 1: Latent space compression

Video is expensive to process directly at the pixel level a single 4K frame contains ~8.3 million pixels, and a 10-second video at 24fps is 240 frames. That's 2 billion pixel values to process per generation.

To make this tractable, video diffusion models use a Variational Autoencoder (VAE) to compress frames into a lower-dimensional "latent space." A 4K frame that takes megabytes of raw pixel data is compressed to a latent representation hundreds of times smaller, while preserving the essential structure (shapes, colors, textures).

The model does all its generation work in this compressed latent space, then uses the VAE decoder to reconstruct high-resolution frames at the end.

Step 2: Controlled noise removal

During training, the model sees millions of videos with noise added at different levels. It learns to predict and remove that noise. At generation time, you start with pure random noise in latent space and the model runs through 20–50 "denoising steps," removing noise iteratively until a coherent video emerges.

Your text prompt guides each denoising step via a technique called Classifier-Free Guidance (CFG). The model simultaneously predicts a noise-free version conditioned on your prompt and an unconditioned version, then moves in the direction that increases agreement with your prompt. Higher CFG scale = stricter adherence to the prompt.

Step 3: Temporal consistency

The hardest problem in AI video is making objects stay stable across frames not flickering, morphing, or disappearing. This is called temporal consistency.

Modern video diffusion models address this with 3D attention: instead of only attending to spatial relationships within a single frame, the model's attention mechanism spans across the time dimension too. Every frame can "see" every other frame when computing what it should look like.

This is why Gemini Omni clips look smooth and physically plausible the temporal attention layers are what prevent the frame-to-frame incoherence of early AI video systems.

Image and video conditioning

Text is not the only way to steer a generation. With image-to-video, your uploaded still is encoded and used as the first frame the model denoises forward from that anchor, so the output keeps your subject, composition, and lighting while adding motion.

With video-to-video, an existing clip conditions the generation: the model preserves the source motion and structure while the text prompt drives a new style or look. In Gemini Omni, both run through the same model as text-to-video the only difference is whether you attach a reference image, a reference video, or neither.