How AI Video Generation Works: Clip Length, Resolution and Audio by Model

1. A video is a block of numbers, squeezed into a latent

To a model, a clip is a stack of frames, each a grid of pixels with three color channels. The Wan technical report writes its input as a tensor of (1 + T) frames × height × width × 3.¹⁴ Generating that many pixels directly is expensive, so nearly every current system works in a compressed latent space instead, the approach the Diffusion Transformer (DiT) paper describes as latent diffusion: an autoencoder learns to compress images into smaller representations, the diffusion model learns to generate those, and a decoder turns them back into pixels.¹³

For video the autoencoder is a 3D variational autoencoder (VAE) that compresses time as well as space. Wan-VAE shrinks a clip to [1 + T/4, H/8, W/8] with 16 channels, a 4 × 8 × 8 compression; its first frame is compressed only spatially, which lets the same model handle still images.¹⁴ The later Wan2.2-VAE reaches 4 × 16 × 16.⁹

2. Generation starts from noise and removes it step by step

The model begins with a latent of pure random noise the shape of the target clip and refines it over a series of steps. Wan uses flow matching, a variant of diffusion: in training, each example is a straight-line blend between noise and a real video latent, x_t = t·x₁ + (1 − t)·x₀, and the network learns to predict the velocity x₁ − x₀ that points from noise toward the video.¹⁴ At generation time the model follows that predicted direction from noise to a clean latent over many small steps, then hands the result to the VAE decoder, which turns it back into frames.¹⁴

3. The denoiser is a transformer over spacetime patches

Early image diffusion models used a convolutional U-Net as the denoiser. The 2022 DiT paper replaced it with a transformer that operates on latent patches: the latent is cut into p × p patches and flattened into a sequence of tokens, and a smaller patch means a longer sequence and more compute.¹³ It found that DiTs with more compute per forward pass, through a deeper or wider transformer or more input tokens, consistently produced better images (lower FID).¹³

Video models extend the patches through time. Wan's patchifying module is a 3D convolution with a (1, 2, 2) kernel, giving a sequence of length L = (1 + T/4) × H/16 × W/16 that the transformer processes with full spatio-temporal attention, so patches attend to each other across frames as well as within them.¹⁴ Wan2.2 splits the denoiser into two experts (a mixture of experts): a high-noise expert for the early steps, focusing on overall layout, and a low-noise expert for the later steps, refining detail; each has about 14B parameters, with 14B of the 27B total active per step.⁹

4. The prompt steers every step through cross-attention

The text prompt never becomes pixels itself. A separate text encoder turns it into a sequence of embeddings, and the transformer reads them at every denoising step. Wan uses the multilingual umT5 encoder, with a text embedding sequence 512 tokens long, and injects it through cross-attention layers in each transformer block; other designs, the report notes, instead concatenate the text tokens with the video tokens (MM-DiT).¹⁴ APIs extend the same idea to images: MiniMax H3, for example, accepts a first frame, a last frame or reference images, videos and audio alongside the text prompt.⁶

5. Why clip length and resolution are capped

The sequence the transformer must process grows with both frames and pixels. Applying Wan's formula to the report's own example of 81 frames at 1280 × 720 gives (1 + 80/4) × (720/16) × (1280/16) = 21 × 45 × 80 = 75,600 tokens (our arithmetic from the paper's formula). Doubling the frame count roughly doubles the sequence; doubling both height and width quadruples it. Wan's authors say such extended sequence lengths "substantially reduce training throughput" and that memory use forces small batch sizes, which is why they trained first on low-resolution images, then on 5-second clips at 192 px and 16 fps, then 480 px and finally 720 px.¹⁴

The vendors' limits reflect the same trade-off. Google's Veo 3.1 on Vertex AI offers 4-, 6- or 8-second clips at 24 FPS, up to 4K,⁸ while Runway's Gen-4.5 API takes 2 to 10 seconds at 1280 × 720.⁴ At the long end, ByteDance lists 4 to 30 seconds for Seedance 2.5 at up to 1080p,¹ and Alibaba lists up to 30 seconds at up to 1080P for Wan 3.0.² Seedance 2.0 shows the trade within one family: up to 4K, but 15 seconds.¹

6. Native audio is a second stream generated in step

Many text-to-video diffusion models "remain silent," as the LTX-2 authors put it, so sound has to be added afterward. Models with native audio generate the soundtrack in the same pass. Lightricks' open LTX-2 paper describes one design in detail: an asymmetric dual-stream transformer with a 14B-parameter video stream and a 5B-parameter audio stream, coupled through bidirectional audio-video cross-attention layers with temporal positional embeddings, so each stream can follow the other in time.¹⁵

In the commercial APIs this shows up as a switch. BytePlus documents Seedance's generate_audio option as producing voice, sound effects and background music matched to the prompt and the picture, delivered as mono audio.¹ Veo 3.1 lists sound generation as supported,⁸ xAI describes lip-synced speech for Grok Imagine Video 1.5,⁷ and Kuaishou says Kling Video 3.0 generates speech in English, Chinese, Japanese, Korean and Spanish, including multi-character dialogue.⁵

7. Longer videos come from extension, not one giant generation

Because each generation is capped, longer sequences are built by continuing a finished clip: the model is conditioned on the end of the previous clip and generates the next stretch. Veo 3.1 on Vertex AI lists "extend videos" as a supported capability,⁸ and BytePlus documents video extension as one of Seedance 2.5's reference-to-video tasks.¹ Inside the model, Wan-VAE's causal design helps: it encodes and decodes video chunk by chunk, at most 4 frames per chunk, carrying cached features from earlier chunks so that it can handle arbitrarily long videos without running out of memory.¹⁴

8. Where Sora went

OpenAI's Sora 2 was one of the best-known examples of the approach; OpenAI's guide describes it as built on research into multimodal diffusion, with clips of up to 20 seconds and 1080p output on sora-2-pro, and with video extensions.¹¹ On March 24, 2026, OpenAI told developers it would remove the Sora 2 models and the Videos API, and both were shut down on Sept 24, 2026, with no recommended replacement.¹⁰ The guide is now kept for historical reference only.¹¹