Level 6Lesson 51⏱️ 45 min

AI Video Generation

Text-to-video and image-to-video with Veo, Kling and Runway - plus the hard limits you must design around

Video Is the Fastest-Moving Frontier

In early 2025, AI video was a novelty - a few seconds of warped, silent footage. By 2026, multiple models produce native 4K with synchronized audio, multi-shot storyboards, and cinematic camera moves. It is the single most rapidly improving area of generative media, and the most expensive.

Currency warning: Video model rankings change monthly. The principles in this lesson are stable; the specific model names are a mid-2026 snapshot. Always re-check before a paid project.

Two Modes: Text-to-Video vs Image-to-Video

Text-to-video

Describe a scene, get a clip. Fast and creative, but you have less control over exactly how the first frame looks.

Image-to-video (usually better)

Generate a perfect still first (in your favorite image tool), then animate it. You control the look precisely, then add motion. This is the pro default.

The winning workflow: Nail the frame as an image (cheap, controllable), then feed it to a video model as the starting frame. You get the best of both: image-level control plus motion.

The Leading Video Models (2026)

Google Veo 3.1   Best all-rounder. Strong prompt adherence,
                 native audio, 4K landscape + portrait.
                 → narrative scenes, establishing shots, ads.

Kling 3.0        Cinematic motion (hair, liquids, fabric),
                 multi-shot storyboard mode with audio across cuts.
                 → story sequences, dynamic motion.

Runway Gen-4.5   Granular CONTROL: camera moves, motion brush,
                 reference-driven character consistency.
                 → when you need to direct, not just describe.

Note: OpenAI announced Sora's app/API are being retired in 2026.
Don't build a workflow that depends on it.
2026 leap: Most top models now generate audio with the video- dialogue, ambient sound, effects - in a single pass. A year ago that was zero.

Prompting for Motion

Video prompts add a dimension images don't have: movement and camera. Describe the subject, then the action, then the camera behavior.

[SCENE] + [SUBJECT ACTION] + [CAMERA MOVE] + [STYLE/MOOD]

"A lighthouse on a cliff at dusk. Waves crash below in slow motion.
 The camera slowly pushes in from a wide aerial.
 Cinematic, moody, volumetric light."

Camera vocabulary that works:
  push in / pull out / pan left / tilt up / orbit / tracking shot
  static locked-off / handheld / dolly / crane / aerial drone
Keep actions simple. One clear motion per clip beats three competing ones. Complex choreography is where video models still break down.

The Hard Limits (Design Around These)

CLIP LENGTH    Most generate ~5-10 seconds at a time.
               Long video = many clips stitched in an editor.

CONSISTENCY    Characters/locations drift between clips.
               Use reference images + the same seed to anchor.

PHYSICS        Hands, fast motion, object permanence, and
               "things passing behind things" still glitch.

TEXT           On-screen words are unreliable - add them in
               your editor afterward, not in the prompt.

COST           Per-second pricing. A 30s piece is 3-6 clips,
               each regenerated several times = real money.
Plan for stitching. Real AI videos are assembled from many short generations in a normal editor (CapCut, Premiere, DaVinci). Generation is one step in a pipeline, not the whole thing.

A Realistic Short-Video Pipeline

1 - Script & shot list

Break the idea into 4-6 distinct shots, each ≤ 8 seconds.

2 - Generate a still for each shot

Lock the look in an image tool. Keep a character/style reference for consistency.

3 - Animate each still (image-to-video)

One clear motion + camera move per clip. Regenerate until acceptable.

4 - Assemble in an editor

Cut clips together, add titles/text, music and voiceover (next lesson), color grade.

Hands-On: Animate a Still

Hands-on (20 min): Take one image you generated in Lesson 49-50 and feed it to an image-to-video tool (Runway, Kling, or a Veo-powered app - free tiers exist). Prompt a single simple motion: "slow camera push-in, gentle wind in the grass." Generate, then try a second motion on the same still. Notice how much control the starting image gave you versus pure text-to-video - and where the model still glitches. That gap is exactly what your editor and shot-planning exist to cover.
Lesson 51 Quick Reference
Image-to-video

Generate a perfect still, then animate it - the controllable pro default

Veo 3.1

Top all-rounder in 2026: native audio, 4K, strong prompt adherence

Kling 3.0

Cinematic motion and multi-shot storyboards with synced audio

Runway Gen-4.5

Best granular control - camera moves, motion brush, character consistency

Camera vocabulary

push in, pan, tilt, orbit, tracking, dolly, aerial - describe the move explicitly

Clip length limit

~5-10 sec per generation; long video = stitched clips in an editor

Per-second cost

Video is priced per second of output - storyboard cheaply before generating

Sora retirement

OpenAI's Sora app/API are being phased out in 2026 - don't depend on it

← Advanced Image Workflows
Unlocks in ~12 min of reading