A Production Pipeline for Generative Video: From Audio Track to Release-Grade Master

Context

Generative video models have reached the point where a single frame looks like a staged shoot. But a frame is not a film. Between a model that produces striking five-second fragments and a released video for a three-minute track lies a production distance: storyboarding, matching motion to the music, editing, color grading, controlling generation artifacts, and mastering for the distribution platform. The market mostly demonstrates the former and rarely shows the latter.

The studio had a steady stream of source material — music tracks and series of still artworks for them — and a need to release full-length videos from that material regularly, not as one-off experiments. A single success with generative video does not scale: without a formalized process, each new video starts from zero, repeats mistakes already made, and pays for them again — in generation credits and in days of work.

There is also a purely engineering layer of difficulty. Generative engines are non-deterministic: the same frame-and-prompt pair can yield cinematic motion or a character's melting face. The API infrastructure is unstable — the provider's own file upload turned out to be closed during the project. The cost of each generation is fixed in credits, so every rejected take is a direct expense. The pipeline had to minimize waste before material was sent to generation, not after.

Objective

Build a repeatable production process — track plus artworks in, master video out — that transfers to any new material without redesign. Not a set of ad-hoc scripts for a specific release, but a pipeline with fixed stages, verifiable intermediate artifacts, and a known cost for each step.

The key quality requirements: the edit must "hear" the track — cuts land on the beat grid, the vocalist appears on screen only in the intervals where the voice actually sounds; motion within the frame must stay cinematic with no morphing of faces or figures; and the output master is assembled to the platform's format, including the non-trivial case of vertical sources in a horizontal release.

The constraints we worked within: one image-to-video generation yields a hard limit of roughly five seconds, so any video's runtime is assembled from such blocks; the credit economics make reshoots expensive; the editing machine had no librosa, so beat analysis required a custom numpy implementation; and the ffmpeg build lacked drawtext, ruling out the standard path for text overlays.

Approach

The pipeline is split into isolated stages, each closed out by a dedicated tool and leaving a verifiable artifact: frame preparation, cleanup and upscaling, motion generation, the track's audio map, editing, color grading, mastering. An error is localized to its own stage and does not require restarting the whole process — essential when generation carries a fixed cost.

Frame preparation is formalized into rules that cut waste before generation even starts. Collage artworks are sliced into cells programmatically with PIL, with a mandatory inner inset of about 8 pixels — without it, the frames and skew of the original storyboard bleed into the animated video. A separate rule covers characters: a face survives animation only if the source shows it sharp, frontal, and with minimal head movement; frames failing that filter are either cleaned up or never sent to generation at all.

The generative layer is built on the kie.ai API with a deliberate choice of engines by quality-to-cost ratio. Kling 2.6 image-to-video is the default engine: the best source fidelity and detail at 55 credits per five seconds, with output around 1652×1252 preserving the source's proportions. Veo is the alternative for special cases: veo3_fast at about 45 credits, full veo3 at about 290. Small and distorted storyboard cells are run through GPT-4o Image in image-to-image mode before animation — about 14 credits per clean frame. A separate branch — face-swap on insightface — was tried and rejected: the swap produces a blurry face at 128-pixel resolution, insufficient for release quality.

So that the edit "hears" the track, an audio map is built for each composition. Demucs splits the track into stems — vocals, drums, bass, other; the RMS of the vocal stem yields precise intervals of voice presence. Whisper, run over the vocal stem, gives timecodes for every sung line — shots are synchronized to them. The beat grid is computed by a custom numpy implementation: spectral flux plus tempo autocorrelation. On real tracks the method produced 70 BPM with a 0.86-second beat and 112 BPM with a 0.534-second beat — accurate enough to land every cut on the beat.

Architecture

The pipeline's tooling layer is a set of specialized Python scripts with clear roles. Storyboarding and per-scene motion prompts are prepared by a dedicated tool; submission runs through a submit script that uploads the frame and creates a Kling task via the jobs API with the prompt, source image, and duration; task statuses are polled by a poll script via recordInfo. The audio map and the edit each have their own tools. Every script can be run and verified independently.

An infrastructure quirk: the provider's own upload was closed and returned 403, so source frames are delivered to generation through an external file host with direct URLs — the submit script handles this transparently for the operator. The credit balance is checked via the API before launching a generation batch: each video's economics are computed in advance, not from the charges after the fact.

The editing core is built on ffmpeg with strict parameter discipline. All clips are normalized to a single crop-fill 1920×1080 format, 24 frames per second, aligned SAR. The Ken Burns effect is implemented via zoompan with the input pre-scaled to roughly 2150×1210 — animated crop was rejected because its frame size is constant and the graphs fail with errors, while zoom without pre-scaling produces jitter. Shot durations are multiples of the beat, and cuts land on vocal entrances. A hard engine constraint is also handled: Kling outputs exactly about 5.04 seconds, so the editing script trims a longer shot to the source's actual length instead of trying to stretch it.

A separate architectural line is format adaptation. For a release where 18 vertical artworks had to ship in a 1920×1080 horizontal master, an ambilight scheme was built: the video sits centered at its native 608×1080 resolution, and the side panels are filled with a blurred, darkened continuation of the video itself — downscale, gblur, stretch to full frame, reduce brightness. On top: a muted showfreqs audio spectrum, mirrored and tinted dark red, and a showwaves waveform along the bottom. Text infographics are overlaid as a transparent PNG layer prepared with PIL — the workaround for the missing drawtext. Final assembly is split into two stages: editing the center track with hard cuts and grade, then compositing the panels, visualizations, and sound.

Quality control and the economics of waste

The main source of waste in generative video is morphing: under an aggressive push-in or zoom, the engine "melts" characters' faces and figures. The problem was closed with prompt discipline: the generation brief includes an explicit lock — static camera on a tripod, no zoom, face stable and identical to the first frame, no deformation or melting. Six problem fragments of one release were regenerated under this protocol, and the rule entered the pipeline standard.

The second class of problems is transitions. Layering zoompan over an already-moving generated frame produces strobing; even a soft xfade dissolve over moving i2v fragments reads as image ghosting. The pipeline's final rule: clean hard cuts on the beat only, assembly via concat with no overlays — motion within the frame is carried by the generation itself, the edit is responsible for rhythm. The decision simplified the filter graphs and raised perceived quality at the same time.

Color grading is verified on a single frame before a full render — a rule paid for in practice: the first grade pass on one master drifted into magenta because of an admixed blue cast and a bloom that lifted the blacks. The working recipe for a red-and-black palette: colorbalance without blue, eq with contrast 1.14 and gamma 0.95, bloom on the red channel only via lowered green and blue curves, a vignette, and moderate grain. The noise filter is known to inflate the file substantially, so a compact crf 21 copy is built for distribution. Render stability was closed separately: a single heavy ffmpeg graph with full-size glow and triple use of the audio track without asplit died with an out-of-memory SIGKILL — assembly was split into two passes with explicit audio branching.

Outcome

The pipeline has shipped six full-length videos assembled through a single process. The benchmark master in complexity is a video for a roughly 195-second track: 18 vertical artworks animated in Kling 2.6, a final 1920×1080 master of about 192 seconds containing 64 cuts landed on a 112 BPM beat grid, with a two-act dramatic structure — a linear story, then a reprise with an accelerated cutting rhythm and a climax at the track's peak.

The process is codified into eight reusable production skills covering the full cycle: direction and storyboarding, asset preparation, beat-synced editing, cinematic motion, transitions, color grading, sound design, and platform mastering. A new video starts not from a blank page but from a standard toolkit and ruleset that already embed every mistake paid for earlier — from the inset when slicing collages to the ban on overlays atop a moving frame.

For the studio, this is a proven production pattern for generative video with known per-shot economics: the cost of every stage is expressed in credits and computed before launch — 55 credits per five-second Kling shot, about 14 per frame cleanup — so a full master's generation budget is known before the first take is submitted. The same pipeline — storyboarding, generation, audio map, deterministic editing — transfers from music releases to commercial formats: promos, advertising videos, digital-campaign content.

What we built

• Frame preparation

  Programmatic slicing of collage artworks with PIL using an inner inset of about 8 pixels, eliminating frames and skew in the video. A suitability filter for character frames: a sharp frontal face, minimal head movement.

• Cleanup and upscaling

  Small and distorted storyboard cells are run through GPT-4o Image in image-to-image mode (about 14 credits per frame) and enter generation already clean.

• Generative layer

  Kling 2.6 image-to-video as the default engine (55 credits per 5 seconds, around 1652×1252), Veo as the alternative. Submission via the jobs API, status polling via recordInfo, frame delivery through an external file host to bypass the closed upload.

• Track audio map

  Demucs splits the track into stems, the vocal stem's RMS yields voice-presence intervals, Whisper yields timecodes for sung lines. The beat grid is a custom numpy implementation: spectral flux plus tempo autocorrelation.

• Beat-synced editing

  ffmpeg: crop-fill 1920×1080, 24 fps, shot durations in multiples of the beat, hard cuts on vocal entrances. Ken Burns via zoompan with pre-scaled input; shots trimmed to the actual ~5.04-second generation length.

• Grade and mastering

  Color grading with colorbalance plus eq, single-channel bloom, vignette, grain; a mandatory single-frame grade check before the full render. A compact crf 21 distribution copy.

• Format adaptation

  An ambilight scheme for vertical sources in a horizontal master: a blurred continuation of the video on the side panels, a muted audio spectrum and waveform, text as a transparent PNG layer. Two-pass assembly with asplit against OOM.

• Process codification

  Eight reusable production skills: direction, asset preparation, beat-synced editing, cinematic motion, transitions, grading, sound design, mastering.

Engineering challenges