Writing · Case study · 2026 · AI Video Pipeline
CUTLINE:a director, not a template engine
One sentence in, one finished MP4 out - directed by a pipeline of up to 12 stages per slideshow, not a template engine. The pipeline commits the editorial decisions before it touches a frame.
fig. 00 - results
- 11-12
- fixed-order pipeline stages
- 0
- signups, all inbound
- Up to 3
- image providers per shot, then a placeholder
- 0
- creative knobs
Section 01
The problem
Short-form video has eaten attention, but the production loop has not compressed.
Script, storyboard, b-roll, cut, caption, render - every step has a tool, the assembly is still manual, and by the time the idea is on screen it has aged. The "AI video" category mostly automates the cut, not the editorial work that decides what to cut to.
The naive AI version - "type what you want, we'll generate a video" - collapses into template fill-ins. Same Ken Burns pan over the same Unsplash photo, same kinetic-type intro, same captions. The output is generic because the system is generic: it picked a layout, not a narrative. CUTLINE takes the opposite bet.
The product is "describe, receive," not "configure, render."
Section 02
Thesis
Director layer, not template engine. The pipeline commits to editorial decisions before it touches a frame.
- 01Director layer, not template engine
From one sentence it infers audience, goal, tone, complexity, and duration; plans a narrative arc whose beat count scales with the duration (3 to 12 beats); breaks each beat into 1-3 shots with per-shot purpose, motion hint, and text density; writes the script aligned to shot boundaries; sources or generates the imagery; composes the MP4. The user does not pick a template, voice, or layout. The system makes those calls.
- 02One sentence in, no creative knobs
Optional brand kit and uploaded assets enrich the pipeline; they do not steer it. A coffee brand can upload a logo and product photos, drop in two hex colors, and set banned_phrases and required_phrases on a brand_kits row - the director still chooses the shot list and the cuts.
- 03Pipeline over agent
A fresh slideshow runs eleven stages, twelve when assets or brand colors are supplied. The order is fixed. The intent, narrative, shot, script, TTS and image stages call outside models or image providers, so the same prompt can give a different video. Fixed stage boundaries beat agent loops for debugging and retries: each stage's start, finish, duration and error code are written to render_events, so most failed renders point at the stage that failed. An agent loop hides that. Spend is capped per user per month when a job is admitted, and each job's cost is estimated per category (LLM, voice, video, images), not per stage.
- 04Worker separate from app
Rendering is CPU-heavy. Serverless functions time out, and even when they don't, billing-by-execution is the wrong shape for long jobs. The Next.js app handles UI, API, and job enqueue; a separate BullMQ worker pipelines and renders.
Section 03
Architecture
The app enqueues; the worker renders. The app runs on Vercel; the worker runs on a long-running host (Railway / Render / Fly), sharing the same Redis.
Browser
App - Vercel
Worker - long-running host
Section 04
The slideshow stages
A fresh slideshow render runs eleven stages; asset analysis, the twelfth, runs only when assets or brand colors are supplied. The order is fixed, and each stage's start, finish, duration and error code are written to render_events.
Direct
Voice & motion
Visuals & render
Section 05
The talking-character branch
When mode === talking_object the renderer detours through one of three modes on two providers (Google Veo and HeyGen), depending on talkingObjectStyle and talkingRealMode.
Cartoon
Studio framing
Cinematic
Section 06
Why this is hard
- 01Three talking-character modes, three failure semantics
VEO's RAI filter can reject a chunk. The VEO client throws a distinct VeoContentFilteredError, which the retry classifier marks non-retryable. The chunk loop retries the identical prompt once, then runs up to two LLM reword passes on the narration while keeping the visual prompt intact. A chunk still blocked after that is dropped, and the loop moves on to the next one. If every chunk is blocked, the job fails with an actionable message.
- 02HeyGen Photo Avatar quota under at-least-once submissions
Lower-tier HeyGen accounts cap stored Photo Avatars at 3. The upload path keys a SHA-256 cache on image bytes so identical inputs short-circuit. On code:401028 (quota full), the orchestrator lists the account, partitions avatars into orphans vs cached, and bulk-deletes orphans oldest-first in parallel batches (concurrency 10) with LRU eviction over cached as backup.
- 03Idempotency without a job-state table
X-Idempotency-Key paired with an in-process Map and a per-key Promise lock serializes concurrent submissions with the same key and returns the original { jobId }; 24h retention. In-memory by design - BullMQ already owns job lifecycle, and duplicating that into Postgres creates two sources of truth.
- 04Cancellation across a long-running async pipeline
Cancel writes a Redis SET; every stage reads it before starting work. On hit the worker throws and the same cleanup path runs as on success or failure. Cancellation is eventual, not preemptive - latency-to-cancel is bounded by the current stage's duration.
- 05Plan limits enforced at two layers
Free gets 3 videos a month and Beginner 10, counted as videos finished. Professional and Enterprise have no video count, but every plan has a monthly dollar spend cap, and talking-character video is capped at 90 seconds a month on Professional and 400 on Enterprise, plus purchased top-ups. The UI shows Pro badges and lock states. Plan limits are checked server-side before work is admitted: the generate handler reads the plan and usage before anything reaches the BullMQ queue. A tampered request body can't get past that check.
- 06Image sourcing falls back, then refuses to ship blank plates
One missing image shouldn't throw away a job that has already run most of its stages. Each shot without an uploaded photo walks a fallback chain: Unsplash, then Pexels, then DALL-E 3 (skipped on Free and preview renders), then the same three again with the first three words of the query, then a placeholder. The query is derived per shot from intent + script via OpenRouter, falling back to words from the topic. A failed link moves on to the next; a missing OpenRouter key stops the job. But a video of mostly blank plates isn't worth shipping: on a paid plan, if more than half the shots end on the placeholder, the job fails. Free renders are stock-only and have no floor.
Section 07
Image sourcing has a floor
Every shot ends up with an image, even if it's the placeholder. A paid render does not: if more than half the shots land on the placeholder, the job fails instead of shipping blank plates. Free, stock-only renders have no floor.
Per-shot fallback chain
Section 08
Design decisions & tradeoffs
Pipeline, not LLM agent
- Why: fixed stage boundaries beat agent loops for debugging and retries.
- Tradeoff: less emergent behaviour, more handcrafted prompts per stage. Visibility over magic.
Worker on a long-running host
- Why: CPU-heavy renders belong off the request path, and per-execution billing is the wrong shape for long jobs.
- Tradeoff: deploy is two services (Vercel app + worker) sharing one Redis.
In-memory idempotency + Redis cancellation
- Why: BullMQ already owns job lifecycle; a Postgres job-state table creates two sources of truth.
- Tradeoff: the idempotency cache resets on app restart - a duplicate POST with the same key within 24h after a restart creates a second job. The request ID is carried through logs for support correlation.
Placeholder as last resort, with a floor
- Why: one missing image shouldn't throw away a job that has already run most of its stages, so a shot that finds nothing gets a placeholder.
- Tradeoff: a render that is mostly placeholders isn't worth shipping. On a paid plan, over half the shots on the placeholder and the job fails. Free, stock-only renders have no floor.
user_plan_overrides separate from the auth table
- Why: entitlement is a thin map keyed by user_id; identity is Better Auth's job. Coupling them locks billing to one auth provider.
- Tradeoff: each plan lookup is its own SELECT on user_plan_overrides, up to three per generate request - a small cost for an auth-provider swap that doesn't touch billing.
Section 09
Failure modes
VEO content-safety block on chunk N
- Detected as VeoContentFilteredError. The identical prompt is retried once, then an LLM is asked up to two times to reword that chunk's narration with the same meaning.
- A chunk still blocked is dropped and the rest keep generating, and the user is told how many segments were skipped. If every chunk is blocked, the job fails with an actionable message.
HeyGen 401028 Photo Avatar quota full
- Bulk auto-cleanup partitions orphans vs cached, deletes oldest orphans in parallel batches, then retries the upload once.
- If retry still hits 401028, the user-facing error is generic; the operator log carries the technical detail.
Image provider 5xx / 429
- Provider calls that throw on a 429, a 5xx or a timeout are retried twice by default (1s, then 2s); then the shot falls through Unsplash, Pexels, DALL-E 3, a shorter-query pass and the placeholder.
- The render completes unless it's on a paid plan and more than half the shots end on the placeholder; then the job fails.
Worker process killed mid-render
- Per-job temp dir is deleted on success, failure, and cancel via one shared cleanup path.
- An hourly cleanup job is the backstop for orphaned temp files. The same job enforces MP4 retention: published videos are kept 24 hours on Free and 90 days on paid plans.
Dodo webhook replay
- Events are keyed by type plus an ID (payment, subscription, refund or dispute), with the product, the billing cycle or both added for some subscription events. processed_webhook_events.event_key is the primary key, and plan grants and revokes claim it with INSERT ... ON CONFLICT DO NOTHING before the plan changes.
- A replayed event finds the key taken and returns. Top-up credits run first, guarded by a Redis key that expires after 90 days.
Section 10
Security model
Browser hardening
- CSP with frame-ancestors none, object-src none, base-uri self, form-action self, upgrade-insecure-requests.
- HSTS (2y, includeSubDomains, preload), X-Frame-Options DENY, nosniff, Referrer-Policy strict-origin-when-cross-origin - on every response.
API keys
- Format clk_<48 hex> from crypto.randomBytes(24).
- Stored as SHA-256 of the full key, with a 16-char prefix kept as the indexed lookup column. Plaintext is returned only at creation.
Dodo webhook signature
- The @dodopayments/nextjs handler verifies the Standard Webhooks signature with DODO_PAYMENTS_WEBHOOK_KEY before any handler runs.
- Missing or invalid signature -> 401.
SSRF guard on callbackUrl
- Validates scheme (http/https only) and rejects private and internal hostnames in every environment: localhost, .internal and .local names, cloud metadata hosts, RFC1918, CGNAT and link-local IPv4, and IPv6 loopback, ULA and fe80 link-local. ALLOW_LOCALHOST_WEBHOOK=true turns the check off.
- It matches the hostname as written: it does not resolve DNS or re-check redirects.
- Fire-and-forget, 5-second timeout, no retries.
Prompt-injection rejection
- validateGenerateInput rejects topics matching the injection-pattern set.
- Returns a field-level VALIDATION_FAILED.
Rate limiting
- Redis-backed fixed window, keyed per client IP or per API key.
- generate 5/h (120/h for API-key callers) and 50/day, upload 20/h, status 60/min.
- If Redis is down, the limiter fails open and lets the request through.
Section 11
Testing
The pipeline's correctness lives in pure functions, so that's where the tests concentrate - with one path that calls the real generate endpoint and polls the job it creates.
- 01Unit coverage on the load-bearing logic
304 Vitest cases across 33 files, including the two integration tests below, cover input validation, the retry classifier, the pipeline orchestrator, and the generate route. The same stage boundaries that make the pipeline debuggable are what make it testable.
- 02Opt-in integration against a live stack
Two integration tests drive the real POST /api/generate and poll job status against a running worker and Redis; they self-skip (describe.skipIf) when Redis isn't configured, so they run only when the dependency is actually present.
- 03The deliberate tradeoff
No browser test suite and no CI gate yet - the integration tests are run locally when the worker stack is up, and the unit layer is where regressions get caught first.