We Doubled the Quality of Our AI Editing Pipeline by Adding One Thing

We doubled the quality of our AI editing pipeline by adding one thing.

A critique loop.

At Threadline Studio we build AI tools for documentary filmmakers. Our pipeline reads transcripts and assembles a rough cut. For months it was single-pass. One prompt, one output, done.

Results were always fine. Never great.

So we added a second step: the model reviews its own edit like a producer would, then revises based on its own notes. Two to three rounds, then output.

But the real insight wasn't adding the loop. It was how we structured the feedback.

Our first version used numerical scores. "Pacing: 7/10. Narrative: 6/10." The model would look at these and make vague improvements. A 6 in pacing doesn't tell you what to fix.

Then we thought about how actual producers give notes.

No producer says "pacing is a 6." They say "the opening drags and we lose Sarah's thread after the second act." Specific. Actionable. Pointed.

We switched the critique to generate qualitative notes. Which clips run too long. Where the narrative loses focus. Whether the ending resolves or just stops.

The quality jump was immediate. The revision pass finally had real instructions to work with.

Three things we learned:

Self-critique is surprisingly cheap. One extra LLM call adds maybe 30 seconds and a few cents. The ROI is enormous.

Qualitative feedback dramatically outperforms numerical scoring. "The opening drags" beats "opening: 6/10" every time.

Three iterations is the ceiling. After that, changes go lateral, not upward.

If you're shipping first-pass LLM output in production, try adding a critique loop. It's the highest-leverage improvement we've made to our pipeline.

First drafts find the story. Second drafts tell it.

What's the single highest-leverage change you've made to an LLM pipeline in production?