Can Fine Cutting Be Handed to AI? The Boundary Isn't Model Precision — It's What Defines the Cut

The verdict up front: "can fine cutting be automated" is not one question but two. Once you separate cuts by what defines their position — signal or meaning — the first has been shipping at frame accuracy for years and the second has zero products, with the blocker sitting nowhere near the model. More importantly: there is still no public way to score whether a cut is good. So this isn't a battle being slowly won. It's a battle with no scoreboard.

1. First, pin down "fine cut" versus "rough cut"

Without pinning the definition, this topic immediately slides into "AI can auto-edit video," which carries no information. The industry sequence is: assembly → rough cut (structure and selection) → fine cut (frame level) → picture lock.

• Rough cut = pick the material, order it, build the structure. The unit of decision is the clip; tolerance is seconds.
• Fine cut = frame-level trimming of in/out points, pacing and breath, J/L cut audio-video offset, match cuts, shot-scale rhythm, the emotional curve, beat timing. The unit of decision is the frame (41.7ms at 24fps), and tolerance is often ±2–3 frames.

Almost all "AI editing" marketing conflates these two layers. And the exit condition of fine cutting has only been partially formalized [Narrowed]: picture lock itself is not a machine-verifiable fact — it's a declaration made by a director or producer under scheduling and contractual pressure. What has been formalized is its consequence: once locked, downstream work is scheduled and billed against frozen frame numbers (VFX pull lists with head/tail handles, composer spotting sessions writing cue sheets against timecode, audio turnover carrying 2–10 second handles). Any change after lock must produce a change list so downstream can auto-conform.

One correction I would have gotten wrong myself: this diff is not an O(n) per-clip comparison. Insert one segment and every downstream timecode shifts, so you must first do sequence alignment (LCS/Myers class, keyed on reel + source timecode). Worse, the alignment is semantically ambiguous — "trim 12 frames off the head and slip" versus "delete the old shot and insert a different take" can produce a frame-identical new timeline while implying completely different conform instructions. This is exactly why commercial conform tools ship with interactive adjudication UIs and why assistant editors still QC change lists by hand. And "empty diff" is neither sufficient for, nor equivalent to, "the picture didn't change": VFX version swaps, reframes, timewarps, an equal-length dissolve changing duration, titles, and color all move zero timecode.

2. The real dividing line: what defines this cut

This is the highest-information finding of the round, and the one all 14 dimensions' corrections independently converged on [Solid]:

Whether something can be automated depends not on how "fine" the operation is, but on whether the position of the cut is defined by a detectable physical event, or by meaning, emotion, and style.

Signal-defined boundaries — word onsets and offsets, silence, shot changes, beats — already have frame-level or sub-frame annotations, metrics, off-the-shelf tools, and leaderboards. Forced alignment is evaluated with millisecond boundary error as its primary metric. Shot boundary detection is per-frame binary classification. Beat tracking's standard tolerance is ±70ms. All of these are at or below the size of one frame.

Meaning-defined boundaries — "the moment he starts to hesitate," "right as the emotion lands," "cut when the punchline finishes" — have no loss, no leaderboard, and more fundamentally no way to construct reliable frame-level ground truth: inter-annotator agreement on such boundaries sits at the ±1 second scale. This is not "nobody has trained it." Evaluation and supervision fail first, in principle.

This line cuts across the fine-cut checklist rather than running along an easy/hard axis:

3. The part that is automated gets its frame accuracy from somewhere other than AI

This is the most easily misread point [Solid]. For all four rows marked "Automated" above, none of the frame-level precision comes from a video LLM. It all comes from deterministic detectors:

• Word boundaries: CTC forced alignment (20ms frame stride for wav2vec2-class aligners, ~10ms for phoneme-level ones), finer than the 41.7ms frame at 24fps. This is the most widely deployed automatic fine-cut operation in the world, and it underpins text-based editing and filler-word removal in mainstream NLEs.
• Shot boundaries: per-frame transition probability, output as integer frame numbers.
• Beats: beat and downbeat tracking, standard tolerance ±70ms.
• Silence: loudness gating plus VAD, exported straight to NLE project files.

The LLM's place in this chain is selection and ranking at the symbolic layer — it looks at a handful to a few dozen already-textualized candidates per second, not at 24 arbitrary frames. An LLM never needs to "say frame 37," and shouldn't be the thing that does. Every shipping frame-accurate AI edit today has this two-layer shape: second-level semantic selection, frame-level deterministic snapping.

Two discounts on that "automated," though:

Discount one: what's called "pacing" is actually a constant. [Narrowed] The open-source silence remover's breath control is a single scalar defaulting to 0.2 seconds; the commercial "shorten word gaps" feature is one slider for the whole file. What they expose is g' = f(g) — a memoryless scalar map whose target depends only on the gap's own length, with exactly zero conditional dependence on semantic position, syntactic boundary, or emotion. And the result isn't a "flattened" gap distribution — it's one quantized to a single spike: anything under threshold survives untouched, everything else collapses to the same value. That is, by definition, not pacing.

Discount two: the domain gap is bigger than you'd think. Shot boundary detection scores about 0.96 F1 on professional footage but only 77.9 on a UGC short-video dataset — precisely the domain where AI editing actually gets deployed. The measurement primitive itself loses roughly 18 points before you start.

4. Three layers of blockage, none of which is "the model isn't strong enough"

4.1 Missing counterfactual data [Solid — the hardest finding of the round]

Every public corpus records only the version that shipped. Not one dataset records the in/out points that were rejected for the same footage, which means no model has ever seen a paired preference label for "this cut 6 frames earlier versus 6 frames later — which is better." Public work can only do contrastive learning between real cuts and randomly displaced fake ones — and "real versus random" is vastly easier than "good versus plausible-but-worse."

There's a directly citable number here: the representative work on learning cut placement from finished films achieves top-1 accuracy of 8.18% at a ±1 snippet (roughly 8 frames) tolerance, against a 0.60% random baseline. At a tolerance far looser than fine cutting requires, top-1 is 8%.

Notably this is not "unobtainable": that same work's human study had 10 professional editors do pairwise "which cut is better" comparisons — the protocol is validated, it just stopped at dozens of examples. Scaling to the 10⁵ range needs roughly 600–800 editor-hours, on the order of US$40–80k — cheaper than a mid-sized preference dataset. So the accurate statement is that nobody wants to fund it for a small market, not that it's physically impossible.

4.2 No objective function, and no verifier [Solid]

As of 2026, there exists no public benchmark that takes "is this editing craft good" as the thing being measured. Every existing asset substitutes a different question — imitation of the released cut, i.e. "did you pick the same frame as that editor did."

How fatal that substitution is has already been demonstrated in an adjacent field: video summarization used multi-reference evaluation with several human-annotated versions of the same footage, and the result was that randomly generated summaries scored comparably to published methods. Multiple references didn't rescue the metric — they exposed that it had degenerated.

Sharper evidence comes from the one public benchmark that lists J cut, L cut, match cut and friends as explicit categories. On the easy version of the task — cut position already given, only the type needs classifying — the best baseline reaches mean AP of just 47.9%, with match cut at 2.43%.

Two corrections are mandatory here, or you'll cite it wrong:

• [Narrowed] Per-class AP can't be read at face value. The rank correlation between each class's AP and its base rate in the data is ρ=0.903. So "reaction cut 83%, speaker-change 77% — those are high" is mostly base rate doing the work; their lift over base rate is the lowest in the table.
• [Solid] What's genuinely damning is the match-cut model ladder. Going from a linear probe to the strongest configuration, overall mAP climbs 23.9 → 47.9, cut-away goes 14.8 → 63.0, and even smash cut at a 4% base rate goes 10.7 → 24.3 — while match cut moves 1.1 → 2.3 and otherwise doesn't budge. That says the missing ingredient isn't capacity, isn't sample count, and isn't class imbalance (an ablation with a loss designed for long-tail multi-label made it slightly worse). The representation simply has no term for cross-shot compositional correspondence. The input pipeline crops to 112×112 and destroys composition first — and composition correspondence is the definition of a match cut.

4.3 Sampling economics kills reinforcement learning [Inference, arithmetic verified]

If you can't write the objective analytically, can you use real engagement as the reward? Run the numbers. To detect a 1 percentage point difference in retention via A/B, you need roughly 3.9×10⁴ plays per arm; for 0.5 points, 1.6×10⁵. The true effect of a 3-frame trim is almost certainly far below 0.5 points, and a 10-minute talking-head piece has 200–500 cuts. One round of per-cut credit assignment across a whole piece would need 10⁷–10⁸ plays — more than the lifetime view count of most target channels. This is sampling economics, not compute.

4.4 The criterion itself is non-local [Solid, with one common misquote to remove]

The most-cited weighting in editing is Walter Murch's Rule of Six: emotion 51% / story 23% / rhythm 10% / eye-trace 7% / two-dimensional plane 5% / three-dimensional space 4%. It's routinely used to argue that "only the bottom 16% is formalizable."

[Narrowed] That usage is a category error. Murch gives a lexicographic priority ordering (his own phrasing is to sacrifice your way up, item by item, from the bottom), not additive shares. Summing 7+5+4 into a "16% capability ceiling" is mathematically meaningless. What it actually demonstrates is something else — that no hand-written, decomposable objective function for edit quality exists, because the top two entries were never defined as measurable quantities within that vocabulary.

And it has a direct consequence: an A/B on ±6 frames, conducted outside the context of the whole scene, mostly measures noise. Which means that even if you obtained every EDL diff from v1 through v40 of some film, the labels would be specific to that editor and that scene, and might not transfer.

5. The claim verification overturned: "1fps is a hard wall" is wrong

This one deserves its own section, because it's the claim I was most likely to get wrong myself [Narrowed].

The usual argument runs: mainstream video LLMs ingest at 1 frame per second, so 96% of the frames in 24fps footage never enter the context, and frame-level fine cutting is therefore walled off at the level of input representation. That argument doesn't hold. 1 fps is a default, not a ceiling — the mainstream API's video metadata carries a configurable sampling-rate field (valid up to 24), and on the open-source side the sampling rate is entirely the caller's choice. 24fps footage can go into context at 100%.

Cost isn't the wall either: scanning an hour at 1fps is roughly 900k tokens, in the small-change range; even maxed to 24fps at low resolution, an hour is about 5.7M tokens. What stops it is neither money nor "pixels can't get in."

The blockers that actually survive are three, and they're more interesting:

1. Output contract: the API returns no frame numbers, the model has no frame-index handle, and the documented convention is to write times as MM:SS. The sampling-rate ceiling of 24 also means 25/30/50/60fps footage can't be fully covered even maxed out. Getting frame numbers requires the caller to maintain their own extraction-to-frame mapping.
2. Temporal localization accuracy: the vendors' own stated ceiling has consistently been "second-level." The strictest public evaluation tier in the entire temporal-grounding field stops at IoU=0.95, while frame-level accuracy corresponds to IoU≈0.99–0.997 — no benchmark, loss, or leaderboard points at that range. For scale: on segments averaging 8 seconds, IoU=0.7 permits a 34-frame displacement.
3. Label grid: mainstream datasets quantize ground truth to 2-second clips, and human annotators disagree on boundaries at the second scale anyway. Evaluating IoU≈0.99 against labels with ±1s of noise is measuring the noise.

So the correct formulation is: a bare model emitting frame-level in/out points = can't (mechanism: second-level localization accuracy plus no frame-number output contract); a tool-equipped agent performing frame-level trims = can, but the frame accuracy is supplied by deterministic tools. Attributing the mechanism to "1fps sampling" invites the wrong rebuttal ("just raise the frame rate, then") — when what actually seals the road is labels and objective functions.

One counterexample is especially persuasive: back in 2017, a SIGGRAPH paper used hand-written film-idiom cost functions and dynamic programming to produce fully automatic, frame-level dialogue-scene editing decisions, with zero neural video understanding anywhere in the loop. Nine years on, the decision layer in shipping products is still hand-written heuristics. The bottleneck was never model intelligence.

6. "AI can't drive editing software" is an expired claim

The most counterintuitive finding of the round is at the execution layer [Narrowed — the draft badly underestimated this]. As of mid-2026, frame-level timeline manipulation is already open to agents:

• In one major NLE's newer scripting interface, video and audio clips each independently expose actions to set start, end, and in/out points, addressable by frame number plus frame rate. So "nudge this cut 3 frames earlier" is a single call, and "L cut, audio enters 8 frames early" is two lines of script — because the audio and video are two separate objects to begin with.
• The MCP ecosystem aimed at NLEs is now substantial, with individual projects exposing 270–340 tools and still under active development; there are even AI-native editors shipping their own MCP servers.
• At the interchange-format level, J/L cuts are first-class: FCPXML expresses a split edit with two attributes on the same clip (audioStart and audioDuration) — the spec's own comment says exactly that — with no track splitting required.

In other words: AI can already emit a frame-accurate timeline containing J/L cuts today. It just doesn't know whether to offset, or by how much. The bottleneck has moved wholesale to judgment.

And the strongest corroboration is silence: the flagship NLE's latest release lists more than a dozen AI features on its own site — footage search, speech generation, focus adjustment, face age and retouching, slate identification, sharpening, motion deblur — and not one of them touches cut timing. The entire AI budget for that cycle went into retrieval, restoration, beautification, and generation.

Which means that at the fine-cut layer there is currently no gap between marketing and reality — because nobody is marketing it. The vendors aren't even overclaiming. That says more than any benchmark.

7. If you want to build this yourself: a pipeline that runs today

Given all of the above, there's only one viable architecture — and notably, it doesn't require waiting for any model to improve:

1. Deterministic detection layer (the source of frame accuracy): word-level forced alignment, shot boundary detection, VAD/silence, beat grid, speaker diarization. All open source, mature, cheap.
2. Symbolic selection layer (the LLM's correct position): turn the previous layer's output into timecoded text candidates and let the model do nothing but select, rank, and segment. An hour of footage as "word-level transcript + shot table" is roughly 12–20k tokens, about 70× cheaper than 1.08M video tokens — and breath, filler words, and talking-head pacing are exactly the decisions that can be made frame-accurately on that cheap path.
3. Write-back layer: emit an interchange format, or drive the NLE scripting API directly.

Format choice has a counterintuitive conclusion [Narrowed]: don't use OTIO as your delivery format. Its core schema has no audio level automation, no keyframes, no A/V link semantics, and no speed ramps. Anything reducible to a point in time survives losslessly (frame-level in/out points, the J/L offset itself, beat sync), but the audio envelopes and ramps that fine cutting depends on are silently dropped at the adapter boundary. The right use is OTIO as internal intermediate representation (rational time base, small schema, JSON that's easy to generate and diff), with FCPXML generated directly at the delivery layer.

Avoid EDL more strongly still: it does not store the frame rate in the file at all (not an implementation gap — the spec has no such field), and it can't carry a second video track, keyframes, or speed ramps, with only 3-digit event numbers.

The ranking of real-world failure modes is also the reverse of intuition [Narrowed]. By damage, worst first:

1. Rate semantics errors: treating 23.976 as 24 costs 86 frames per hour; treating drop-frame as non-drop costs 108 frames per hour.
2. Interval semantics errors: some formats' out is a closed interval, others' end is open — a systematic one-frame error at every single cut. This is the classic cross-format off-by-one.
3. Variable frame rate footage: phone and screen recordings have no constant frame duration, so "frame accurate" is undefined before conform.
4. Numeric representation precision — dead last. Microsecond representation error is about 1.5×10⁻⁵ frames; the claim that "accumulating a few hundred cuts drifts to a whole frame" is off by roughly 100×. The real risk is a rounding-mode mismatch when the renderer converts back, not exhausted precision.

Put differently: time-base incompatibility is a deterministic, unit-testable bookkeeping problem the industry solved twenty years ago. It is not the killer. The perception layer's error floor is 100–1000× larger.

8. A few details worth remembering on their own

• Beat sync usually fails by walking in phase, not by jitter. [Solid] Beat period and frame period are generally incommensurable: 128 BPM at 24fps is 11.25 frames per beat, so the quantization error walks back and forth between ±0.5 frames (±20.8ms) on a 4-beat cycle. Human detection threshold for irregularity in an isochronous sequence is about 2–5% of the interval, and this lands at roughly 4.4% — over threshold, and perceived as speeding up and slowing down. So a single global audio nudge only changes the constant term and cannot fix the phase-walking component; the real fixes are retiming the tempo to be commensurable with the frame rate, or sub-frame retiming of the picture.
• "You can hear a 10ms error" uses the wrong threshold. [Narrowed] ±10–20ms is the auditory-to-auditory discrimination threshold, not the audiovisual one. The audiovisual threshold is an order of magnitude larger and asymmetric: broadcast standards allow roughly 40ms of audio lead but 60ms or more of audio lag. A single cut's half-frame quantization error is actually below the single-event audiovisual detection threshold.
• Professional breath removal doesn't delete — it attenuates. [Solid] The craft practice is to duck breaths by 6–12dB, not cut them out. Automated tools' hard cuts plus fixed micro-fades create both a room-tone discontinuity and a broadband click — that's half the reason automated silence removal "sounds mechanical." The other half: every tool cuts audio and video on the same frame when removing gaps, and not one derives a J/L offset automatically.
• Filler-word removal really does shift cost — but state the direction correctly. [Narrowed] The audio seam is already fixed automatically (a 2–4 frame crossfade over steady room tone is perceptually invisible). The picture side can't reuse that trick, because pose is non-stationary and a dissolve produces a double exposure rather than a cover. Optical-flow jump-cut repair is mathematically undefined at occlusions (a hand crossing the face, a blink) and large head turns — not imprecise, but with no corresponding pixels to move. Net wall-clock time still drops substantially; what rises is the share of remaining time spent inspecting seams.
• Coverage sets the ceiling, and this is pre-AI craft that was already validated. [Inference] In a single continuous take from one camera, any operation requiring a picture change has zero degrees of freedom — not combinatorial explosion, but an empty candidate set, so no algorithm has anything to choose from. Front-loading the narrative formula into a shot list is the highest-leverage intervention in this entire problem; AI's genuine contribution is turning that list into an ingest-time matching key, which reduces editing to an assignment problem.

9. What this research did not cover

Honest labeling matters, especially with the search budget exhausted:

• February–July 2026 is a complete blind spot. Every conclusion touching product versions, feature names, or pricing may already be stale — particularly the NLE AI feature lists, which turn over quarterly.
• Vendor-internal assets are invisible. NLE vendors and post houses hold private "rushes ↔ finished timeline" pairings — the only party that could plausibly possess frame-level supervision. If any of them already has paired data at the 10⁵ scale, section 4.1 collapses immediately. That is the single most refutable claim in this piece.
• Many accuracy figures are "as claimed in papers/docs," not reproduced. No detector was run on real footage this round, so real-world degradation is unknown.
• No public benchmark of professional editors' in/out points on raw footage exists. So the core capability of the fine-cut layer has no metric at all today — which is both the largest knowledge gap and possibly the most valuable proprietary asset to build.

Main sources (all passed three-perspective verification): craft and workflow — Murch, In the Blink of an Eye (lexicographic priority, not additive weights), and the downstream contracts of picture lock, change lists, and conform; computational editing — Leake et al., SIGGRAPH 2017 (frame-level automatic dialogue-scene editing with zero neural video understanding), Berthouzoz et al., SIGGRAPH 2012; learning cut placement — Pardo et al., ICCV 2021, Learning to Cut by Watching Movies (255K cuts; top-1 8.18% at ~8-frame tolerance), MovieCuts, ECCV 2022 (173,967 clips / 10 classes, mean AP 47.9, match cut 2.43, model ladder 1.1→2.3), AVE, ECCV 2022, Netflix Match Cutting, WACV 2023 (candidate retrieval, explicitly leaving frame choice to the editor); evaluation methodology — Otani et al., CVPR 2019 (random summaries score comparably to published methods under multi-reference F1), Otani et al., BMVC 2020 (strong priors in temporal-localization datasets); temporal grounding — QVHighlights (ground truth quantized to 2-second clips), Charades-STA, ActivityNet Captions, the Qwen2.5-VL technical report (second-level event localization); signal layer — CTC forced alignment and phoneme-level aligners (10–20ms), TransNetV2 (BBC 96.2 / RAI 93.9 / ClipShots 77.9), beat tracking (mir_eval ±70ms tolerance), PodcastFillers (non-lexical filler event F1 92.8); sync thresholds — EBU R37 and ITU-R BT.1359-1 (asymmetric audiovisual thresholds); formats and interfaces — the FCPXML DTD (audioStart/audioDuration expressing split edits), the OTIO core schema, the CMX3600 spec, NLE scripting interfaces and the MCP ecosystem. 56 load-bearing conclusions entered three-perspective adversarial verification; all landed on "directionally sound, wording requires correction," and this piece presents the corrected versions, with grades and recency risk labeled throughout.