MEDit-Bench: A Dataset for Evaluating Message-Driven Narrative Video Editing
Code will be made available upon acceptance.
Abstract
Video editing is fundamentally message-driven: even from the same source footage, the selected shots change depending on the narrative the editor wishes to convey. Benchmarks for a closely related task, video summarization, typically provide a single reference ground truth per video and do not account for this diversity.
We present MEDit-Bench, a dataset and benchmark for evaluating message-driven video editing. MEDit-Bench pairs long-form videos with multiple editing messages and multiple professionally produced edits per message, demonstrating that different messages yield substantially different edits from the same source. We define an evaluation protocol combining temporal IoU and an LLM-as-a-judge preference to holistically evaluate narrative quality.
We additionally annotate each message with ambiguity and contextfulness scores, and show that both dimensions negatively correlate with model performance, establishing message difficulty as a meaningful stratification factor. Experiments with state-of-the-art MLLMs and reinforcement fine-tuned baselines show that while strong models approach human temporal alignment at lenient thresholds, all models fall behind humans at stricter criteria. Preference evaluation confirms a large quality gap among model sizes, and a human perceptual study shows that human edits are still preferred over model outputs.
What MEDit-Bench Offers
- Message-conditioned dataset: Each source video is paired with 3 distinct editing messages, reflecting different narrative goals; 60 videos × 3 messages × 3 editors = 540 edited videos.
- Professional editing annotations: Each (video, message) pair is independently edited by 3 professional editors, yielding 540 cut sequences with per-cut start/end timestamps and rationales.
- Dual evaluation protocol: Temporal IoU metrics (R@θ, F1@θ, mIoU for θ ∈ {0.3, 0.5, 0.7}) combined with LLM-as-a-judge preference scoring using Gemini-3-pro.
- Message difficulty labels: All 180 messages are annotated with ambiguity (1–5) and contextfulness (1–5) scores, enabling fine-grained difficulty stratification.
- MEDitor: GRPO-trained model (MEDitor-4B and MEDitor-8B) using editing-specific rewards; MEDitor-8B surpasses all zero-shot open-source baselines and rivals commercial models.
- Human perceptual study: 1,620 side-by-side comparisons between human and AI edits across five film-editing dimensions.
Data Collection
MEDit-Bench is constructed in three stages:
- Source video selection: 60 videos of length 7–18 minutes selected from LongVideoBench and VideoMME.
- Editing message creation: For each video, 3 distinct editing messages are written that describe the narrative the editor wishes to convey. Messages are designed to be (i) visually communicable without relying on audio or dialogue, (ii) diverse in intent across the three messages, and (iii) varied in difficulty from clear and concrete to ambiguous or context-dependent.
- Professional annotation: Each pair is given to 3 professional editors who independently produce a cut-only edit (1–3 minutes), yielding 60 × 3 × 3 = 540 edited videos. Editors are also asked to submit a rationale for every cut; shot orderings and rationales are released alongside the dataset for future research.
Comparison with Existing Datasets
Unlike prior datasets that provide a single target edit per source video without an explicit message, MEDit-Bench provides multiple messages and multiple editor outputs per message. Cross-Annotator F1-Avg is 0.42 (comparable to SumMe: 0.31; TVSum: 0.54), while Cross-Message F1-Avg is only 0.15—confirming that changing the message substantially changes the edit.
| Dataset | Message | Source Videos | Edited Videos | Avg. Duration |
|---|---|---|---|---|
| SumMe | × | 25 | 25 | 2m40s |
| VSUMM | × | 50 | 50 | 1m30s |
| TVSum | × | 50 | 50 | 4m12s |
| MEDit-Bench (Ours) | ✓ | 60 | 540 | 13m44s |
Statistics of the MEDit-Bench dataset. Source videos range from 461–1,076 s (avg. 824 s); edited videos range from 28–329 s (avg. 132 s, avg. compression ratio 16.5%). MEDit-Bench contains 6,231 segments total; average cut length is 11.40 s (median 5.09 s).
Message Quality Labels
To characterize the inherent difficulty of editing messages, we annotate all 180 messages with two properties:
- Ambiguity (1–5): How open to interpretation is the message? Highly ambiguous messages admit multiple equally valid narrative interpretations.
- Contextfulness (1–5): How much external domain knowledge is required? Messages with high contextfulness presuppose familiarity with specific topics.
Both dimensions negatively correlate with model performance: ambiguity r = −0.15 (p < 0.001), contextfulness r = −0.18 (p < 0.001). The most ambiguous messages yield mIoU roughly 34% lower than low-ambiguity messages.
Distribution of message quality labels. Most messages are moderate-to-high ambiguity and low-to-moderate contextfulness.
Mean mIoU as a function of message ambiguity (left) and contextfulness (right). Error bars show standard error. Both dimensions negatively correlate with mIoU (*p < 0.001).
Baseline Methods
Zero-Shot MLLMs
We treat frozen MLLMs (GPT-5, Gemini-3-pro, Qwen3-VL, etc.) as direct predictors: given a source video and an editing message, the model outputs a sequence of cut timestamps with no task-specific training. This probes how well general video understanding transfers to message-driven editing.
MEDitor: GRPO with Editing Rewards
We fine-tune Qwen3-VL models — which we call MEDitor — using Group Relative Policy Optimization (GRPO) with a combined reward that captures both coverage (temporal IoU between the union of predicted and reference cuts) and cut-level F1 (one-to-one greedy matching rewarding the correct number and placement of cuts).
Training data is constructed without expensive professional annotation by distilling pseudo-labels from a frontier MLLM over 977 long videos from MLVU, yielding 2,931 pseudo-labeled (video, message, edit) triples. This scalable pipeline enables GRPO training that substantially improves over zero-shot open-source baselines.
Experimental Results
Evaluation Protocol
We define a dual evaluation protocol:
- Temporal Alignment (R@θ, F1@θ, mIoU): Union-based and cut-level temporal IoU metrics evaluated at θ ∈ {0.3, 0.5, 0.7}.
- LLM-as-a-Judge Win Rate: Forced-choice preference between model and human edits using Gemini-3-pro as the judge, with order-swapped double evaluation to mitigate position bias.
Main Results
We evaluate state-of-the-art MLLMs (GPT-5, Gemini-3-pro, Qwen3-VL) in zero-shot settings and reinforcement fine-tuned baselines (Qwen3-VL+GRPO) trained on pseudo-labeled data.
| Model | R@0.3 | R@0.5 | R@0.7 | F1@0.3 | F1@0.5 | F1@0.7 | mIoU | Pref. (AI vs. Human) |
|---|---|---|---|---|---|---|---|---|
| Random | 0.030 | 0.002 | 0.000 | 0.121 | 0.054 | 0.017 | 0.112 | 16.7% |
| Commercial models | ||||||||
| GPT-5 | 0.348 | 0.065 | 0.009 | 0.273 | 0.151 | 0.064 | 0.253 | 77.8% |
| GPT-5-mini | 0.157 | 0.033 | 0.000 | 0.229 | 0.112 | 0.041 | 0.186 | 31.1% |
| Gemini-3-pro | 0.263 | 0.054 | 0.004 | 0.255 | 0.152 | 0.083 | 0.235 | 72.8% |
| Gemini-2.5-pro | 0.144 | 0.033 | 0.006 | 0.180 | 0.102 | 0.051 | 0.152 | 28.3% |
| Open-source models | ||||||||
| Qwen3-VL-4B | 0.032 | 0.009 | 0.002 | 0.075 | 0.036 | 0.014 | 0.119 | 25.2% |
| Gemma-4-E4B | 0.111 | 0.017 | 0.000 | 0.156 | 0.071 | 0.024 | 0.169 | 30.6% |
| Qwen3-VL-8B | 0.144 | 0.030 | 0.004 | 0.173 | 0.099 | 0.041 | 0.165 | 26.1% |
| InternVL3.5-8B | 0.020 | 0.000 | 0.000 | 0.106 | 0.042 | 0.011 | 0.080 | 7.4% |
| Qwen3-VL-32B | 0.248 | 0.070 | 0.013 | 0.234 | 0.126 | 0.061 | 0.227 | 40.7% |
| MEDitor — GRPO fine-tuned (ours) | ||||||||
| MEDitor-4B (Ours) | 0.244 | 0.069 | 0.013 | 0.250 | 0.146 | 0.071 | 0.229 | 52.9% |
| MEDitor-8B (Ours) | 0.304 | 0.100 | 0.026 | 0.258 | 0.150 | 0.061 | 0.241 | 54.9% |
| 🌟 Human | 0.432 | 0.133 | 0.033 | 0.277 | 0.184 | 0.109 | 0.290 | — |
Pref. = LLM-as-a-judge win rate vs. human edits (green: AI wins, red: human wins). LLM-as-a-judge results are for reference.
Key Findings:
- All models outperform the random baseline, but fall well short of the human ceiling, confirming MEDit-Bench is a challenging benchmark.
- Performance degrades sharply at stricter thresholds: at θ=0.7, the best model achieves only 0.026 vs. human 0.033.
- Among zero-shot models, larger commercial models generally outperform open-source ones.
- MEDitor-8B achieves the highest R@0.5 (0.100) and R@0.7 (0.026), surpassing all zero-shot open-source models and rivaling commercial models, with its LLM preference win rate rising from 26.1% to 54.9%.
- Gemini-3-pro's unusually high LLM preference score (72.8%) may reflect self-enhancement bias since the same model family acts as judge.
Human Perceptual Study
To validate the quality gap beyond automatic metrics, we conducted a perceptual study comparing professional human edits to Gemini-3-pro outputs. For each of 90 (video, message) pairs, three independent evaluators watched both edits side-by-side and assessed them on five film-editing aspects: (A) message transmission, (B) cut order, (C) shot continuity, (D) contextual depth, and (E) narrative pacing.
Human perceptual study results (1,620 evaluations). Left: human edits preferred in 56.7% of first-choice and 56.8% of final-choice evaluations. Right: mean scores (1–5) per aspect; the largest gap appears in cut order and message transmission.
Qualitative Analysis
Qualitative comparison on two cases where all evaluators unanimously preferred the human edit. Each pair shows the human edit (top) and Gemini edit (bottom) alongside the diagnosed failure mode. Top (“The white rabbit who loves nature”): the abstract message causes Gemini to select clips irrelevant to the intended narrative. Bottom (“The revenge of the white rabbit”): Gemini omits the contextual buildup needed to make the reversal meaningful.
These cases illustrate the core challenge: current MLLMs can identify approximately relevant scenes but struggle with narrative precision—the fine-grained shot selection and ordering that conveys meaning through sequence rather than content alone.
BibTeX
@article{ogata2026meditbench,
title={MEDit-Bench: A Dataset for Evaluating Message-Driven Narrative Video Editing},
author={Ogata, Katsuya and Pang, Zongshang and Otani, Mayu and Nakashima, Yuta},
journal={arXiv preprint arXiv:2607.25300},
year={2026}
}