Evaluating video generative edits using pixel-level comparisons fails because AI models reconstruct entirely original visual textures rather than replicating target baseline frames. To address this breakdown, Xiaomi’s MiLM Plus team released PROVE (Perceptual RemOVal cohErence), an open-source Apache 2.0 PyTorch repository accepted at ACM MM 2026 for automated video object removal benchmarking.
PROVE introduces two reference-free metrics—RC-S for spatial coherence and RC-T for temporal consistency—that use sliding-window Maximum Mean Discrepancy over DINOv2 features, specifically employing the DINOv2-giant feature extractor model specification. Running at 134.6 ms/frame on a single NVIDIA RTX 4090 GPU, RC-S executes 13.7× faster than the legacy CFD metric, which takes 1842.8 ms/frame, while also outperforming ReMOVE at 180.7 ms/frame.
Against human quality rankings, RC-S achieved a 0.59 average Kendall’s τ and a 0.66 Spearman’s ρ, dramatically outpacing ReMOVE (0.26 τ / 0.29 ρ) and CFD (0.16 τ / 0.18 ρ). According to research published by Xiaomi’s team, this benchmark breakthrough establishes a fast, automated standard for continuous integration without relying on ground-truth video pairings.
Evaluating Spatial and Temporal Coherence Without Reference Videos
Traditional pixel metrics degrade severely when assessing generative video inpainting. On the RORD dataset, legacy metrics yield poor human agreement: PSNR achieves a Kendall’s τ of 0.01, SSIM drops to -0.22, LPIPS yields -0.23, and m-LPIPS scores 0.19. Global metrics like Temporal Consistency (TC) and Temporal Flicker (TF) dilute localized artifacts by averaging calculations over an entire frame.
For the RC-S metric calculation, this dilution is prevented by expanding the erased target’s bounding box by 1/3 along each side length. Should a sliding window lie fully within the mask, RC-S references every background feature contained in the crop. For temporal analysis, RC-T calculates feature drift by cropping adjacent frames using the union of their masks, measuring feature shifts exclusively within the intersection of the two frames. Xiaomi’s research paper reports that removing the crop step makes the RC-T metric insensitive to injected corruption, and RC-T ignores frame pairs that have no shared restored region, resulting in no valid window contribution for those frames.
Mathematical component choices heavily influence correlation accuracy. Swapping MMD for cosine similarity incurs a Kendall’s τ penalty cost of 0.07, while dropping the sliding window entirely carries a Kendall’s τ penalty cost of 0.11. Selecting the optimal vision backbone is equally critical: DINOv2 reaches an average Kendall’s τ of 0.59, compared to DINOv3 at 0.51 and SAM at 0.44.
For the correlation study, human feedback was collected from twenty participants who evaluated four models for each sample via the Borda count method. On the RORD-Val benchmark, RC-S correctly selected the clean image over blurred and region-swapped edits in 100% of cases, compared to 60.06% for ReMOVE and 49.27% for CFD. Furthermore, blur bias analysis on the ROSE-Bench dataset demonstrates that ReMOVE and CFD incorrectly identify blurred patches as higher quality due to artificial variance reduction.
| Metric Tool | Key Difference | Best For |
|---|---|---|
| RC-S (PROVE) | Sliding-window MMD over DINOv2-giant features; 134.6 ms/frame runtime. | Reference-free spatial alignment with high human correlation (0.59 τ). |
| ReMOVE | Feature distance scoring prone to blur bias; 180.7 ms/frame runtime. | Legacy perceptual masking (fails on synthetic patch blurs). |
| CFD | Contextual feature distribution metric; slow 1842.8 ms/frame runtime. | Offline research baselines (exhibits low 0.16 τ human alignment). |
Benchmark Performance and Real-World Dataset Validation
The Xiaomi team validated RC-S across diverse evaluation datasets, recording consistent performance over existing approaches. Correlation scores for RC-S across specific benchmarks demonstrate its generalization: RORD (0.31), OBER-Wild (0.57), DAVIS (0.60), ROSE (0.61), PROVE-M (0.70), and PROVE-H (0.76). On OBER-Wild, DAVIS, and ROSE, legacy baselines ReMOVE and CFD trailed significantly in human correlation alignment.
To enable standard testing, Xiaomi released the PROVE-Bench suite. It features PROVE-M, containing 80 paired 1080p, 81-frame ground-truth clips, alongside PROVE-H, comprising 100 hard ungrounded real-world video clips. This suite tests models against complex motion and background occlusion without artificial lab constraints.
On the public PROVE leaderboard, state-of-the-art models demonstrate varying strengths. SVOR (1.3B) leads combined RC-S with a score of 0.5197, while EffectErase leads RC-T with a score of 0.2525. These scores reflect how dedicated spatial and temporal evaluation metrics expose model-specific trade-offs in real-world video editing tasks.
📊 Key Numbers
- RC-S Average Kendall’s τ: 0.59 (vs ReMOVE 0.26, CFD 0.16)
- RC-S Average Spearman’s ρ: 0.66 (vs ReMOVE 0.29, CFD 0.18)
- RORD-Val Clean Image Selection Rate: RC-S 100% (vs ReMOVE 60.06%, CFD 49.27%)
- Execution Speed per Frame (RTX 4090): RC-S 134.6 ms (vs ReMOVE 180.7 ms, CFD 1842.8 ms)
- SVOR (1.3B) Combined RC-S Score: 0.5197 on PROVE-Bench leaderboard
- EffectErase RC-T Score: 0.2525 on PROVE-Bench leaderboard
- DINOv2 Model Kendall’s τ: 0.59 (vs DINOv3 0.51, SAM 0.44)
- Sliding Window Ablation Cost: Kendall’s τ drops by 0.11 when omitted
- MMD Loss Replacement Cost: Kendall’s τ drops by 0.07 when swapped for cosine distance
- PSNR Kendall’s τ (RORD): 0.01
- SSIM Kendall’s τ (RORD): -0.22
- LPIPS Kendall’s τ (RORD): -0.23
- m-LPIPS Kendall’s τ (RORD): 0.19
- RC-S Correlation by Dataset: RORD (0.31), OBER-Wild (0.57), DAVIS (0.60), ROSE (0.61), PROVE-M (0.70), PROVE-H (0.76)
🔍 Context
The research paper from Xiaomi’s MiLM Plus team was accepted at ACM MM 2026, establishing external academic peer review for the PROVE evaluation framework. Traditional video quality evaluation relied on full-reference metrics like PSNR or LPIPS, which fail when generative models synthesize plausible background details that do not match the exact pixels of a masked original. PROVE fills this operational gap by offering a reference-free evaluation suite that correlates with human judgment without requiring pixel-perfect ground truth. This advance accelerates automated continuous integration pipelines for generative media. In contrast to hand-built heuristic scripts and slow contextual distribution metrics, PROVE provides a fast PyTorch implementation under the Apache 2.0 license.
💡 AIUniverse Analysis
Our reading: The primary technical achievement of PROVE lies in its local feature distribution matching via sliding-window MMD. By isolating evaluation to bounding boxes expanded by one-third of their side length, RC-S prevents unedited background regions from masking subtle spatial defects inside the erased area. Operating at 134.6 ms per frame on consumer hardware makes automated quality gating viable within production CI/CD workflows.
However, this tight spatial scoping creates a critical limitation. Because RC-S operates strictly within expanded bounding boxes, peripheral editing artifacts—such as long cast shadows, unedited surface reflections, or ambient lighting changes outside the crop—go completely unpenalized. Furthermore, relying on heavy DINOv2-giant feature extractors requires substantial GPU memory, preventing real-time on-device evaluation on mobile phones and restricting the tool to server-side asynchronous test harnesses.
For PROVE to become the universal benchmark for generative video editing over the next 12 months, its developers must extend its perceptual field to detect global lighting and shadow dynamics without re-introducing global metric dilution.
⚖️ AIUniverse Verdict
✅ Promising. RC-S achieves a 0.59 Kendall’s τ correlation with human rankings while running 13.7× faster than CFD, proving its viability for automated video evaluation pipelines despite current boundary shadow constraints.
🎯 What This Means For You
Founders & Startups: AI media startup founders can implement automated model evaluation and continuous integration testing without spending capital to collect expensive paired ground-truth benchmark datasets.
Developers: Computer vision developers can integrate reference-free quality gating into MLOps pipelines using Xiaomi’s single-CLI PyTorch repository executing at 134.6 ms per frame.
Enterprise & Mid-Market: Video production and e-commerce enterprises can systematically evaluate competitive third-party editing APIs and optimize model quantization without manual human quality testing.
General Users: Everyday smartphone gallery users will experience fewer persistent visual glitches, unhandled floating shadows, and temporal flickering in AI generative removal features.
⚡ TL;DR
- What happened: Xiaomi released PROVE, an open-source ACM MM 2026 suite featuring RC-S and RC-T metrics for evaluating video object removal without reference clips.
- Why it matters: RC-S correlates with human quality ratings at 0.59 Kendall’s τ—more than double legacy metrics ReMOVE and CFD—while running at 134.6 ms per frame on an NVIDIA RTX 4090 GPU.
- What to do: Computer vision teams should adopt PROVE-Bench to automate MLOps evaluation pipelines for generative editing without collecting paired ground-truth datasets.
📖 Key Terms
- RC-S
- A reference-free spatial coherence metric introduced in PROVE that measures feature distribution consistency across erased target areas.
- RC-T
- A reference-free temporal consistency metric that tracks feature drift across adjacent frame intersections in video edits.
- PROVE-Bench
- An evaluation benchmark suite containing grounded paired clips and hard ungrounded real-world videos for testing object removal.
- Maximum Mean Discrepancy
- A statistical distance metric used to compare feature distributions between restored target windows and surrounding background regions.
- DINOv2
- A vision transformer feature extractor model whose high-level embeddings supply the perceptual foundation for PROVE metrics.
Analysis based on reporting by MarkTechPost. Original article here.