Think, Then Verify: A Hypothesis–Verification Multi-Agent Framework for Long Video Understanding

Zheng Wang¹, Haoran Chen¹, Haoxuan Qin¹, Zhipeng Wei³, Tianwen Qian², Cong Bai^1†,

¹Zhejiang University of Technology, ²East China Normal University,
³UC Berkeley ^†Corresponding author

Abstract

Long video understanding is challenging due to dense visual redundancy, long-range temporal dependencies, and the tendency of chain-of-thought and retrieval-based agents to accumulate semantic drift and correlation-driven errors. We argue that long-video reasoning should begin not with reactive retrieval, but with deliberate task formulation: the model must first articulate what must be true in the video for each candidate answer to hold. This thinking-before-finding principle motivates VideoHV-Agent, a framework that reformulates video question answering as a structured hypothesis–verification process. Based on video summaries, a Thinker rewrites answer candidates into testable hypotheses, a Judge derives a discriminative clue specifying what evidence must be checked, a Verifier grounds and tests the clue using localized, fine-grained video content, and an Answer agent integrates validated evidence to produce the final answer. Experiments on three long-video understanding benchmarks show that VideoHV-Agent achieves state-of-the-art accuracy while providing enhanced interpretability, improved logical soundness, and lower computational cost.

Difference between prior agents and VideoHV-Agent

Figure 1: From correlation-based search to hypothesis verification: prior agents search and aggregate related clips, while VideoHV-Agent verifies testable clues with focused visual evidence.

Method Overview

VideoHV-Agent recasts long-video understanding as a hypothesis–verification process, thus achieving “thinking then verify” principle:

Context Summarization

This design preserves detailed information when necessary while keeping the overall context compact and efficient to process.

Two-step Reasoning

Our approach adopts a two-step hypothesis–verification process that first formulates hypotheses beyond coarse summaries and then verifies them with fine-grained visual evidence, enabling accurate and efficient long-video reasoning.

Evidence Integration

In the final stage, the Answer agent integrates validated evidence to re-evaluate candidates and produce the final answer through explicit, evidence-grounded reasoning.

Figure 2: Overview of the proposed VideoHV-Agent framework. The framework first (a) summarizes the long video captions, then performs (b) two-step reasoning where a Thinker and a Judge agent rewrite options into hypotheses and a discriminative clue, and a Verifier agent grounds this clue to collect visual evidence, finally an Answer agent integrates the evidence to (c) answer the question.

Comparative Experiments

VideoHV-Agent achieved SOTA among zero-shot methods on Egoschema, NextQA and IntentQA datasets. It shows that video agents perform well on problems such as temporal order understanding, common sense understanding, and character behavior intention. For the NextQA ATP-hard subset, our accuracy improved significantly. VideoHV-Agent is not only effective at solving simple problems, but also performs even better on difficult ones. This demonstrates that VideoHV-Agent has the ability to solve complex problems. This is also consistent with our hypothesis verification method, which focuses more on causal reasoning. VideoHV-Agent shows outstanding performance on both splits, highlighting the robustness of the method.

Case Study

we qualitatively demonstrate the effectiveness of VideoHV-Agent. Given multiple possible cloth joint ways, the Thinker formulates testable hypotheses and specifies the visual clue needed to distinguish them. When inspecting early frames (3–8), the Verifier finds insufficient evidence and marks the clue as unverified rather than guessing. In a second round, it examines later frames (31–35), detects clear sewing actions (e.g., illuminated machine, fabric under the presser foot, manual alignment), and reaches a verified judgment consistent with the ground truth. This case demonstrates targeted evidence retrieval, robust uncertainty handling, and transparent reasoning that directly ties visual observations to logical decisions.

Figure 4: Qualitative study of event understanding in long videos. VideoHV-Agent uses hypothesis–verification to locate decisive evidence, highlighting its ability to avoid search purposefully and ground conclusions in explicit visual proof.

BibTeX

@misc{wang2026thinkverifyhypothesisverificationmultiagent, title={Think, Then Verify: A Hypothesis-Verification Multi-Agent Framework for Long Video Understanding}, author={Zheng Wang and Haoran Chen and Haoxuan Qin and Zhipeng Wei and Tianwen Qian and Cong Bai}, year={2026}, eprint={2603.04977}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2603.04977}, }