Online Video Agent Harness for Long Video Understanding

Sen Yang*, Boqiang Duan*, Jing Yang, Weihao Bo, Jie Liu, Boyuan Tong, Ze Feng, Wenkang Zhang
Jingdong Wang, Hua Wu

Baidu Inc.
* Equal contribution.

Abstract

Long-video understanding often behaves like a visual needle-in-a-haystack problem: query-relevant evidence is sparsely distributed, while packing dense frames into a single VLM context incurs context rot and high cost. We present VideoXAgent, a purely online video-agent harness that starts from the raw video file and user query, plans and decomposes the task, invokes specialized expert tools on demand, and aggregates multimodal evidence with conflict resolution. Tools are guided by a data-driven taxonomy of atomic capabilities (scripts, VLMs, domain models). Across Video-MME-Long, LongVideoBench-Long, LVBench, and MINERVA, the agent is competitive with frontier LMMs under ~50k agent context—even on hour-long videos—and can bring a visually weak or text-only orchestrator into this range.

Method

Purely online: start from the raw video at query time, with no offline index. A capability taxonomy guides tool design; the harness handles planning, invocation, and budget control.

Offline vs online agent paradigms
Offline indexing first vs. purely online, on-demand exploration.
System architecture overview
System: plan and decompose → expert tools → multimodal evidence aggregation.

1. Online orchestration

Inputs are the video path and the query. The agent invokes tools on query-relevant spans on demand, without query-agnostic full-video preprocessing.

2. Capability mining

From MINERVA expert reasoning traces we mine 22 atomic capabilities in 5 categories, used as a blueprint for the tool space rather than an ad hoc toolkit.

3. Expert tools + harness

60+ tools (scripts / VLMs / detection, OCR, ASR, face, etc.). A ReAct loop with objective-evidence prompting and budget warnings curbs hallucination and non-termination.

Visual Perception Temporal Relation Spatial Relation Audio-Visual High-Level Reasoning

Experiments

Headline setup: Claude-Opus-4.6 as orchestrator with Gemini-3.1-Pro-preview VLM tools. About 50k agent context, far below dense frame packing or full-video preprocessing.

85.1%VideoMME-Long · avg 51k
78.1%LongVideoBench-Long · avg 55k
76.5%LVBench (vision-only) · avg 55k
65.7%MINERVA · avg 37k
Method Type VideoMME-L LVBench Context scale
Seed-2.0-proLMM85.5%76.4%~112k
Gemini-3.1-Pro-previewLMM86.2%—~179k
HAVENAgent82.8%71.7%*Thousand-frame preprocess
VideoXAgentAgent85.1%76.5%~51–55k · tens of frames

*HAVEN LVBench result is from the visual-only ablation shown in its paper's Table 3.

MINERVA: A visually weak or text-only orchestrator plus VLM tools matches frontier LMMs at a smaller context than 1,024-frame packing.

Method Type Agent VLM Acc. Context avg (min–max) Agent Steps
Gemini-2.5-Pro-ThinkingLMM——64.7%†65.5k (256f)—
Gemini-2.5-Pro-ThinkingLMM——66.2%†262.1k (1,024f)—
Gemini-3-Pro-previewLMM——65.0%‡——
Doubao-Seed-1.8LMM——62.4%——
Doubao-Seed-2.0-proLMM——66.5%——
VideoXAgentAgentGLM-5.2Qwen3.6-35B-A3B65.4%33.7k (17.7k–189.9k)35.6
VideoXAgentAgentClaude-Opus-4.6Qwen3.6-35B-A3B65.7%37.1k (25.0k–97.6k)27.5

†From the original MINERVA paper (1,515 questions, with ASR; context from reported frames at 256 tokens/frame). ‡Gemini-3-Pro-preview from the Doubao-Seed-1.8 paper. GLM-5.2 is text-only; Claude-Opus-4.6 is visually weak—gains come from the harness, not from packing the video into one context.

LVBench accuracy comparison (wo. subtitle or audio)
Video-MME-Long (w. subtitle): accuracy vs. context cost.

Trajectories

Real agent runs. The first is the paper case study; the rest are logged traces with tool-produced frames. Switch tabs to inspect each path—without packing the full video into one context.

VideoMME, qVZOKel-gpE. Coarse localization, then subtitle and local visual cross-checks.

Watch on YouTube · qVZOKel-gpE
https://www.youtube.com/watch?v=qVZOKel-gpE
GT / Pred = D Q. On the third day of training, why is the man in the green top so far behind the contestant in the white top?
A. Slacking off · B. Windy / difficult · C. Fell and hurt · D. Bike broke down, repairs took time
User
Video path + the question above.
Tool call
get_video_duration
Observation
2242s (37:22), 1280×720, audio available.
Tool call
check_subtitle_exists
Observation
Matching SRT found, 749 cues.
Agent
In parallel: localize “third day of training” from subtitles, and build a scene-level overview.
Parallel
locate_event_by_text("third day of training")
caption_video_scenes(max_scenes=30)
Observation
Subtitle hit [905–912s] “energy training day three”. Scene 13 (~875–960s) opens Day 3 of the ride.
Tool call
get_video_subtitle([1050s–1340s])
Observation
[17:55] “I just fell down… I'm fine.” · [19:38] “I lost my two lowest gears.” · [21:01] “he's not here yet” · [21:16] “I fixed it—took my wheel out.” Competing cues for B, C, and D appear together.
Agent
Subtitles favor D, but a fall and wind also appear. Run local visual verification on [1255–1335s].
Tool call
query_segment_detail(1255s, 1335s, num_frames=6)
Observation
White-top waits on the track; green-top arrives later by bicycle, consistent with a repair delay.
20:55 white-top waiting 20:55 White-top waits
21:24 green-top arrives 21:24 Green-top arrives later
22:00 follow-up 22:00 After arrival
Answer
D — bike broke down and the repairs took time. B and C are incidental context, not the primary cause.

Division of labor: coarse temporal routing narrows the search → subtitles and vision recover complementary evidence → the answer is deferred until conflicts are resolved.