FLUID: Flexible, Evidence-Grounded Situated Task Guidance

Yunqi Zhao*, Run Peng*, Ze Dong, Ryan Lu, Yinpei Dai, Joyce Chai
University of Michigan
*Equal contribution

TL;DR FLUID annotates why, whether and when human instructors intervene during real task guidance and what they say, evaluates real-time systems against that evidence instead of a fixed reference, and builds an agent that decides whether, when and how to speak from the same evidence.

0:57Press play: the timeline follows the video. Click any block to jump there.

Abstract

Real-time task guidance is inherently flexible: when an agent guides a user through a task, multiple responses and timings can be appropriate, and sometimes the best action is to remain silent. Existing benchmarks often miss this flexibility by treating synthesized dialogues as fixed targets for what to say and when to say it. We introduce FLUID (Flexible, Evidence-Grounded Situated Task Guidance), a unified framework for modeling, evaluating, and enabling flexible interaction in multimodal task guidance. Using naturally occurring human-human guidance from WTaG and HoloAssist, FLUID grounds interventions in their supporting evidence, necessity, and timing flexibility, and uses this representation both to evaluate flexible guidance and to build a real-time agent for task guidance. Our agent achieves the strongest overall balance of response correctness, intervention precision, timely guidance, and response latency among evaluated systems, while still leaving room for improvement in avoiding speech overlap. Together, FLUID moves situated task guidance beyond rigid reference matching toward more natural and adaptive human-AI interaction.

FLUID-Modeling: Grounded Evidence in Task Guidance

FLUID links each instructor intervention to the user actions or utterances that triggered it, whether the intervention was necessary, and when it would be useful. These annotations capture four layers of task guidance: user utterances, actions and states, evidence and intervention windows, and instructor responses. Explore real examples from WTaG through synchronized video and timelines to see how guidance is grounded in unfolding interactions.

FLUID-Evaluation on Task Guidance

Fixed-reference evaluation uses the same timing tolerance around each human instructor response, regardless of when evidence becomes available. FLUID instead evaluates interventions within evidence-derived windows, assigning a timing score based on when guidance is delivered.

Conventional: Fixed Reference Window

Conventional fixed reference window, reactive: full score inside fixed offsets before and after the reference utterance, which starts where the user turn ends00.51scoretime →Reactive guidancebeforeaftertusr = treftlatest
Full score within fixed offsets, even before the user finishes speaking.
Conventional fixed reference window, proactive: the same fixed offsets around the reference utterance; the early start, the evidence onset and the latest useful time are ignored00.51scoretime →Proactive guidancebeforeaftertearlytevidtreftlatest
Fixed timing ignores when evidence actually becomes available.

FLUID: Evidence-Derived Window

FLUID evidence-derived window, reactive: full score when the user turn ends at t usr, decreasing to zero at t latest00.51scoretime →Reactive guidancetusrtlatest
Full score after the user finishes speaking, decreasing toward the latest useful time.
FLUID evidence-derived window, proactive: partial score between the early start t early and the evidence onset t evid, full at the onset, decreasing to zero at t latest00.51scoretime →Proactive guidancetearlytevidtlatest
Partial score between the early start and the evidence onset, full at the onset, then decreasing toward the latest useful time.

Timely Correctness combines content correctness with the timing score. Necessary interventions are evaluated for timely responses, while optional interventions may be left unanswered without penalty.

FLUID-Agent: From Interaction Dynamics to Real-Time Guidance

FLUID-Agent: perception, task memory, user utterance understanding and intervention decision around one vision-language model

FLUID-Agent is a real-time system for flexible, evidence-grounded task guidance: an asynchronous agentic architecture around one Qwen3.6-35B-A3B vision-language model, which serves perception, utterance understanding and intervention reasoning through task-specific prompting. Its components share a task memory of task-relevant evidence and interaction context, from which the agent decides whether, when and how to intervene, or to stay silent, as the interaction unfolds.

Main results

Evidence-grounded evaluation on the 43 WTaG and 72 HoloAssist test sessions; the small ± values are 95 % confidence half-widths. Bold marks the best per metric and dataset. Hover a metric’s name for its definition.

Acknowledgments

This work was supported in part by the Johns Hopkins Artificial Intelligence and Technology Collaboratory (JH AITC), funded by the National Institute on Aging of the National Institutes of Health under grant P30AG073104. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

BibTeX

@misc{zhao2026fluid,
  title  = {FLUID: Flexible, Evidence-Grounded Situated Task Guidance},
  author = {Zhao, Yunqi and Peng, Run and Dong, Ze and Lu, Ryan and Dai, Yinpei and Chai, Joyce},
  year   = {2026},
  note   = {Preprint}
}