EgoSpeak: Learning When to Speak for Egocentric Conversational Agents in the Wild

Overview

Speaking is easy. Knowing when is not.

A real-world conversational agent cannot wait for a tidy pause before it talks. Conversations in the wild overlap, stall, and restart; speaker roles blur. EgoSpeak predicts, at every frame of an egocentric video stream, whether the camera wearer is about to start speaking — treating the wearer's own speaking moments as free supervision for natural turn-taking.

The model watches exactly what the agent would see and hear, processes it causally in real time, and outputs a continuous speak probability that an agent can act on the moment it crosses a threshold.

Egocentricfirst-person view — the stream is exactly what the agent perceives
RGBworks from raw pixels when audio or nonverbal cues are unreliable
Onlinecausal, real-time — only the past is visible at every step
Untrimmedlong streams with silence and sporadic turns, not pre-cut clips

Prior utterance-initiation and turn-taking methods cover at most two of these four capabilities; EgoSpeak is built for all of them.

Task

Predict the turn — don't detect the silence.

Commercial dialogue systems typically speak after detecting a fixed silence interval, which leaves an agent roughly 200 ms to respond. Psycholinguistics tells a different story: humans start preparing their response while the other person is still talking.

≈200 msresponse window left by silence-threshold detection

600–1500 mshow far ahead humans actually plan their speech (Levinson & Torreira)

up to 2 show far ahead EgoSpeak anticipates speech initiation, in 0.2 s steps

EgoSpeak therefore frames the problem as anticipation: at each timestep it classifies the next 10 future steps (up to 2 s) as background, target speaker speaking, or other person speaking — before the turn shift happens.

Egocentric video frames along a timeline moving through no speech, another person speaking, and the camera wearer speaking, with an agent deciding 'I should speak now'. — teaserEgoSpeak models the conversation from the camera wearer's viewpoint, mirroring how a real-world agent would perceive a dynamic, multi-speaker scene — and when it should jump in.

A transcript with timestamps converted into a row of per-frame one-hot labels colored gray, amber, and violet. — labelsTranscript timestamps become per-frame one-hot labels — gray [1,0,0] background, amber [0,1,0] target speaker, violet [0,0,1] other speaker — so no manual frame-level annotation is needed.

Framework

One causal pass over an untrimmed stream.

Video is downsampled to 20 FPS and encoded at 5 predictions per second: RGB features from a Kinetics-400-pretrained ResNet-50, audio features from wav2vec2, concatenated per frame. On top, three online backbones are evaluated — a Transformer (LSTR), a GRU, and a Mamba model — trained with a cross-entropy objective, akin to next-token prediction in language modeling.

Pipeline diagram: an audio waveform and RGB frames enter an online processing model, which outputs per-class probabilities and a speak-probability curve crossing a threshold labeled 'start speaking'. — methodAt every step the online model classifies near-future frames as background, other person speaking, or target speaker speaking. An agent triggers speech once the speak probability crosses a threshold.

Dataset

YT-Conversation: pretraining from the wild.

Turn-taking corpora usually come from labs or video calls — expensive to annotate and far tamer than real conversation. YT-Conversation instead harvests podcasts, interviews, and casual face-to-face dialogues from YouTube, pseudo-labeled automatically with voice activity detection at 200 ms resolution. Only YouTube IDs are released, so creators keep control of their content.

414in-the-wild conversational videos

41 hof footage, from clips 1 to 60 minutes long

0 hof manual annotation — per-frame labels come from VAD

A three-by-three grid of frames from YouTube conversations: podcast studios, interview sets, and living-room chats. — yt-conversationSample frames: podcasts, interviews, and informal dialogues spanning very different rooms, framings, and speaker counts. Available on Hugging Face.

Results

Anticipation beats waiting for silence.

On two egocentric benchmarks — EasyCom (controlled, around-a-table) and Ego4D (in-the-wild, audio-visual diarization split) — EgoSpeak's predictive models roughly double the target-speaker average precision of a silence-based rule, even though the evaluation is deliberately biased in that rule's favor. The silence rule lands at random-baseline level.

52.7 vs 26.6 EasyCom target-speaker AP — Transformer (A+V) vs silence-based rule (random: 27.2)

66.8 vs 27.7 Ego4D target-speaker AP — Transformer (A+V) vs silence-based rule (random: 26.1)

Average mAP (%) over the 0.2–2.0 s anticipation horizon, mean ± standard error over five seeds. ^P = pretrained on YT-Conversation. Anticipation decays gracefully with distance: the EasyCom Transformer (A+V), for example, goes from 73.8 mAP at 0.2 s to 53.0 at 2.0 s.
Model	Modality	EasyCom	Ego4D
Transformer	A	56.9 ± 0.05	69.2 ± 0.03
	V	51.0 ± 0.08	58.0 ± 0.27
	A+V	58.7 ± 0.13	69.0 ± 0.24
	A+V^P	58.5 ± 0.26	68.9 ± 0.18
GRU	A	57.0 ± 0.30	69.2 ± 0.25
	V	51.7 ± 0.29	57.9 ± 0.61
	A+V	60.6 ± 0.17	68.2 ± 0.42
	A+V^P	57.0 ± 0.29	68.3 ± 0.18
Mamba	A	55.4 ± 0.62	67.9 ± 0.37
	V	50.9 ± 0.21	57.7 ± 0.28
	A+V	57.4 ± 0.26	67.5 ± 0.18
	A+V^P	55.8 ± 0.43	65.8 ± 0.23

Audio+visual input wins on EasyCom (GRU reaches 60.6 mAP); on the noisier Ego4D, audio alone is already strong at 69.2 mAP. Vision-only models trail but stay clearly above chance — enough to matter when audio fails. YT-Conversation pretraining brings modest overall change, with its clearest gain on the other person speaking class (+0.7 AP on EasyCom, +1.5 on Ego4D).

Five egocentric frames above a plot of predicted speak scores; the audio-visual model's curve rises inside the wearer's speaking regions and stays low while another person speaks. — qualitativePredicted speak scores on an EasyCom clip. The audio-only model (red) fires on *anyone's* speech; the audio-visual model (purple) rises mainly inside the wearer's own turns (amber regions) and stays low while others speak (violet region). RGB alone (blue) barely separates the two.

Analysis

What the models tell us.

Line chart of per-frame mAP versus long-term memory length for short-term windows of 16, 32, and 64 frames; the 16-frame curve is highest. — context lengthMore long-term memory steadily helps (up to 4096 frames), but growing the short-term window from 16 to 64 frames consistently *hurts* — models don't exploit short-term information well.

Plot of normalized attention weight versus distance from the prediction frame, collapsing to near zero within about ten frames. — attentionEncoder attention, averaged over the test set, collapses onto the most recent frames — utterance initiation is decided mostly from local context.

Motion helps

EasyCom, average mAP with the gain from adding optical flow (F) in parentheses.
Model	Modality	Avg. mAP
Transformer	A (+F)	57.0 (+8.0)
	V (+F)	51.2 (+6.4)
	A+V (+F)	58.8 (+9.6)
GRU	A (+F)	57.1 (+3.9)
	V (+F)	51.2 (+4.2)
	A+V (+F)	61.2 (+0.9)
Mamba	A (+F)	55.6 (+2.6)
	V (+F)	51.3 (+3.9)
	A+V (+F)	57.6 (+2.7)

Fast enough to interrupt you

Throughput on a single RTX 3090 (EasyCom). All three backbones run in real time; the GRU processes nearly 14k frames per second.
Model	FPS	Params	GFLOPs
Transformer	99.8	67.21M	129.48
RNN (GRU)	13,939.5	34.6M	206.52
Mamba	12,009.3	83.1M	610.93

Plot of prediction score over time: the score rises before amber target-speaking regions but stays flat before a thin red backchannel region. — failure caseAn honest miss: before a short backchannel ("mm-hm", red) the speak probability barely moves, unlike the clear ramps before full speaking turns (amber). Backchannels behave differently from turn-taking — a target for future work.

Citation

BibTeX

@inproceedings{kim-etal-2025-egospeak,
  title     = {{EgoSpeak}: Learning When to Speak for Egocentric Conversational Agents in the Wild},
  author    = {Kim, Junhyeok and Kim, Min Soo and Chung, Jiwan and Cho, Jungbin and
               Kim, Jisoo and Kim, Sungwoong and Sim, Gyeongbo and Yu, Youngjae},
  booktitle = {Findings of the Association for Computational Linguistics: NAACL 2025},
  publisher = {Association for Computational Linguistics},
  year      = {2025}
}