Real-time InteractionMOSS-Live lets AI keep listening, converse naturally, and handle tasks within the same conversation. It understands new input while speaking and uses meaning to decide when to respond, wait, or accept an interruption. Across multiple turns and topic changes, it retains context and picks up unfinished explanations.
The system combines speech, live video, screen sharing, and environmental audio. Its foreground and background models work together to process complex reasoning, tool use, and task execution asynchronously. Users can assign tasks while continuing to talk; once results are ready, MOSS-Live brings them back into the conversation at a suitable moment.
Core Capabilities
Full-duplex conversation and semantic interruption
Listening and speaking happen at the same time. MOSS-Live uses meaning and context to distinguish backchannels, thinking pauses, and genuine interruptions. It continues through brief acknowledgments such as “mm-hmm” or “right,” but stops promptly when the user asks a new question and responds to the new request.
Ignoring unrelated speech and interacting with multiple speakers
During group conversations, navigation announcements, or background speech, it determines whether speech is directed at it, reducing false responses to unrelated audio. It tracks topics, intended recipients, and changing information throughout group discussions.
Multi-turn memory and resuming after interruptions
Users can ask questions, add constraints, or change topics at any time. MOSS-Live maintains context and resumes from the interrupted point when users return to an earlier topic.
Real-time audiovisual understanding
Video calls and screen sharing bring visual content, speech, and environmental sounds into the conversation. Users can ask about real scenes and let MOSS-Live use what it sees to understand, explain, and look up relevant information.
Tool use and asynchronous background execution
Look up information, plan trips, and conduct research while talking. Longer tasks run in the background as the foreground conversation continues. Shared context keeps task progress and results in sync so they can be delivered at an appropriate moment.
Streaming simultaneous interpretation and proactive interaction
It produces the target language while continuously receiving source-language speech, without waiting for the entire utterance to end. It can also intervene proactively under user-defined conditions, such as correcting a recitation error or offering a reminder when a meeting discussion meets a specified condition.
Technical Architecture
MOSS-Live coordinates real-time interaction with complex task processing, allowing long-running tasks to progress alongside the current conversation.
- Foreground interaction model: continuously perceives speech, visuals, and environmental information, handling understanding, spoken interaction, and decisions about responding, waiting, or accepting interruptions.
- Background model: handles complex reasoning, tool use, and task execution, asynchronously processing tasks and context passed on by the foreground model.
- Coordination layer: shares memory, context, and task state, keeping the foreground model informed of background progress so results can return naturally to the conversation.
Demos
Demo 1 | A group meeting: join the discussion, look up information, and summarize conclusions
At an event-planning meeting, MOSS-Live joins as an AI participant and looks up the weather and concurrent events in Shanghai and Hangzhou based on the discussion. The meeting continues during searches, and it reports results at an appropriate moment. It speaks through acknowledgments but stops for new questions. After several rounds, it can return to earlier topics and summarize preliminary conclusions and follow-up tasks.
Capabilities shown: multi-speaker interaction, distinguishing acknowledgments from interruptions, contextual memory, tool use, and proactive feedback.
Demo 2 | Planning a road trip: change your mind and keep the itinerary moving
While planning a two-day trip to Zhoushan, a user switches from hiking to relaxing by the sea, then adds questions about ferry tickets, parking fees, and accommodation. MOSS-Live updates the itinerary while searching for hotels in the background. When asked to “pick up where you left off with the island plan,” it resumes the earlier explanation without treating navigation announcements as new user instructions.
Capabilities shown: evolving requirements, resuming after interruptions, background tasks, and ignoring unrelated speech.
Demo 3 | Exploring Shanghai: see, talk, and understand the world around you
An international visitor talks to MOSS-Live on a phone while planning a route through the Bund and Nanjing Road. When the visitor points the camera at a building, MOSS-Live identifies the Peace Hotel from the live view and explains its history. As the conversation turns to room rates and dining, it looks up relevant information, then returns to the unfinished story of the green roof.
Capabilities shown: English conversation, real-time visual understanding, information retrieval, acknowledgment recognition, and contextual continuity.
Evaluation Results
We compile results from six benchmarks—FDB v1, FDB v1.5, FDB v2, FDB v3, Big-Bench Audio, and τ-Voice—covering full-duplex interaction, tool use, speech understanding, and multi-turn dialogue, and compare them with several voice models.
Full-duplex capabilities
In the compiled results, MOSS-Live ranks first among participating models on all six full-duplex metrics shown below. It better distinguishes brief pauses, acknowledgments, and unrelated speech, maintaining conversational flow while reducing unnecessary interruptions and false responses.
Tool use and multi-turn tasks
From individual tool calls to multi-turn task execution, MOSS-Live ranks among the leading models in both evaluations: its FDB v3 tool-call success rate is 51%, tied for the highest score, and its τ-Voice multi-turn task success rate is 57.93%, the second highest among participating models.
Speech understanding and multi-turn dialogue
Across speech understanding and multi-turn dialogue, MOSS-Live ranks among the leading models on several evaluations in this comparison, with particular strengths in conversational continuity, information tracking, and error correction. Together, these abilities support sustained conversation: understanding the current question while retaining context and following changes in the discussion.
