MOSS-Transcribe Takes First in Speech Understanding and MOSS-TTS Second in Speech Generation at the NVVSpeech Challenge

At the ISCSLP 2026 NVVSpeech Challenge, MOSS-Transcribe, MOSS's speech transcription model, took first place in Track 1 (speech understanding) with a score nearly twice the official baseline of 33.32. Our text-to-speech model MOSS-TTS placed second in Track 2 (speech generation) with a final score of 55.20, just behind Tencent-HY. Our papers for both tracks have been accepted to ISCSLP 2026, and we will present them in Penang, Malaysia, in November.

Laughs, coughs, and sighs carry rich emotional and tonal information, yet conventional speech recognition discards almost all of these nonverbal vocalizations (NVVs). The NVVSpeech Challenge is built around these sounds, with one track for listening and one for speaking. The speech understanding track asks a model to transcribe Mandarin and English speech and insert each NVV it hears as a tag at the right position in the transcript. The speech generation track works the other way around and tests whether a model can actually make the sound when the text calls for it: given tags in the text, the model must produce the matching laugh, sigh, or other sound at the specified position.

MOSS-Transcribe, a roughly 2.4B-parameter model distilled on-policy from a 30B teacher, outputs the transcript and its NVV tags in a single generation pass, with no separate event classifier. MOSS-TTS generates a complete utterance containing the specified NVVs in a single inference pass, without splicing in laughter or other audio clips and without any per-sample fixes.

Tracks and Timeline of the ISCSLP 2026 NVVSpeech Challenge

The NVVSpeech Challenge (Understanding and Generation of Speech with Non-Verbal Vocalizations) is a special challenge session of ISCSLP 2026, the 15th International Symposium on Chinese Spoken Language Processing. It focuses on nonverbal vocalizations: sounds such as laughs, coughs, sighs, and gasps that people make while speaking but that are not words. The same sentence can convey a different attitude depending on whether it opens with a laugh or a sigh.

Both tracks cover Mandarin Chinese and English. As defined by the organizers, Track 1 (speech understanding) asks models to detect, classify, and localize nonverbal vocalizations in speech; Track 2 (speech generation) asks models to synthesize speech that naturally incorporates specified nonverbal vocalizations. The two tracks share one set of 16 labels: breath, sniff, laugh, cry, cough, throat clearing, sneeze, sigh, gasp, snore, yawn, hum, moan, hiss, lipsmack, and burp.

ISCSLP 2026 accepted the challenge as a special session on June 15, 2026, and registration opened on June 22. In July, the organizers released baseline models for both tracks and distributed evaluation sets to registered teams. Final system submissions closed on August 2, when the leaderboard was frozen. System description papers were due on August 9, and acceptance notices went out on August 31. On October 6, the organizers published the session program and officially launched the leaderboard; on October 19, they made the test sets and evaluation code public. After the challenge, the organizers released an overview paper comparing 10 systems in terms of data construction, model adaptation, data resampling, candidate selection, and preference optimization.

We competed in both tracks under the team name "Moss-Transcribe," with OpenMOSS as our affiliation. MOSS-Transcribe was our Track 1 entry, and MOSS-TTS was our Track 2 entry. OpenMOSS is an industry–academia research team for large models, jointly established by the Shanghai Innovation Institute, the Fudan University NLP Lab, and MOSS.

Speech Understanding Track: MOSS-Transcribe Takes First Place

Track 1 Task and Scoring

The Track 1 evaluation set contains 985 Mandarin recordings and 961 English recordings. Scoring is strict: a tag with the right class but the wrong position counts as both a miss and a false alarm. Under the scoring rules published in the overview paper, each language's score combines three weighted components:

Event F1, weighted 70%: precision and recall are computed from hits, misses, and false alarms, and F1 is their harmonic mean (higher is better). It is micro-averaged, meaning hits, misses, and false alarms are summed across all 16 event classes before the calculation.
mNTD, weighted 20%: measures the position offset between predicted and reference tags (lower is better) and is scored as 1 − mNTD.
Tagged transcription error rate, weighted 10%: CER (character error rate) for Mandarin and WER (word error rate) for English, with each tag counted as one character or word (lower is better). It is scored as 1 − error rate. Heavy insertion errors can push the error rate above 100%, so it is capped at 100%.

The weighted sum, multiplied by 100, gives the per-language score, and the bilingual score is the average of the Mandarin and English scores. Tag position affects the first two components: a prediction in the wrong position does not count as a hit in event F1, and mNTD measures position offset directly. Together, these two components carry 90% of the weight. The remaining 10%, the tagged error rate, also counts tags, so missing, extra, or wrong tags all raise CER and WER.

MOSS-Transcribe's Approach: A 2.4B Model That Outputs Tagged Transcripts Directly

Under these rules, transcription alone falls far short. In our local tests, a general-purpose automatic speech recognition (ASR) model with good transcription accuracy scored only 8.04 because it could not output valid tags. MOSS-Transcribe is MOSS's speech transcription model, and it writes the text and the event tags in the same generation pass. We trained it in three steps:

We trained a 30B teacher model on large-scale normalized data. The teacher is more than ten times the size of MOSS-Transcribe.
We used 1,000 labeled samples to teach MOSS-Transcribe the tag format.
We ran on-policy distillation on unlabeled challenge audio. MOSS-Transcribe, as the student, first generates its own transcript, and the 30B teacher then provides token-level guidance conditioned on that same prefix.

MOSS-Transcribe's Results: Nearly Twice the Official Baseline

In the official final ranking, MOSS-Transcribe placed first in Track 1, more than 1 point ahead of the runner-up, with a bilingual score nearly twice the official baseline (33.32).

Official leaderboard for NVVSpeech Challenge Track 1 (speech understanding)

The official leaderboard also reports MOSS-Transcribe's metrics for each language:

Event F1 is above 0.63 in both languages and slightly higher in English. Mandarin has a lower mNTD, which means smaller tag position offsets. Both CER and WER here count the tags, so they cannot be compared with text-only ASR results. CER and WER are also measured in different units, so the Mandarin and English error rates should not be compared directly.

Speech Generation Track: MOSS-TTS Places Second

Track 2 Task and Scoring

In Track 2, the input is text with inline tags and the output is a complete utterance. A sentence may contain several NVVs, placed at the beginning, at the end, or between any two words. The model must not read the tag names aloud or blur the words around a tag. Each language has 800 tagged test prompts.

Each system is scored on five dimensions in each language:

NVV Accuracy, weighted 30%: whether NVVs appear with the specified type, count, and position.
NVV Perceptual Effect, weighted 25%: whether NVVs are clearly audible and easy to recognize.
Overall Naturalness, weighted 15%: whether the full utterance is natural, fluent, and acoustically continuous.
Overall Quality, weighted 15%: whether there is noticeable noise or distortion.
Overall Expression, weighted 15%: whether the delivery fits the text, the target NVVs, and the conversational context.

The ranking does not rely on automatic scores alone. The organizers first used a large audio-language model (LALM) to rate the five dimensions, producing the objective score, and then ran human listening tests on the top-ranked systems to produce the subjective score. Both scores are averaged across Mandarin and English, and the final score is 0.30 × objective + 0.70 × subjective.

MOSS-TTS's Approach and Results: Highest Objective Score Among Competing Teams, Second Overall

MOSS-TTS is MOSS's text-to-speech (TTS) model. In the challenge, it generated each utterance in a single inference pass, producing the NVVs together with the surrounding words. It did not splice in laughter or other audio clips afterward, and it used no per-sample fixes.

In the official final ranking, MOSS-TTS placed second in Track 2 with 55.20, just behind Tencent-HY (55.51). The overview paper notes that automatic and human evaluations can rank systems differently. Among the competing teams, MOSS-TTS had the highest objective score, 1.93 points above Tencent-HY, while its subjective score was 1.28 points lower. With the two weighted at 30% and 70%, Tencent-HY finished first by 0.31 points.

Official leaderboard for NVVSpeech Challenge Track 2 (speech generation). We submitted MOSS-TTS to this track under the team name "Moss-Transcribe." NVV-Gen is the official baseline. VocalMind is the organizers' post-trained reference system and is not part of the official ranking

Bilingual scores on the official leaderboard (higher is better):

MOSS-TTS scored higher than the official baseline on all three, with a final score 1.46 points higher. In the human listening sub-scores (on a 1–5 scale), MOSS-TTS scored 2.773 on NVV Accuracy and 2.440 on NVV Perceptual Effect, compared with the baseline's 2.671 and 2.327.

MOSS-Transcribe and MOSS-TTS Papers Accepted to ISCSLP 2026 for Presentation in Penang

Our papers for both tracks have been accepted to ISCSLP 2026. The Track 1 paper, "Large-Scale Teacher Training and Tokenizer-Aligned On-Policy Distillation for Bilingual NVV Recognition," describes MOSS-Transcribe's teacher training and on-policy distillation. The Track 2 paper, "Annotation- and ASR-Assisted Data Construction for MOSS-TTS in NVVSpeech Track 2," describes how we built the data MOSS-TTS used in the challenge.

ISCSLP 2026 takes place November 14–17 in Penang, Malaysia. The NVVSpeech Challenge session is on November 15 and includes invited talks, flash presentations of challenge papers, the challenge summary and award ceremony, and a poster session. We will present both papers at the session and look forward to talking with fellow researchers there.

MOSS-Transcribe and MOSS-TTS approach nonverbal vocalization from opposite ends: one listens, the other speaks. We will keep extending what MOSS-Transcribe can hear beyond the words themselves, and keep making the nonverbal vocalizations MOSS-TTS produces more accurate, more natural, and more humanlike.