Picture a kitchen on a Tuesday morning. One person asks the speaker to add oat milk to the grocery list. Another says "actually, make it two." A kid yells something from the hallway, and the dishwasher starts up.
A transcript of that scene is a pile of correct sentences with no owners. Whose oat milk? Who changed the order? Was the shout from the hallway meant for the assistant at all?
For a decade, voice assistants have mostly dodged this question because they lived in a one-person, one-device world. You said the wake word, it listened, you got your answer. But ambient AI doesn’t work that way. It has to figure out who is speaking as the conversation happens, often without knowing ahead of time how many people are in the room or who they are, and it has to do it while dealing with interruptions, background noise, and people talking over one another.
Voice assistants have gotten very good at understanding what people say. The next challenge is understanding who is saying it. That is a much harder problem, and it is the problem that matters for voice devices. This is the problem that speaker diarization hopes to solve.
In late September, NVIDIA released a model that goes right at this problem: Nemotron 3 Diarization. Here's what it is and why we think it matters.
What is diarization?
Every conversation carries two layers of information: what was said and who said it. Speech recognition handles the first. Speaker diarization handles the second. It answers the question "who spoke when?" in an audio recording or live conversation, marking the time intervals in which each speaker is active, including the moments when people talk at the same time.
A simple example.
Without diarization:
“I’ll send the report.”
“Can you include the figures?”
“Yes, by Friday.”
With diarization:
Speaker 0: “I’ll send the report.”
Speaker 1: “Can you include the figures?”
Speaker 0: “Yes, by Friday.”
The words haven't changed. But suddenly the conversation has structure.
On its own, diarization doesn't produce words. Pair its speaker timestamps with a speech-to-text model and you get a speaker-attributed transcript, where every line has an owner. Without that, you can't reliably tell who made a commitment, who objected, or who interrupted.
That structure is useful for meeting transcripts, summaries, and search. But for an always-on voice device, it becomes something more fundamental. If an assistant is going to operate in a shared space, it needs to understand not just what was said, but which voice said it and when. And unlike a recorded meeting, it needs to make that determination while the conversation is happening.
Why Nemotron 3 stand outs
In September, NVIDIA released Nemotron 3 Diarization, an open-weight model designed specifically to identify speaker activity in real-world audio.
The model is relatively small at around 100 million parameters, which makes it small enough to be practical, but it supports up to eight speakers and can operate in both streaming and offline settings. The same model can be configured for different latency and accuracy requirements.
The model is built on NVIDIA's Sortformer architecture and uses an approach called an Arrival-Order Speaker Cache to keep speaker assignments consistent as new chunks of audio arrive. As the conversation continues, the model uses that context to avoid accidentally swapping speaker labels between chunks of audio.
That may sound like an implementation detail, but it addresses one of the hardest parts of streaming diarization. Here’s what makes it so interesting:
It handles overlap natively. Nemotron 3 is designed to handle overlapping audio - the model outputs a probability for each of up to eight speakers, every 10 milliseconds. If two people talk at once, the model can mark both speakers as active rather than forcing the audio into a single speaker channel.
It keeps speakers straight over time. Streaming diarization has a classic failure: speaker A gets one label in this chunk of audio and a different one in the next. Nemotron 3 solves this by labeling speakers in the order they first appear and carrying a small memory of earlier speakers forward. The first voice it hears stays speaker 0 for the whole conversation.
It doubles the speaker count. The previous streaming model topped out at four speakers. This one supports eight, which covers a lot more of real life: dinner tables, team standups, and family rooms.
It’s fast and accurate. Nemotron 3 ranked first among 12 systems and 17 configurations, with a 14.72% diarization error rate (DER), versus 19.3% for the next-best system. It's also much faster, processing audio roughly six times faster than its predecessor in NVIDIA's batched throughput tests.
In other words, Nemotron 3 isn't just making transcripts better after a conversation is over. It is helping give voice systems what they need to understand a conversation as it happens.
Why it matters for ambient AI
Real spaces and conversations are messy. People interrupt each other. Someone starts talking before another person has finished. Multiple people can speak at the same time. A speaker might disappear for 30 seconds and then return.
A good diarization system needs to keep track of all of that while operating quickly enough to be useful in a live conversation. For voice systems to gain context about the interaction happening around them, they need to:
Know who's talking to it. A voice assistant in a shared space hears side conversations, TV, and kids. Per-speaker activity lets an assistant tell the person who just asked a question from the person chatting in the background. Baseten points out that the same signal can drive turn-taking: spotting the primary speaker, noticing when a second voice interrupts, and flagging overlap in roughly 300 milliseconds at the lowest-latency setting.
Know when not to talk. An assistant that jumps in on every pause is worse than no assistant. Knowing whether a pause is a speaker finishing or a different person starting is the difference between helpful and annoying.
Have memory that makes sense. "Remind me about that" only works if the assistant knows who "me" is and what "that" referred to. Speaker-attributed context is what lets a note-taker, a family-organizer, or a meeting companion keep the right things attached to the right people.
That shift, from understanding speech to understanding conversations, is key to building spatial intelligence. The promise of ambient AI depends on that context.
This is where diarization like Nemotron 3 becomes interesting. Diarization doesn't make an assistant intelligent on its own, but it gives the rest of the system a much richer representation of the conversation it is trying to understand. Instead of treating a room as a single source of audio, the assistant can begin to reason about the people within it.
In other words, speaker awareness isn't a feature to bolt on later. It sits underneath nearly everything an ambient system wants to do.
It’s also why hardware matters. Good diarization starts with good audio, which is part of why we built the DevKit around a multi-microphone array designed to pick up sound across a room.
Why builders are excited
The most interesting thing about Nemotron 3 may be what happens after diarization.
Diarization by itself doesn't understand what anyone said. It produces speaker activity and timestamps. ASR turns the audio into words. Put the two together and you get a speaker-attributed transcript that downstream systems can actually reason about. For developers, this means:
Multi-party becomes a design assumption. Not every device will be one-to-one. Smart speakers in homes and assistants in shared spaces: designing for several speakers from the start, instead of patching it in later, becomes realistic.
Turn-taking gets smarter. Per-speaker activity is a much richer signal than "is there sound?" It makes it possible to respond to the right person and stay quiet during someone else's conversation.
Attribution becomes the default. Notes, summaries, and action items that say who committed to what are far more useful than anonymous ones.
We're already seeing some of these ideas appear in projects built around the model, providing a useful building block for meeting assistants, voice agents, and search and summarization.
Meet2Notes, a local-first AI meeting assistant, has made Nemotron 3 its default diarization engine, using it to identify up to eight speakers in locally processed recordings. Argmax has integrated Nemotron 3 into its Pro SDK 3 to provide real-time speaker attribution and a pre-diarized transcription workflow.
And NVIDIA itself is putting diarization into a broader voice-agent stack. Its open-source voice-agent projects combine speech recognition, diarization, language models, text-to-speech, tool calling, and orchestration into a single pipeline.
That is perhaps the clearest signal of where this technology is heading: diarization is becoming infrastructure for voice agents rather than a standalone transcription feature.
From hearing words to reading the room
For years, the question for voice AI was "does it understand the words correctly?" The next question is "does it understand the room?" Knowing who spoke when is one of the first real steps toward an answer, and having it as an open model that runs on affordable hardware means it's available on devices built for the shared spaces that people actually live and work in.
The story isn't that one new model has solved speaker diarization. It hasn't. There are still tradeoffs around latency, noise, far-field audio, and speaker identity. Nemotron's speaker labels are anonymous, rather than identifying a person by name. Applications need additional information or speaker-verification systems if they want to map those labels to specific people.
The bigger story is that an increasingly capable set of open speech models is making it possible to build more sophisticated voice systems locally that can read the room in real time. The next generation of voice interfaces won't just need to understand what was said. They'll need to understand who said it, when they said it, who else was speaking, and what was happening around them.
Nemotron 3 is another sign that the ambient future is on the way.
Join us in building the future of ambient AI
At OpenHome, we’re building a future where AI is freed from chat to become an ambient intelligence layer woven into everyday life. OpenHome provides the hardware, platform, and community that enable developers to create voice-first experiences for the physical spaces people work and live in.
Through our open-source Voice SDK, local models, sensor layer, and DevKit, OpenHome turns a home device into an intelligent, conversational interface that can listen, sense, understand, and act in the real world.
To learn more, join our Discord, read our blog, and check out our developer docs.