A talking AI character needs a rehearsal of the pauses

The interesting part happens between the lines

A character can have an excellent voice and still feel impossible to talk to. It answers before you finish. It keeps talking after you correct it. It remembers the joke but forgets that the conversation ended. For anyone making an interactive exhibit, a game companion or a playful installation, those awkward moments deserve as much design attention as the dialogue.

On October 30, NVIDIA employee Rick Grandy described MonsterHouse, a Halloween projection that combines speech recognition, a language model, speech synthesis and animation. The account describes prepared responses, cached audio and an operator interface, including a microphone fallback. It is a creator’s documented project, not evidence that an ordinary laptop can reproduce the same experience. The described workstation uses two professional GPUs.

Our interpretation is that the useful lesson lies in directing the encounter. The rest of this article proposes an original rehearsal method for a small conversational prototype. These are suggested design exercises, not tests we ran on MonsterHouse or measurements of NVIDIA’s software.

Start by imagining a visitor who has never seen your demo. They do not know when the microphone is listening, whether the voice can be interrupted or what happens when the network stalls. Your first job is to make those moments understandable. A brilliant response arriving at the wrong time can be less useful than a simple response whose timing makes sense.

Source: NVIDIA: Building a Haunted House with NVIDIA Nemotron Open Models ↗ · NVIDIA: Nemotron Voice Agent reference implementation and deployment requirements ↗ · ELEGOO: UNO R3 Most Complete Starter Kit ↗ · ELEGOO: Official starter kit tutorial ↗

Write the encounter before writing the personality

Suppose you are designing an audio guide for a fictional collection of strange rocks. A visitor can ask about one specimen, hear a short answer and leave. Before choosing whether the guide sounds scholarly or mischievous, write the encounter in ordinary verbs: invite, listen, answer, offer one follow-up, close.

Give each verb an observable condition. Invite means a greeting is available without requiring someone to guess a wake phrase. Listen means there is a clear indication that the visitor has the floor. Answer means the system speaks a bounded response. Close means the interaction ends without chasing the visitor with another unsolicited question.

Now add three interruptions to that imaginary visit. The visitor changes the subject halfway through a question. Someone nearby speaks while the visitor is thinking. The visitor says goodbye while the guide is still preparing an answer. Decide what should happen in each case before asking a model to improvise.

This exercise reveals where personality is allowed to act. A mischievous guide might make a dry comment about a rock, but it should not interpret goodbye as an invitation to argue. A character’s style can shape a sentence without deciding whether the application keeps listening.

Keep the first version deliberately narrow. One exhibit, one visitor and one question are enough to expose timing problems. Adding a camera, additional speakers and several characters creates more situations to direct before you have learned whether the simplest encounter works. Write those ambitions down for later rather than letting them quietly become requirements for the first rehearsal.

Give silence a meaning the visitor can understand

Silence can mean many different things: the system is listening, it is thinking, it missed the question or it has stopped working. If all four states look identical, the visitor has to guess. That guess often becomes another utterance, which may itself confuse the prototype.

For a rehearsal, use a small set of visible status cues. You might sketch a listening indicator, a preparing indicator and an idle indicator on paper. Their purpose is to test whether a person understands the turn, not to decorate the interface. Ask a friend to point to the moment when they think the system is ready for another question.

Do not make a loading sound pretend to be an answer. A neutral acknowledgement can buy time, but a character saying it understands something before the system has processed it creates a promise you may not be able to keep. Draft acknowledgements that describe the process rather than the conclusion.

For example, a fictional guide could say that it is checking the exhibit notes. It should not announce that the visitor has found a rare mineral before it has resolved which specimen is being discussed. The first line can be playful while remaining honest about what is happening.

Also decide how long you are willing to leave someone waiting. This is an editorial choice for your prototype, not a universal latency threshold. Choose a limit appropriate to the setting and write a fallback for reaching it. A quiet reading room and a noisy school fair may need different interaction rhythms. The relevant question is whether the visitor can recover without learning your software stack.

Make an interruption cancel the right things

Treat an interruption as more than a request to lower the speaker volume. Imagine that a response is being generated, speech has been queued and an animation has already begun. If the visitor corrects the question, which of those pending outputs should still be allowed to appear?

A useful paper exercise is to assign each turn a number. Every draft sentence, queued clip and visual cue belongs to that turn. When the visitor starts a replacement turn, cross out the previous one. Anything still associated with the crossed-out turn should need an explicit reason to continue.

This is our proposed bookkeeping method, not a claim about the internal implementation of the NVIDIA project. Its value is that it makes a confusing bug describable: the screen displayed a cue from the previous turn after the visitor had moved on. A designer and developer can discuss that behavior without debating whether the model was intelligent enough.

Rehearse several interruption types separately. A correction such as “I meant the blue stone” should update the subject. A request such as “say that again” should repeat something already approved rather than generate a new explanation by accident. “Stop” should stop the encounter according to a clear application rule.

NVIDIA’s Voice Agent repository documents support for streaming conversations and interruption handling, but its deployment profiles have different requirements. Consult the current recipe for the exact model, hardware and authentication combination. A feature in a reference implementation is a starting point for a prototype, not a guarantee that your assembled character handles every interruption correctly.

Keep physical behavior outside the character’s improvisation

If your installation includes a physical indicator, begin with a simulated one. Have the application print a small permitted state such as idle, listening or speaking. Do not begin by giving the language model free-form control over hardware commands.

For a separate beginner exercise, the ELEGOO UNO R3 Most Complete Starter Kit (affiliate link) is an optional way to explore manufacturer-taught microcontroller examples; ELEGOO provides a board kit and official lessons. A paper indicator or software-only simulation is enough for this rehearsal, and the UNO is not the computer running the voice models.

As an Amazon Associate I earn from qualifying purchases. This optional recommendation is not a claim of hands-on testing.

Keep the electronics learning task independent from the conversation experiment. Follow the exact kit tutorial and component instructions rather than asking an assistant to invent wiring. Limit any initial integration to a harmless indicator that represents an application state you have already checked. There is no need for motors, heaters, locks or mains equipment in this exercise.

Write down what the indicator is allowed to mean. A light that means speaking should follow the actual playback state, not merely the arrival of a generated sentence. Otherwise it can suggest that the character is talking when the audio device has failed. The physical cue should reflect observed application behavior.

This separation also helps troubleshooting. If the software display says speaking but the indicator does not, inspect the connection between those layers. If neither says speaking, investigate earlier in the turn. You have reduced one vague complaint about an unresponsive character to two smaller questions with evidence you can collect.

Build a tiny performance scorecard

A rehearsal scorecard should describe encounters rather than model rankings. For the fictional rock guide, record whether a visitor got a relevant answer, whether an interruption stopped old output, whether goodbye ended the encounter and whether a fallback was understandable.

Add timing observations, but define their endpoints. Time from the last audible word of the question to the first audible word of the answer is different from time spent generating text. Neither alone measures whether the visitor felt interrupted. Record the behavior alongside the duration so a fast failure does not look like a success.

Choose a handful of deliberately awkward scripts. One visitor pauses mid-sentence. Another asks the same question twice. A third changes their mind. A fourth says nothing after the greeting. These are proposed rehearsal cases, not a representative study of how all visitors behave.

Keep first-time and repeated encounters separate. A prototype might appear responsive after everything is loaded but feel broken when started fresh. If you observe that difference, write it down rather than blending both experiences into one flattering average. Likewise, record a disconnected microphone as a failed encounter even if the language model itself was ready.

Give each failure an owner. The greeting could need rewriting, the status cue could be unclear or the application could be playing a stale response. Naming the next change is more useful than giving the whole character a single score out of ten. Repeat only the relevant cases after a change, then run the complete small rehearsal again before showing the prototype to new visitors.

Design an operator role before inviting an audience

An operator is part of an interactive installation’s design. Decide what they can see, what they can stop and what they can replace. A basic rehearsal might require only a mute control, a reset control and a prepared closing line. Those controls should be obvious to the person supervising the experience.

Make the recovery procedure short enough to use under pressure. If a visitor starts sharing personal information, the operator should not need to search a configuration file to stop playback. If the character repeats itself, the operator should be able to end that encounter cleanly and explain what happens next.

For a prototype, avoid saving visitor recordings by default unless they are necessary for an agreed evaluation. Use consenting participants and explain what will be collected. A public performance involving children needs a more considered plan than an invitation to help debug a microphone. You can learn a great deal from scripted adult rehearsals before reaching that setting.

Do not make the fallback an embarrassing admission of defeat. A fictional guide can simply say it is taking a break, while a visible message tells the visitor how to continue. The important feature is that the human operator can take responsibility for the experience without competing with an autonomous monologue.

After the rehearsal, ask the operator which moment felt least controllable. That answer may identify a more valuable next improvement than adding another model capability. A character that is easy to supervise is easier to bring into an actual event.

You can test the encounter design without deploying any model. One person plays the visitor, another reads approved character lines and a third moves the paper status indicator. Use the same interruption scripts you intend to try with software.

If the human-operated version is confusing, simplify it before adding generation. Perhaps the greeting invites too many possible questions. Perhaps the character keeps offering follow-ups when the visitor wants to leave. Perhaps the indicator changes before the audience can interpret it. These are design problems you can resolve cheaply.

Then choose one part that benefits from generation. It could be adapting a brief answer to a visitor’s wording while keeping the factual content fixed. Preserve the rest of the encounter so you can see what the generative component changes. Record unexpected behavior without presenting the exercise as a scientific benchmark.

The goal is an interaction someone can enter, understand and leave. A talking house is a memorable demonstration, but the transferable skill is directing a conversation with clear turns, observable states and a human who can bring it to a graceful end.