When Someone Interrupts Your Voice Agent, Stop More Than the Audio

The booking that kept talking after the caller changed their mind

Imagine calling a delivery service while walking toward a train station. You want to move a parcel delivery to Friday. The assistant starts explaining Thursday's available slots, and you interrupt: “Sorry, Friday, not Thursday.”

It stops speaking. You assume it understood. Then it resumes with the end of the old sentence: “...between two and four.” A moment later, it says Friday is available. You are no longer sure which day it booked.

This fictional call is a walkthrough of voice-agent coordination, not a report from a production customer. The awkwardness comes from several systems moving on different clocks: the model is generating, the player is buffering, the microphone is detecting speech, and the booking service may already be processing an action.

A text chatbot can leave a contradictory paragraph on the screen for the user to inspect. Voice disappears as it is spoken. The caller has to remember the sequence while deciding whether the assistant actually stopped.

I would reproduce the call with event timestamps before changing the voice or prompt. When was speech detected? When did playback stop? Which response ID produced the audio that resumed? Was a booking proposal created, and did any external action already happen?

Those questions turn “the assistant feels confused” into a set of observable coordination failures. The desired experience is simple: when the caller interrupts, the product should stop the old turn and make the next state understandable.

Keep three timelines separate

Track what the model generated, what the client played, and what the backend executed. Those timelines are related, but they are not identical.

Audio may be buffered on the server and the device. A generated response may be longer than the part actually played. A tool call may complete after the caller starts speaking.

Assign response identifiers and action identifiers so events can be matched to the correct turn. A late audio packet from an old response should not resume playback after the new conversation has started.

Cancel the right work

On interruption, stop or flush the relevant playback buffer and cancel ongoing generation when your transport supports it. Then reconcile conversation state to the speech that was actually delivered, using the provider's documented mechanisms.

OpenAI's Realtime API exposes interruption-related controls and conversation events. Exact responsibilities differ by transport and client implementation, so check whether your stack handles playback reconciliation automatically or expects the client to report it.

Do not treat canceling generation as canceling a booking. If an external action is already in progress, the application needs to track its outcome separately.

The mute button that passes the first test

In the first patch, the client mutes the speaker whenever the microphone detects speech. A quick test looks good: the assistant goes quiet as soon as someone interrupts.

The next test uses a slow connection. Old audio arrives after the mute is released and starts playing again. The team adds a buffer flush. That fixes playback, but the model's conversation still contains the full response, including a confirmation the caller never heard.

Now the assistant says, “As I mentioned, Thursday is confirmed.” From its context, that seems consistent. From the caller's experience, it is baffling. The system has confused generated speech with delivered speech.

I would track the response being played and the playback position that the client can actually establish. On interruption, the application needs to coordinate cancellation and conversation reconciliation using the transport's supported mechanisms. It should not invent precise listening information it does not have, but it should avoid assuming that every generated word was heard.

Meanwhile, the booking service needs its own treatment. If an action was only proposed, the proposal can be superseded. If it was submitted, the agent has to check its outcome. If it was confirmed, changing the date is another business action. None of those states can be safely inferred from whether the audio stopped.

This is why a good interruption test includes side effects. A voice assistant can sound perfectly responsive while leaving a duplicate or unwanted booking behind. The pleasant conversation is only the visible part of the workflow.

A booking needs its own state machine

In this example, the assistant should propose a Friday slot, confirm the customer's intent where required, and submit one booking operation. The operation record must survive an interrupted conversation.

If Thursday's change was already accepted, Friday may require a compensating change rather than pretending the first action never happened. Tell the caller what is known. “I am checking whether Thursday was confirmed” is better than promising Friday while two requests race.

Use stable operation IDs and destination-side duplicate protection where available. Voice interfaces make repeated requests common; people repeat themselves when they hear silence.

Silence is not always a turn boundary

A caller can pause to find an order number. Background speech can sound like an interruption. Someone may say “uh-huh” while the assistant is speaking without trying to take the turn.

Tune turn detection against realistic audio for your use case. Keep the application able to recover from false starts. If the assistant cuts off every short acknowledgment, a fast system can still feel exhausting to talk to.

I would avoid solving this with a universal silence threshold copied from a demo. Evaluate accents, noisy rooms, slow speech, and poor connections with consented or synthetic test material.

Test what the caller actually experiences

Interrupt near the beginning, middle, and end of a response. Repeat while a tool is pending. Inject delayed audio packets and disconnect the client immediately after approval.

Check time until audible playback stops, whether old audio returns, whether the next answer understands the correction, and whether the booking ledger contains the intended action exactly once.

Also test a caller who changes their mind twice. The agent should preserve a comprehensible history of decisions instead of blending all three dates into one confusing summary.

Good barge-in is not a cosmetic feature. It is coordination between playback, model state, and real-world actions. When those agree, the conversation feels natural because the system truly stopped doing the thing the caller interrupted.

Replaying the call with clearer boundaries

The caller interrupts again. The client stops the old playback and rejects late packets associated with that response. The assistant receives the correction with a conversation state that reflects what was actually delivered as closely as the transport allows.

The booking operation remains visible. If nothing was submitted, the assistant can offer Friday. If Thursday is already confirmed, it can explain that it is checking the change rather than announcing a result prematurely.

There is no need for a long technical explanation on the phone. “Got it—Friday. I’m checking the booking before I change it” gives the caller a useful picture of the state. The engineering complexity should support that plain sentence.

For evaluation, listen to complete calls rather than isolated turns. A system that handles one interruption well may struggle when the caller pauses, corrects a postcode, and changes a date in the same conversation. Test background voices and brief acknowledgments too; not every sound is a request to take over.

The best outcome is not the shortest possible response time under ideal conditions. It is a call where the person knows what the assistant heard, what it is checking, and what it actually changed.

When voice agents work well, they seem effortless. Getting there requires taking the messy timing seriously: generated words, audible words, and completed actions are three different things.

Sources & further reading

Worked scenarios are illustrative. Technical references were checked on September 10, 2026.

• OpenAI: Realtime client events ↗

Working through a similar problem?

Tell me what you are building and where it gets stuck.

Let’s talk ↗