Speech interfaces: assess the audio operating case
A speech interface is worth building when a named group of users must complete one task hands-free or eyes-free, and a misheard word can be caught before it changes a record. Test it on consented recordings from real users, measure whether the task succeeds rather than how clean the transcript looks, and keep a non-voice route for every task.
Submit a project Assess a workflow firstReviewed by Huy Nguyen (David), CEO · Updated 6 Oct 2026 · 9 min read
OutsourcingVN is operated by Netbase JSC, which scopes and builds conversational software, so we have an interest in your decision. The method applies whoever builds yours. This guide is for product leads, operations owners and CTOs considering voice for field work, contact centres, kiosks, clinical or warehouse notes, or accessibility.
What this guide covers
- Which task should voice handle?
- How does the audio pipeline work?
- How do you build the assessment?
- A worked example: a negation in a part number
- What has Netbase delivered near this area?
- Which questions should you ask a supplier?
- What failure modes should you watch for?
- Common questions
- How this guide is sourced
- Bring one task and some real audio
Which task should voice handle?
Write one task in plain language: logging a service visit, finding a customer record, dictating a note, filling a form field or answering a caller's question. Name who speaks, where they are, what system changes as a result and what a misunderstood word costs. Voice earns its place when hands or eyes are busy (a technician holding a part, a driver, a picker with a scanner) or when typing is a barrier for the user.
Keep three records apart: the recording, the candidate transcript and the confirmed business record. Only the confirmed record may trigger a consequential step such as ordering a part, closing a ticket or changing an appointment. Everything in the design follows from where that confirmation happens and who gives it.
How does the audio pipeline work?
A speech interface is a chain, and each link has its own failure. Voice activity detection decides when someone is speaking. Speech recognition turns audio into text, either streaming as the person talks or in one pass afterwards. Normalisation turns "four four seven one bee" into "4471-B" and spoken dates into dates. An intent step, rule-based or model-based, decides what the user wants and which values fill the task. Speech synthesis reads a response back when the user cannot look at a screen.
| Choice | Fits when | What must be proved |
|---|---|---|
| Cloud speech API, streaming | Conversational turns need fast partial results and data may leave your environment | Latency at your sites, data-retention terms, custom vocabulary support |
| Open model such as Whisper, self-hosted | Recordings must stay in your environment, or dictation is processed after capture | Accuracy on your accents and microphones, serving capacity, update ownership |
| On-device recognition | Connectivity is poor or audio must not leave the device | Accuracy after model compression, battery and heat |
| Push-to-talk with confirmation screen | Errors are costly and users can glance at a screen | Correction speed and the non-voice fallback |
| No voice interface | Noise, privacy or error cost outweigh the benefit | A recorded reason and the alternative input |
OpenAI's Whisper README describes a general-purpose model trained on a large, varied audio set and notes that performance varies by language. Treat any published accuracy figure as someone else's audio. Telephone audio, a noisy plant floor and a headset in a quiet office behave very differently, and part numbers and proper names are where general models struggle most. Custom vocabulary or phrase hints help, and they need their own test cases.
How do you build the assessment?
-
Collect consented real audio
Record target users on target devices at target sites, with background noise, accents, hesitations, corrections and overlapping speech. Agree retention with the data owner first.
-
Label the business outcome
For each recording, write the correct task result and the critical values (numbers, names, negations), not only an ideal transcript.
-
Measure task success and critical values
Word error rate hides the errors that matter; count how often the task completes correctly and how often a critical value is wrong.
-
Design confirmation and correction
Read back or show critical values before commit, and let users fix one value without repeating everything.
-
Test the non-voice route
Every core task works by keyboard, touch or assisted support, including in error states, in line with WCAG 2.2.
-
Decide with owners
The workflow owner accepts task success, the data owner accepts audio handling, and an accessibility reviewer accepts the alternatives.
(opens the full-size diagram in a new tab)
A worked example: a negation in a part number
A field technician wearing gloves dictates a repair note: "Replace valve 4471-B, not the 4417, customer declined the second visit."
Recognition returns "Replace valve 4471-B not the 4417 customer declined the second visit" with no punctuation. The intent step extracts part 4471-B, but a naive version also extracts 4417 as a second part, because it ignored "not the". The assessment's labelled case expects exactly one part and a declined follow-up, so the case fails.
The fix has two parts. The extraction rule treats "not" and "instead of" as exclusions and flags any utterance containing a negation near a part number for confirmation. The interface then reads back: "Replacing 4471-B. Follow-up visit cancelled. Confirm?" The technician says "confirm", and the confirmed record goes to the service system with the recording reference attached for the supervisor. The case joins the permanent test set under "negations near critical values". The service manager, as workflow owner, reviews flagged confirmations weekly to see whether the rule is catching real errors or slowing technicians down.
What has Netbase delivered near this area?
Recognition models are chosen per project too: Netbase works with commercial and open-source AI models, so a self-hosted open model and a cloud speech API can be compared on the same labelled audio. Netbase applies the ISO/IEC 42001 AI management system framework to its own AI delivery practice, which covers its own ways of working rather than a client's voice system.
-
In delivered work
A text chatbot with intent and conversation flow
Netbase built a WhatsApp Business AI chatbot with intent and conversation-flow handling, an LLM API (GPT-4 in the stack) and CRM synchronisation. It is a text channel; the intent and dialogue design carries over to voice, the audio layer does not.
-
In delivered work
Milestone delivery from prototype to support
The same chatbot was delivered in milestones from design and prototype to documentation, knowledge transfer and 30 days of support over 4-8 weeks.
-
Available capability
Speech recognition, confirmation design and synthesis
Building the audio pipeline, custom vocabulary and read-back confirmation is offered through a scoped assessment on your recordings.
Which questions should you ask a supplier?
-
How will you test on our audio?
Consented recordings from our users, devices and sites, labelled by outcome
-
Which values must be confirmed before commit?
A list of critical values and the read-back or screen confirmation for each
-
How do you handle names and part numbers?
Custom vocabulary or lookup against our records, with test cases
-
What happens in noise or silence?
The interface asks again, defers or routes to a person; no action is taken
-
Where is audio stored, and for how long?
Named storage, retention period, deletion method and who may listen
-
What is the non-voice route?
The same task by keyboard, touch or assisted support, tested
-
How is latency kept acceptable in conversation?
Streaming recognition, partial results and a measured turn-taking delay at our sites
-
What changes trigger retesting?
New microphones, model versions, languages or downstream systems
What failure modes should you watch for?
-
Critical value misheard
- Early signal
- Corrections cluster on numbers or names
- Who acts
- Workflow owner adds confirmation; technical owner adds vocabulary
-
Negation or cancellation missed
- Early signal
- Actions taken that the user declined
- Who acts
- Technical owner adds exclusion rules and cases
-
Noise at a new site
- Early signal
- Confirmation requests spike at one location
- Who acts
- Operations owner reviews microphones; cases added
-
Users abandon voice
- Early signal
- Non-voice route usage rises
- Who acts
- Product owner reviews task design
-
Recordings kept too long
- Early signal
- Storage audit finds expired audio
- Who acts
- Data owner enforces deletion
-
Accent or language gap
- Early signal
- Task success lower for one group
- Who acts
- Product owner extends the audio set and retests
Plan the next step for your project
Common questions
On its own it is misleading. A transcript can score well and still get the one number that matters wrong, or score poorly on filler words that change nothing. Measure task success and critical-value accuracy on labelled cases, and use word error rate only to compare models on the same audio.
Keep them only as long as a defined purpose needs, such as dispute review or improving the test set, and only with consent. Many teams keep the confirmed record and a short-lived recording reference, then delete audio on a schedule the data owner sets and audits.
Multilingual models cover many languages, but accuracy varies by language, accent and audio quality. Include every language and accent group in the labelled set and compare task success per group. If one group performs worse, add confirmation for that group or keep the non-voice route primary.
No. A voice assistant can use the same retrieval service as chat, provided answers are short enough to speak and critical details are confirmed. The retrieval guide covers source authority, and customer support automation covers routing callers to staff.
When the task changes money, safety, health or legal records, when confirmation fails twice, or when the user asks for a person. The human-in-the-loop guide covers placing those review points so they catch errors without stalling ordinary tasks.
How this guide is sourced
Statements about Netbase map to approved claims backed by attested company facts listed under Sources, in their registered wording. External guidance comes from the Whisper README and WCAG 2.2, each dated below. The technician example is illustrative and describes no client. The chatbot record above is a text channel; speech work is offered through the scoped assessment described above. This page does not replace an accessibility review with your own users. See the methodology for how claims are reviewed.
Bring one task and some real audio
A first conversation goes furthest with one task, the people who would speak it, and a few consented recordings from the real environment, including a noisy one. Submit a project with those, or read the emerging technology adoption guide, the edge deployment assessment, the multimodal document guide and the other guides first; an AI Workflow Blueprint is the assessment route when the task is still open. OutsourcingVN is operated by Netbase JSC and is Netbase's own outsourcing-services platform.
Work behind this guide
Netbase designed and built an AI chatbot on WhatsApp Business that handles intents and conversation flows and keeps lead and customer data in step with the client's CRM, delivered in milestones from prototype to knowledge transfer.
Keep ReadingRelated services and solutions
AI workflow blueprint: decide where automation belongs before you build
A paid two-to-four-week project that ends in a scope you can approve, change or stop.
Learn More