Skip to content
All game projects
Prototype Voice-Controlled Racing Unity · C#

Voice Racer

A racing prototype exploring microphone-driven interaction, responsive HUD feedback and arcade-style gameplay mechanics.

Voice Racer project overview

Overview

Voice Racer tests how spoken input can become an immediate game control. The prototype pairs microphone interaction with a dense but readable racing HUD, configurable voice skills and rapid arcade feedback.

Development areas

  • ▹Voice and microphone input
  • ▹Racing gameplay loop
  • ▹Responsive HUD feedback
  • ▹Configurable voice-triggered skills

Gameplay showcase

See the prototype in motion

Work in progress

Captured from the current Unity build: live speech-to-text in the listening bar, item effects (boost, bomb, turbo) and the HUD reacting in real time.

Interaction hypothesis

Can a spoken phrase be as fast and fair as a button?

Every item in Voice Racer is fired by reading a phrase out loud mid-race. The prototype tests whether speech can feel immediate and fair: tolerant of accents and mis-hearings, but not so loose that background chatter fires items by accident.

Input pipeline

How a voice command fires

  1. 01

    Listening

    Mic level sampled every frame (RMS → dBFS)

  2. 02

    Utterance

    Voice-activity detection splits speech into short segments

  3. 03

    Recognition

    On-device streaming model (Sherpa-ONNX) or Whisper; Cantonese, Mandarin and English

  4. 04

    Phrase buffer

    Recent segments are joined so a paused sentence still counts; cleared after 10 s of silence

  5. 05

    Match

    Character + Jyutping fuzzy match against the active phrase set

    Matched — ≥ 78% similar and loud enough: the item fires, then cools down (e.g. 20 s speed boost, 25 s turbo)
    Not matched — The HUD shows what was heard and the similarity score; keep listening

Back to listening; the buffer resets whenever an item fires

Engineering notes

Making speech usable as a game control

The concerns that shaped the input layer:

  • Mis-hearings: matching works per character and on Cantonese Jyutping romanisation, so a near-homophone (密 for 麥) still counts while a genuinely different phrase does not.
  • False positives: a loudness gate (RMS ≥ 0.02, about −34 dBFS) stops a mumbled phrase that happens to transcribe correctly from firing an item.
  • Pauses mid-sentence: voice-activity detection splits speech into fragments, so the matcher scores an accumulating buffer instead of only the latest fragment.
  • Tuning: a live debug panel shows the heard text, per-phrase similarity and mic dB, to calibrate thresholds against real voices and microphones.
  • Accessibility and fallback: every item also has a keyboard key (1–5), and players can rewrite the phrases in per-language profiles.

Starting grid

Characters & karts

Kai, Boost Ace
Kai's kart

Boost Ace

Kai

“Leave the straight to me!”

Mimi, Item Star
Mimi's kart

Item Star

Mimi

“Cute voice. Serious chaos.”

Yuna, Drift Pro
Yuna's kart

Drift Pro

Yuna

“Every corner has a rhythm.”

Sol, Tactic Tech
Sol's kart

Tactic Tech

Sol

“Calculated. Then accelerated.”

What I built

  • Mapped microphone input to discrete racing actions and skills
  • Designed visual feedback for recognised, missed and cooling-down commands
  • Built an arcade HUD around speed, position and voice abilities
  • Tested the control idea as a short, repeatable race loop

Key challenges

  • Handling background noise and different speaking volumes
  • Giving instant feedback when a command is not recognised
  • Keeping voice input useful without removing conventional steering skill

Next steps

  • Add calibration and accessibility presets
  • Test alternative command vocabularies
  • Measure input latency across desktop microphones

Project gallery

Sol, the main racer, in his purple smart-cube kart (3D character render)Sunshine Metro track concept: harbour skyline, sky bridges and wide bendsConcept gameplay screen: collecting a voice item in the city courseConcept gameplay screen: boost activated through a canyon course

Interested in games, interactive technology or indie development?

Get in touch