Audio AI: Transcription, Diarization, and Analysis

Why Audio AI Matters Now

Most enterprise audio is dark data. Calls, meetings, interviews, and earnings calls are recorded but rarely analyzed at scale because listening is linear: one hour of audio takes one hour to review. Text is random-access: a one-hour call becomes a 2,000-word transcript you can search, summarize, and feed into an LLM in milliseconds.

The pipeline has matured enough that transcription accuracy on clean English speech is now comparable to professional human transcribers. The remaining engineering challenge is not accuracy on ideal audio but robustness on real-world audio: accented speech, cross-talk, low-bitrate recordings, and domain-specific vocabulary like medical terminology or financial jargon.

How Whisper-Class ASR Works

ASR stands for Automatic Speech Recognition. Whisper (released by OpenAI, arXiv:2212.04356) is an ASR model trained on a large corpus of multilingual web audio paired with transcripts. It set a new benchmark for zero-shot multilingual transcription accuracy and became the dominant reference architecture in this class.

Step 1: Raw audio to mel spectrogram. Audio is a waveform: amplitude over time. To give a neural network something it can process, the waveform is converted into a mel spectrogram, a two-dimensional representation of frequency content over time. The horizontal axis is time, the vertical axis is frequency (compressed to the mel scale, which approximates human hearing), and pixel brightness represents energy at that frequency at that moment. Think of it as a visual fingerprint of sound.

Step 2: The encoder. The mel spectrogram is fed into a transformer encoder, which learns to represent the acoustic content of each time segment as a high-dimensional vector. The encoder captures which sounds are present and in what sequence, without yet producing words.

Step 3: The decoder. A transformer decoder takes the encoder's representations and generates a text sequence, one token at a time. It is conditioned on the language token (which language to transcribe in) and can optionally translate directly into English even when the source audio is in another language.

Training data. Whisper was trained on audio scraped from the web paired with corresponding text. The breadth of that data, covering many accents, languages, recording conditions, and domains, is why it generalizes better than models trained only on curated studio recordings. Domain-specific vocabulary remains a weak point and is typically addressed through fine-tuning on in-domain audio or post-processing with a language model that knows the vocabulary.

Speaker Diarization: Who Said What

Transcription answers "what was said." Diarization answers "who said it." They are distinct problems solved by different models, then combined in the final output.

The diarization problem. Given an audio recording with multiple speakers, segment it into contiguous time intervals and assign each interval a speaker label (Speaker A, Speaker B, and so on). The model does not know in advance how many speakers there are or who they are. It discovers speaker identity from the audio alone.

How it works. Diarization systems extract speaker embeddings, compact numerical representations of a speaker's vocal characteristics, from short audio segments, typically half a second to two seconds long. These embeddings are sometimes called d-vectors or x-vectors depending on the architecture. The system then clusters these embeddings: segments that sound like the same person cluster together. The cluster boundaries define when a speaker change occurred.

Combining transcription and diarization. After running ASR and diarization separately, the two outputs are aligned by timestamp. Each word from the ASR output is assigned to whichever speaker the diarization model assigned to that moment in the audio. The result is a speaker-labeled transcript: Speaker A [00:01:14]: "I think we should move the launch to Q3."

Where diarization fails. Cross-talk, where two people speak simultaneously, is the hardest case. Diarization systems typically assign overlapping speech to one speaker or the other rather than both. Very short speaker turns, under a second, are also unreliable. In enterprise deployments, human review of the diarization output is common for high-stakes use cases like legal depositions or medical consultations.

Real-time transcription uses streaming ASR: audio is processed in short windows (often 100-500ms) and partial results are emitted continuously, with corrections as more context arrives. Batch transcription processes the complete audio file after the event ends, which allows the model to use full bidirectional context and typically produces higher accuracy. For most enterprise analytics use cases, batch is sufficient and significantly cheaper to operate.

Post-Transcript LLM Pipeline

The transcript itself is a lightly structured text document. An LLM turns it into structured intelligence: a summary, a list of action items, key decisions made, sentiment per speaker, open questions, and follow-up owners. The prompt design for meeting analysis follows consistent patterns.

Pattern 1: Role-framed extraction. Tell the LLM its role and the input type before presenting the transcript. "You are analyzing a sales call transcript. Extract all action items with their owner and deadline. Format as JSON."

Pattern 2: Per-speaker analysis. Because the transcript includes speaker labels, you can request sentiment or engagement signals per person. "For each speaker, characterize their overall sentiment as positive, neutral, or negative and list two to three phrases that support your assessment."

Pattern 3: Structured output with a schema. Specifying a JSON schema in the prompt reduces parsing errors and makes downstream pipeline steps simpler. Define the fields: summary (string), action_items (array of {owner, task, deadline}), decisions (array), open_questions (array).

What to watch for. LLMs can hallucinate decisions or action items that were implied but not explicitly stated. The prompt should ask the model to only extract items that were explicitly mentioned, not inferred. A human review step for high-stakes decisions (budget commitments, staffing changes) is a reasonable control.

Latency Tradeoffs in Enterprise Deployments

The choice between streaming and batch transcription is an architectural decision, not a product preference. The right answer depends on where the output is consumed and when.

Real-time streaming is appropriate when the human or system that consumes the transcript needs it during the conversation. Contact center agent assist, live captions for accessibility, and real-time compliance flagging are genuine real-time use cases. Streaming requires a persistent audio connection to the ASR service and substantially higher infrastructure cost per hour of audio.

Batch processing covers the majority of enterprise audio analytics use cases. A call recorded at 2pm does not need its transcript until the manager reviews it the next morning. Running batch jobs on completed recordings is cheaper, more accurate, and easier to scale. Most organizations start with batch and add real-time capability only for specific workflows that require it.

A practical split. Run real-time transcription for agent-facing use cases where the agent needs live guidance. Run batch transcription for all QA, coaching, analytics, and compliance review workflows. This split reduces infrastructure cost substantially while covering both use cases.