This article is a technical investigation of skipping ASR transcription to understand and act directly on audio vectors. Its conclusions were adversarially verified across multiple sources (the TMLR 2025 survey of spoken language models, several ICASSP/EMNLP/Interspeech papers and their official repositories, OpenAI's official announcements and developer docs, among others). Every claim is labeled [Fact] / [Research] / [Judgment]; vendor self-reported numbers and unreviewed preprints are flagged as such — discount accordingly. This is a snapshot of a fast-moving field (mid-2026), not a technology-selection verdict.
The one-sentence conclusion up front: on closed command sets, this approach is mature today and simply better; for open-ended tasks it has been productized, but you accept the cost of "roughly one in three complex calls failing, with no reliable transcript" — and "bypassing text" may itself be an illusion.
1. What's Possible: Two Tiers to Keep Separate
Closed command sets (fixed intents): mature — and winning
[Research, corroborated across sources] "Audio in, intent out, no transcript ever produced" is not just feasible in closed domains — it can beat the cascade:
- Fluent Speech Commands (31 intents, smart-home domain): a fully ASR-free end-to-end model reaches a SOTA error rate of 1.15%, edging out the 1.20% of a model with ASR pretraining;
- Skit-S2I (14 intents, banking domain): end-to-end beats the cascade across the board (0.946–0.956 vs 0.923–0.933), with one sixth to one twenty-seventh the parameter count (19.8M–87M vs 522M–593M);
- wav2vec 2.0 vectors fed straight into simple classifiers beat the SOTA ASR-transcription path on three tasks with no more than 100 hours of labeled data each (including cardiac-arrest detection on real 911 calls).
But this is also where the hard boundary sits: hand an ASR-free model phrasing it never saw in training and the error rate jumps from 1% to 15%–57% — the paper itself concedes this is an "inherent limitation of end-to-end ASR-free spoken language understanding." It wins on fixed phrasings, dies on rephrasings.
Open-ended tasks: productized, at a price
[Fact] The OpenAI Realtime API went GA on 2025-08-28 (gpt-realtime): officially positioned as a single model that consumes audio and produces audio directly (rather than an STT→LLM→TTS three-stage chain), with native in-session support for function calling, remote MCP, SIP telephony, interruptions, and agent handoff; enterprise deployments at Zillow, T-Mobile, and StubHub are cited in the announcement. This is the shipped form of "speech executing tasks directly."
⚠️ But "bypassing text" may be an illusion: implicit transcription
[Research, 2026-02 preprint, medium confidence] This is the single most important finding here. Mechanistic analysis (logit lens) of two open-source speech LLMs, Ultravox and Qwen2-Audio, shows that a literal transcript can be read directly out of the LLM's hidden layers; and when concept erasure (LEACE) is used to delete the text-prediction directions from those hidden layers, both models' accuracy collapses to 0.0%–0.3% across all tasks, while erasing the same number of random dimensions has almost no effect (−1.5% to −4.0%). Conclusion: so-called "audio vectors straight into the LLM" still passes through text-like representations on the causal path — it merely hides the ASR inside the hidden layers; it does not genuinely bypass text.
That work is a single-author, unreviewed preprint, and its mechanistic analysis covers only two open-source architectures; closed models such as GPT-4o Realtime were not tested — for them, "implicit transcription" is a reasonable suspicion, not an established fact. Two independent concurrent works point in the same direction.
[Fact, against-interest evidence] One piece of corroboration: OpenAI's own docs concede that the "input transcript" the speech-to-speech path gives you is generated by an asynchronous side-channel ASR and "should be treated as a reference only, not what the model actually heard."
2. What's Already in Use
| Form | Status | Notes |
|---|---|---|
| Wake words / command words (KWS) | [Fact] Shipped at scale for years | At its core, this is transcript-free "audio → action" execution |
| Open-source audio→intent | [Fact] Public since 2019 | Official Lugosch et al. repo: decode_intents(waveform) decodes the structured intent {activate, lights, kitchen} straight from the waveform — zero transcription at inference. But training still relies on text forced alignment over 960 hours of LibriSpeech — "no text" holds only at runtime; the training pipeline remains steeped in text supervision (this is the general pattern) |
| Realtime native speech + function calling | [Fact] GA, in enterprise use | See above; tool execution is triggered directly within the voice session |
| Text-free speech generation / direct speech translation | [Research] | wav2vec vectors plus shallow probes fail badly on open-ended generation (English→German BLEU 6.1 vs 15.5 for ground-truth text); yet the same year, fully fine-tuned end-to-end speech translation was already competitive with cascades — classification can win; generation only works with brute-force scale |
3. Three Motivations for "Bypassing Text" — the Evidence Is Uneven
- No error cascading (ASR wrong → NLU necessarily wrong) — [hard data, but bounded]. On the SLURP benchmark, a cascade running at 16% word error rate loses only about 5% intent accuracy versus gold transcripts (entity tasks drop more: SLU-F1 78.19→70.84). The evidence in the opposite direction is actually more solid: on purely out-of-vocabulary (OOV) named entities, the direct-vector path errs 23% versus 36% for the transcription path — the robustness advantage of skipping transcription on rare entities has hard numbers behind it.
- Preserving paralinguistic information (tone / emotion / speaker / prosody) — [holds up]. This is currently end-to-end's only consistent win (TMLR 2025 survey: cascades typically win on semantic tasks and lag on speaker and paralinguistic tasks).
- Lower latency — [unverified]. In all the evidence that survived scrutiny, the latency advantage exists only as assertions in vendor announcements and papers; not a single public controlled measurement made it through. Do not treat this one as settled.
4. Drawbacks and Risks (the Important Part)
- On semantic tasks, cascades still generally win — and the gap widens under noise. [Research, multi-source] The TMLR 2025 survey aggregates six benchmarks; VoiceBench independently corroborates: a naive ASR→LLM pipeline beats every end-to-end voice assistant by more than 10 points on spoken instructions. Under added noise (0 dB babble noise): the Whisper cascade drops only 0.5%–4.2% while four end-to-end speech LLMs drop 3.9%–12.7% — enough to flip a clean-condition end-to-end advantage by up to 7.6 percentage points.
- Roughly one in three complex function calls fails. [Vendor self-reported] gpt-realtime scores 66.5% on the audio version of ComplexFuncBench (up from 49.7% for the 2024-12 predecessor) — note this is OpenAI's internal, private eval. Independent academic corroboration: tool calling through the audio entry point runs systematically about 4.8 percentage points below the text entry point.
- Auditability is a structural weakness both sides acknowledge. [Fact, conceded by both camps] The very papers advocating transcription-free approaches state outright that "automatic transcription provides an important interpretability layer for modern speech applications"; more telling still, OpenAI's official docs — from the company selling Realtime — recommend chained architectures for customer-service and approval-style workflows, precisely because of "persistent transcripts plus deterministic logic between stages." A vendor recommending against its own new product is against-interest evidence — the most credible kind.
- Paired-data scarcity is a hard constraint. [Research] Speech→intent corpora are orders of magnitude smaller than text intent data (Skit-S2I: just 13.8 hours from 11 speakers; SLURP: 58 hours), and dialect and accent coverage is inherently thin — as of late 2022, a spoken language understanding dataset for Indian accents did not even exist.
- Deterministic gates are hard to insert. A text path can be regex/schema validated and reviewed before anything is released; on the audio-vector path, guardrails can only fall back to the tool-call argument layer (fortunately, that layer is still structured JSON).
5. Pragmatic Conclusions
[Judgment, derived from the verified findings above]
- Closed command sets (wake words / in-car commands / smart home / fixed IVR menus): pure end-to-end is the right choice today — more accurate, fewer parameters, and no transcript needed by construction.
- Open-ended voice agents: for high-reliability scenarios that need compliance auditing, choose the cascade (even OpenAI recommends this); for low-latency natural conversation where the business can tolerate errors, choose native Realtime — but do three things without fail: (1) turn on asynchronous transcription for audit logs (keep it even knowing it is only a reference); (2) funnel all critical execution through server-side schema-validated function calls (tool arguments are structured JSON, so text-layer guardrails still hold at that layer); (3) add a confirmation turn for high-risk actions.
- Over the next two to three years: native audio models will keep chipping away at the cascade on latency and paralinguistics; but the demand for auditing and deterministic gates will keep the hybrid form — audio in, structured text out — around for a long time. "Fully text-free" is more likely to stay confined to closed command sets and research papers.
Primary sources: the TMLR 2025 survey "On The Landscape of Spoken Language Models", arXiv 2602.17598 (the cascade-equivalence hypothesis, preprint), Borgholt et al. ICASSP 2022, Lugosch et al. 2019 and its official repository, SLURP (EMNLP 2020), Skit-S2I, VoiceBench, plus the OpenAI gpt-realtime announcement and the official voice agents docs. The research ran through 104 retrieval and verification agents, with three-vote adversarial verification for every conclusion; the [Fact] / [Research] / [Judgment] tiers, vendor numbers, and preprint status are labeled item by item.