Speech Engine

Overview

ElevenLabs Speech Engine adds voice capabilities to any chat agent. ElevenLabs handles speech-to-text and text-to-speech while your server provides the LLM logic. The SDK manages connection lifecycle, turn-taking, and interruption detection so you can focus on your agent’s behavior.

How it works

Speech Engine connects your server to ElevenLabs over WebSocket. Each connection represents one conversation.

1. A user speaks in the browser. ElevenLabs captures the audio and transcribes it.
2. The transcript is sent to your server along with the full conversation history.
3. Your server passes the transcript to your LLM and streams the response back.
4. ElevenLabs converts the text to speech and plays it in the browser.

When to use Speech Engine

Speech Engine is designed for developers who want to bring their own LLM and control the conversation logic on their own server. Use it when you need to:

• Add voice to an existing text-based chat agent
• Use a specific LLM, fine-tuned model, or custom inference pipeline
• Keep full control over conversation routing, context management, and tool calling
• Integrate voice into an existing server application (Express, FastAPI, etc.)

If you want a fully hosted solution where ElevenLabs provides the LLM, knowledge base, and tools, use ElevenAgents instead.

Key features

• Any LLM - use OpenAI, Anthropic, Google Gemini, or any model that produces text. The SDK auto-extracts text from OpenAI, Anthropic, and Gemini stream formats.
• Interruption handling - when the user speaks mid-response, the SDK cancels the in-flight LLM request automatically via an AbortSignal (TypeScript) or task cancellation (Python).
• Streaming - responses are streamed to the browser as they are generated. Pass a string, an async iterable, or a native LLM stream object.
• Turn-taking - the SDK manages conversation turns, so your server only needs to respond to transcripts.

IP whitelisting

If your server is behind a firewall or uses IP-based access controls, you can whitelist the static egress IPs from which ElevenLabs WebSocket connections originate.

For additional security, you can whitelist the following static egress IPs from which ElevenLabs requests originate:

If you are using a data residency region then the following IPs will be used:

If your infrastructure requires strict IP-based access controls, adding these IPs to your firewall allowlist will ensure you only receive requests from ElevenLabs’ systems.

FAQ