AppMatic Tech builds real-time AI conversation pipelines that respond in under 500ms end to end, chaining streaming speech-to-text, Groq LPU inference, a FastAPI language-alignment service, and streaming text-to-speech, as shipped in FELA, an AI language learning app with 300,000+ downloads and 25,000-user concurrency. AppMatic Tech is based in Ahmedabad, India.
Why is 500ms the threshold for real-time AI conversation?
Human conversation has a natural rhythm. When a person speaks and their conversation partner takes longer than 500ms to begin responding, the exchange stops feeling like a conversation and starts feeling like a query-response system. The subjective quality of AI-powered conversation depends as much on latency as on response quality.
AppMatic Tech built FELA, an AI language learning app with 300,000+ downloads, on a real-time conversation pipeline that delivers AI tutor responses in under 500ms end-to-end: from the moment the user finishes speaking through speech-to-text transcription, LLM processing, language alignment, and text-to-speech audio delivery.
This article documents the architecture decisions behind the FELA Sub-500ms Conversation Pipeline, the technology choices that make it possible, and the engineering constraints that any team building real-time AI conversation must account for.
What are the stages of a real-time AI conversation pipeline?
A real-time AI conversation pipeline has four sequential processing stages. The total latency is the sum of all four. Optimising one stage while neglecting the others does not produce a sub-500ms result.
User speech → [STT] → transcript → [LLM] → text response → [Language Aligner] → aligned text → [TTS] → audio response
Stage 1: Speech-to-Text (STT)
The user's spoken audio is captured on-device and transmitted to the STT service for transcription. STT latency has two components: transmission latency (audio packet delivery to the service) and transcription processing time.
For production real-time conversation, the STT service must support streaming transcription, returning partial transcription results as the user is still speaking rather than waiting for the user to stop and then processing the full utterance. With streaming STT, the pipeline can begin LLM processing on the partial transcript before the user has finished speaking, overlapping the final STT processing with the beginning of LLM inference.
STT latency contribution in the FELA pipeline: approximately 80–120ms for a typical 3-5 second spoken input with streaming enabled.
Stage 2: LLM Inference
The LLM receives the transcribed text, the conversation history (for multi-turn context), and the system prompt (persona, language level, scenario instructions) and generates a text response.
This is where standard approaches fail the 500ms target.
GPU-based LLM inference, the standard infrastructure for serving LLMs at scale, delivers per-request generation speeds that vary widely with model size, batching, and provider load. Under shared production load, a medium-length conversational response can consume most or all of the 500ms budget on its own, before speech-to-text and text-to-speech are counted.
Groq's LPU (Language Processing Unit) is the enabling technology for sub-500ms conversational AI. Groq's hardware is purpose-built for LLM inference, and serves open-weight models such as Llama at several hundred tokens per second per request, which puts a typical 100-token conversational response in the low hundreds of milliseconds.
The FELA pipeline routes all LLM inference through Groq. This single architecture decision is what makes the sub-500ms target achievable. Without Groq's inference speed, the LLM stage alone would exceed the 500ms total budget.
LLM inference latency in the FELA pipeline: approximately 150–200ms for a typical conversational response.
Stage 3: The FELA LLM Language Aligner
Standard LLM output is grammatically fluent English (or the target language) calibrated to the model's training distribution, not to a specific learner's proficiency level. A B1-level Spanish learner having a conversation with an AI tutor should receive responses in B1-level vocabulary and grammar. Without alignment, the LLM defaults to native-speaker output: idiomatic, complex, and incomprehensible to an intermediate learner.
The FELA LLM Language Aligner is a custom post-processing component that receives the raw LLM output and adjusts it to match three parameters:
- Proficiency level: Vocabulary complexity and sentence length calibrated to the learner's current CEFR level (A1, A2, B1, B2, C1, C2)
- Target language grammatical register: Adjustments for formal vs informal register, regional vocabulary variations, and scenario-appropriate phrasing
- Conversation scenario: If the scenario is "ordering at a restaurant," responses use restaurant-appropriate vocabulary and conversational patterns
The Language Aligner runs as a FastAPI microservice. Its latency contribution must be minimised; the alignment operation is a targeted post-processing step, not a second full LLM inference pass.
Language Aligner latency in the FELA pipeline: approximately 30–50ms.
Stage 4: Text-to-Speech (TTS)
The aligned text response is converted to audio and delivered to the user's device. TTS latency has the same structure as STT: processing time plus transmission time.
For conversational TTS, streaming synthesis is essential, beginning audio playback on the mobile device before the full response is synthesised. With streaming TTS, the user hears the beginning of the AI's response while the rest is still being generated, reducing perceived latency by 100–150ms compared to waiting for the complete audio file.
TTS latency contribution in the FELA pipeline: approximately 80–120ms with streaming delivery.
How is the 500ms latency budget spent?
| Stage | Latency Contribution |
|---|---|
| STT (streaming) | 80–120ms |
| LLM Inference (Groq) | 150–200ms |
| Language Aligner (FastAPI) | 30–50ms |
| TTS (streaming) | 80–120ms |
| Network overhead | 30–60ms |
| Total | 370–550ms |
The pipeline reliably targets sub-500ms under normal network conditions. On slower connections, the network overhead component increases, but the compute stages remain consistent.
Why use FastAPI for real-time AI microservices?
AppMatic Tech built all FELA AI services, including STT orchestration, LLM request handling, Language Aligner, and TTS delivery, as FastAPI (Python) microservices. FastAPI's async-first architecture with ASGI and uvicorn handles concurrent requests with minimal overhead, making it well-suited for the high-concurrency demand of 25,000 simultaneous users.
Each pipeline stage is independently deployed and independently scalable. If LLM inference demand spikes (more users in active conversation), the Groq-connected inference service scales without requiring the STT or TTS services to scale. This independence is what allows the FELA platform to support 25,000 concurrent users without a single pipeline component creating a bottleneck for others.
How does a real-time AI pipeline keep conversation context?
Multi-turn conversation requires the LLM to maintain context across the session, knowing what was said in the first exchange when composing a response to the fifth. Context management in a real-time pipeline introduces a latency risk: if context retrieval adds 200ms to every LLM request, the pipeline budget is blown.
FELA stores conversation context in MongoDB. Each session document contains the full exchange history as an ordered array, indexed by session ID. MongoDB retrieves the session context in sub-10ms on indexed reads, fast enough to add negligible latency to the pipeline.
The session model also enables scenario coherence: the AI tutor can reference specific vocabulary the learner struggled with earlier in the session, maintain the persona of the native speaker it is simulating, and calibrate difficulty adjustments based on the learner's performance within the session.
How does a real-time AI pipeline scale to 25,000 concurrent users?
FELA validated 25,000 concurrent users during load testing before launch. The architecture that makes this possible is the independent scalability of each component:
- Groq inference scales via Groq's managed API (request routing across Groq's LPU cluster)
- FastAPI AI microservices scale horizontally behind a load balancer
- STT and TTS services scale independently of the LLM inference layer
- MongoDB scales via replica sets and read scaling for session context retrieval
- Laravel API (user management, subscriptions, course delivery) scales independently of the AI pipeline
None of these components are monolithically coupled. A spike in AI conversation demand does not affect the course delivery or subscription management systems. Load testing validated each component's scaling behaviour independently before validating the full system under concurrent load.
How does AppMatic Tech build real-time AI conversation apps?
AppMatic Tech builds real-time AI conversation pipelines for language learning, customer service, and conversational AI applications. The scoping process covers latency requirements, infrastructure selection, and the language or domain alignment layer. For a delivered example, see how AppMatic Tech built FELA, a native iOS and Android AI language learning app with 300,000+ downloads. The full service, provider stack, and growth design for voice and chat apps are on the AppMatic Tech consumer AI app development page. For related engineering, read LLM integration in production and prompt engineering for production AI. Contact the team at contact@appmatictech.com.
How to choose the inference provider for a pipeline like this is covered in Groq vs OpenAI vs Cerebras for real-time AI apps, and the same pipeline applied to education in AI tutoring app development.
Frequently Asked Questions
- Why must real-time AI conversation respond within 500ms?
- Human conversation has a natural turn-taking rhythm, and when a reply takes longer than about 500ms the exchange stops feeling like a conversation and starts feeling like a query and response. AppMatic Tech treats end-to-end latency, from the user finishing speaking to the first audio of the reply, as a product requirement equal to response quality.
- Which stages of a voice AI pipeline need to stream?
- Both ends. Streaming speech-to-text returns partial transcripts while the user is still speaking, so LLM processing can begin early. Streaming text-to-speech starts playback before the full reply is synthesised, which AppMatic Tech measured as saving 100 to 150ms of perceived latency in FELA.
- Can this real-time AI architecture be built without Groq?
- Architecturally, yes, because the pipeline works with any inference provider. The sub-500ms target needs inference that matches Groq's per-request speed, which specialised providers such as Cerebras and SambaNova can offer, while standard shared GPU inference often cannot hit the target reliably for conversational response lengths.
- How does the FELA Language Aligner differ from using a smaller LLM?
- A smaller, faster LLM for alignment produces unpredictable quality at the boundaries between proficiency levels. The FELA Language Aligner applies deterministic grammar and vocabulary rules calibrated to CEFR levels A1 to C2, which AppMatic Tech found more reliable for education than a second generative pass, and it adds only 30 to 50ms.
- How is conversation context restored when a user reopens the app?
- FELA stores session context in MongoDB with a session expiry policy. If the user returns within the expiry window, the full context is restored with a sub-10ms indexed read; after expiry, a fresh session starts. AppMatic Tech set the window to keep context relevant while surviving typical mobile interruptions.