AI Emotional Intelligence Assistant
A production-grade real-time emotionally intelligent virtual assistant that delivers human-like voice interaction with synchronized facial expressions and avatar animations. Implements a full speech-to-speech pipeline enhanced with emotional reasoning — the assistant doesn't just answer, it empathizes.
The Problem
Current AI voice assistants (Siri, Alexa, ChatGPT Voice) deliver factually accurate but emotionally flat responses. In mental health, eldercare, and customer support contexts, emotionally congruent responses significantly improve user outcomes and trust — yet no production system delivered this at scale.
The Solution
A complete speech-to-speech pipeline: Whisper ASR captures user voice → emotional context extracted from speech tone and content → LLM generates an emotionally-calibrated response → neural TTS renders it with prosodic emotional markers → Rhubarb synchronizes avatar lip movement. WebSocket streaming enables sub-2s end-to-end latency.
Key Features
Challenges & Learnings
Extracting emotional context reliably from short utterances (< 5 words) was unreliable with general-purpose sentiment models. A specialized emotion-tagging prompt layer was added between ASR output and the LLM to force explicit emotional state labeling before response generation. Avatar expression blending required custom interpolation to prevent jarring transitions between emotional states.
Business Impact
Directly applicable to mental health companion apps, eldercare platforms, and enterprise customer support systems where emotional congruence increases user trust and retention. Architecture is modular enough to swap ASR, TTS, and avatar components for domain-specific optimizations.
Future Direction
Emotion-adaptive response length (shorter when user is distressed), multimodal emotion detection from video (facial expression + voice + text fusion), and clinical trial deployment in a mental health support context.
Tech Stack
Quick Links
Want to see more of my work?
Back to Projects