ConvoBuddy
ConvoBuddy is a research study that deployed three state-of-the-art conversational AI models as Telegram chatbots to empirically evaluate their response quality, coherence, and user perception. It provides structured comparative evidence for LLM selection in conversational AI product design.
The Problem
The conversational AI landscape in 2023 offered many model options (DialoGPT, GPT-2, LLaMA) but lacked rigorous, production-context comparative studies. Teams building conversational products had to rely on benchmark scores that didn't reflect real user interaction patterns on platforms like Telegram.
The Solution
Three Telegram bots, each backed by a different model, were deployed to real users over a structured study period. User interactions were logged with consent, and responses were evaluated across coherence, relevance, engagement, and user satisfaction metrics. Statistical analysis identified model-specific strengths and failure modes.
Key Features
Challenges & Learnings
LLaMA inference on available hardware required aggressive quantization (4-bit GGUF) which affected response quality in measurable ways. Ensuring fair comparison across models required identical prompt engineering, identical conversation history management, and controlled study conditions — challenging when models have different context window sizes.
Research Contributions
Provides empirical evidence for real-world conversational performance differences between DialoGPT, GPT-2, and LLaMA in a production Telegram context. Findings inform LLM selection criteria for conversational AI products targeting casual chat use cases.
Future Direction
Extension to include GPT-4, Claude, and Gemini variants; longitudinal engagement study (4+ weeks); and domain-specific fine-tuned variants of each model.
Tech Stack
Quick Links
Want to see more of my work?
Back to Projects