A multimodal memory recall AI assistant for dementia patients, built on a RAG pipeline with facial recognition and voice interaction so patients can recall memories through natural conversation.
MemoryBox is a multimodal memory recall AI assistant that helps dementia patients revisit family photos through natural conversation instead of navigating an app. A patient can speak to the system, and it identifies the people and context in a photo, answers questions about it, and pulls in relevant conversation history to keep the interaction grounded and personal. The goal is to give patients an accessible, low friction way to reconnect with memories without needing to operate a phone or tablet themselves.
I built the core AI pipeline end to end. Photo and conversation retrieval runs through a RAG pipeline backed by a Qdrant vector database with Qwen embeddings, so the assistant can pull the right memory into context instead of relying on the model's own recall. Voice input runs through Whisper and a speech recognition interface so the patient can talk to the assistant naturally, and responses are generated using OpenRouter function calling so the model can trigger real actions, like pulling up a specific photo, rather than only producing text. I implemented the Python facial recognition and image detection layer that identifies people in each photo, and a SQLite backed conversation history so the assistant remembers what has already been discussed across a session.