All projects
Multimodal AIActive
Sentinelle
A multimodal voice agent that estimates cognitive load in a clinical context.
Personal project2026
- Multimodal AI
- RAG
- CNN-LSTM
- Whisper
- TTS
- MLOps
Context
Sentinelle explores an applied research question: can we estimate, continuously and without an intrusive sensor, a practitioner’s cognitive load during a clinical task?
Problem
Existing cognitive-load indicators usually need dedicated sensors that don’t fit everyday clinical use. The system needed an estimate usable with lighter multimodal signals, and a natural voice interaction that stays in the clinician’s workflow.
Architecture
- Cognitive-load estimation: a CNN-LSTM model (MobileNetV2 + LSTM) trained on multimodal signals from several reference datasets — AVCAffe, RAVDESS, PoseTrack, BIWI.
- RAG agent: Mistral / Ollama + LangChain + ChromaDB, to ground the copilot’s answers in reference clinical documentation.
- Speech-to-speech voice pipeline: speech recognition (Whisper), processing, then speech synthesis (Coqui TTS), for a copilot that stays in the conversation.
- FastAPI backend and React dashboard for monitoring, with an MLOps layer for the model lifecycle.
Stack
Python for signal processing and the CNN-LSTM models (MobileNetV2 + LSTM), Whisper for speech recognition, Coqui TTS for synthesis, Mistral/Ollama + LangChain + ChromaDB for RAG, FastAPI + React for the backend and dashboard.
Results
- A working end-to-end speech-to-speech pipeline in a test environment.
- A CNN-LSTM model trained and validated across four reference multimodal datasets.
Screenshots
Screenshots coming soon.