Trpkoski, Mitko and Nedelkovski, Igor and Bocevska, Andrijana (2026) Adaptive Emotional Digital Avatar for Speech and Facial Animation in the Macedonian Language. In: 2026 61st International Scientific Conference on Information, Communication and Energy Systems and Technologies (ICEST), 01-03 July 2026, Nis, Serbia.
Full text not available from this repository.Abstract
This paper presents an interactive multimodal system for generating speech and photorealistic facial animations with explicit emotional expression in Macedonian, a low-resource language. The proposed approach integrates emotion recognition, emotion-controlled speech synthesis, and audio-driven facial animation within a fully automated pipeline. Emotion detection is performed using a dual-channel model combining Wav2Vec2-based speech analysis with a text classifier as a fallback mechanism. Speech synthesis is controlled via Azure Neural Text-to-Speech (TTS) with SSML-based prosody manipulation, while facial animation is generated using SadTalker and DECA models. The system is evaluated using objective lip synchronization metrics (LSE-C) and subjective Mean Opinion Score (MOS). Experimental results confirm that the adaptive multimodal pipeline leads to measurable improvements in naturalness and emotional expressiveness in Macedonian speech synthesis compared to a neutral baseline. In particular, emotions with slower and more stable speech patterns, such as sadness, achieve lower synchronization error, while highly dynamic emotions such as anger present greater challenges. An additional efficiency coefficient is introduced to quantify computational cost relative to generated output duration. The research confirms that the adaptive multimodal flow leads to a measurable improvement in naturalness and emotional expressiveness in Macedonian speakers compared to classical neutral systems.
| Item Type: | Conference or Workshop Item (Paper) |
|---|---|
| Subjects: | Scientific Fields (Frascati) > Natural sciences > Computer and information sciences |
| Divisions: | Faculty of Information and Communication Technologies |
| Depositing User: | Prof. d-r. Andrijana Bocevska |
| Date Deposited: | 22 Aug 2026 19:38 |
| Last Modified: | 22 Aug 2026 19:38 |
| URI: | https://eprints.uklo.edu.mk/id/eprint/11704 |
Actions (login required)
 |
View Item |