Multimodal audio-visual information fusion using canonical-correlated Graph Neural Network for energy-efficient speech enhancement

Registro completo de metadados
MetadadosDescriçãoIdioma
Autor(es): dc.contributorUniversity of Wolverhampton-
Autor(es): dc.contributorUniversidade Estadual Paulista (UNESP)-
Autor(es): dc.contributorBasque Research & Technology Alliance (BRTA)-
Autor(es): dc.contributorUniversity of the Basque Country (UPV/EHU)-
Autor(es): dc.contributorEdinburgh Napier University-
Autor(es): dc.contributorDeepCI-
Autor(es): dc.creatorPassos, Leandro A.-
Autor(es): dc.creatorPapa, João Paulo-
Autor(es): dc.creatorDel Ser, Javier-
Autor(es): dc.creatorHussain, Amir-
Autor(es): dc.creatorAdeel, Ahsan-
Data de aceite: dc.date.accessioned2025-08-21T16:45:25Z-
Data de disponibilização: dc.date.available2025-08-21T16:45:25Z-
Data de envio: dc.date.issued2023-07-29-
Data de envio: dc.date.issued2023-07-29-
Data de envio: dc.date.issued2023-01-31-
Fonte completa do material: dc.identifierhttp://dx.doi.org/10.1016/j.inffus.2022.09.006-
Fonte completa do material: dc.identifierhttp://hdl.handle.net/11449/247622-
Fonte: dc.identifier.urihttp://educapes.capes.gov.br/handle/11449/247622-
Descrição: dc.descriptionThis paper proposes a novel multimodal self-supervised architecture for energy-efficient audio-visual (AV) speech enhancement that integrates Graph Neural Networks with canonical correlation analysis (CCA-GNN). The proposed approach lays its foundations on a state-of-the-art CCA-GNN that learns representative embeddings by maximizing the correlation between pairs of augmented views of the same input while decorrelating disconnected features. The key idea of the conventional CCA-GNN involves discarding augmentation-variant information and preserving augmentation-invariant information while preventing capturing of redundant information. Our proposed AV CCA-GNN model deals with multimodal representation learning context. Specifically, our model improves contextual AV speech processing by maximizing canonical correlation from augmented views of the same channel and canonical correlation from audio and visual embeddings. In addition, it proposes a positional node encoding that considers a prior-frame sequence distance instead of a feature-space representation when computing the node's nearest neighbors, introducing temporal information in the embeddings through the neighborhood's connectivity. Experiments conducted on the benchmark ChiME3 dataset show that our proposed prior frame-based AV CCA-GNN ensures a better feature learning in the temporal context, leading to more energy-efficient speech reconstruction than state-of-the-art CCA-GNN and multilayer perceptron.-
Descrição: dc.descriptionMinisterio de Ciencia e Innovación-
Descrição: dc.descriptionEusko Jaurlaritza-
Descrição: dc.descriptionEngineering and Physical Sciences Research Council-
Descrição: dc.descriptionCMI Lab School of Engineering and Informatics University of Wolverhampton, England-
Descrição: dc.descriptionDepartment of Computing São Paulo State University, Bauru-
Descrição: dc.descriptionTECNALIA Basque Research & Technology Alliance (BRTA), Bizkaia-
Descrição: dc.descriptionUniversity of the Basque Country (UPV/EHU), Bizkaia-
Descrição: dc.descriptionSchool of Computing Edinburgh Napier University, Scotland-
Descrição: dc.descriptionDeepCI, Scotland-
Descrição: dc.descriptionDepartment of Computing São Paulo State University, Bauru-
Descrição: dc.descriptionEngineering and Physical Sciences Research Council: EP/T021063/1-
Formato: dc.format1-11-
Idioma: dc.languageen-
Relação: dc.relationInformation Fusion-
???dc.source???: dc.sourceScopus-
Palavras-chave: dc.subjectCanonical correlation analysis-
Palavras-chave: dc.subjectGraph Neural Networks-
Palavras-chave: dc.subjectMultimodal learning-
Palavras-chave: dc.subjectPositional encoding-
Palavras-chave: dc.subjectPrior frames neighborhood-
Título: dc.titleMultimodal audio-visual information fusion using canonical-correlated Graph Neural Network for energy-efficient speech enhancement-
Tipo de arquivo: dc.typelivro digital-
Aparece nas coleções:Repositório Institucional - Unesp

Não existem arquivos associados a este item.