Bi-Attention Enhanced Representation Learning for Image-Text Matching (2023)

Pattern Recognition, Vol. 139, 2023, Item 109367

We present a 3D facial reconstruction system aimed at restoring babies' 3D facial geometry from uncalibrated photos, BabyNet. Because the 3D facial geometry of babies differs significantly from that of adults, baby-specific facial reconstruction systems are needed. BabyNet consists of two stages: 1) a 3D convolving graph autoencoder learns a latent space of the baby's 3D face shape; and 2) a 2D encoder that maps photographs to 3D latent space based on representative features extracted using transfer learning. This allows us to recreate a 3D face from 2D images using the pre-trained 3D decoder. We evaluate BabyNet and show that 1) methods based on adult datasets fail to model babies' 3D facial geometry, demonstrating the need for a baby-specific method, and 2) BabyNet outperforms classic model-fitting methods, even if it is one baby-specific 3D method is morphable model, like BabyFM, is used.

Pattern Recognition, Volume 136, 2023, Item 109259

Multimodal sentiment analysis (MSA), which aims to detect the mood expressed by speakers in videos using textual, visual, and auditory cues, has attracted significant research attention in recent years. However, textual, visual, and auditory modalities often contribute differently to mood analysis. In general, text contains more intuitive emotional information and outperforms non-linguistic modalities in MSA. Finding a strategy to exploit this property to obtain a fusion representation that contains more mood-related information while preserving relationships between and within the modality becomes a major challenge. To this end, we propose a novel method called Text Enhanced Transformer Fusion Network (TETFN) that learns text-oriented pairwise cross-modal mappings to obtain effective unified multimodal representations. In particular, textual information is included in learning emotional nonverbal representations through text-based multi-head attention. In addition to maintaining consistency information through cross-modal mapping, the differentiated information between modalities is also preserved through unimodal label prediction. In addition, the pre-trained vision model Vision-Transformer is used to extract visual features from the original videos to preserve both global and local information of a human face. Extensive experiments with the benchmark datasets CMU-MOSI and CMU-MOSEI demonstrate the superior performance of the proposed TETFN compared to state-of-the-art methods.

Information Processing & Management, Volume 58, Issue 1, 2021, Item 102432

The image-text retrieval task has been a popular research topic and is attracting growing interest as it bridges the computer vision and natural language processing communities and involves two distinct modalities. Although many methods have made great strides in the image-text task, it remains a challenge as it is difficult to learn the correspondence between two heterogeneous modalities. In this article, we propose a multi-level representation learning for the image-text retrieval task that uses semantic, structural, and contextual information to improve the quality of the visual and textual representation. To leverage information at the semantic level, we first extract the high-frequency nouns, adjectives, and numbers as semantic labels, and adopt a convolutional neural multi-label framework to encode the information at the semantic level. To examine the structural-level information of image-text pairs, we first construct two graphs to encode the visual and textual information in terms of the corresponding modality, and then apply triplet-loss graph matching to resolve the discrepancy between the reduce modalities. To further improve the retrieval results, we use the context-level information from two modalities to refine the ranking and improve the retrieval quality. Extensive experiments with Flickr30k and MSCOCO, two commonly used datasets for image-text retrieval, have demonstrated the superiority of our proposed method.

Knowledge-Based Systems, Volume 226, 2021, Article 107138

How to eliminate the semantic gap between multimodal data and effectively fuse multimodal data is the key problem of cross-modal retrieval. The abstractness of the semantics makes the semantic representation one-sided. To obtain complementary semantic information for samples with the same semantics, we construct a local graph for each instance and use a Graph Feature Extractor (GFE) to reconstruct the sample representation based on the adjacency relationship between the sample itself and its neighbors. Due to the problem that some cross-modal methods only focus on paired-sample learning and can no longer integrate cross-modal information from the other modalities, we propose a cross-modal graph attentional strategy to generate the graph attentional representation for each sample on the local graph its corresponding paired sample. To eliminate heterogeneous gaps between modalities, we merge the features of the two modalities using a recurrent gated memory network to select prominent features from other modalities and filter out unimportant information to obtain a more differentiated feature representation in the shared latent space. Experiments on four benchmark datasets demonstrate the superiority of our proposed model compared to state-of-the-art cross-modal methods.

Neurocomputing, Band 535, 2023, S. 1-12

Recently, image-text matching has been intensively researched to bridge vision and language. Previous methods examine an intermodality relationship between an image-text pair from the single-view feature. However, it is difficult to discover all the abundant information based on a single intermodal relationship. In this paper, a novel Multi-View Inter-Modality Representation with Progressive Fusion (MIRPF) is developed to study intermodality relationships from multi-view features. The multi-view strategy offers more complementary and global semantic cues than single-view approaches. In particular, the multi-view intermodality representation network is built to generate multiple intermodality representations that provide different views to discover the latent image-text relationships. In addition, the progressive fusion module is performed to progressively fuse intermodal features, fully exploiting the inherent complementarity between different views. Extensive experiments on Flickr30K and MSCOCO demonstrate the superiority of MIRPF over several existing approaches. The code is available at: https://github.com/jasscia18/MIRPF.

Expert Systems with Applications, Volume 216, 2023, Article 119415

Cross-modal retrieval requires a query from one modality to retrieve relevant results from another modality, and its main problem is how to learn cross-modal similarity. Note that the full semantic information of a given concept is widely scattered across the multimodal and multigranular data and cannot be fully captured by most existing methods to accurately learn cross-modal similarity. Therefore, we propose a multimodal and multigranular hierarchical semantic enhancement network (M${}^2$HSE), which includes two stages to obtain more complete semantic information by fusing complementarity in multimodal and multigranular data. In stage 1, two classes of cross-modal similarity (primary similarity and auxiliary similarity) are computed more comprehensively in two sub-meshes. In particular, the primary similarities of two sub-networks are fused to perform the cross-modal search, while the auxiliary similarity provides a valuable complement to the primary similarity. In Stage 2, the multi-spring balance loss is proposed to more flexibly optimize cross-modal similarity. Exploiting this loss, the most representative samples are selected to establish the multi-spring equilibrium system, which adaptively optimizes the cross-modal similarities until the equilibrium state is reached. Extensive experiments performed on public benchmark datasets clearly demonstrate the effectiveness of our proposed method and demonstrate its competitiveness over the prior art.