NIRNL Framework: CMP and NIR Module Overview
Figure 1: The framework of our proposed NIRNL. Our framework comprises two core modules operating in parallel: the Cross-modal Margin Preserving (CMP) module and the Neighbor-aware Instance Refining (NIR) module. The CMP module refines the global structure of the embedding space, promoting proximity between positive pairs (indicated by light yellow and light pink) while enforcing separation of negative pairs (indicated by dark yellow and dark pink). For clarity, only image samples are visualized in the NIR module. The NIR module initially computes the Wasserstein Barycenter of samples and generates soft labels through KNN. It subsequently partitions the dataset into pure, hard, and noisy subsets by evaluating the consistency between soft labels and ground-truth labels. Finally, we design three different loss functions for each subset to dig up as much semantic information as possible.
논문 컨텍스트
Paper title: Neighbor-aware Instance Refining with Noisy Labels for Cross-Modal Retrieval Abstract: In recent years, Cross-Modal Retrieval (CMR) has made significant progress in the field of multi-modal analysis. However, since it is time-consuming and labor-intensive to collect large-scale and well-annotated data, the annotation of multi-modal data inevitably contains some noise. This will degrade the retrieval performance of the model. To tackle the problem, numerous robust CMR methods have been developed, including robust learning paradigms, label calibration strategies, and instance selection mechanisms. Unfortunately, they often fail to simultaneously satisfy model performance ceilings, calibration reliability, and data utilization rate. To overcome the limitations, we propose a novel robust cross-modal learning framework, namely Neighbor-aware Instance Refining with Noisy Labels (NIRNL). Specifically, we first propose Cross-modal Margin Preserving (CMP) to adjust the relative distance between positive and negative pairs, thereby enhancing the discrimination between sample pairs. Then, we propose Neighbor-aware Instance Refining (NIR) to identify pure subset, hard subset, and noisy subset through cross-modal neighborhood consensus. Afterward, we construct different tailored optimization strategies for this fine-grained partitioning, thereby maximizing the utilization of all available data while mitigating error propagation. Extensive experiments on three benchmark datasets demonstrate that NIRNL achieves state-of-the-art performance, exhibiting remarkable robustness, e Passages referencing this figure: Figure 1: The framework of our proposed NIRNL. As shown in Fig. 1 , NIRNL is composed of two core modules, i.
