IAM-CL2R Compatible Learning Framework — Poster

Paper context

Paper title: Stationary Representations: Optimally Approximating Compatibility and Implications for Improved Model Replacements Abstract: A CVPR 2024 poster detailing the IAM-CL2R framework for compatible learning representations. It defines compatibility, proves stationarity implies optimal approximation, and shows empirical results on CIFAR100/10 with heatmaps and distance graphs. Paper body (method & results): Stationary Representations: Optimally Approximating Compatibility and Implications for Improved Model Replacements Niccol`o Biondi Federico Pernici Simone Ricci Alberto Del Bimbo DINFO (Department of Information Engineering), University of Florence, Italy, MICC (Media Integration and Communication Center), name.surname@unifi.it Abstract Learning compatible representations enables the inter- changeable use of semantic features as models are updated over time. This is particularly relevant in search and re- trieval systems where it is crucial to avoid reprocessing of the gallery images with the updated model. While recent research has shown promising empirical evidence, there is still a lack of comprehensive theoretical understanding about learning compatible representations. In this paper, we demonstrate that the stationary representations learned by the d-Simplex fixed classifier optimally approximate compat- ibility representation according to the two inequality con- straints of its formal definition. This not only establishes a solid foundation for future works in this line of research but also presents implications that can be exploited in practical learning scenarios. An exemplary application is the now- standard practice of downloading and fine-tuning new pre- trained models. Specifically, we show the strengths and criti- cal issues of stationary representations in the case in which a model undergoing sequential fine-tuning is asynchronously replaced by downloading a better-performing model pre- trained elsewhere. Such a representation enables seamless delivery of retrieval service (i.e., no reprocessing of gallery images) and offers improved performance without opera- tional disruptions during model replacement. Code available at: https://github.com/miccunifi/iamcl2r. 1. Introduction By learning powerful internal feature representations from data, Deep Neural Networks (DNNs) [1–4] have made tremendous progress in some of the most challenging search tasks such as face recognition [5–9], person re- identification [10–12], image retrieval [13–15] and this sig- nificance also extends to a variety of other data modalities [16, 17]. Although all of the works mentioned above have focused on learning feature representations from static and, 𝜙𝑇 task 2 task 𝑇 task 1 𝜙2 𝜙1 Stationary Representation (⟹ Compatible) 𝜙 Figure 1. Improved Asynchronous Model Compatible Lifelong Learning Representation (IAM-CL2R pronounced “I am clear”). In the process of lifelong learning, a model is sequentially fine-tuned and asynchronously replaced with improved third-party models that are pre-trained externally. Stationary representations ensure seamless retrieval services and better performance, without the need to reprocess gallery images. more recently, dynamic datasets [18–21], the now-standard practice is downloading and fine-tuning representations from models pre-trained elsewhere [22, 23]. These “third-party” pre-trained models often incorporate new data, utilize alter- native architectures, adopt different loss functions or more in general provide novel methodologies. Whether applied individually or combined, these advancements aim to en- capsulate the field’s rapid progress within a single unified model [24]. This greatly facilitates the exploitation of in- ternally learned semantic representations, particularly as models, datasets, and computational infrastructure continue to expand in size, complexity, and cost [25, 26]. The challenge of fully exploiting such standard practice in retrieval/search systems has to deal with the underlying 1 arXiv:2405.02581v1 [cs.CV] 4 May 2024 problem of compatible learning [27–29]. That is the desire to align the representation of different models trained with different data, initialization seeds, loss functions, or alterna- tive architectures—either individually or in combination. In such applications, maintaining alignment is crucial to mini- mize the need for repeated reprocessing of gallery images for feature extraction each time a new pre-trained model becomes available [24]. Reprocessing is not only computa- tionally intensive but may also be unsustainable for extensive gallery sets [25, 26, 30] or unfeasible if the original images are no longer accessible due to privacy concerns [31]. This holds across various typical galleries: social networks update millions of images every month, while in robotics and auto- motive domains, the update rate can be as rapid as hundreds of images every second. Similarly, in textual domains, books can be structured into chapters, paragraphs, and sentences, enabling the capture of semantic relationships between these segments. While a similar organizational principle can be structured for the web with LLMs [17, 32], the challenge lies in the impracticality of reprocessing such extensive content with each advancement in representation models. Although recent research has shown the effectiveness of compatible representation learning [27–29, 33–41], there is still a lack of comprehensive theoretical understanding about compati- bility. This paper introduces a theorem that demonstrates how the stationary representations proposed in [42, 43] optimally approximate compatibility according to the two inequality constraints of its formal definition as provided in [27]. This not only establishes a solid foundation for future works, but also presents implications that can be exploited fine- tuning third-party models without the need of reprocessing gallery images. Specifically, we show that a continuously fine-tuned model can be asynchronously replaced by down- loading a higher-performing, pre-trained model from an ex- ternal source. Due to stationarity (and therefore optimal com- patibility), such a replacement provides seamless retrieval services with improved performance, eliminating the need for image gallery reprocessing. We refer to this scenario as Improved Asynchronous Model Compatible Lifelong Learn- ing Representation (IAM-CL2R pronounced “I am clear”). Fig. 1 illustrates the relationship between sequential fine- tuning and model replacement. Furthermore, as will be elab- orated in the related work section, our foundation draws connections with the Neural Collapse phenomenon [44] and its associated theory. Our second contribution is related to a specific challenge that arises: the tendency of the old and the new replaced mod- els to align at their first-order statistics, an inherent property of stationary representation. Consequently, cross-entropy based prediction errors alone, when fine-tuning the represen- tation, may not fully capture higher-order dependencies. To address this issue while preserving compatibility, we show that learning stationary representations using a convex com- bination of the cross-entropy loss and the infoNCE loss [45] is equivalent to training under one of the compatibility in- equality constraints in [27]. This combined loss, termed Higher-Order Compatibility (HOC), distinguishes itself from the use of cross-entropy alone by capturing higher-order de- pendencies and optimally approximating compatibility. 2. Related Work Neural Collapse. Neural Collapse (NC) is an empirical phenomenon that demonstrates the alignment between fea- tures and the classifier in a symmetric configuration [44]. Specifically, each class feature vector and its corresponding class prototype vector align with each other (i.e., collapse onto the same vector), forming a regular Simplex geometry in a subspace of the representation space. This particular configuration, which results in maximal separation of the collapsed vectors, is also referred to as a regular Simplex ETF (Equiangular Tight Frame). As training progresses and the training phase goes beyond zero classification error, the network increasingly approaches collapse. Notably, this also agrees with the double descent generalization regime ob- served within the same training phase [46]. The two phe- nomena together indicate a form of stable steady-state for the internal representations of Deep Neural Networks. Prior to the observation of neural collapse, other re- search applied the steady-state of the Simplex geometry directly from the beginning of training. The fixed classi- fier with mutually orthogonal prototypes, introduced in [47], firstly demonstrates no degradation in classification perfor- mance. Building on this initial model, the regular polytope fixed classifiers—such as the d-Simplex, d-Cube, and d- Orthoplex—advance the concept further by observing sta- tionary and maximally separated representations, as intro- duced in [48] and further detailed in [42]. Prior to these developments, [49] delved into the early energy-based in- vestigations of symmetric and maximal separation in the representation space. The distinction between the natural emergence of a regular Simplex ETF and intentionally fixing the regular Simplex geometry at the beginning of training is that prior fixing can preserve regions in the representation space for future classes, as introduced in [50] and more re- cently in [51] and [52]. Our work takes advantage of this preservation for future classes, allowing third-party repre- sentation models to be trained from scratch and fine-tuned, while mitigating the interference in the representation space of the classes involved in both processes. As neural collapse is related to the interaction between the neural network’s final and penultimate layers, it offers a tool to examine training dynamics and convergence, as in- troduced in [53] and [54] under the name of Unconstrained Feature Model (UFM) and Layered Peeled Model (LPM), respectively. In both [55] and [56], the favorable conver- 2 gence of fixing the final classifier according to the UFM is demonstrated. In [54], it is shown that training on imbal- anced datasets does not necessarily result in NC. Additional observations from [57] suggest that NC can emerge in both imbalanced and long-tail scenarios when the classifier is fixed to a d-Simplex geometry. Further detailed results on NC are presented in [58]. Our proof is based on the assump- tion from the UFM and LPM that the backbone has sufficient expressiveness to allow for the independent study of each feature. Our proof is also based on the assumption of d- Simplex fixed classifier, whose inherent symmetry allows to reduce the extent of the analysis to a single pairwise class interaction, as it causes all the interactions to be identical. Compatible Representations Learning. Compatible rep- resentations broadly refer to the ability to align different learned representations, as discussed in [59–63]. The dis- tinction outlined in [27] is that the alignment of models should be achieved without wasting the information learned from new data. This capability is typically evaluated in a query and gallery setting, where query and gallery features are extracted from two different representation models. The model for the query is trained using