Promptable Multi-Granularity Image-Text Embeddings
Figure 1: As shown on the (a), prior methods fail on fine-grained local tasks due to inability to accept mask input, While existing promptable methods struggle to handle long texts due to token limitations, as shown in (b). Only our method succeeds in yielding robust embeddings for image and text at any granularity.
논문 컨텍스트
Paper title: PixCLIP: Achieving Fine-grained Visual Language Understanding via Any-granularity Pixel-Text Alignment Learning Abstract: While the Contrastive Language-Image Pretraining(CLIP) model has achieved remarkable success in a variety of downstream vison language understanding tasks, enhancing its capability for fine-grained image-text alignment remains an active research focus. To this end, most existing works adopt the strategy of explicitly increasing the granularity of visual information processing, e.g., incorporating visual prompts to guide the model focus on specific local regions within the image. Meanwhile, researches on Multimodal Large Language Models(MLLMs) have demonstrated that training with long and detailed textual descriptions can effectively improve the model's fine-grained vision-language alignment. However, the inherent token length limitation of CLIP's text encoder fundamentally limits CLIP to process more granular textual information embedded in long text sequences. To synergistically leverage the advantages of enhancing both visual and textual content processing granularity, we propose PixCLIP, a novel framework designed to concurrently accommodate visual prompt inputs and process lengthy textual descriptions. Specifically, we first establish an automated annotation pipeline capable of generating pixel-level localized, long-form textual descriptions for images. Utilizing this pipeline, we construct LongGRIT, a high-quality dataset comprising nearly 1.5 million samples. Secondly, we replace CLIP's original text encoder with the LLM and propose a three-branch pixel-text alignment l Passages referencing this figure: com/StuHude/PixCLIP/ Introduction Figure 1: As shown on the (a), prior methods fail on fine-grained local tasks due to inability to accept mask input, While existing promptable methods struggle to handle long texts due to token limitations, as shown in (b).
