SoftRankCLIP: Soft Top-K Aggregation for Efficient Image-Text Retrieval

However, common diagonal contrastive training may underuse the positive relationships available in captioned datasets, where each image is associated with multiple valid descriptions. We present SoftRankCLIP, a lightweight fine-tuning formulation that combines a multi-positive contrastive objective with a smooth negative-aware separation loss and a temperature-weighted top-$K$ positive score.

All additional objectives are used only during training, leaving the encoder architecture and retrieval procedure unchanged at inference. On Flickr30k and MS-COCO, the complete objective obtains higher retrieval scores than the corresponding pretrained references and the alternative fine-tuning objectives evaluated in our study. Ablation, backbone-transfer, and efficiency results further indicate that the training formulation is portable across the tested encoders and introduces moderate training overhead.