AraCLIP: Cross-Lingual Learning for Effective Arabic Image Retrieval

In our work, we introduces Arabic Contrastive Language-Image Pre-training (AraCLIP), a model designed for Arabic image retrieval tasks, building upon the Contrastive Language-Image Pre-training (CLIP) architecture. AraCLIP leverages Knowledge Distillation to transfer cross-modal knowledge from English to Arabic, enhancing its ability to understand Arabic text and retrieve relevant images. Unlike existing multilingual models, AraCLIP is uniquely positioned to understand the intricacies of the Arabic language, including specific terms, cultural nuances, and contextual constructs. By leveraging the CLIP architecture as our foundation, we introduce a novel approach that seamlessly integrates textual and visual modalities, enabling AraCLIP to effectively retrieve images based on Arabic textual queries. We offer an online demonstration allowing users to input Arabic prompts and compare AraCLIP's performance with state-of-the-art multilingual models. We conduct comprehensive experiments to evaluate AraCLIP's performance across diverse datasets, including Arabic XTD-10, amd Arabic Flicker 8k. Our results showcase AraCLIP's superiority in image retrieval accuracy, demonstrating its effectiveness in handling Arabic queries. AraCLIP represents a significant advancement in cross-lingual image retrieval, offering promising applications in Arabic language processing and beyond.

The overall view of the cross-lingual Arabic framework involves three main stages as shown below:

In stage 1, we illustrates the English text model and the image model on contrastive learning. These models are strongly connected in the embedding space since this is our assumption from the beginning. Therefore, if we give the model an image and its caption, their embedding should show a high cosine similarity. Also, these models are already trained using contrastive learning to be able to capture the similarity between image-text pairs based on their embedding. Note that the English text model and image model can be separated.

In stage 2, we conduct the model training where we get the English text model (teacher model) from the first stage and a pre-trained Arabic model (student model), Arabert model in this work. The input for the teacher model will be the English caption and the input for the student model will be the translated and pre-processed Arabic caption. Then, the Arabic text model (student model) will produce an embedding vector of the Arabic caption that is similar to the embedding vector of the English caption that was produced from English model. Then, we minimize the Mean Squared Error (MSE) between the resulting output embeddings of these models.

It is important to highlight that this approach is different from the original training objective of CLIP, which focuses on training the model to establish correlations through cosine similarity calculations between pairs of images and texts. Nevertheless, it is feasible to directly employ cosine similarity in the context of teacher learning; however, prior research has demonstrated that the minimization of MSE yields a more informative signal for learning. On this stage, we are using the teacher-learning (Knowledge Distillation) training, as the student model will learn the features from the teacher, so it could be connected to the image model features.

Stage 3 shows the the evaluation of the Arabic text trained model (student model) with the image model, to check the performance in understanding text images. In our work, we evaluated it using different datasets (Arabic XTD-10 and Arabic flicker 8k) based on different metrics.