CLIP yielded impressive results on zero-shot transfer learning tasks and is<br>considered as a foundation model like BERT or GPT3. CLIP vision models that<br>have a rich representation are pre-trained using the InfoNCE objective and<br>natural language supervision before they are fine-tuned on particular tasks.<br>Though CLIP excels at zero-shot transfer learning, it suffers from an<br>explaining away problem, that is, it focuses on one or few features, while<br>neglecting other relevant features. This problem is caused by insufficiently<br>extracting the covariance structure in the original multi-modal data. We<br>suggest to use modern Hopfield networks to tackle the problem of explaining<br>away. Their retrieved embeddings have an enriched covariance structure derived<br>from co-occurrences of features in the stored embeddings. However, modern<br>Hopfield networks increase the saturation effect of the InfoNCE objective which<br>hampers learning. We propose to use the InfoLOOB objective to mitigate this<br>saturation effect. We introduce the novel "Contrastive Leave One Out Boost"<br>(CLOOB), which uses modern Hopfield networks for covariance enrichment together<br>with the InfoLOOB objective. In experiments we compare CLOOB to CLIP after<br>pre-training on the Conceptual Captions and the YFCC dataset with respect to<br>their zero-shot transfer learning performance on other datasets. CLOOB<br>consistently outperforms CLIP at zero-shot transfer learning across all<br>considered architectures and datasets.