CLIP yielded impressive results on zero-shot transfer learning tasks and isconsidered as a foundation model like BERT or GPT3. CLIP vision models thathave a rich representation are pre-trained using the InfoNCE objective andnatural language supervision before they are fine-tuned on particular tasks.Though CLIP excels at zero-shot transfer learning, it suffers from anexplaining away problem, that is, it focuses on one or few features, whileneglecting other relevant features. This problem is caused by insufficientlyextracting the covariance structure in the original multi-modal data. Wesuggest to use modern Hopfield networks to tackle the problem of explainingaway. Their retrieved embeddings have an enriched covariance structure derivedfrom co-occurrences of features in the stored embeddings. However, modernHopfield networks increase the saturation effect of the InfoNCE objective whichhampers learning. We propose to use the InfoLOOB objective to mitigate thissaturation effect. We introduce the novel "Contrastive Leave One Out Boost"(CLOOB), which uses modern Hopfield networks for covariance enrichment togetherwith the InfoLOOB objective. In experiments we compare CLOOB to CLIP afterpre-training on the Conceptual Captions and the YFCC dataset with respect totheir zero-shot transfer learning performance on other datasets. CLOOBconsistently outperforms CLIP at zero-shot transfer learning across allconsidered architectures and datasets.