Flow-based propagation and spatiotemporal Transformer are two mainstream<br>mechanisms in video inpainting (VI). Despite the effectiveness of these<br>components, they still suffer from some limitations that affect their<br>performance. Previous propagation-based approaches are performed separately<br>either in the image or feature domain. Global image propagation isolated from<br>learning may cause spatial misalignment due to inaccurate optical flow.<br>Moreover, memory or computational constraints limit the temporal range of<br>feature propagation and video Transformer, preventing exploration of<br>correspondence information from distant frames. To address these issues, we<br>propose an improved framework, called ProPainter, which involves enhanced<br>ProPagation and an efficient Transformer. Specifically, we introduce<br>dual-domain propagation that combines the advantages of image and feature<br>warping, exploiting global correspondences reliably. We also propose a<br>mask-guided sparse video Transformer, which achieves high efficiency by<br>discarding unnecessary and redundant tokens. With these components, ProPainter<br>outperforms prior arts by a large margin of 1.46 dB in PSNR while maintaining<br>appealing efficiency. <<<

CLIP yielded impressive results on zero-shot transfer learning tasks and is<br>considered as a foundation model like BERT or GPT3. CLIP vision models that<br>have a rich representation are pre-trained using the InfoNCE objective and<br>natural language supervision before they are fine-tuned on particular tasks.<br>Though CLIP excels at zero-shot transfer learning, it suffers from an<br>explaining away problem, that is, it focuses on one or few features, while<br>neglecting other relevant features. This problem is caused by insufficiently<br>extracting the covariance structure in the original multi-modal data. We<br>suggest to use modern Hopfield networks to tackle the problem of explaining<br>away. Their retrieved embeddings have an enriched covariance structure derived<br>from co-occurrences of features in the stored embeddings. However, modern<br>Hopfield networks increase the saturation effect of the InfoNCE objective which<br>hampers learning. We propose to use the InfoLOOB objective to mitigate this<br>saturation effect. We introduce the novel "Contrastive Leave One Out Boost"<br>(CLOOB), which uses modern Hopfield networks for covariance enrichment together<br>with the InfoLOOB objective. In experiments we compare CLOOB to CLIP after<br>pre-training on the Conceptual Captions and the YFCC dataset with respect to<br>their zero-shot transfer learning performance on other datasets. CLOOB<br>consistently outperforms CLIP at zero-shot transfer learning across all<br>considered architectures and datasets. <<<

This paper presents $\textbf{R}$epresentation-$\textbf{C}$onditioned image<br>$\textbf{G}$eneration (RCG), a simple yet effective image generation framework<br>which sets a new benchmark in class-unconditional image generation. RCG does<br>not condition on any human annotations. Instead, it conditions on a<br>self-supervised representation distribution which is mapped from the image<br>distribution using a pre-trained encoder. During generation, RCG samples from<br>such representation distribution using a representation diffusion model (RDM),<br>and employs a pixel generator to craft image pixels conditioned on the sampled<br>representation. Such a design provides substantial guidance during the<br>generative process, resulting in high-quality image generation. Tested on<br>ImageNet 256$\times$256, RCG achieves a Frechet Inception Distance (FID) of<br>3.31 and an Inception Score (IS) of 253.4. These results not only significantly<br>improve the state-of-the-art of class-unconditional image generation but also<br>rival the current leading methods in class-conditional image generation,<br>bridging the long-standing performance gap between these two tasks. Code is<br>available at https://github.com/LTH14/rcg. <<<