Denoising diffusion probabilistic models (DDPM) are powerful hierarchical<br>latent variable models with remarkable sample generation quality and training<br>stability. These properties can be attributed to parameter sharing in the<br>generative hierarchy, as well as a parameter-free diffusion-based inference<br>procedure. In this paper, we present Few-Shot Diffusion Models (FSDM), a<br>framework for few-shot generation leveraging conditional DDPMs. FSDMs are<br>trained to adapt the generative process conditioned on a small set of images<br>from a given class by aggregating image patch information using a set-based<br>Vision Transformer (ViT). At test time, the model is able to generate samples<br>from previously unseen classes conditioned on as few as 5 samples from that<br>class. We empirically show that FSDM can perform few-shot generation and<br>transfer to new datasets. We benchmark variants of our method on complex vision<br>datasets for few-shot learning and compare to unconditional and conditional<br>DDPM baselines. Additionally, we show how conditioning the model on patch-based<br>input set information improves training convergence.