Variational autoencoders are powerful algorithms for identifying dominantlatent structure in a single dataset. In many applications, however, we areinterested in modeling latent structure and variation that are enriched in atarget dataset compared to some background---e.g. enriched in patients comparedto the general population. Contrastive learning is a principled framework tocapture such enriched variation between the target and background, butstate-of-the-art contrastive methods are limited to linear models. In thispaper, we introduce the contrastive variational autoencoder (cVAE), whichcombines the benefits of contrastive learning with the power of deep generativemodels. The cVAE is designed to identify and enhance salient latent features.The cVAE is trained on two related but unpaired datasets, one of which hasminimal contribution from the salient latent features. The cVAE explicitlymodels latent features that are shared between the datasets, as well as thosethat are enriched in one dataset relative to the other, which allows thealgorithm to isolate and enhance the salient latent features. The algorithm isstraightforward to implement, has a similar run-time to the standard VAE, andis robust to noise and dataset purity. We conduct experiments across diversetypes of data, including gene expression and facial images, showing that thecVAE effectively uncovers latent structure that is salient in a particularanalysis.