Towards Understanding Deep Generative Models for Continuous Data Augmentation

Score-based models, a class of Deep Generative models, are considered state-of-the-art for image generation. The key idea behind such models is to gradually add noise to the data and then use a neural network to recover the original data by reversing the noising process. From a computational perspective, existing score-based models can be efficiently trained only if the forward or the corruption process comprises Gaussian noise, i.e., can be computed in closed form. On this, we first propose the use of continuous data augmenting methods, which provide the alternative to Gaussian noise. We also propose a new framework, named Intermediate Generator Optimization (IGO), that explicitly models such Non-Gaussian forward processes, by using the relationship between the corruption process and the layers in a deep Generative model. The main advantage of our framework is that it can be incorporated into any standard autoencoder pipeline for generative tasks. We provide implementation details to apply our framework on benchmark image generation and point-cloud denoising models, as well as the downstream task of Generative PCA.