Generative Adversarial Networks for Semantic Image Synthesis and Unconditional Synthesis with Limited Data

Abstract

Generative modeling of images is an important task aimed at synthesizing new images that are indistinguishable from real samples. The ability to generate diverse, realistic- looking images holds numerous applications, ranging from artistic content creation to data augmentation for other computer vision tasks. In this thesis, we study generative adversarial networks (GANs), which have gained popularity as the leading image synthesis framework due to their exceptional performance. GANs consist of a generator and a discriminator that are engaged in an adversarial game. In this game, the discriminator learns to correctly classify incoming real and fake images, while the generator is trained to fool the discriminator by producing more realistic samples. Although GANs have made significant advancements in recent years, more research is needed to further enhance their scalability across diverse datasets, as well as the quality and diversity of their synthesis. To this end, this thesis presents several new GAN methods that expand the literature on GANs in various aspects. <br /> Firstly, we address the task of semantic image synthesis, generating realistic images from semantic label maps. For this task, our work introduces a new segmentation-based discriminator that provides a strong training signal for the generator, eliminating the need for additional losses and tricks used in prior models. Compared to previous GAN approaches, our proposed method achieves a synthesis with higher image quality and diversity, while showing much better scalability to datasets with severe class imbalances. <br /> Secondly, we explore the training of unconditional GAN models in low data regimes. Previously, under limited data, GAN models suffered from training instabilities and memorization issues, which limited their application in restricted image domains. To address this, we propose new GAN approaches for various limited-data scenarios, including traditional one-shot and few-shot learning regimes. The advancements of our methods include new training schemes, improved architectures of discriminators, and novel regularization terms for generators. Our proposed methods enable high-quality and diverse synthesis from extremely small datasets, on which prior GAN models could not be trained successfully. <br /> Overall, this thesis advances the field of generative adversarial networks by introducing new GAN models that improve over prior work in the image synthesis quality, diversity, and applicability across various image domains. The proposed approaches demonstrate superior performance in semantic image synthesis and unconditional training with limited data, making GANs more powerful and effective for a wide range of computer vision applications.