Revisiting and Incorporating GANs and Diffusion Models in High-Fidelity Speech Synthesis

Abstract
Comparison with other models
Model Generalization
Ablation Study

Abstract

Generative adversarial networks and denoising probabilistic models have recently achieved impressive performances in image and audio synthesis. After revisiting their success in conditional speech synthesis, we find that 1) GANs sacrifice sample diversity for quality and speed, 2) diffusion models exhibit outperformed sample quality and diversity while requiring a large number of iterative refinements. Achieving high-quality and diverse speech synthesis at a low computational cost has become an open problem for all neural synthesizers. In this work, we propose to converge advantages from GANs and diffusion models by incorporating both classes, introducing dual-empowered modeling perspectives: 1) DiffGAN-Wave, a diffusion model whose denoising process is parametrized by conditional GANs, and the non-Gaussian denoising distribution makes it much more stable to implement the reverse process with large steps sizes; and 2) GANDiff-Wave, a generative adversarial network whose forward process is constructed by multiple denoising diffusion iterations, which exhibits better sample diversity than traditional GANs. Experimental results show that both variants enjoy an efficient 4-step sampling process and demonstrate superior sample quality and diversity. Audio samples are available at https://RevisitSpeech.github.io