Audio samples from "Data-augmented cross-lingual synthesis in a teacher-student framework"
Authors: Marcel de Korte, Jaebok Kim, Aki Kunikoshi, Adaeze Adigwe, Esther Klabbers
Abstract:Cross-lingual synthesis can be defined as the task of letting a speaker generate fluent synthetic speech in another language. This is a challenging task, and resulting speech can suffer from reduced naturalness, accented speech, and/or loss of essential voice characteristics. Previous research shows that many models appear to have insufficient generalization capabilities to perform well on every of these cross-lingual aspects. To overcome these generalization problems, we propose to apply the teacher-student paradigm to cross-lingual synthesis. While a teacher model is commonly used to produce teacher forced data, we propose to also use it to produce augmented data of unseen speaker-language pairs, where the aim is to retain essential speaker characteristics. Both sets of data are then used for student model training, which is trained to retain the naturalness and prosodic variation present in the teacher forced data, while learning the speaker identity from the augmented data. Some modifications to the student model are proposed to make the separation of teacher forced and augmented data more straightforward. Results show that the proposed approach improves the retention of speaker characteristics in the speech, while managing to retain high levels of naturalness and prosodic variation.
Submitted to InterSpeech 2022
Contents
Naturalness and speaker similarity experiments
The samples below refer to Section 5.1 of the paper and are examples of sentences used for evaluation of speaker similarity and naturalness.
Reference source language speaker samples:
Text: He did retain his sole set clarinet solo on Rockin' In Rhythm.
Polish reference source speaker:
Baseline:
Proposed:
German reference source speaker:
Baseline:
Proposed:
Swedish reference source speaker:
Baseline:
Proposed:
Text: Bars were to be attached to denote the area or mission which qualified for recognition.
Polish reference source speaker:
Baseline:
Proposed:
German reference source speaker:
Baseline:
Proposed:
Swedish reference source speaker:
Baseline:
Proposed:
Text: Via the Broad River, it is part of the Savannah River watershed.
Polish reference source speaker:
Baseline:
Proposed:
German reference source speaker:
Baseline:
Proposed:
Swedish reference source speaker:
Baseline:
Proposed:
Naturalness of intonation experiments
The samples below refer to Section 5.2 of the paper and are examples of sentences used for evaluation of naturalness of intonation.
5.2.1: Comparing speech from proposed model with augmented speech
Text: Members of the grid also collaborate on their Grid wiki.
Speech from proposed model:
Augmented speech from teacher model:
Text: Now why should a man with three hundred and fifty thousands dollars bother to imitate four Hawaiians?
Speech from proposed model:
Augmented speech from teacher model:
Text: The route, through remote country, was otherwise seldom used.
Speech from proposed model:
Augmented speech from teacher model:
5.2.2: Evaluating the effect of the proposed model changes
Text: It is an extremely aggressive carnivore that will eat anything in its sight.
Speech from proposed model:
Speech from baseline model:
Text: In September, Deng Xiaoping also visited the North Korean capital Pyongyang.
Speech from proposed model:
Speech from baseline model:
Text: Stage three is initiative versus guilt, which occurs during play age.
Speech from proposed model:
Speech from baseline model:
5.2.3: Evaluating the effect of in-lingual augmented data
Text: They are getting more organised, they learnt from what the special and security forces are doing.
Speech from proposed model:
Speech from model without in-lingual augmented data:
Text: I on the other hand, would often go back and finish rhymes that I started.
Speech from proposed model:
Speech from model without in-lingual augmented data:
Text: Two eighteenth century organs are spectacularly flamboyant.
Speech from proposed model:
Speech from model without in-lingual augmented data:
5.2.4: Comparing teacher forced and augmented generation
Text: Some Kind of Trouble is the third studio album from British singer songwriter James Blunt.
Speech from proposed model, teacher forced mode:
Speech from proposed model, augmented mode:
Text: Virgil is the second son of guitarist Steve Howe, of Yes
Speech from proposed model, teacher forced mode:
Speech from proposed model, augmented mode:
Text: At about the same time, the museum possessed a fossil of an ichthyosaur.