Training using SDP (and with DP by ratio?)

[isdanni]

This is a follow up to the previous discussion threads regarding stochastic duration predictor in #11 and #68 (comment), as well as with the reference of Bert-VITS2:

Regarding training using SDP, I have a few feedbacks:

1. A few months ago my experiments using use_sdp at earlier steps(100K ~ 500K) show below the average results compared to those trained without use_sdp, the audios did not sound natural and certain pronunciations are not clear. Now I plan to transfer learn a more well-trained checkpoint with SDP(like mentioned in the thread above), would be curious to hear anyone who has done similar experiments.

2. I am curious to learn if adding sdp_ratio and training both SDP and DP simultaneously would offer any improvements to results. Not sure about how much code changes but would love to add a pr if this sounds good to you!

3. About train both SDP & DP together and compare the result to save time(necessary of adversarial duration predictor #11 (comment)), if we train from scratch using this method my assumption is it does not sound good compared to two stage training.

4. DurationPredictor works very well from my experience, but is there any improvement can be done regarding both DP models?

==========================

A summary of my experience using use_sdp so far(will update later when I have more results):

• train using SDP from scratch: does not sound good at all.
• train without SDP from scratch: sound natural, best performing checkpoint to date
• train without SDP from scratch, then continue training using SDP: ?
• train with both SDP & DP by ratio: ?

[HuuHuy227]

Can you update the result of train without SDP from scratch, then continue training using SDP and train with both SDP & DP by ratio ?

[isdanni]

Can you update the result of train without SDP from scratch, then continue training using SDP and train with both SDP & DP by ratio ?

It’s still training at the moment. I will update as soon as there are some notable results

[HuuHuy227]

Hi isdanni, have you try WavLMDiscriminator like Bert-Vits2? WavLM is pre-trained on English and i wonder whether it can be used efficiently on other language?

[isdanni]

Hi isdanni, have you try WavLMDiscriminator like Bert-Vits2? WavLM is pre-trained on English and i wonder whether it can be used efficiently on other language?

no i haven't, i'm training EN only so haven't experienced much with multilingual set up. If u tried it please post here would love to hear the result :)

[HuuHuy227]

Hi isdanni, have you try WavLMDiscriminator like Bert-Vits2? WavLM is pre-trained on English and i wonder whether it can be used efficiently on other language?

no i haven't, i'm training EN only so haven't experienced much with multilingual set up. If u tried it please post here would love to hear the result :)

Well, I tried on my language and it produced very poor wavs. Now I'm trying a pre-trained phoneme as an input phoneme encoder.

[isdanni]

Well, I tried on my language and it produced very poor wavs. Now I'm trying a pre-trained phoneme as an input phoneme encoder.

Good to know! Tbh I'm not sure if WavLMDiscriminator would provide any notable improvements on top of the existing discriminator. I'll update here if I get a chance to try it.