Ability to define prompts/speakers in text corpus

[C0untFloyd]

When Bark finally breaks the 13s limit on text, it would be cool to be able to define fixed voices for text blocks. That way, you could enable scripted dialogues and they would be reproduceable, instead of the (albeit funny) random walks now. What do you guys think?
The hardest point might be the definition markup itself. The first which came to my mind would be XML notation, something like:

<voice name="speaker_en_09">Bark is the best, forget the rest!</voice><voice name="speaker_en_07">Who makes up these bullshit ideas?</voice>

Perhaps I'll start experimenting with this in my fork this weekend if I find the time.

[djmaze]

Might make sense to follow the SSML standard in this regard.

[C0untFloyd]

Thanks, didn't know there is already a standard. https://www.w3.org/TR/speech-synthesis11/#S3.2.1 seems to fit nicely.

[C0untFloyd]

I started a very basic parser/generator only supporting the voice element so far. But that way you can at least define voices for lines to speak. Here is a quick result
drunk.webm

from this input

<?xml version="1.0"?> <speak version="1.0" xmlns="http://www.w3.org/2001/10/synthesis" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.w3.org/2001/10/synthesis http://www.w3.org/TR/speech-synthesis/synthesis.xsd" xml:lang="en-US"> <voice name="en_speaker_3">Look at that drunk guy!</voice> <voice name="en_speaker_9">Who is he?</voice> <voice name="en_speaker_3">WOMAN: [clears throat] 10 years ago, he proposed me and I rejected him.</voice> <voice name="en_speaker_9">Oh my God [laughs] he is still celebrating</voice> </speak>

[bghira]

I have implemented a different method in my Discord bot at bghira/discord-tron-master

import re

def process_line(line, characters):
    pattern = r"\{([^}]+)\}:?"
    match = re.search(pattern, line)

    if match:
        actor = match.group(1)
        line = re.sub(pattern, "", line).strip()
        character_voice = characters.get(actor, {}).get("voice", None)
    else:
        character_voice = None

    return line, character_voice

actors = {
    "MAN": {"voice": "hi_speaker_1"},
    "WOMAN": {"voice": "hi_speaker_5"},
}

prompt = """{MAN}: Hello, lady!
{WOMAN} Hello, sir!
"""

# Then, to use it:
lines = prompt.split("\n")

for line in lines:
    line_segment, character_voice = process_line(line, actors)
       generate_audio(line_segment, history_prompt=character_voice)..
        combined_audio = np.array([], dtype=np.int16)

        for audio in audio_segments:
            # Concatenate the audio
            combined_audio = np.concatenate((combined_audio, audio))
# Then, save combined_audio.

[bghira]

{narrator} Gary and his friends, JIM and KATE, are driving to the beach for a day of fun in the sun. Gary is driving, while Jim and Kate are in the backseat, looking out the window.
{JIM} I can't wait to get to the beach! I'm gonna work on my tan and build a sandcastle!
{KATE} I'm just happy to get out of the city for a while.
{GARY} Me too. It's nice to take a break from the hot dog business and enjoy the ocean.

allows for this kind of formatted script input.