Final week, Meta Platforms’ synthetic intelligence analysis arm launched Voicebox, a machine studying mannequin that may generate speech from textual content. What units Voicebox aside from different text-to-speech fashions is its means to carry out many duties that it has not been skilled for, together with modifying, noise elimination, and magnificence switch.

The mannequin was skilled utilizing a particular methodology developed by Meta researchers. Whereas Meta has not launched Voicebox resulting from moral considerations about misuse, the preliminary outcomes are promising and may energy many purposes sooner or later.

‘Movement Matching’

Voicebox is a generative mannequin that may synthesize speech throughout six languages, together with English, French, Spanish, German, Polish, and Portuguese. Like massive language fashions, it has been skilled on a really normal activity that can be utilized for a lot of purposes. However whereas LLMs attempt to study the statistical regularities of phrases and textual content sequences, Voicebox has been skilled to study the patterns that map voice audio samples to their transcripts. 

Such a mannequin can then be utilized to many downstream duties with little or no fine-tuning. “The objective is to construct a single mannequin that may carry out many text-guided speech era duties by means of in-context studying,” Meta’s researchers write in their paper (PDF) describing the technical particulars of Voicebox.

The mannequin was skilled Meta’s “Flow Matching” method, which is extra environment friendly and generalizable than diffusion-based learning methods utilized in different generative fashions. The method allows Voicebox to “study from diversified speech knowledge with out these variations having to be rigorously labeled.” With out the necessity for handbook labeling, the researchers have been in a position to prepare Voicebox on 50,000 hours of speech and transcripts from audiobooks.

The mannequin makes use of “text-guided speech infilling” as its coaching objective, which implies it should predict a phase of speech given its surrounding audio and the entire textual content transcript. Principally, it implies that throughout coaching, the mannequin is supplied with an audio pattern and its corresponding textual content. Elements of the audio are then masked and the mannequin tries to generate the masked half utilizing the encompassing audio and the transcript as context. By doing this again and again, the mannequin learns to generate natural-sounding speech from textual content in a generalizable method.

Replicating voices throughout languages, modifying out errors in speech, and extra

In contrast to generative fashions which can be skilled for a selected utility, Voicebox can carry out many duties that it has not been skilled for. For instance, the mannequin can use a two-second voice pattern to generate speech for brand new textual content. Meta says this functionality can be utilized to deliver speech to people who find themselves unable to talk or customise the voices of non-playable recreation characters and digital assistants.

Voicebox additionally performs model switch in numerous methods. For instance, you’ll be able to present the mannequin with two audio and textual content samples. It’s going to use the primary audio pattern as model reference and modify the second to match the voice and tone of the reference. Curiously, the mannequin can do the identical factor throughout totally different languages, which could possibly be used to “assist folks talk in a pure, genuine method — even when they don’t communicate the identical languages.”

The mannequin may also do a wide range of modifying duties. For instance, if a canine barks within the background when you’re recording your voice, you’ll be able to present the audio and transcript to Voicebox and masks out the phase with the background noise. The mannequin will use the transcript to generate the lacking portion of the audio with out the background noise. 

The identical method can be utilized to edit speech. For instance, when you’ve got misspoken a phrase, you’ll be able to masks that portion of the audio pattern and cross it to Voicebox together with a transcript of the edited textual content. The mannequin will generate the lacking half with the brand new textual content in a method that matches the encompassing voice and tone.

One of many attention-grabbing purposes of Voicebox is voice sampling. The mannequin can generate varied speech samples from a single textual content sequence. This functionality can be utilized to generate artificial knowledge to coach different speech processing fashions. “Our outcomes present that speech recognition fashions skilled on Voicebox-generated artificial speech carry out nearly in addition to fashions skilled on actual speech, with 1 p.c error charge degradation versus 45 to 70 p.c degradation with artificial speech from earlier text-to-speech fashions,” Meta writes.

Voicebox has limits too. Because it has been skilled on audiobook knowledge, it doesn’t switch effectively to conversational speech that’s informal and incorporates non-verbal sounds. It additionally doesn’t present full management over totally different attributes of the generated speech, similar to voice model, tone, emotion, and acoustic situation. The Meta analysis workforce is exploring strategies to beat these limitations sooner or later.

Mannequin not launched

There may be rising concern in regards to the threats of AI-generated content material. For instance, cybercriminals lately tried to rip-off a girl by calling her and utilizing AI-generated voice to impersonate her grandson. Superior speech synthesis methods similar to Voicebox could possibly be used for comparable functions or different nefarious deeds, similar to creating pretend proof or manipulating actual audio.

“As with different highly effective new AI improvements, we acknowledge that this know-how brings the potential for misuse and unintended hurt,” Meta wrote on its AI weblog. As a consequence of these considerations, Meta didn’t launch the mannequin however offered technical particulars on the structure and coaching course of within the technical paper. The paper additionally incorporates particulars a few classifier mannequin that may detect speech and audio generated by Voicebox to mitigate the dangers of utilizing the mannequin.