Striving for Open Supply Modular GPT4-o with Hugging Face’s Speech To Speech

Picture by Editor | Midjourney

Within the growth of AI expertise, many superb closed-source fashions are locked behind firm doorways, and we are able to solely entry them if we’re concerned in inner work.

In distinction, the group has tried to try for the extent of closed-source fashions by constructing open-source fashions and permitting everybody to enhance the fashions. One of many initiatives that we have to learn about is the Hugging Face’s Speech-to-Speech.

What’s the Hugging Face’s Speech-to-Speech venture, and why ought to you already know?

Let’s focus on.

Hugging Face’s Speech-to-Speech Challenge

 
The Hugging Face’s Speech-to-Speech Project is a modular venture that makes use of the Transformers library to combine a number of open-source fashions into the speech-to-speech pipeline.

The venture goals to satisfy the GPT4-o functionality by leveraging the open-source mannequin, designed to be simply modified and assist many developer wants.

The pipeline consists of a number of mannequin functionalities in a cascading method, together with:

1. Voice Exercise Detection (VAD)
2. Speech to Textual content (STT)
   • Any Whisper mannequin
   • Lightning Whisper MLX
   • Paraformer – FunASR
3. Language Mannequin (LM)
   • Any Instruction-Mannequin in Hugging Face Hub
   • max-lm
   • OpenAI API
4. Textual content to Speech (TTS)
   • Parler-TTS
   • MeloTTS
   • ChatTTS

This doesn’t imply you might want to use each mannequin obtainable above, however the pipeline requires 4 fashions above to run accurately.

The primary goal of the pipeline above is to remodel any speech given to it into one other form, similar to speech in a distinct language or tone.

Let’s arrange the venture in your setting to check the pipeline.

Challenge Setup

First, we have to clone the GitHub repository into your setting. The next code will aid you do this.

git clone https://github.com/huggingface/speech-to-speech.git
cd speech-to-speech

Given the setup above, you may set up the required packages with pip. The really helpful technique is to make use of uv, however you may set up them utilizing pip.

pip set up -r necessities.txt

If you’re utilizing a Mac, use the next code.

pip set up -r requirements_mac.txt

Make sure that your set up is completed earlier than we proceed. It’s additionally really helpful that you just use a digital setting so it doesn’t disturb your primary setting.
 

Challenge Utilization

There are a number of really helpful methods to implement the pipeline. The primary one is utilizing the Server/Shopper Method.

To try this, you may run the next code to run the pipeline in your server.

python s2s_pipeline.py --recv_host 0.0.0.0 --send_host 0.0.0.0

Then, run the next code domestically to obtain microphone enter and the generated audio output.

python listen_and_play.py --host 

Moreover, you need to use the next arguments if you’re utilizing a Mac to make use of it native.

python s2s_pipeline.py --local_mac_optimal_settings --host 

For those who desire that technique, you can too use Docker. Nevertheless, you would wish the NVIDIA Container Toolkit to run it. With the setting prepared, you solely have to run the next code.

That’s how you would run the pipeline; let’s take a look at some arguments you may discover with the Hugging Face Speech-to-Speech pipeline.
 

Further Arguments

Every of the STT (Speech-to-Textual content), LM (Language Mannequin) and TTS (Textual content-to-Speech) has pipeline arguments with the prefix
stt, lm or tts.

For instance, that is tips on how to run the pipeline utilizing CUDA.

python s2s_pipeline.py --lm_model_name microsoft/Phi-3-mini-4k-instruct --stt_compile_mode reduce-overhead --tts_compile_mode default --recv_host 0.0.0.0 --send_host 0.0.0.0 

Within the code above, we explicitly determine which Language Mannequin (LM) to make use of whereas controlling the opposite fashions’ behaviour.

The pipeline additionally helps multi-language use instances, which embrace English, French, Spanish, Chinese language, Japanese, and Korean.

We are able to add the language argument with the next code for computerized language detection.

python s2s_pipeline.py --stt_model_name large-v3  --language auto --mlx_lm_model_name mlx-community/Meta-Llama-3.1-8B-Instruct

Implementing a selected language (e.g. Chinese language) with the next code can be attainable.

python s2s_pipeline.py --stt_model_name large-v3 --language auto --mlx_lm_model_name mlx-community/Meta-Llama-3.1-8B-Instruct

You possibly can try their arguments repository for the list to see if they suit your use cases.
 

Conclusion

 
Within the pursuit of attaining the closed-source mannequin functionality, Hugging Face tries to emulate a venture referred to as Speech-to-Speech. The venture makes use of fashions from the Hugging Face Transformers library on the hub to create a pipeline that may carry out Speech-to-Speech duties. On this article, we have now explored how the venture is structured and tips on how to set it up.

I hope this has helped!
 
 

Cornellius Yudha Wijaya is an information science assistant supervisor and knowledge author. Whereas working full-time at Allianz Indonesia, he likes to share Python and knowledge suggestions by way of social media and writing media. Cornellius writes on a wide range of AI and machine studying matters.