javascript - Converting real-time audio to phonemes - Stack Overflow

admin管理员组

文章数量:1022989

Using a microphone as an input for real-time audio. How do I extract the currently said phoneme from the audio? I need it for lipsyncing 2d characters.

Basically, my approach would be to:

1. Fetch the real-time audio using a microphone
2. Detect the current phoneme that is being pronounced from the audio.

I have tried looking everywhere for an example or library that could solve this type of problem. Most libraries don't seem to output phonemes from audio.

There is a website that explains how they used machine learning to solve this, however without any code or tutorial on how to do it. .08685/

There is also this cool speech recognition tool called Pocketsphinx, but I cannot seem to find an example of it using Phoneme Recognition yet.

Using a microphone as an input for real-time audio. How do I extract the currently said phoneme from the audio? I need it for lipsyncing 2d characters.

Basically, my approach would be to:

1. Fetch the real-time audio using a microphone
2. Detect the current phoneme that is being pronounced from the audio.

I have tried looking everywhere for an example or library that could solve this type of problem. Most libraries don't seem to output phonemes from audio.

There is a website that explains how they used machine learning to solve this, however without any code or tutorial on how to do it. https://www.arxiv-vanity./papers/1910.08685/

There is also this cool speech recognition tool called Pocketsphinx, but I cannot seem to find an example of it using Phoneme Recognition yet.

• javascript
• python
• speech-recognition
• phoneme

Share Improve this question Follow edited May 18, 2023 at 21:46 NectoJ asked May 18, 2023 at 21:20 NectoJNectoJ 9022 silver badges88 bronze badges

Add a ment  | 

1 Answer 1

Sorted by: Reset to default 5

The way I would approach this is to get the word from the audio using Whisper or a similar STT service (the Python Speech Recognition Library is the go-to at the moment), then I would use the CMU Dict Library to provide phonemes for each word.

The phonemes are given using the CMU dictionary - for example DH for the θ phoneme - the th sound in this and that. That is, they are not given in IPA pronunciation - so you may need another layer if you need the phonemes in IPA format. If you need IPA formatted phonemes, then consider the IPA2 library.

Using a microphone as an input for real-time audio. How do I extract the currently said phoneme from the audio? I need it for lipsyncing 2d characters.

Basically, my approach would be to:

1. Fetch the real-time audio using a microphone
2. Detect the current phoneme that is being pronounced from the audio.

I have tried looking everywhere for an example or library that could solve this type of problem. Most libraries don't seem to output phonemes from audio.

There is a website that explains how they used machine learning to solve this, however without any code or tutorial on how to do it. .08685/

There is also this cool speech recognition tool called Pocketsphinx, but I cannot seem to find an example of it using Phoneme Recognition yet.

Using a microphone as an input for real-time audio. How do I extract the currently said phoneme from the audio? I need it for lipsyncing 2d characters.

Basically, my approach would be to:

1. Fetch the real-time audio using a microphone
2. Detect the current phoneme that is being pronounced from the audio.

I have tried looking everywhere for an example or library that could solve this type of problem. Most libraries don't seem to output phonemes from audio.

There is a website that explains how they used machine learning to solve this, however without any code or tutorial on how to do it. https://www.arxiv-vanity./papers/1910.08685/

There is also this cool speech recognition tool called Pocketsphinx, but I cannot seem to find an example of it using Phoneme Recognition yet.

• javascript
• python
• speech-recognition
• phoneme

Share Improve this question Follow edited May 18, 2023 at 21:46 NectoJ asked May 18, 2023 at 21:20 NectoJNectoJ 9022 silver badges88 bronze badges

Add a ment  | 

1 Answer 1

Sorted by: Reset to default 5

The way I would approach this is to get the word from the audio using Whisper or a similar STT service (the Python Speech Recognition Library is the go-to at the moment), then I would use the CMU Dict Library to provide phonemes for each word.

The phonemes are given using the CMU dictionary - for example DH for the θ phoneme - the th sound in this and that. That is, they are not given in IPA pronunciation - so you may need another layer if you need the phonemes in IPA format. If you need IPA formatted phonemes, then consider the IPA2 library.

本文标签： javascriptConverting realtime audio to phonemesStack Overflow