Wav2vec2-base-960h
Kahlil Algya
I was trying to test Wav2Vec2Tokenizer.from_pretrained("facebook/wav2vec2-base-960h") on my own wav files (generated by pyaudio) and the generated transcription is far away from what I expected. However when I tested it with the samples downloaded from this page (facebook/wav2vec2-base-960h Hugging Face), it works really well no matter if it is a flac or wav file. I actually also tried uploading my own audio file to the demo page (facebook/wav2vec2-base-960h Hugging Face) and it worked very well. I am wondering if there are any pre-processing steps I missed that the HF sever side does take before the audio is read and fed to the model?
Transformer models are changing the world of machine learning, starting with natural language processing, and now, with audio and computer vision. Hugging Face's mission is to democratize good machine learning and give anyone the opportunity to use these new state-of-the-art machine learning models.Together with Amazon SageMaker and AWS have we been working on extending the functionalities of the Hugging Face Inference DLC and the Python SageMaker SDK to make it easier to use speech and vision models together with transformers.You can now use the Hugging Face Inference DLC to do automatic speech recognition using MetaAIs wav2vec2 model or Microsofts WavLM or use NVIDIAs SegFormer for semantic segmentation.
If you are going to use Sagemaker in a local environment (not SageMaker Studio or Notebook Instances). You need access to an IAM Role with the required permissions for Sagemaker. You can find here more about it.
Setting up the development environment and permissions needs to be done for the automatic-speech-recognition example and the semantic-segmentation example. First we update the sagemaker SDK to make sure we have new DataSerializer.
We use the facebook/wav2vec2-base-960h model running our recognition endpoint. This model is a fine-tune checkpoint of facebook/wav2vec2-base pretrained and fine-tuned on 960 hours of Librispeech on 16kHz sampled speech audio achieving 1.8/3.3 WER on the clean/other test sets.
Before we are able to deploy our HuggingFaceModel class we need to create a new serializer, which supports our audio data. The Serializer are used in Predictor and in the predict method to serializer our data to a specific mime-type, which send to the endpoint. The default serialzier for the HuggingFacePredcitor is a JSNON serializer, but since we are not going to send text data to the endpoint we will use the DataSerializer.
The .deploy() returns an HuggingFacePredictor object with our DataSerializer which can be used to request inference. This HuggingFacePredictor makes it easy to send requests to your endpoint and get the results back.
We succesfully managed to deploy Wav2vec2 to Amazon SageMaker for automatic speech recognition. The new DataSerializer makes it super easy to work with different mime-types than json/txt, which we are used to from NLP.
With this support we can now build state-of-the-art speech recognition systems on Amazon SageMaker with transparent insights on which models are used and how the data is processed. We could even go further and extend the inference part with a custom inference.py to include custom post-processing for grammar correction or punctuation.
Automatic speech recognition (ASR) is a commonly used machine learning (ML) technology in our daily lives and business scenarios. Applications such as voice-controlled assistants like Alexa and Siri, and voice-to-text applications like automatic subtitling for videos and transcribing meetings, are all powered by this technology. These applications take audio clips as input and convert speech signals to text, also referred to as speech-to-text applications.
Wav2Vec2 is a transformer-based architecture for ASR tasks. The following diagram shows its simplified architecture. The model is composed of a multi-layer convolutional network (CNN) as a feature extractor, which takes an input audio signal and outputs audio representations. They are fed into a transformer network to generate contextualized representations. This part of training can be self-supervised; the transformer can be trained with unlabeled speechand learn from it. Then the model is fine-tuned on labeled data with the Connectionist Temporal Classification (CTC) algorithm for specific ASR tasks. The base model we use in this tutorial is Wav2Vec2-Base-960h, fine-tuned on 960 hours of Librispeech on 16 kHz sampled speech audio.
AudioUtterance inherited from the AudioAnnotation ontology class is a span-based annotation, normally used to represent an utterance in audio dialogue. Using pyannote/speaker-segmentation, we will divide the input audio clip into individual speaker segments and store them in audio_utter.
Utterance inherited from the Annotation ontology class is also a span-based annotation. The Utterance is a different ontology from AudioUtterance. It is designed for text-based data. We will use it to store the text transcription segments from the facebook/wav2vec2-base-960h ASR model in text_utter.
We will use Link to connect the audio_utter as the parent to text_utter as the child. It helps us to keep the connection between the audio segment and text transcription segment respectively, which we can use later in this tutorial to visualize our outputs.
In the rapidly evolving landscape of technology, automatic speech recognition (ASR) stands out as a groundbreaking advancement that has the potential to reshape how we interact with our devices. Among the plethora of models facilitating this transformation, Wav2Vec2, introduced by Meta AI Research in September 2020, has emerged as a frontrunner. This model, thanks to its innovative architecture, has significantly accelerated progress in self-supervised pre-training for speech recognition. Its popularity is evidenced by its impressive download statistics on the Hugging Face Hub, where it garners over a quarter of a million downloads monthly. However, one stumbling block that developers and researchers frequently encounter is the model's handling of lengthy audio files.
Dealing with extensive audio files presents a unique set of challenges. At its core, Wav2Vec2 leverages transformer models, which, despite their numerous advantages, have a limitation in processing long sequences. This limitation stems not from the use of positional encodings, which Wav2Vec2 does not employ, but from the quadratic increase in computational complexity with respect to sequence length. Consequently, attempting to process an hour-long file, for instance, would overwhelm even the most robust GPUs, such as the NVIDIA A100, leading to inevitable crashes.
Recognizing this challenge, the community has devised innovative strategies to make ASR feasible for files of any length or for live inference scenarios. These strategies revolve around the clever use of the Connectionist Temporal Classification (CTC) architecture that underpins Wav2Vec2. By exploiting the specific characteristics of CTC, we can achieve remarkably accurate speech recognition results, even with files that would traditionally be considered too long for processing.
The most straightforward approach involves dividing the lengthy audio files into smaller, more manageable chunks, such as segments of 10 seconds. This method, while computationally efficient, often results in suboptimal recognition quality, especially around the boundaries of the chunks.
A more sophisticated strategy employs chunking with stride, allowing for overlapping chunks. This technique ensures that the model has adequate context in the center of each chunk, significantly improving the quality of speech recognition.
Further refinements are possible with models augmented with a language model (LM), boosting word error rate (WER) performance without the need for fine-tuning. The integration of an LM directly with the logits allows for seamless application of the chunking with stride technique, enhancing the model's accuracy.
Leveraging the single-pass, fast-processing capability of CTC models like Wav2Vec2, live inference becomes a practical reality. By feeding the pipeline data in real-time and applying strategic striding, the model can deliver immediate transcription results, enhancing user experience in live scenarios.
This introduction aims to shed light on the transformative potential of Wav2Vec2 in the realm of automatic speech recognition. By addressing the challenges associated with processing lengthy audio files and live data streams, we unlock new possibilities for user interaction and accessibility. Through continuous innovation and strategic application of the model's capabilities, we can push the boundaries of what's possible in ASR technology, making it more versatile and effective than ever before.
The realm of Automatic Speech Recognition (ASR) has witnessed significant advancements, thanks to the advent of models like Wav2Vec2, developed by Meta AI Research. This model, since its introduction in September 2020, has revolutionized the approach to self-supervised pretraining for speech recognition. It has not only garnered attention for its innovative architecture but also for its impressive ability to understand and transcribe human speech with remarkable accuracy.
One of the inherent limitations when dealing with transformer-based models, such as Wav2Vec2, is their handling of long sequences. These models, despite their prowess, encounter constraints related to sequence length. This is not due to the use of positional encodings, as one might expect, but rather the quadratic cost associated with attention mechanisms. The computational demand skyrockets with an increase in sequence length, making it impractical to process hour-long audio files on standard hardware configurations.
93ddb68554