Fix M-CTC-T chunking #18949

samwaterbury · 2022-09-09T05:25:01Z

Summary

Chunking doesn't currently work for the M-CTC-T architecture, which this PR attempts to fix. The change is fairly minor but I am not an expert on how M-CTC-T works, so definitely open to feedback. In my usage, it works as intended.

Paging @patrickvonplaten since I think you originally implemented the chunking mechanism. 🙂

I will add some PR comments explaining the changes.

Testing

I added one test to cover the ASR pipeline with M-CTC-T. I also ran these tests:

RUN_SLOW=True RUN_PIPELINE_TESTS=True pytest \
    tests/models/mctct \
    tests/pipelines/test_pipelines_automatic_speech_recognition.py \
    tests/pipelines/test_pipelines_common.py

Some of these tests fail, but I was able to confirm they're also failing on the main branch. Here are the failures:

FAILED tests/pipelines/test_pipelines_common.py::CommonPipelineTest::test_iterator_data_tf - tensorflow.python.framework.errors_impl.InternalError: Exception encountered when calling layer...
FAILED tests/pipelines/test_pipelines_common.py::PipelineUtilsTest::test_load_default_pipelines_tf - tensorflow.python.framework.errors_impl.InternalError: Exception encountered when calli...
FAILED tests/pipelines/test_pipelines_automatic_speech_recognition.py::AutomaticSpeechRecognitionPipelineTests::test_chunking_fast_with_lm - AssertionError: 'e<s>eh' != '<s> <s'
FAILED tests/pipelines/test_pipelines_automatic_speech_recognition.py::AutomaticSpeechRecognitionPipelineTests::test_large_model_pt_with_lm - AssertionError: 'ctc' != 'ctc_with_lm'
FAILED tests/pipelines/test_pipelines_automatic_speech_recognition.py::AutomaticSpeechRecognitionPipelineTests::test_with_lm_fast - AssertionError: 'ctc' != 'ctc_with_lm'
FAILED tests/pipelines/test_pipelines_automatic_speech_recognition.py::AutomaticSpeechRecognitionPipelineTests::test_with_local_lm_fast - AssertionError: 'ctc' != 'ctc_with_lm'
FAILED tests/models/mctct/test_modeling_mctct.py::MCTCTModelIntegrationTest::test_inference_ctc_normal - RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 14.76 GiB tot...
FAILED tests/models/mctct/test_modeling_mctct.py::MCTCTModelIntegrationTest::test_inference_ctc_normal_batched - RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 14.76...
FAILED tests/models/mctct/test_modeling_mctct.py::MCTCTModelIntegrationTest::test_inference_ctc_robust_batched - RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 14.76...

Some of these look to just be issues with my machine (memory errors).

samwaterbury · 2022-09-09T05:33:27Z

src/transformers/pipelines/automatic_speech_recognition.py

@@ -30,18 +30,33 @@
    from ..models.auto.modeling_auto import MODEL_FOR_CTC_MAPPING, MODEL_FOR_SPEECH_SEQ_2_SEQ_MAPPING




tl;dr of changes to this function:

This function takes a list of strides stride. It previously took a single number ratio. Now, it takes a list of numbers ratio to match the list of strides. This lets the ratio differ between strides.

samwaterbury · 2022-09-09T05:33:27Z

src/transformers/pipelines/automatic_speech_recognition.py

            attention_mask = model_inputs.pop("attention_mask", None)
-            outputs = self.model(input_values=input_values, attention_mask=attention_mask)
+            outputs = self.model(input_values, attention_mask=attention_mask)
            logits = outputs.logits

            if self.type == "ctc_with_lm":
                out = {"logits": logits}
            else:
                out = {"tokens": logits.argmax(dim=-1)}
            if stride is not None:


Here is the main change. The previous approach of calculating the inputs_to_logits_ratio from the convolutional layers of the model is still used when possible. For M-CTC-T, essentially the new approach is:

Calculate the ratio of the original length of the input features (stride[0]) divided by the length of the output logits (outputs.logits.shape[1])

Assume that this ratio holds for the strides

samwaterbury · 2022-09-09T05:33:27Z

tests/pipelines/test_pipelines_automatic_speech_recognition.py

@@ -377,6 +379,33 @@ def test_chunking_fast(self):
        self.assertEqual(output, [{"text": ANY(str)}])
        self.assertEqual(output[0]["text"][:6], "ZBT ZC")

+    @require_torch


Heads up: this new test uses speechbrain/m-ctc-t-large, which is a 4.1 GB model.

I'm not familiar with the testing infrastructure, so if that's undesirable there may be other options for testing. However, there is no readily available official/popular "smaller" version of this model.

This should be fine thanks :-)

samwaterbury · 2022-09-09T05:33:27Z

src/transformers/pipelines/automatic_speech_recognition.py

-            outputs = self.model(input_values=input_values, attention_mask=attention_mask)
+            outputs = self.model(input_values, attention_mask=attention_mask)


M-CTC-T takes an argument input_features instead of input_values, so I made this a positional argument instead of a keyword argument. I'm happy to add some if/else logic to keep it a keyword argument if desired, but this is a bit easier to read.

samwaterbury · 2022-09-09T05:33:27Z

tests/pipelines/test_pipelines_automatic_speech_recognition.py

+        # And also for multiple batched inputs
+        n_repeats = 2
+        audio_tiled = np.tile(audio, n_repeats)
+        output = speech_recognizer([audio_tiled], batch_size=2)
+        self.assertEqual(output, [{"text": ANY(str)}])
+        self.assertEqual(output[0]["text"][:26], "A man said to the universe")


Flagging this because I can't for the life of me understand why the other ASR tests are constructed this way:

n_repeats = 2 audio_tiled = np.tile(audio, n_repeats) output = speech_recognizer([audio_tiled], batch_size=2)

If audio is an array with shape (N,), then audio_tiled is an array with shape (2 * N,), and output is a list of length 1. It seems to me like a correct test for batched inputs would instead look like:

output = speech_recognizer([audio, audio], batch_size=2)

Which indeed works and produces an output which is length 2. Am I crazy? 😄

Hmm - I'm not 100% sure here - was the idea to test a long input? cc'ing @Narsil here - was the test intended?

samwaterbury · 2022-09-09T05:33:27Z

tests/pipelines/test_pipelines_automatic_speech_recognition.py

+        model = MCTCTForCTC.from_pretrained("speechbrain/m-ctc-t-large")
+        processor = MCTCTProcessor.from_pretrained("speechbrain/m-ctc-t-large")
+        speech_recognizer = AutomaticSpeechRecognitionPipeline(
+            feature_extractor=processor.feature_extractor,
+            tokenizer=processor.tokenizer,
+            model=model,
+            chunk_length_s=10.0,
+            framework="pt",
+        )


On a side note, I wasn't able to load this model with pipeline directly because there is a typo in one of the model artifacts: https://huggingface.co/speechbrain/m-ctc-t-large/blob/main/preprocessor_config.json

It's MCTCFeatureExtractor instead of MCTCTFeatureExtractor (missing the T).

Thanks for the info - updating!

HuggingFaceDocBuilderDev · 2022-09-09T05:35:31Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

samwaterbury · 2022-09-09T05:58:06Z

In addition to running tests, as a sanity check I ran the code snippet from this blog post:

from transformers import pipeline

pipe = pipeline(model="facebook/wav2vec2-base-960h")
pipe("very_long_file.mp3", chunk_length_s=10)

and it works as expected. I also modified it for M-CTC-T and it still works:

from transformers import AutomaticSpeechRecognitionPipeline, MCTCTForCTC, MCTCTProcessor

model = MCTCTForCTC.from_pretrained("speechbrain/m-ctc-t-large")
processor = MCTCTProcessor.from_pretrained("speechbrain/m-ctc-t-large")
pipe = AutomaticSpeechRecognitionPipeline(
    feature_extractor=processor.feature_extractor,
    tokenizer=processor.tokenizer,
    model=model,
    framework="pt",
)

print(pipe("very_long_file.mp3", chunk_length_s=10))

patrickvonplaten · 2022-09-09T15:02:21Z

Thanks for the PR - this all looks good to me! cc @sanchit-gandhi and @Narsil for a quick second review :-)

src/transformers/pipelines/automatic_speech_recognition.py

patrickvonplaten · 2022-09-09T15:04:15Z

src/transformers/pipelines/automatic_speech_recognition.py

@@ -258,7 +273,7 @@ def preprocess(self, inputs, chunk_length_s=0, stride_length_s=None):
            # XXX: Carefuly, this variable will not exist in `seq2seq` setting.
            # Currently chunking is not possible at this level for `seq2seq` so
            # it's ok.
-            align_to = self.model.config.inputs_to_logits_ratio
+            align_to = getattr(self.model.config, "inputs_to_logits_ratio", 1)


@Narsil ok for you or should we add 1 to the model's config?

I personally wouldn't add 1 to the config because we actually don't know what the "inputs to logits ratio" is. In other models like wav2vec, inputs_to_logits_ratio is a property that calculates the product of the strides of the convolutional layers, but we can't do that with M-CTC-T. As far as I can tell, it needs to be calculated dynamically.

This is assigning align_to, which is used on the subsequent lines to try and make sure that the chunk length and stride lengths are even multiples of that ratio. But in the case we don't know what that ratio is yet, setting align_to = 1 simply aligns the chunk length and stride lengths to the nearest frame instead.

patrickvonplaten · 2022-09-09T15:05:36Z

src/transformers/pipelines/automatic_speech_recognition.py

    # Shape is [B, SEQ] for tokens
    # [B, SEQ, V] for logits

    new_strides = []
-    for input_n, left, right in stride:
-        token_n = int(round(input_n * ratio))
+    for (input_n, left, right), stride_ratio in zip(stride, ratio):


Ok for me! Thanks for making it backward compatible. @Narsil what do you think?

patrickvonplaten · 2022-09-09T15:06:34Z

src/transformers/pipelines/automatic_speech_recognition.py

+                # If the proportion of input feature frames to logit frames can be statically
+                # determined based on convolution strides, use that. Otherwise, we need to
+                # calculate it here dynamically.
+                inputs_to_logits_ratio = getattr(self.model.config, "inputs_to_logits_ratio", None)


Hmm this is a bit inconsistent with the line above where the default is 1 and not None

@Narsil could you take a look here?

Replied to your comment above to clarify the inconsistency but basically:

Above we're assigning align_to, and if we don't know the inputs-to-logits-ratio, 1 is the only reasonable choice (that I can think of)

But here we're actually rescaling the strides, and so if we don't know the inputs-to-logits-ratio, we'll need to calculate it

patrickvonplaten · 2022-09-09T15:08:38Z

Thanks for the PR - that's a great addition! Generally this all looks more or less all good to me! Just a bit unsure about the changes in the pipeline, but ok for me since all tests pass. @Narsil what do you think?

Also cc @sanchit-gandhi for info

sanchit-gandhi · 2022-09-12T14:41:21Z

Thanks for the PR @samwaterbury! LGTM - happy with the changes to computing the inputs : logits ratio :-)

patrickvonplaten · 2022-09-13T23:32:36Z

@Narsil if you have 10 minutes it'd be super nice to get your review here :-)

samwaterbury · 2022-09-22T23:44:19Z

Thanks all! 🙂 @Narsil any chance for your 👀

samwaterbury added 2 commits Sep 1, 2022

Fix chunking for M-CTC-T

3685f46

Fix batching and add test

1bda51e

samwaterbury reviewed Sep 9, 2022

View changes

patrickvonplaten reviewed Sep 9, 2022

View changes

src/transformers/pipelines/automatic_speech_recognition.py Outdated Show resolved Hide resolved

patrickvonplaten reviewed Sep 9, 2022

View changes

src/transformers/pipelines/automatic_speech_recognition.py Outdated Show resolved Hide resolved

patrickvonplaten reviewed Sep 9, 2022

View changes

Rename input_values to inputs

82041c0

Fix M-CTC-T chunking #18949

Fix M-CTC-T chunking #18949

samwaterbury commented Sep 9, 2022

samwaterbury Sep 9, 2022

samwaterbury Sep 9, 2022

samwaterbury Sep 9, 2022

patrickvonplaten Sep 9, 2022

samwaterbury Sep 9, 2022

samwaterbury Sep 9, 2022

patrickvonplaten Sep 9, 2022

samwaterbury Sep 9, 2022 •

edited

patrickvonplaten Sep 9, 2022

HuggingFaceDocBuilderDev commented Sep 9, 2022

samwaterbury commented Sep 9, 2022 •

edited

patrickvonplaten commented Sep 9, 2022

patrickvonplaten Sep 9, 2022

samwaterbury Sep 9, 2022

patrickvonplaten Sep 9, 2022

patrickvonplaten Sep 9, 2022

patrickvonplaten Sep 9, 2022

samwaterbury Sep 9, 2022

patrickvonplaten commented Sep 9, 2022 •

edited

sanchit-gandhi commented Sep 12, 2022

patrickvonplaten commented Sep 13, 2022

samwaterbury commented Sep 22, 2022

		@@ -30,18 +30,33 @@
		from ..models.auto.modeling_auto import MODEL_FOR_CTC_MAPPING, MODEL_FOR_SPEECH_SEQ_2_SEQ_MAPPING

		outputs = self.model(input_values=input_values, attention_mask=attention_mask)
		outputs = self.model(input_values, attention_mask=attention_mask)

Fix M-CTC-T chunking #18949

Are you sure you want to change the base?

Fix M-CTC-T chunking #18949

Conversation

samwaterbury commented Sep 9, 2022

Summary

Testing

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

samwaterbury Sep 9, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

HuggingFaceDocBuilderDev commented Sep 9, 2022

samwaterbury commented Sep 9, 2022 • edited

patrickvonplaten commented Sep 9, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

patrickvonplaten commented Sep 9, 2022 • edited

sanchit-gandhi commented Sep 12, 2022

patrickvonplaten commented Sep 13, 2022

samwaterbury commented Sep 22, 2022

samwaterbury Sep 9, 2022 •

edited

samwaterbury commented Sep 9, 2022 •

edited

patrickvonplaten commented Sep 9, 2022 •

edited