WhisperTextStreamer token_ids must be a non-empty array of integers

[SpeedyGonzaless]

System Info

@huggingface/transformers 3.4.2

Environment/Platform

• Website/web-app
• Browser extension
• Server-side (e.g., Node.js, Deno, Bun)
• Desktop app (e.g., Electron)
• Other (e.g., VSCode extension)

Description

I am using AutomaticSpeechRecognitionPipeline (automatic-speech-recognition) and when I try to define new WhisperTextStreamer using tokenizer from this pipeline I get error:
"token_ids must be a non-empty array of integers"

This problem was not happening on versions before 3.4.0

Reproduction

Define pipeline:

const transcriber = pipeline('automatic-speech-recognition', 'onnx-community/whisper-small', {
                dtype: {
                    encoder_model:
                        this.model === "onnx-community/whisper-large-v3-turbo"
                            ? "fp16"
                            : "fp32",
                    decoder_model_merged: 'q4',
                },
                device: 'webgpu',
                progress_callback,
            });

And then try to define WhesperTextStreamer:

const streamer = new WhisperTextStreamer(transcriber.tokenizer, {
        time_precision,
        on_chunk_start: (x) => {
            const offset = (chunk_length_s - stride_length_s) * chunk_count;
            chunks.push({
                text: "",
                timestamp: [offset + x, null],
                finalised: false,
                offset,
            });
        },
        token_callback_function: () => {
            start_time = start_time || performance.now();
            if (num_tokens++ > 0) {
                tps = (num_tokens / (performance.now() - start_time)) * 1000;
            }
        },
        callback_function: (x) => {
            if (chunks.length === 0) return;
            chunks.at(-1).text += x;
            console.log('chunk', chunks.at(-1).text);
            chrome.runtime.sendMessage({
                status: 'update',
                data: {chunks, tps},
            });
        },
        on_chunk_end: (x) => {
            const current = chunks.at(-1);
            current.timestamp[1] = x + current.offset;
            current.finalised = true;
        },
        on_finalize: () => {
            start_time = null;
            num_tokens = 0;
            chunk_count++;
        },
    });

[xenova]

Hi there 👋

and when I try to define new WhisperTextStreamer using tokenizer from this pipeline I get error:
"token_ids must be a non-empty array of integers"

Does this mean the error occurs at construction, or when running the pipeline the first time?

There may be an edge-case where the model stops generating, but we attempt to decode (leading to an empty input), which I can try investigate. Do you have sample input or input file that causes this error?

---

One possibility (unlikely) is that it may be the case that you are not awaiting the creation of the pipeline? The error message would be a bit strange in this case though 🤔

- const transcriber = pipeline('automatic-speech-recognition', 'onnx-community/whisper-small', {
+ const transcriber = await pipeline('automatic-speech-recognition', 'onnx-community/whisper-small', {
// ...
const streamer = new WhisperTextStreamer(transcriber.tokenizer, {
// ...

[SpeedyGonzaless]

Hi there 👋

and when I try to define new WhisperTextStreamer using tokenizer from this pipeline I get error:
"token_ids must be a non-empty array of integers"

Does this mean the error occurs at construction, or when running the pipeline the first time?

There may be an edge-case where the model stops generating, but we attempt to decode (leading to an empty input), which I can try investigate. Do you have sample input or input file that causes this error?

One possibility (unlikely) is that it may be the case that you are not awaiting the creation of the pipeline? The error message would be a bit strange in this case though 🤔

• const transcriber = pipeline('automatic-speech-recognition', 'onnx-community/whisper-small', {

• const transcriber = await pipeline('automatic-speech-recognition', 'onnx-community/whisper-small', {
  // ...
  const streamer = new WhisperTextStreamer(transcriber.tokenizer, {
  // ...

It reproduces during running the pipeline.
You can reproduce it using your repository (https://github.com/xenova/whisper-web/tree/experimental-webgpu)
Just update "@huggingface/transformers" to version "3.5.0" and use any file. I used this one:

video_en.mp4

[PierreMesure]

Update: I just checked with 3.5.1 and the problem is still not solved.

@xenova, @fs-eire, @guschmue if you want to reproduce, you can try with my fork of Whisper-web and upgrade from 3.3.3 to any newer version. Any audio file and Whisper model will cause the problem.

Related issue: xenova/whisper-web#60