Residual adapters in sherpa-onnx for streaming ASR

[aanchan]

I am working on a project that needs highly personalized models, but per speaker data is limited. In the literature, residual adapters are popular for the problem I am working on to be trained on a per speaker (or a per cohort basis, assuming a speaker is from a particular cohort having a certain kind of atypicality). Whisper models allow for adapters and with the PEFT library, adapter merging is trivial. With the 30s delay though I understand Whisper is not for streaming. NeMO models are also supported by sherpa-onnx and do come with adapter support: https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/core/adapters/intro.html. I've played around with adapting a couple of streaming models. I wanted to know what approach might make sense to incorporate adapters:

1. Modify the model export and sherpa-onnx runtime to allow for model adapters?
2. Write a custom script with NeMO to merge adapters and then leave the sherpa-onnx runtime untouched.

Any advice would be appreciated. Is there any past experiments with adapters on sherpa-onnx or anywhere on the roadmap?

[csukuangfj]

Is it for training or inference?

[aanchan]

Training affects the inference. I do want to use adapters in training. Once trained for inference one has the option of merging the adapters as in PEFT with Whisper ( https://t.co/cAVf1OBhYZ), in this case sherpa-onnx runtime needs no modification. Whisper isn't great for streaming.

For models like NeMO merging of adapters into the model isn't supported. So one would need to 1. modify the sherpa-onnx runtime to support adapters OR 2. Write custom NeMO model merging. I wanted to know whether sherpa-onnx will support adapters in runtime or if we should do 2. Work with Nemo and write up custom merge scripts prior to inference?

[csukuangfj]

If you can open-source an onnx model and a python script showing how to use this model, we can support it in sherpa-onnx.
(Provided that the model has reasonable accuracy and can benefit the community)