Autoscaling for whisper

[Franck Tanoh]

Hi everyone! 

Has anyone in the community implemented AWS autoscaling for Whisper GPU? 

We've now moved to whisper for our transcriptions, but since processing is between 9am-6pm, it's a waste of money leaving AWS GPU instances there with not much to do.

Happy to hear your thoughts. 

Thanks,

Franck

[Franck Tanoh]

Hi everyone!

We've decided to go the lambda function way, a python script to trigger the autoscaling based on whisper job type. 

The issue we have though is that opencast does not create for example whisper jobs if whisper service does not exist (correct me if I'm wrong), meaning we have to have at least 1 GPU running all the time. 

Is there a way to create a kind of dummy whisper service, which doesn't process jobs but allows opencast to at least queue the jobs. That way, the autoscaling will come in and process the job.

Alternately, if you have some ideas on how we can trigger the GPU based on transcription requests or jobs, please comment. 

Thanks, 

Franck

[Katrin Ihler]

Hi Franck,

I guess you could so some trickery with job loads - have a whisper service running on a node with a rather low load, and set the job load to be too high to actually be dispatched to that node or something? I think it's possible, but it ain't gonna be pretty.

Katrin

Opencast-DevOps Teamlead
elan e.V.
Karlstr. 23
D-26123 Oldeburg

elan-ev.de

To unsubscribe from this group and stop receiving emails from it, send an email to users+un...@opencast.org.

[Matthias Neugebauer]

Hi Franck,

Maybe the exec-* operations can help with that? Other people are using that to trigger Whisper jobs outside of Opencast workers as well.

Best regards
Matthias Neugebauer

---

shio solutions GmbH
Hafenweg 11a
48155 Münster
Germany

Mail: mtn...@shio.solutions
Fon: +49 251 97007987
Website: https://shio.solutions

Geschäftsführer: Matthias Neugebauer
Handelsregister: Amtsgericht Münster HRB 21404
Sitz: Münster

> On 1. Oct 2024, at 17:05, Franck Tanoh <fta...@gmail.com> wrote:
>
> Hi everyone!
>
> We've decided to go the lambda function way, a python script to trigger the autoscaling based on whisper job type.
> The issue we have though is that opencast does not create for example whisper jobs if whisper service does not exist (correct me if I'm wrong), meaning we have to have at least 1 GPU running all the time.
> Is there a way to create a kind of dummy whisper service, which doesn't process jobs but allows opencast to at least queue the jobs. That way, the autoscaling will come in and process the job.
>
> Alternately, if you have some ideas on how we can trigger the GPU based on transcription requests or jobs, please comment.
>
> Thanks,
> Franck
>
>
>
>
>
> On Mon, Sep 16, 2024 at 1:26 PM Franck Tanoh <fta...@gmail.com> wrote:
> Hi everyone!
>
> Has anyone in the community implemented AWS autoscaling for Whisper GPU?
> We've now moved to whisper for our transcriptions, but since processing is between 9am-6pm, it's a waste of money leaving AWS GPU instances there with not much to do.
>
> Happy to hear your thoughts.
>
> Thanks,
> Franck
>
>

[Lars Kiesow]

Yes, I think the execute operation is a good way if you want to script
that. The alternative would be to build that in similar to the other
cloud transcription services.
–Lars

[Franck Tanoh]

Thanks all for your reply. 

We tried setting the job load very high, but Opencast being too clever :-) will just reset it to maximum job load based on the cpu core.

Can you please expand on how the 'exec' can be used? Any example? 

Thanks 

Franck

[Greg Logan]

A note for anyone considering doing something like this with current (16.x era and older) codebases: Don't.  You're going to run into all kinds of unexpected behaviour:

https://github.com/opencast/opencast/issues/5837

G

[Christoph Reifers]

Hi Franck,

have a look here: https://github.com/opencast/helper-scripts/tree/master/whisper-AI-exec-script

-- Christoph

[Franck Tanoh]

Very good example, thanks Christoph.

Franck