Fail to pickup previously encoded audio file with speech-to-text-woh
84 views
Skip to first unread message
Martin Schamberger
May 23, 2023, 3:48:33 AM5/23/23
to Opencast Users
Hi,
I'm trying to add a WhisperC++ engine to the speech-to-text-impl. Different to classical Whisper you have to feed a wav file as input to WhisperC++ - therefore I have an encode-job before running speech-to-text:
```
id: common-partial-publish-enrich-stt
title: Operations for speech-to-text
operations:
- id: encode
# fail-on-error: true
exception-handler-workflow: common-partial-error
description: " └─ * Extract wav audio for speech to text"
configurations:
- source-flavor: "*/themed"
- target-flavor: "*/audio+stt"
- encoding-profile: audio-whisper
- target-tags: archive
- id: speechtotext
# fail-on-error: true
exception-handler-workflow: common-partial-error
description: " └─ * Generates subtitles for video and audio files (whispercpp)"
configurations:
- source-flavor: "*/audio+stt"
- target-flavor: captions/vtt+auto-#{lang}
- target-element: attachment
- language-code: de
- target-tags: archive,subtitle,engage-download
```
The wav file is encoded, but the speechtotext-woh cannot pick up the wav file:
```
2023-05-23T08:49:05,098 | INFO | (ComposerServiceImpl:572) - Starting parallel encode with profile audio-whisper with job load 1
2023-05-23T08:49:07,139 | INFO | (EncoderEngine:216) - Executing encoding command: [ffmpeg, -nostdin, -nostats, -i, /data/opencast/staging/workspace/mediapackage/47fb375d-5ad8-4a6f-9a6e-a2237aa73a42/e9494333-cf33-498d-af52-1a9932e9cf2a/e9494333-cf33-498d-af52-1a9932e9cf2a.mp4, -vn, -ar, 16000, -ac, 1, -c:a, pcm_s16le, /data/opencast/staging/workspace/mediapackage/47fb375d-5ad8-4a6f-9a6e-a2237aa73a42/e9494333-cf33-498d-af52-1a9932e9cf2a/e9494333-cf33-498d-af52-1a9932e9cf2a_359d202b-f87a-4619-87c3-5e2ebc2d08e7-stt.wav]
2023-05-23T08:49:07,163 | INFO | (EncoderEngine:491) - vendor_id : [0][0][0][0]
2023-05-23T08:49:07,163 | INFO | (EncoderEngine:491) - vendor_id : [0][0][0][0]
2023-05-23T08:49:07,168 | INFO | (EncoderEngine:460) - Identified output file /data/opencast/staging/workspace/mediapackage/47fb375d-5ad8-4a6f-9a6e-a2237aa73a42/e9494333-cf33-498d-af52-1a9932e9cf2a/e9494333-cf33-498d-af52-1a9932e9cf2a_359d202b-f87a-4619-87c3-5e2ebc2d08e7-stt.wav
2023-05-23T08:49:07,168 | INFO | (EncoderEngine:491) - ISFT : Lavf60.3.100
2023-05-23T08:49:07,168 | INFO | (EncoderEngine:491) - vendor_id : [0][0][0][0]
2023-05-23T08:49:07,356 | INFO | (EncoderEngine:240) - Tracks {video=/data/opencast/staging/workspace/mediapackage/47fb375d-5ad8-4a6f-9a6e-a2237aa73a42/e9494333-cf33-498d-af52-1a9932e9cf2a/e9494333-cf33-498d-af52-1a9932e9cf2a.mp4} successfully encoded using profile 'audio-whisper'
2023-05-23T08:49:07,512 | INFO | (ComposerServiceImpl:533) - Copied the encoded file to the workspace at https://admin.oc.univie.ac.at/files/collection/composer/522538_0.wav
2023-05-23T08:49:12,610 | INFO | (ComposerServiceImpl:553) - Deleted the local copy of the encoded file at /data/opencast/staging/workspace/mediapackage/47fb375d-5ad8-4a6f-9a6e-a2237aa73a42/e9494333-cf33-498d-af52-1a9932e9cf2a/e9494333-cf33-498d-af52-1a9932e9cf2a_359d202b-f87a-4619-87c3-5e2ebc2d08e7-stt.wav
2023-05-23T08:49:19,432 | INFO | (WhisperCppEngine:130) - Executing WhisperC++'s transcription command: [whispercpp, -ovtt, -bs 5, --model, /usr/share/ggml/ggml-base.bin, --output-file, /data/opencast/staging/workspace/collection/subtitles/tmp_522540_e9494333-cf33-498d-af52-1a9932e9cf2a.vtt, -l, de, -f, /data/opencast/staging/workspace/mediapackage/47fb375d-5ad8-4a6f-9a6e-a2237aa73a42/f43a7a1b-8d9e-4c69-a63d-0f652761c238/e9494333-cf33-498d-af52-1a9932e9cf2a.wav]
2023-05-23T08:49:19,472 | INFO | (ServiceRegistryJpaImpl:2266) - State set to WARNING for current service org.opencastproject.speechtotext on host https://oc-worker2.oc.univie.ac.at
2023-05-23T08:49:19,527 | ERROR | (AbstractJobProducer$JobRunner:343) - Error handling operation 'speechtotext':
org.opencastproject.speechtotext.api.SpeechToTextServiceException: Error while generating subtitle from https://admin.oc.univie.ac.at/files/mediapackage/47fb375d-5ad8-4a6f-9a6e-a2237aa73a42/f43a7a1b-8d9e-4c69-a63d-0f652761c238/e9494333-cf33-498d-af52-1a9932e9cf2a.wav
at org.opencastproject.speechtotext.impl.SpeechToTextServiceImpl.process(SpeechToTextServiceImpl.java:165) ~[?:?]
at org.opencastproject.job.api.AbstractJobProducer$JobRunner.call(AbstractJobProducer.java:313) [!/:?]
at org.opencastproject.job.api.AbstractJobProducer$JobRunner.call(AbstractJobProducer.java:272) [!/:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: org.opencastproject.speechtotext.api.SpeechToTextEngineException: org.opencastproject.speechtotext.api.SpeechToTextEngineException: WhisperC++ produced no output
at org.opencastproject.speechtotext.impl.engine.WhisperCppEngine.generateSubtitlesFile(WhisperCppEngine.java:155) ~[?:?]
at org.opencastproject.speechtotext.impl.SpeechToTextServiceImpl.process(SpeechToTextServiceImpl.java:155) ~[?:?]
... 6 more
Caused by: org.opencastproject.speechtotext.api.SpeechToTextEngineException: WhisperC++ produced no output
at org.opencastproject.speechtotext.impl.engine.WhisperCppEngine.generateSubtitlesFile(WhisperCppEngine.java:150) ~[?:?]
at org.opencastproject.speechtotext.impl.SpeechToTextServiceImpl.process(SpeechToTextServiceImpl.java:155) ~[?:?]
... 6 more
```
The whispercpp command's input does not exist:
```
[opencast@oc-worker2 /]$ ls /data/opencast/staging/workspace/mediapackage/47fb375d-5ad8-4a6f-9a6e-a2237aa73a42/f43a7a1b-8d9e-4c69-a63d-0f652761c238/e9494333-cf33-498d-af52-1a9932e9cf2a.wav
ls: Zugriff auf '/data/opencast/staging/workspace/mediapackage/47fb375d-5ad8-4a6f-9a6e-a2237aa73a42/f43a7a1b-8d9e-4c69-a63d-0f652761c238/e9494333-cf33-498d-af52-1a9932e9cf2a.wav' nicht möglich: No such file or directory
```
The command runs successfully when feeding an existing wav-file:
```
[opencast@oc-worker2 ~]$ whispercpp -ovtt -bs 5 --model /usr/share/ggml/ggml-base.bin \
--output-file /data/opencast/staging/workspace/collection/subtitles/tmp_522540_e9494333-cf33-498d-af52-1a9932e9cf2a.vtt \
-l de \
-f /data/opencast/staging/archive/u_stream/9f24e806-6716-4606-b8ce-4e274e13ad16/3/3a834f68-28d7-4c3b-815a-aa65d9cf0603.wav
whisper_init_from_file_no_state: loading model from '/usr/share/ggml/ggml-base.bin'
whisper_model_load: loading model
[...]
output_vtt: saving output to '/data/opencast/staging/workspace/collection/subtitles/tmp_522540_e9494333-cf33-498d-af52-1a9932e9cf2a.vtt.vtt'
```
I'm trying to add a WhisperC++ engine to the speech-to-text-impl. Different to classical Whisper you have to feed a wav file as input to WhisperC++ - therefore I have an encode-job before running speech-to-text:
```
id: common-partial-publish-enrich-stt
title: Operations for speech-to-text
operations:
- id: encode
# fail-on-error: true
exception-handler-workflow: common-partial-error
description: " └─ * Extract wav audio for speech to text"
configurations:
- source-flavor: "*/themed"
- target-flavor: "*/audio+stt"
- encoding-profile: audio-whisper
- target-tags: archive
- id: speechtotext
# fail-on-error: true
exception-handler-workflow: common-partial-error
description: " └─ * Generates subtitles for video and audio files (whispercpp)"
configurations:
- source-flavor: "*/audio+stt"
- target-flavor: captions/vtt+auto-#{lang}
- target-element: attachment
- language-code: de
- target-tags: archive,subtitle,engage-download
```
The wav file is encoded, but the speechtotext-woh cannot pick up the wav file:
```
2023-05-23T08:49:05,098 | INFO | (ComposerServiceImpl:572) - Starting parallel encode with profile audio-whisper with job load 1
2023-05-23T08:49:07,139 | INFO | (EncoderEngine:216) - Executing encoding command: [ffmpeg, -nostdin, -nostats, -i, /data/opencast/staging/workspace/mediapackage/47fb375d-5ad8-4a6f-9a6e-a2237aa73a42/e9494333-cf33-498d-af52-1a9932e9cf2a/e9494333-cf33-498d-af52-1a9932e9cf2a.mp4, -vn, -ar, 16000, -ac, 1, -c:a, pcm_s16le, /data/opencast/staging/workspace/mediapackage/47fb375d-5ad8-4a6f-9a6e-a2237aa73a42/e9494333-cf33-498d-af52-1a9932e9cf2a/e9494333-cf33-498d-af52-1a9932e9cf2a_359d202b-f87a-4619-87c3-5e2ebc2d08e7-stt.wav]
2023-05-23T08:49:07,163 | INFO | (EncoderEngine:491) - vendor_id : [0][0][0][0]
2023-05-23T08:49:07,163 | INFO | (EncoderEngine:491) - vendor_id : [0][0][0][0]
2023-05-23T08:49:07,168 | INFO | (EncoderEngine:460) - Identified output file /data/opencast/staging/workspace/mediapackage/47fb375d-5ad8-4a6f-9a6e-a2237aa73a42/e9494333-cf33-498d-af52-1a9932e9cf2a/e9494333-cf33-498d-af52-1a9932e9cf2a_359d202b-f87a-4619-87c3-5e2ebc2d08e7-stt.wav
2023-05-23T08:49:07,168 | INFO | (EncoderEngine:491) - ISFT : Lavf60.3.100
2023-05-23T08:49:07,168 | INFO | (EncoderEngine:491) - vendor_id : [0][0][0][0]
2023-05-23T08:49:07,356 | INFO | (EncoderEngine:240) - Tracks {video=/data/opencast/staging/workspace/mediapackage/47fb375d-5ad8-4a6f-9a6e-a2237aa73a42/e9494333-cf33-498d-af52-1a9932e9cf2a/e9494333-cf33-498d-af52-1a9932e9cf2a.mp4} successfully encoded using profile 'audio-whisper'
2023-05-23T08:49:07,512 | INFO | (ComposerServiceImpl:533) - Copied the encoded file to the workspace at https://admin.oc.univie.ac.at/files/collection/composer/522538_0.wav
2023-05-23T08:49:12,610 | INFO | (ComposerServiceImpl:553) - Deleted the local copy of the encoded file at /data/opencast/staging/workspace/mediapackage/47fb375d-5ad8-4a6f-9a6e-a2237aa73a42/e9494333-cf33-498d-af52-1a9932e9cf2a/e9494333-cf33-498d-af52-1a9932e9cf2a_359d202b-f87a-4619-87c3-5e2ebc2d08e7-stt.wav
2023-05-23T08:49:19,432 | INFO | (WhisperCppEngine:130) - Executing WhisperC++'s transcription command: [whispercpp, -ovtt, -bs 5, --model, /usr/share/ggml/ggml-base.bin, --output-file, /data/opencast/staging/workspace/collection/subtitles/tmp_522540_e9494333-cf33-498d-af52-1a9932e9cf2a.vtt, -l, de, -f, /data/opencast/staging/workspace/mediapackage/47fb375d-5ad8-4a6f-9a6e-a2237aa73a42/f43a7a1b-8d9e-4c69-a63d-0f652761c238/e9494333-cf33-498d-af52-1a9932e9cf2a.wav]
2023-05-23T08:49:19,472 | INFO | (ServiceRegistryJpaImpl:2266) - State set to WARNING for current service org.opencastproject.speechtotext on host https://oc-worker2.oc.univie.ac.at
2023-05-23T08:49:19,527 | ERROR | (AbstractJobProducer$JobRunner:343) - Error handling operation 'speechtotext':
org.opencastproject.speechtotext.api.SpeechToTextServiceException: Error while generating subtitle from https://admin.oc.univie.ac.at/files/mediapackage/47fb375d-5ad8-4a6f-9a6e-a2237aa73a42/f43a7a1b-8d9e-4c69-a63d-0f652761c238/e9494333-cf33-498d-af52-1a9932e9cf2a.wav
at org.opencastproject.speechtotext.impl.SpeechToTextServiceImpl.process(SpeechToTextServiceImpl.java:165) ~[?:?]
at org.opencastproject.job.api.AbstractJobProducer$JobRunner.call(AbstractJobProducer.java:313) [!/:?]
at org.opencastproject.job.api.AbstractJobProducer$JobRunner.call(AbstractJobProducer.java:272) [!/:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: org.opencastproject.speechtotext.api.SpeechToTextEngineException: org.opencastproject.speechtotext.api.SpeechToTextEngineException: WhisperC++ produced no output
at org.opencastproject.speechtotext.impl.engine.WhisperCppEngine.generateSubtitlesFile(WhisperCppEngine.java:155) ~[?:?]
at org.opencastproject.speechtotext.impl.SpeechToTextServiceImpl.process(SpeechToTextServiceImpl.java:155) ~[?:?]
... 6 more
Caused by: org.opencastproject.speechtotext.api.SpeechToTextEngineException: WhisperC++ produced no output
at org.opencastproject.speechtotext.impl.engine.WhisperCppEngine.generateSubtitlesFile(WhisperCppEngine.java:150) ~[?:?]
at org.opencastproject.speechtotext.impl.SpeechToTextServiceImpl.process(SpeechToTextServiceImpl.java:155) ~[?:?]
... 6 more
```
The whispercpp command's input does not exist:
```
[opencast@oc-worker2 /]$ ls /data/opencast/staging/workspace/mediapackage/47fb375d-5ad8-4a6f-9a6e-a2237aa73a42/f43a7a1b-8d9e-4c69-a63d-0f652761c238/e9494333-cf33-498d-af52-1a9932e9cf2a.wav
ls: Zugriff auf '/data/opencast/staging/workspace/mediapackage/47fb375d-5ad8-4a6f-9a6e-a2237aa73a42/f43a7a1b-8d9e-4c69-a63d-0f652761c238/e9494333-cf33-498d-af52-1a9932e9cf2a.wav' nicht möglich: No such file or directory
```
The command runs successfully when feeding an existing wav-file:
```
[opencast@oc-worker2 ~]$ whispercpp -ovtt -bs 5 --model /usr/share/ggml/ggml-base.bin \
--output-file /data/opencast/staging/workspace/collection/subtitles/tmp_522540_e9494333-cf33-498d-af52-1a9932e9cf2a.vtt \
-l de \
-f /data/opencast/staging/archive/u_stream/9f24e806-6716-4606-b8ce-4e274e13ad16/3/3a834f68-28d7-4c3b-815a-aa65d9cf0603.wav
whisper_init_from_file_no_state: loading model from '/usr/share/ggml/ggml-base.bin'
whisper_model_load: loading model
[...]
output_vtt: saving output to '/data/opencast/staging/workspace/collection/subtitles/tmp_522540_e9494333-cf33-498d-af52-1a9932e9cf2a.vtt.vtt'
```
Therefore I guess it's not an error in the whispercpp-engine itself but a misconfiguration of the workflow?
Any suggestions?
Best regards, Martin
Maximiliano Lira Del Canto
May 23, 2023, 4:29:48 AM5/23/23
to Opencast Users, majo...@gmail.com
Hi Martin,
I can explain how the whisper (original) module works.
How The original whisper works:
When you use Whisper STT WoH, Opencast requires two files, the VTT and the JSON file.
The VTT is the transcription and the JSON is used for tagging the language.
The JSON file is not required when:
- You set the language beforehand.
- You activate the translation to English.
Now, the files that opencast search are:
{{Opencast_workspace_path}}/{{Name_of_file}}.vtt
{{Opencast_workspace_path}}/{{Name_of_file}}.json
In the example from whisper c++ that you are showing, the output file is tmp_522540_e9494333-cf33-498d-af52-1a9932e9cf2a.vtt.vtt <-- There is a double file ending.
The subtitle files in opencast are deleted because the cleaning operation deletes it from the workspace.
I strongly recommend you to use the debugger for this case.
Martin Schamberger
May 30, 2023, 3:19:19 PM5/30/23
to Opencast Users, Maximiliano Lira Del Canto, Martin Schamberger
Hi,
thank you for your explanation and hints!
You're right, I was messing with the output path. I've made a second attempt inlcuding PR 4869.
The whispercpp cli differs to some extent from the whisper cli - especially awaiting an output file basename (without file extension) instead of an output dir - see attached txt.
whispercpp also supports json output. So the whispercpp engine is mostly copy&pasted from the whisper and vosk engines (plus renaming variables and adjusting command parameters).
Output file path should now be fixed, but I'm still having the issue, that the whispercpp command does not seem to execute - although an exit code 0 is returned.
I've tried to debug this on our Opencast test instance (multi-node):
Logs admin:
2023-05-30T19:12:42,740 | INFO | (SpeechToTextWorkflowOperationHandler:128) - Start speech-to-text workflow operation for media package 7c758e34-a6aa-4211-ba5c-d04d6f7e863e
2023-05-30T19:12:42,740 | INFO | (SpeechToTextWorkflowOperationHandler:141) - Found 1 track(s) with source flavor '*/audio+stt'.
2023-05-30T19:12:42,740 | INFO | (SpeechToTextWorkflowOperationHandler:178) - Generating subtitle for 'https://admin.oc.univie.ac.at/files/mediapackage/7c758e34-a6aa-4211-ba5c-d04d6f7e863e/4bd3e0a9-c205-4556-8521-55966c7e50ca/90737cb1-6529-4ae9-935d-b7bb75fd8ee2.wav'...
2023-05-30T19:12:42,740 | INFO | (SpeechToTextServiceRemoteImpl:78) - Generating subtitle for https://admin.oc.univie.ac.at/files/mediapackage/7c758e34-a6aa-4211-ba5c-d04d6f7e863e/4bd3e0a9-c205-4556-8521-55966c7e50ca/90737cb1-6529-4ae9-935d-b7bb75fd8ee2.wav
2023-05-30T19:12:42,771 | INFO | (SpeechToTextServiceRemoteImpl:88) - Completed transcription for https://admin.oc.univie.ac.at/files/mediapackage/7c758e34-a6aa-4211-ba5c-d04d6f7e863e/4bd3e0a9-c205-4556-8521-55966c7e50ca/90737cb1-6529-4ae9-935d-b7bb75fd8ee2.wav
2023-05-30T19:12:47,781 | ERROR | (WorkflowOperationWorker:140) - Workflow operation 'operation:'speechtotext, state:'FAILED'' failed
2023-05-30T19:12:42,740 | INFO | (SpeechToTextWorkflowOperationHandler:141) - Found 1 track(s) with source flavor '*/audio+stt'.
2023-05-30T19:12:42,740 | INFO | (SpeechToTextWorkflowOperationHandler:178) - Generating subtitle for 'https://admin.oc.univie.ac.at/files/mediapackage/7c758e34-a6aa-4211-ba5c-d04d6f7e863e/4bd3e0a9-c205-4556-8521-55966c7e50ca/90737cb1-6529-4ae9-935d-b7bb75fd8ee2.wav'...
2023-05-30T19:12:42,740 | INFO | (SpeechToTextServiceRemoteImpl:78) - Generating subtitle for https://admin.oc.univie.ac.at/files/mediapackage/7c758e34-a6aa-4211-ba5c-d04d6f7e863e/4bd3e0a9-c205-4556-8521-55966c7e50ca/90737cb1-6529-4ae9-935d-b7bb75fd8ee2.wav
2023-05-30T19:12:42,771 | INFO | (SpeechToTextServiceRemoteImpl:88) - Completed transcription for https://admin.oc.univie.ac.at/files/mediapackage/7c758e34-a6aa-4211-ba5c-d04d6f7e863e/4bd3e0a9-c205-4556-8521-55966c7e50ca/90737cb1-6529-4ae9-935d-b7bb75fd8ee2.wav
2023-05-30T19:12:47,781 | ERROR | (WorkflowOperationWorker:140) - Workflow operation 'operation:'speechtotext, state:'FAILED'' failed
Logs worker:
2023-05-30T19:12:44,815 | INFO | (WhisperCppEngine:132) - Using language de from workflows
2023-05-30T19:12:44,815 | INFO | (WhisperCppEngine:137) - Executing WhisperC++'s transcription command: [whispercpp, /data/opencast/staging/workspace/mediapackage/7c758e34-a6aa-4211-ba5c-d04d6f7e863e/4bd3e0a9-c205-4556-8521-55966c7e50ca/90737cb1-6529-4ae9-935d-b7bb75fd8ee2.wav, --model, /usr/share/ggml/ggml-base.bin, -ovtt, -oj, -bs 5, --output-file, /data/opencast/staging/workspace/collection/subtitles/tmp_567380_90737cb1-6529-4ae9-935d-b7bb75fd8ee2, --language, de]
2023-05-30T19:12:44,815 | INFO | (WhisperCppEngine:138) - preparedOutputFile: /data/opencast/staging/workspace/collection/subtitles/tmp_567380_90737cb1-6529-4ae9-935d-b7bb75fd8ee2.vtt
2023-05-30T19:12:44,815 | INFO | (WhisperCppEngine:139) - mediaFile: /data/opencast/staging/workspace/mediapackage/7c758e34-a6aa-4211-ba5c-d04d6f7e863e/4bd3e0a9-c205-4556-8521-55966c7e50ca/90737cb1-6529-4ae9-935d-b7bb75fd8ee2.wav
2023-05-30T19:12:44,815 | INFO | (WhisperCppEngine:140) - preparedOutputFileNoExt: /data/opencast/staging/workspace/collection/subtitles/tmp_567380_90737cb1-6529-4ae9-935d-b7bb75fd8ee2
2023-05-30T19:12:44,821 | INFO | (WhisperCppEngine:152) - WhisperC++ process finished with exit code 0
2023-05-30T19:12:44,822 | INFO | (WhisperCppEngine:168) - Transcription failed closing WhisperC++ transcription process for: /data/opencast/staging/workspace/mediapackage/7c758e34-a6aa-4211-ba5c-d04d6f7e863e/4bd3e0a9-c205-4556-8521-55966c7e50ca/90737cb1-6529-4ae9-935d-b7bb75fd8ee2.wav
2023-05-30T19:12:44,851 | INFO | (ServiceRegistryJpaImpl:2268) - State set to WARNING for current service org.opencastproject.speechtotext on host https://oc-worker2.oc.univie.ac.at
2023-05-30T19:12:44,890 | ERROR | (AbstractJobProducer$JobRunner:343) - Error handling operation 'speechtotext':
2023-05-30T19:12:44,815 | INFO | (WhisperCppEngine:137) - Executing WhisperC++'s transcription command: [whispercpp, /data/opencast/staging/workspace/mediapackage/7c758e34-a6aa-4211-ba5c-d04d6f7e863e/4bd3e0a9-c205-4556-8521-55966c7e50ca/90737cb1-6529-4ae9-935d-b7bb75fd8ee2.wav, --model, /usr/share/ggml/ggml-base.bin, -ovtt, -oj, -bs 5, --output-file, /data/opencast/staging/workspace/collection/subtitles/tmp_567380_90737cb1-6529-4ae9-935d-b7bb75fd8ee2, --language, de]
2023-05-30T19:12:44,815 | INFO | (WhisperCppEngine:138) - preparedOutputFile: /data/opencast/staging/workspace/collection/subtitles/tmp_567380_90737cb1-6529-4ae9-935d-b7bb75fd8ee2.vtt
2023-05-30T19:12:44,815 | INFO | (WhisperCppEngine:139) - mediaFile: /data/opencast/staging/workspace/mediapackage/7c758e34-a6aa-4211-ba5c-d04d6f7e863e/4bd3e0a9-c205-4556-8521-55966c7e50ca/90737cb1-6529-4ae9-935d-b7bb75fd8ee2.wav
2023-05-30T19:12:44,815 | INFO | (WhisperCppEngine:140) - preparedOutputFileNoExt: /data/opencast/staging/workspace/collection/subtitles/tmp_567380_90737cb1-6529-4ae9-935d-b7bb75fd8ee2
2023-05-30T19:12:44,821 | INFO | (WhisperCppEngine:152) - WhisperC++ process finished with exit code 0
2023-05-30T19:12:44,822 | INFO | (WhisperCppEngine:168) - Transcription failed closing WhisperC++ transcription process for: /data/opencast/staging/workspace/mediapackage/7c758e34-a6aa-4211-ba5c-d04d6f7e863e/4bd3e0a9-c205-4556-8521-55966c7e50ca/90737cb1-6529-4ae9-935d-b7bb75fd8ee2.wav
2023-05-30T19:12:44,851 | INFO | (ServiceRegistryJpaImpl:2268) - State set to WARNING for current service org.opencastproject.speechtotext on host https://oc-worker2.oc.univie.ac.at
2023-05-30T19:12:44,890 | ERROR | (AbstractJobProducer$JobRunner:343) - Error handling operation 'speechtotext':
So I'm a bit confused as the timestamp of the admin log line - stating subtitle generation has completed - is before the whispercpp command is composed and executed on the worker?
I'm not sure, if the whispercpp command is executed at all - at least it fails very fast - I'v never seen a glimpse of a whispercpp process in top.
When running the whispercpp command from the logs manually as opencast user on the worker-node (while debugging: stopping before the check of preparedOutputFile, run command manually and resume afterwards) - the workflow finishes (and subtitles are published to the player).
At the moment I have no idea what to check next - any suggestions?
Best regards,
Martin
Martin
PS: In case someone is interested in RPMs for whispercpp: https://github.com/elearning-univie/whispercpp-rpmbuild
Martin Schamberger
Jun 1, 2023, 10:19:43 AM6/1/23
to Opencast Users, Martin Schamberger, Maximiliano Lira Del Canto
Oops, the following conversation unintentionally got lost - I guess I have not clicked "Reply to all"?
---
Hello again,
many thanks for your help! 
I've just tested the whispercpp command as you suggested with the execute workflow operation handler:
```
- id: execute-once
fail-on-error: false
exception-handler-workflow: common-partial-error
description: " └─ * TEST (whispercpp)"
configurations:
- exec: whispercpp
- target-flavor: "captions/vtt+auto-de"
- params: >-
--model /usr/share/ggml/ggml-base.bin
-ovtt -oj -bs 5
--output-file /data/opencast/staging/workspace/collection/subtitles/tmp_#{id} #{flavor(presentation/audio+stt)}
- target-tags: >-
archive,
subtitle,
engage-download
fail-on-error: false
exception-handler-workflow: common-partial-error
description: " └─ * TEST (whispercpp)"
configurations:
- exec: whispercpp
- target-flavor: "captions/vtt+auto-de"
- params: >-
--model /usr/share/ggml/ggml-base.bin
-ovtt -oj -bs 5
--output-file /data/opencast/staging/workspace/collection/subtitles/tmp_#{id} #{flavor(presentation/audio+stt)}
- target-tags: >-
archive,
subtitle,
engage-download
```
The command executes and the vtt and json files are generated successfully:
```
2023-05-31T11:35:05,144 | DEBUG | (ServiceRegistryJpaImpl:1028) - Job {id:579421, operation:Execute_Mediapackage, status:RUNNING} has host 'https://oc-worker2.oc.univie.ac.at': setting processor service to 'org.opencastproject.execute@https://oc-worker2.oc.univie.ac.at'
2023-05-31T11:35:05,148 | DEBUG | (ServiceRegistryJpaImpl:904) - Adding to load cache: Job {id:579421, operation:Execute_Mediapackage, status:RUNNING}, type org.opencastproject.execute, load 1.0, status RUNNING
2023-05-31T11:35:05,149 | DEBUG | (ServiceRegistryJpaImpl:918) - Current host load: 1,0, job load cache size: 1
2023-05-31T11:35:05,149 | DEBUG | (AsyncTimeoutRedirectFilter:61) - Chain processed normally
2023-05-31T11:35:05,183 | DEBUG | (WorkspaceImpl:381) - /data/opencast/staging/workspace/mediapackage/9d5a60d0-2db2-4d42-a373-4d147292d430/5649b589-ccdb-4479-97ab-78013571da93/30cd4526-a613-465b-bfb2-2da960001d87.wav is up to date
2023-05-31T11:35:05,183 | DEBUG | (WorkspaceImpl:383) - Getting https://admin.oc.univie.ac.at/files/mediapackage/9d5a60d0-2db2-4d42-a373-4d147292d430/5649b589-ccdb-4479-97ab-78013571da93/30cd4526-a613-465b-bfb2-2da960001d87.wav directly from working file repository root at /data/opencast/staging/workspace/mediapackage/9d5a60d0-2db2-4d42-a373-4d147292d430/5649b589-ccdb-4479-97ab-78013571da93/30cd4526-a613-465b-bfb2-2da960001d87.wav
2023-05-31T11:35:05,184 | INFO | (ExecuteServiceImpl:481) - Running command whispercpp
2023-05-31T11:35:05,184 | DEBUG | (ExecuteServiceImpl:482) - Starting subprocess whispercpp with arguments --model, /usr/share/ggml/ggml-base.bin, -ovtt, -oj, -bs, 5, --output-file, /data/opencast/staging/workspace/collection/subtitles/tmp_9d5a60d0-2db2-4d42-a373-4d147292d430, /data/opencast/staging/workspace/mediapackage/9d5a60d0-2db2-4d42-a373-4d147292d430/5649b589-ccdb-4479-97ab-78013571da93/30cd4526-a613-465b-bfb2-2da960001d87.wav
2023-05-31T11:35:05,189 | DEBUG | (ExecuteServiceImpl:492) - whisper_init_from_file_no_state: loading model from '/usr/share/ggml/ggml-base.bin'
2023-05-31T11:35:05,189 | DEBUG | (ExecuteServiceImpl:492) - whisper_model_load: loading model
2023-05-31T11:35:05,189 | DEBUG | (ExecuteServiceImpl:492) - whisper_model_load: n_vocab = 51865
2023-05-31T11:35:05,189 | DEBUG | (ExecuteServiceImpl:492) - whisper_model_load: n_audio_ctx = 1500
2023-05-31T11:35:05,189 | DEBUG | (ExecuteServiceImpl:492) - whisper_model_load: n_audio_state = 512
2023-05-31T11:35:05,189 | DEBUG | (ExecuteServiceImpl:492) - whisper_model_load: n_audio_head = 8
2023-05-31T11:35:05,189 | DEBUG | (ExecuteServiceImpl:492) - whisper_model_load: n_audio_layer = 6
2023-05-31T11:35:05,189 | DEBUG | (ExecuteServiceImpl:492) - whisper_model_load: n_text_ctx = 448
2023-05-31T11:35:05,189 | DEBUG | (ExecuteServiceImpl:492) - whisper_model_load: n_text_state = 512
2023-05-31T11:35:05,189 | DEBUG | (ExecuteServiceImpl:492) - whisper_model_load: n_text_head = 8
2023-05-31T11:35:05,189 | DEBUG | (ExecuteServiceImpl:492) - whisper_model_load: n_text_layer = 6
2023-05-31T11:35:05,189 | DEBUG | (ExecuteServiceImpl:492) - whisper_model_load: n_mels = 80
2023-05-31T11:35:05,189 | DEBUG | (ExecuteServiceImpl:492) - whisper_model_load: ftype = 1
2023-05-31T11:35:05,189 | DEBUG | (ExecuteServiceImpl:492) - whisper_model_load: qntvr = 0
2023-05-31T11:35:05,189 | DEBUG | (ExecuteServiceImpl:492) - whisper_model_load: type = 2
2023-05-31T11:35:05,189 | DEBUG | (ExecuteServiceImpl:492) - whisper_model_load: mem required = 310.00 MB (+ 6.00 MB per decoder)
2023-05-31T11:35:05,248 | DEBUG | (ExecuteServiceImpl:492) - whisper_model_load: adding 1608 extra tokens
2023-05-31T11:35:05,249 | DEBUG | (ExecuteServiceImpl:492) - whisper_model_load: model ctx = 140.66 MB
2023-05-31T11:35:05,291 | DEBUG | (ExecuteServiceImpl:492) - whisper_model_load: model size = 140.54 MB
2023-05-31T11:35:05,293 | DEBUG | (ExecuteServiceImpl:492) - whisper_init_state: kv self size = 5.25 MB
2023-05-31T11:35:05,297 | DEBUG | (ExecuteServiceImpl:492) - whisper_init_state: kv cross size = 17.58 MB
2023-05-31T11:35:05,337 | DEBUG | (ExecuteServiceImpl:492) -
2023-05-31T11:35:05,337 | DEBUG | (ExecuteServiceImpl:492) - system_info: n_threads = 4 / 4 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | VSX = 0 | COREML = 0 |
2023-05-31T11:35:05,337 | DEBUG | (ExecuteServiceImpl:492) -
2023-05-31T11:35:05,337 | DEBUG | (ExecuteServiceImpl:492) - main: processing '/data/opencast/staging/workspace/mediapackage/9d5a60d0-2db2-4d42-a373-4d147292d430/5649b589-ccdb-4479-97ab-78013571da93/30cd4526-a613-465b-bfb2-2da960001d87.wav' (960256 samples, 60.0 sec), 4 threads, 1 processors, lang = en, task = transcribe, timestamps = 1 ...
2023-05-31T11:35:05,148 | DEBUG | (ServiceRegistryJpaImpl:904) - Adding to load cache: Job {id:579421, operation:Execute_Mediapackage, status:RUNNING}, type org.opencastproject.execute, load 1.0, status RUNNING
2023-05-31T11:35:05,149 | DEBUG | (ServiceRegistryJpaImpl:918) - Current host load: 1,0, job load cache size: 1
2023-05-31T11:35:05,149 | DEBUG | (AsyncTimeoutRedirectFilter:61) - Chain processed normally
2023-05-31T11:35:05,183 | DEBUG | (WorkspaceImpl:381) - /data/opencast/staging/workspace/mediapackage/9d5a60d0-2db2-4d42-a373-4d147292d430/5649b589-ccdb-4479-97ab-78013571da93/30cd4526-a613-465b-bfb2-2da960001d87.wav is up to date
2023-05-31T11:35:05,183 | DEBUG | (WorkspaceImpl:383) - Getting https://admin.oc.univie.ac.at/files/mediapackage/9d5a60d0-2db2-4d42-a373-4d147292d430/5649b589-ccdb-4479-97ab-78013571da93/30cd4526-a613-465b-bfb2-2da960001d87.wav directly from working file repository root at /data/opencast/staging/workspace/mediapackage/9d5a60d0-2db2-4d42-a373-4d147292d430/5649b589-ccdb-4479-97ab-78013571da93/30cd4526-a613-465b-bfb2-2da960001d87.wav
2023-05-31T11:35:05,184 | INFO | (ExecuteServiceImpl:481) - Running command whispercpp
2023-05-31T11:35:05,184 | DEBUG | (ExecuteServiceImpl:482) - Starting subprocess whispercpp with arguments --model, /usr/share/ggml/ggml-base.bin, -ovtt, -oj, -bs, 5, --output-file, /data/opencast/staging/workspace/collection/subtitles/tmp_9d5a60d0-2db2-4d42-a373-4d147292d430, /data/opencast/staging/workspace/mediapackage/9d5a60d0-2db2-4d42-a373-4d147292d430/5649b589-ccdb-4479-97ab-78013571da93/30cd4526-a613-465b-bfb2-2da960001d87.wav
2023-05-31T11:35:05,189 | DEBUG | (ExecuteServiceImpl:492) - whisper_init_from_file_no_state: loading model from '/usr/share/ggml/ggml-base.bin'
2023-05-31T11:35:05,189 | DEBUG | (ExecuteServiceImpl:492) - whisper_model_load: loading model
2023-05-31T11:35:05,189 | DEBUG | (ExecuteServiceImpl:492) - whisper_model_load: n_vocab = 51865
2023-05-31T11:35:05,189 | DEBUG | (ExecuteServiceImpl:492) - whisper_model_load: n_audio_ctx = 1500
2023-05-31T11:35:05,189 | DEBUG | (ExecuteServiceImpl:492) - whisper_model_load: n_audio_state = 512
2023-05-31T11:35:05,189 | DEBUG | (ExecuteServiceImpl:492) - whisper_model_load: n_audio_head = 8
2023-05-31T11:35:05,189 | DEBUG | (ExecuteServiceImpl:492) - whisper_model_load: n_audio_layer = 6
2023-05-31T11:35:05,189 | DEBUG | (ExecuteServiceImpl:492) - whisper_model_load: n_text_ctx = 448
2023-05-31T11:35:05,189 | DEBUG | (ExecuteServiceImpl:492) - whisper_model_load: n_text_state = 512
2023-05-31T11:35:05,189 | DEBUG | (ExecuteServiceImpl:492) - whisper_model_load: n_text_head = 8
2023-05-31T11:35:05,189 | DEBUG | (ExecuteServiceImpl:492) - whisper_model_load: n_text_layer = 6
2023-05-31T11:35:05,189 | DEBUG | (ExecuteServiceImpl:492) - whisper_model_load: n_mels = 80
2023-05-31T11:35:05,189 | DEBUG | (ExecuteServiceImpl:492) - whisper_model_load: ftype = 1
2023-05-31T11:35:05,189 | DEBUG | (ExecuteServiceImpl:492) - whisper_model_load: qntvr = 0
2023-05-31T11:35:05,189 | DEBUG | (ExecuteServiceImpl:492) - whisper_model_load: type = 2
2023-05-31T11:35:05,189 | DEBUG | (ExecuteServiceImpl:492) - whisper_model_load: mem required = 310.00 MB (+ 6.00 MB per decoder)
2023-05-31T11:35:05,248 | DEBUG | (ExecuteServiceImpl:492) - whisper_model_load: adding 1608 extra tokens
2023-05-31T11:35:05,249 | DEBUG | (ExecuteServiceImpl:492) - whisper_model_load: model ctx = 140.66 MB
2023-05-31T11:35:05,291 | DEBUG | (ExecuteServiceImpl:492) - whisper_model_load: model size = 140.54 MB
2023-05-31T11:35:05,293 | DEBUG | (ExecuteServiceImpl:492) - whisper_init_state: kv self size = 5.25 MB
2023-05-31T11:35:05,297 | DEBUG | (ExecuteServiceImpl:492) - whisper_init_state: kv cross size = 17.58 MB
2023-05-31T11:35:05,337 | DEBUG | (ExecuteServiceImpl:492) -
2023-05-31T11:35:05,337 | DEBUG | (ExecuteServiceImpl:492) - system_info: n_threads = 4 / 4 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | VSX = 0 | COREML = 0 |
2023-05-31T11:35:05,337 | DEBUG | (ExecuteServiceImpl:492) -
2023-05-31T11:35:05,337 | DEBUG | (ExecuteServiceImpl:492) - main: processing '/data/opencast/staging/workspace/mediapackage/9d5a60d0-2db2-4d42-a373-4d147292d430/5649b589-ccdb-4479-97ab-78013571da93/30cd4526-a613-465b-bfb2-2da960001d87.wav' (960256 samples, 60.0 sec), 4 threads, 1 processors, lang = en, task = transcribe, timestamps = 1 ...
```
As suggested I will file an PR from my branch https://github.com/elearning-univie/opencast/tree/stt-whispercpp-engine in a few minutes. It would be great if you could have a look at the code.
Thanks for all your efforts!
Martin
Am Mi., 31. Mai 2023 um 09:15 Uhr schrieb Maximiliano Lira Del Canto <max...@gmail.com>:
Could you try to execute WhisperCpp with the Execute workflow operation handler and enable the debug mode?The Execute WoH has the advantage that can show you the output of the executed program. If there are any problems like user permissions, they will show you immediately.Best,Maximiliano Lira Del CantoKöln, Deutschland.Am Di., 30. Mai 2023 um 23:12 Uhr schrieb Maximiliano Lira Del Canto <max...@gmail.com>:Glad that my tip helped youCan you file a pr with your code? So I can review it and check why it’s not working properly.Thanks for your work!
Martin Schamberger
Jun 1, 2023, 11:21:56 AM6/1/23
to Maximiliano Lira Del Canto, us...@opencast.org
Hi Max,
Sounds great - I'm happy whensoever you find some time! 🙂
There's a repo whisper.cpp on github: https://github.com/ggerganov/whisper.cpp
Unfortunately there are no pre-built binaries for download and you have to build them on your own.
I have no Mac and could not build/test - but whisper.cpp's README.md states it supports CoreML: https://github.com/ggerganov/whisper.cpp#core-ml-support
For users of RedHat-based distributions, this could be useful: https://github.com/elearning-univie/whispercpp-rpmbuild
Best regards,
Martin
Am Do., 1. Juni 2023 um 13:47 Uhr schrieb Maximiliano Lira Del Canto <max...@gmail.com>:
Hi Martin,With "WoH" worked well, we can be sure that the user permissions are okay; I'm now debugging a very nasty bug in the Access policy lists. But I will check your code next.About WhisperCpp, I have yet to look more at it. Can you send me the GitHub link? so I can install it on my Mac to do the debugging,By the way, you have answered me directly instead of to the group. I was mesmerized. Where was your answer? It would be great if you could copy and publish your last mail to the group list.Best regards,
Maximiliano Lira Del CantoKöln, Deutschland.
Am Mi., 31. Mai 2023 um 12:55 Uhr schrieb Martin Schamberger <majo...@gmail.com>:
Hello again,many thanks for your help! 😊
Am Mi., 31. Mai 2023 um 09:15 Uhr schrieb Maximiliano Lira Del Canto <max...@gmail.com>:
Could you try to execute WhisperCpp with the Execute workflow operation handler and enable the debug mode?The Execute WoH has the advantage that can show you the output of the executed program. If there are any problems like user permissions, they will show you immediately.Best,Maximiliano Lira Del CantoKöln, Deutschland.
Am Di., 30. Mai 2023 um 23:12 Uhr schrieb Maximiliano Lira Del Canto <max...@gmail.com>:
Glad that my tip helped you 😊.
Can you file a pr with your code? So I can review it and check why it’s not working properly.Thanks for your work!
--
Reply all
Reply to author
Forward
0 new messages