Opencast stopped processing uploads

34 views
Skip to first unread message

s.la...@googlemail.com

unread,
Aug 13, 2024, 10:17:13 AM8/13/24
to Opencast Users
Hi everyone,

I hope you enjoy the summer.
We have a new problem with our opencast installation (it's still 13, I know, it's outdated).
Since yesterday it stopped processing newly uploaded recordings.
Yesterday we had 6 recordings, which were queued but weren't processed and seemed stuck. So I stopped all workflows and manually deleted all remaining tasks.

Unfortunately, it still does not process new recordings. I'll link the error log from both, admin and worker node, https://pastebin.com/cW9upJTJ.

It seems, the job is being restarted constantly (see screenshot). At the moment only one workflow should be running.

Does anyone have any hint where to look at or what to do?

Regards
Sven

Bildschirmfoto 2024-08-13 um 12.14.56.png 

Greg Logan

unread,
Aug 15, 2024, 10:02:57 PM8/15/24
to us...@opencast.org
Hi Sven,

A few things:
- Verify that only one node is dispatching jobs.  The error message about an object being updated since it was last read is typical of a configuration where multiple service registries are competing, and this *will* cause weird things to happen.
- Verify that all nodes can talk directly to all other nodes via HTTP.  Alternatively, turn off heartbeat checking in org.opencastproject.serviceregistry.impl.ServiceRegistryJpaImpl.cfg by setting heartbeat.interval = 0 in all non-admin nodes.  The bouncing behaviour you're seeing with services going offline then coming back is typical in this configuration case.  This was reported as https://github.com/opencast/opencast/issues/1538
- Check your configuration (and mounts!) for your workspace and working file repository.  These errors are telling you something is wrong:

  1. 2024-08-13T10:58:06,250 | WARN  | (WorkspaceImpl:418) - The working file repository URI and paths don't match. Looking up https://opencast.admin.node/files/mediapackage/54be5fbc-dc95-4272-a4a5-b0e1ee9fcb7d/8ba1ab6e-6a9e-442a-9ea3-ba7782db8f66/dublincore.xml at /opt/opencast/data/videocampus/files/mediapackage/54be5fbc-dc95-4272-a4a5-b0e1ee9fcb7d/8ba1ab6e-6a9e-442a-9ea3-ba7782db8f66/dublincore.xml failed
  2. 2024-08-13T10:58:06,254 | WARN  | (WorkspaceImpl:386) - The working file repository and workspace paths don't match. Looking up https://opencast.admin.node/files/mediapackage/54be5fbc-dc95-4272-a4a5-b0e1ee9fcb7d/8ba1ab6e-6a9e-442a-9ea3-ba7782db8f66/dublincore.xml at /opt/opencast/data/videocampus/files/mediapackage/54be5fbc-dc95-4272-a4a5-b0e1ee9fcb7d/8ba1ab6e-6a9e-442a-9ea3-ba7782db8f66/dublincore.xml failed
  3. 2024-08-13T10:58:06,309 | ERROR | (DublinCoreUtil:79) - Unable to load metadata from catalog 'https://opencast.admin.node/files/mediapackage/54be5fbc-dc95-4272-a4a5-b0e1ee9fcb7d/8ba1ab6e-6a9e-442a-9ea3-ba7782db8f66/dublincore.xml'
  4. org.opencastproject.util.NotFoundException: null

G

To unsubscribe from this group and stop receiving emails from it, send an email to users+un...@opencast.org.

s.la...@googlemail.com

unread,
Aug 16, 2024, 12:06:23 PM8/16/24
to Opencast Users
Hi Greg,

I verified, only the admin node is dispatching jobs.
What made me headache is, the log didn't tell nodes are going offline, it told only nodes coming back online. And the admin node log told me, the admin node itself was coming back online.
To stop opencast messing around, I deleted those failed/failing jobs. After this, suddenly all heartbeat errors disappeared. I really don't know what was going on, because nothing had been changed when the error appeared first time.
I also verified the NFS mounts, which are correct and the same on all nodes.

After deleting setting opencast back into "normal" I tried uploading a new recording, but after ingesting, the workflow is stuck at encoding the preview video.

Here is the new log from the admin node:

### ADMIN NODE LOG ###

2024-08-16T12:13:38,809 | INFO | (ServiceRegistryJpaImpl$JobProducerHeartbeat:2612) - Service org.opencastproject.transcription.microsoftazure@https://opencast.admin.node is back online
2024-08-16T12:13:38,823 | INFO | (ServiceRegistryJpaImpl$JobProducerHeartbeat:2612) - Service org.opencastproject.workflow@https://opencast.admin.node is back online

2024-08-16T12:20:58,051 | INFO | (StaticFileServiceImpl:335) - Start purging temporary storage section of all known organizations
2024-08-16T12:21:10,859 | INFO | (FileUploadServiceCleaner$Runner:120) - Start file upload service cleaner
2024-08-16T12:21:10,859 | INFO | (FileUploadServiceCleaner$Runner:126) - Finished file upload service cleaner
2024-08-16T12:38:58,887 | INFO | (IngestServiceImpl:666) - Created mediapackage 94f35974-2653-4870-8baf-d1e3f1ae0779
2024-08-16T12:38:58,916 | INFO | (IngestServiceImpl:765) - Start adding track 7e1bf8b3-126f-4cfd-a05c-b097165afcd3 from input stream on mediapackage 94f35974-2653-4870-8baf-d1e3f1ae0779
2024-08-16T12:43:28,519 | INFO | (IngestServiceImpl:1716) - Adding element of type Track to mediapackage 94f35974-2653-4870-8baf-d1e3f1ae0779
2024-08-16T12:43:28,522 | INFO | (IngestServiceImpl:783) - Successful added track 7e1bf8b3-126f-4cfd-a05c-b097165afcd3 on mediapackage 94f35974-2653-4870-8baf-d1e3f1ae0779 at URL https://opencast.admin.node/files/mediapackage/94f35974-2653-4870-8baf-d1e3f1ae0779/7e1bf8b3-126f-4cfd-a05c-b097165afcd3/Einfuehrung_SE02_WS2223.mp4
2024-08-16T12:43:28,528 | INFO | (IndexServiceImpl:1220) - Update asset track presenter/source
2024-08-16T12:43:29,191 | INFO | (IngestServiceImpl:1185) - Starting a new workflow with ingested mediapackage 94f35974-2653-4870-8baf-d1e3f1ae0779 based on workflow definition 'moodle'
2024-08-16T12:43:29,191 | INFO | (IngestServiceImpl:1576) - Ingested mediapackage 94f35974-2653-4870-8baf-d1e3f1ae0779 is processed using workflow template 'moodle', specified during ingest
2024-08-16T12:43:29,192 | INFO | (IngestServiceImpl:1366) - No capture event found for id 94f35974-2653-4870-8baf-d1e3f1ae0779
2024-08-16T12:43:29,194 | INFO | (IngestServiceImpl:1208) - Starting new workflow with ingested mediapackage '94f35974-2653-4870-8baf-d1e3f1ae0779' using the specified template 'moodle'
2024-08-16T12:43:29,211 | INFO | (AssetManagerImpl:1331) - Creating new version 0 of media package 94f35974-2653-4870-8baf-d1e3f1ae0779
2024-08-16T12:43:29,624 | INFO | (AssetManagerImpl:443) - Trigger update handlers for snapshot 94f35974-2653-4870-8baf-d1e3f1ae0779, version 0
2024-08-16T12:43:29,956 | INFO | (WorkflowServiceImpl:682) - [>ad5a777f] Scheduling workflow 63783769 for execution
2024-08-16T12:43:34,005 | INFO | (IncludeWorkflowOperationHandler:118) - Insert workflow partial-ingest into the current workflow instance
2024-08-16T12:43:36,046 | INFO | (IngestDownloadServiceRemoteImpl:99) - Downloading Source form mediapackge: 94f35974-2653-4870-8baf-d1e3f1ae0779 to workspace
2024-08-16T12:43:36,085 | INFO | (IngestDownloadServiceRemoteImpl:107) - Starting to download into workspace on remote IngestDownload 94f35974-2653-4870-8baf-d1e3f1ae0779
2024-08-16T12:43:42,155 | INFO | (InspectWorkflowOperationHandler:168) - Inspecting track '7e1bf8b3-126f-4cfd-a05c-b097165afcd3' of 94f35974-2653-4870-8baf-d1e3f1ae0779
2024-08-16T12:43:42,155 | INFO | (MediaInspectionServiceRemoteImpl:159) - Enriching https://opencast.admin.node/files/mediapackage/94f35974-2653-4870-8baf-d1e3f1ae0779/7e1bf8b3-126f-4cfd-a05c-b097165afcd3/einfuehrung_se02_ws2223.mp4 using a remote media inspection service
2024-08-16T12:43:42,192 | INFO | (MediaInspectionServiceRemoteImpl:167) - Completing inspection of media file at https://opencast.admin.node/files/mediapackage/94f35974-2653-4870-8baf-d1e3f1ae0779/7e1bf8b3-126f-4cfd-a05c-b097165afcd3/Einfuehrung_SE02_WS2223.mp4 using a remote media inspection service
2024-08-16T12:43:48,360 | INFO | (SeriesWorkflowOperationHandler:364) - Adding catalog with flavor dublincore/series to mediapackage 94f35974-2653-4870-8baf-d1e3f1ae0779
2024-08-16T12:43:50,297 | INFO | (TagWorkflowOperationHandler:105) - Retagging mediapackage elements
2024-08-16T12:43:52,342 | INFO | (AssetManagerSnapshotWorkflowOperationHandler:116) - Take snapshot of media package 94f35974-2653-4870-8baf-d1e3f1ae0779
2024-08-16T12:43:52,347 | INFO | (AssetManagerImpl:1331) - Creating new version 1 of media package 94f35974-2653-4870-8baf-d1e3f1ae0779
2024-08-16T12:43:52,640 | INFO | (AssetManagerImpl:443) - Trigger update handlers for snapshot 94f35974-2653-4870-8baf-d1e3f1ae0779, version 1
2024-08-16T12:43:54,384 | INFO | (IncludeWorkflowOperationHandler:118) - Insert workflow partial-preview into the current workflow instance
2024-08-16T12:43:56,429 | INFO | (AnalyzeTracksWorkflowOperationHandler:79) - Running analyze-tracks workflow operation on workflow 63783769
2024-08-16T12:43:56,430 | INFO | (AnalyzeTracksWorkflowOperationHandler:120) - Finished analyze-tracks workflow operation adding the properties: {presenter_source_aspect=16/9, presenter_source_media=true, presenter_source_audio=true, presenter_source_resolution_y=1080, presenter_source_resolution_x=1920, presenter_source_video=true, presenter_source_framerate=30.0}
2024-08-16T12:43:58,492 | INFO | (EncodeWorkflowOperationHandler:215) - Encoding track https://opencast.admin.node/files/mediapackage/94f35974-2653-4870-8baf-d1e3f1ae0779/7e1bf8b3-126f-4cfd-a05c-b097165afcd3/einfuehrung_se02_ws2223.mp4 using encoding profile 'mp4-preview.http'
2024-08-16T12:43:58,528 | INFO | (ComposerServiceRemoteImpl:164) - Encoding job 63784399 started on a remote composer

And from the worker node:
### WORKER NODE LOG ###

2024-08-16T12:43:36,077 | INFO | (IngestDownloadServiceEndpoint:99) - starting ingest-download Service
2024-08-16T12:43:38,091 | INFO | (IngestDownloadServiceImpl:264) - Skipping already existing element https://opencast.admin.node/files/mediapackage/94f35974-2653-4870-8baf-d1e3f1ae0779/7e1bf8b3-126f-4cfd-a05c-b097165afcd3/Einfuehrung_SE02_WS2223.mp4
2024-08-16T12:43:38,091 | INFO | (IngestDownloadServiceImpl:264) - Skipping already existing element https://opencast.admin.node/files/mediapackage/94f35974-2653-4870-8baf-d1e3f1ae0779/f3bef7ef-2f01-4686-ab55-ee3b8a26d8b8/dublincore.xml
2024-08-16T12:43:38,091 | INFO | (IngestDownloadServiceImpl:264) - Skipping already existing element https://opencast.admin.node/files/mediapackage/94f35974-2653-4870-8baf-d1e3f1ae0779/d345207d-89dd-4e51-a2db-d33ef59e9133/dublincore.xml
2024-08-16T12:43:38,091 | INFO | (IngestDownloadServiceImpl:264) - Skipping already existing element https://opencast.admin.node/files/mediapackage/94f35974-2653-4870-8baf-d1e3f1ae0779/security-policy-series/xacml.xml
2024-08-16T12:43:38,091 | INFO | (IngestDownloadServiceImpl:264) - Skipping already existing element https://opencast.admin.node/files/mediapackage/94f35974-2653-4870-8baf-d1e3f1ae0779/security-policy-episode/xacml.xml
2024-08-16T12:43:45,543 | INFO | (MediaInspector:294) - Successfully inspected track https://opencast.admin.node/files/mediapackage/94f35974-2653-4870-8baf-d1e3f1ae0779/7e1bf8b3-126f-4cfd-a05c-b097165afcd3/einfuehrung_se02_ws2223.mp4
2024-08-16T12:43:58,523 | INFO | (ComposerServiceImpl:572) - Starting parallel encode with profile mp4-preview.http with job load 1,5

No ffmpeg job is being started on the worker node.

Do you or someone else have any other suggestions?

Rute Santos

unread,
Aug 16, 2024, 1:19:13 PM8/16/24
to us...@opencast.org
Hi Sven,

The message from the worker log:
2024-08-16T12:43:58,523 | INFO | (ComposerServiceImpl:572) - Starting parallel encode with profile mp4-preview.http with job load 1,5

indicates that the job was created in the database, but not dispatched yet.
Check if the worker nodes are online and services look ok via the admin ui Systems->Servers and Services.
Also, look at the job dispatcher: turn the TRACE logging on org.opencastproject.serviceregistry.impl.JobDispatcher. This may give you the reason the job is not dispatched.

Thanks,

Rute

s.la...@googlemail.com

unread,
Aug 17, 2024, 8:24:02 AM8/17/24
to Opencast Users
Hi Rute,

thanks for your advice.
I checked again and the worker node is online (I set all other worker nodes to maintenance mode, to keep it simple) and all services are in normal condition.
Bildschirmfoto 2024-08-17 um 10.00.43.png


Also I enabled TRACE logging for the job dispatcher service.

LOG:
2024-08-17T10:05:09,281 | DEBUG | (JobDispatcher$JobDispatcherRunner:254) - Starting job dispatch
2024-08-17T10:05:09,291 | TRACE | (JobDispatcher$JobDispatcherRunner:416) - Using full list of services for dispatching of Job {id:63785202, operation:ParallelEncode, status:QUEUED} to a service of type 'org.opencastproject.composer'
2024-08-17T10:05:09,296 | DEBUG | (JobDispatcher$JobDispatcherRunner:527) - Trying to dispatch Job {id:63785202, operation:ParallelEncode, status:QUEUED} type 'org.opencastproject.composer' load 1.5 to https://opencast.worker1.node:8443
2024-08-17T10:05:09,324 | DEBUG | (JobDispatcher$JobDispatcherRunner:537) - Service org.opencastproject.composer@https://opencast.worker1.node:8443 is currently refusing to accept jobs of type ParallelEncode
2024-08-17T10:05:09,332 | DEBUG | (JobDispatcher$JobDispatcherRunner:590) - Unable to dispatch Job {id:63785202, operation:ParallelEncode, status:QUEUED}, no service is currently ready to accept the job
2024-08-17T10:05:09,332 | DEBUG | (JobDispatcher$JobDispatcherRunner:439) - Job {id:63785202, operation:ParallelEncode, status:QUEUED} currently cannot be dispatched
2024-08-17T10:05:09,333 | DEBUG | (JobDispatcher$JobDispatcherRunner:332) - Finished job dispatch

I checked the composer configuration which is still standard config. Also the load of 1.5 is much lower than the node could handle.

What I noticed after deleting the test recording, which could not be dispatched, was the following log output from the admin node.
### ADMIN NODE LOG ###
#### DELETE RECORDING ####
2024-08-17T10:14:20,124 | INFO | (WorkflowServiceImpl:882) - [>ad5a777f] Removing temporary files for workflow Workflow {63783781}
2024-08-17T10:14:20,265 | INFO | (WorkflowServiceImpl:882) - [>ad5a777f] Removing temporary files for workflow Workflow {63783781}
2024-08-17T10:14:20,266 | INFO | (WorkingFileRepositoryImpl:269) - [>ad5a777f] Unable to delete non existing media package element fa47a9d9-c090-4e21-a101-336a31c5a5ec@d7b6bfdb-641f-487b-a0f5-9a56dcc5def7
2024-08-17T10:14:20,269 | INFO | (WorkingFileRepositoryImpl:269) - [>ad5a777f] Unable to delete non existing media package element 03925235-75fd-470f-b5f5-39c4818faffa@d7b6bfdb-641f-487b-a0f5-9a56dcc5def7
2024-08-17T10:14:20,272 | INFO | (WorkingFileRepositoryImpl:269) - [>ad5a777f] Unable to delete non existing media package element 0aa81413-5de2-4823-af05-98e7a9661a65@d7b6bfdb-641f-487b-a0f5-9a56dcc5def7
2024-08-17T10:14:20,275 | INFO | (WorkingFileRepositoryImpl:269) - [>ad5a777f] Unable to delete non existing media package element security-policy-episode@d7b6bfdb-641f-487b-a0f5-9a56dcc5def7
2024-08-17T10:14:20,278 | INFO | (WorkingFileRepositoryImpl:269) - [>ad5a777f] Unable to delete non existing media package element security-policy-series@d7b6bfdb-641f-487b-a0f5-9a56dcc5def7
2024-08-17T10:14:20,304 | INFO | (ServiceRegistryJpaImpl:622) - Jobs with IDs '[63783782, 63783783, 63783784, 63783785, 63783786, 63783787, 63783788, 63783789, 63783790, 63783791]' deleted
2024-08-17T10:14:20,308 | INFO | (ServiceRegistryJpaImpl:622) - Jobs with IDs '[63783781]' deleted
2024-08-17T10:14:20,550 | INFO | (AssetManagerImpl:921) - Firing event handlers for event d7b6bfdb-641f-487b-a0f5-9a56dcc5def7, snapshot 0
2024-08-17T10:14:20,664 | INFO | (AssetManagerImpl:921) - Firing event handlers for event d7b6bfdb-641f-487b-a0f5-9a56dcc5def7, snapshot 1
2024-08-17T10:14:20,665 | INFO | (AssetManagerImpl:927) - Firing event handlers for event d7b6bfdb-641f-487b-a0f5-9a56dcc5def7
2024-08-17T10:14:20,696 | WARN | (AssetManagerImpl:592) - Event d7b6bfdb-641f-487b-a0f5-9a56dcc5def7 not found for deletion
2024-08-17T10:14:23,904 | WARN | (JobBarrier$JobStatusUpdater:323) - Error polling job Job {id:63785202, operation:ParallelEncode, status:QUEUED}: Not found!
2024-08-17T10:14:28,905 | ERROR | (WorkflowOperationWorker:140) - Workflow operation 'operation:'encode, state:'FAILED'' failed
org.opencastproject.workflow.api.WorkflowOperationException: java.lang.IllegalStateException: org.opencastproject.util.NotFoundException
at org.opencastproject.workflow.handler.composer.EncodeWorkflowOperationHandler.start(EncodeWorkflowOperationHandler.java:120) ~[?:?]
at org.opencastproject.workflow.impl.WorkflowOperationWorker.start(WorkflowOperationWorker.java:212) ~[!/:?]
at org.opencastproject.workflow.impl.WorkflowOperationWorker.execute(WorkflowOperationWorker.java:117) [!/:?]
at org.opencastproject.workflow.impl.WorkflowServiceImpl.runWorkflowOperation(WorkflowServiceImpl.java:723) [!/:?]
at org.opencastproject.workflow.impl.WorkflowServiceImpl.process(WorkflowServiceImpl.java:1751) [!/:?]
at org.opencastproject.workflow.impl.WorkflowServiceImpl$JobRunner.call(WorkflowServiceImpl.java:2112) [!/:?]
at org.opencastproject.workflow.impl.WorkflowServiceImpl$JobRunner.call(WorkflowServiceImpl.java:2078) [!/:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.lang.IllegalStateException: org.opencastproject.util.NotFoundException
at org.opencastproject.job.api.JobBarrier.waitForJobs(JobBarrier.java:219) ~[!/:?]
at org.opencastproject.workflow.api.AbstractWorkflowOperationHandler.waitForStatus(AbstractWorkflowOperationHandler.java:315) ~[?:?]
at org.opencastproject.workflow.api.AbstractWorkflowOperationHandler.waitForStatus(AbstractWorkflowOperationHandler.java:287) ~[?:?]
at org.opencastproject.workflow.handler.composer.EncodeWorkflowOperationHandler.encode(EncodeWorkflowOperationHandler.java:228) ~[?:?]
at org.opencastproject.workflow.handler.composer.EncodeWorkflowOperationHandler.start(EncodeWorkflowOperationHandler.java:118) ~[?:?]
... 10 more
Caused by: org.opencastproject.util.NotFoundException
at java.util.Optional.orElseThrow(Optional.java:408) ~[?:?]
at org.opencastproject.serviceregistry.impl.ServiceRegistryJpaImpl.getJob(ServiceRegistryJpaImpl.java:828) ~[?:?]
at org.opencastproject.job.api.JobBarrier$JobStatusUpdater.run(JobBarrier.java:295) ~[!/:?]
2024-08-17T10:14:28,906 | WARN | (WorkflowServiceImpl:1768) - [>ad5a777f] Exception while accepting job Job {id:63783791, operation:START_OPERATION, status:RUNNING}
2024-08-17T10:14:28,907 | WARN | (WorkflowServiceImpl:1771) - [>ad5a777f] Marking job Job {id:63783791, operation:START_OPERATION, status:RUNNING} and workflow instance Workflow {63783781} as failed

Donkis Zidan

unread,
Aug 17, 2024, 10:19:20 AM8/17/24
to Opencast Users, s.la...@googlemail.com
There is a growing number of people who  to buy shrooms online. It is often hard to tell which form of products to buy since you cannot see it physically. This is sometimes peculiar to newbies who haven’t tried shrooms before. They just don’t know how to go about it. Shroom Edibles are not so popularly discussed as often as you think. A lot of experienced shroom users have not even had a taste of these Psilonauts Psilocybin Chocolate Bar goodness that offer a premium psychedelic trip.


Shroom edibles refer to food products that have been infused with a measured dosage of psilocybin, the active ingredient found in magic mushrooms. Magic mushrooms, also known as shrooms, have a long history and are renowned for their psychedelic effects. In this article, we will delve into the nature of magic mushrooms, the workings of psilocybin, and the appeal of shroom edibles.
Understanding Magic Mushrooms
Magic mushrooms are a distinct type of fungi that contain the psychoactive compound psilocybin. They have been used for social, religious, and medicinal purposes throughout history. Magic mushrooms are known to produce psychedelic effects similar to other hallucinogens such as LSD.


Buy Albino Penis Envy Mushrooms, or “APE,” is a genetically isolated, non-pigmented strain of the famous Polkadot chocolate. But the lack of pigment gives the albino penile envy mushroom a bright white appearance.


Your best online shop to get platinum quality microdosing psychedelics products online, pain,anxiety pills, and research chemicals.
Be 100% assured about the quality and genuineness of the product, and you will also be able to buy quality psychedelics products at a fair price.


Buy Mushrooms Infused Chocolate Bars

https://t.me/EgUNSNP43FY3MDJk/5356?single


Buy Strawberry Rox

https://t.me/EgUNSNP43FY3MDJk/5325


Buy Muha Meds Disposable

https://t.me/EgUNSNP43FY3MDJk/5302?single


Buy C&G Carts

https://t.me/EgUNSNP43FY3MDJk/5293


Buy Persy Snowcaps 

https://t.me/EgUNSNP43FY3MDJk/5293


Buy Nug Shatter 

https://t.me/EgUNSNP43FY3MDJk/5234


Buy Gold Coast Clear Carts 

https://t.me/EgUNSNP43FY3MDJk/5232


Buy Gelato Pop

https://t.me/EgUNSNP43FY3MDJk/5212


Buy Gelato Online 

https://t.me/EgUNSNP43FY3MDJk/5205


Buy Banana Runtz Online 

https://t.me/EgUNSNP43FY3MDJk/5203


Buy Candy Runtz Online 

https://t.me/EgUNSNP43FY3MDJk/5200


Buy Doja Edibles Online 

https://t.me/EgUNSNP43FY3MDJk/5148


Buy Fryd Cart Online 

https://t.me/EgUNSNP43FY3MDJk/5145


Buy Packman Live Resin x Liquid Diamonds Catridges Online 

https://t.me/EgUNSNP43FY3MDJk/5143


Buy 2G Cake Dispoble Online 

https://t.me/EgUNSNP43FY3MDJk/5143


Buy California Gold Coast Cart Online 

https://t.me/EgUNSNP43FY3MDJk/5140


Buy Bloom Surf Disposable Online 

https://t.me/EgUNSNP43FY3MDJk/5060


Buy Pre-rolls Online 

https://t.me/EgUNSNP43FY3MDJk/5004?single


Buy 2g Persy Cart Online 

https://t.me/EgUNSNP43FY3MDJk/4976?single


Buy Edibles Online 

https://t.me/EgUNSNP43FY3MDJk/4940


Buy Cold Fire Juice Carts Online 

https://t.me/EgUNSNP43FY3MDJk/4928


Buy Whole Melt Tropical Online 

https://t.me/EgUNSNP43FY3MDJk/4797?single


Buy Candy Cake Online 

https://t.me/EgUNSNP43FY3MDJk/4688


Buy Golden Teacher Mushrooms Online 

https://t.me/EgUNSNP43FY3MDJk/4677


Buy Piff Bars 3g Online 

https://t.me/EgUNSNP43FY3MDJk/4642?single


Buy Kaws Moonrocks Online 

https://t.me/EgUNSNP43FY3MDJk/4639?single


Buy Frozen Grapes Ice Caps

https://t.me/EgUNSNP43FY3MDJk/4518?single


Buy Rainbow Runtz Online 

https://t.me/EgUNSNP43FY3MDJk/4486


Buy White Runtz Online 

https://t.me/EgUNSNP43FY3MDJk/4348


Buy Torch 2g Live Resin Disposables Online 

https://t.me/EgUNSNP43FY3MDJk/4280?single


Buy Cookie Carts Online 

https://t.me/EgUNSNP43FY3MDJk/4278?single


Buy Guava Gelato Online 

https://t.me/EgUNSNP43FY3MDJk/4136?single


Buy Sour Diesel Snowball Online 

https://t.me/EgUNSNP43FY3MDJk/4058


Buy psilocybin Mushrooms Online 

https://t.me/EgUNSNP43FY3MDJk/4046


Buy Packman 2g Disposable Online 

https://t.me/EgUNSNP43FY3MDJk/4031?single


Buy Jeetter Juice 2g Disposable Online 

https://t.me/EgUNSNP43FY3MDJk/4016?single


Buy Pluto 2g Carts Online 

https://t.me/EgUNSNP43FY3MDJk/4010?single


Buy Candy Piffs Online 

https://t.me/EgUNSNP43FY3MDJk/4007?single


Buy Moonrocks Online 

https://t.me/EgUNSNP43FY3MDJk/4003?single


Buy Birthday cake Online 

https://t.me/EgUNSNP43FY3MDJk/3915?single


Buy Sourz Liquid Diamonds Edibles Online 

https://t.me/EgUNSNP43FY3MDJk/3837?single


Buy Clarity Shatter Online 

https://t.me/EgUNSNP43FY3MDJk/3835?single


Buy Sauce Bars Online 

https://t.me/EgUNSNP43FY3MDJk/3827?single


Buy Sugar Wax Online 

https://t.me/EgUNSNP43FY3MDJk/3819?single


Where to buy Dmt Carts In USA 

https://t.me/EgUNSNP43FY3MDJk/5349


Order Happy Mushrooms Edibles In USA

https://t.me/EgUNSNP43FY3MDJk/5354?single


Buy Splitz Live Resin Online 

https://t.me/EgUNSNP43FY3MDJk/3738?single


Where to buy Mushrooms Infused Chocolate 

https://t.me/EgUNSNP43FY3MDJk/5375


Buy Polkadot mushrooms chocolate bar

https://t.me/EgUNSNP43FY3MDJk/5374


Buy 1g Disposable Vape Online 

https://t.me/EgUNSNP43FY3MDJk/5371?single


Order Thc Infused Gummies In USA 

https://t.me/EgUNSNP43FY3MDJk/5367?single


Buy Thc Chocolate Edibles Online 

https://t.me/EgUNSNP43FY3MDJk/5365?single


Purchase Mushroom Gummies Online 

https://t.me/EgUNSNP43FY3MDJk/5363?single


Where To Order Mushroom Infused Mints

https://t.me/EgUNSNP43FY3MDJk/5361?single


https://t.me/EgUNSNP43FY3MDJk/5434?single

https://t.me/EgUNSNP43FY3MDJk/5430

https://t.me/EgUNSNP43FY3MDJk/5430

https://t.me/EgUNSNP43FY3MDJk/5428?single

https://t.me/EgUNSNP43FY3MDJk/5423?single

https://t.me/EgUNSNP43FY3MDJk/5423?single

https://t.me/EgUNSNP43FY3MDJk/5423?single

https://t.me/EgUNSNP43FY3MDJk/5416

https://t.me/EgUNSNP43FY3MDJk/5415?single

https://t.me/EgUNSNP43FY3MDJk/5412

https://t.me/EgUNSNP43FY3MDJk/5411

https://t.me/EgUNSNP43FY3MDJk/5410

https://t.me/EgUNSNP43FY3MDJk/5639?single

https://t.me/EgUNSNP43FY3MDJk/5637?single

https://t.me/EgUNSNP43FY3MDJk/5635?single

https://t.me/EgUNSNP43FY3MDJk/5632?single

https://t.me/EgUNSNP43FY3MDJk/5630?single

https://t.me/EgUNSNP43FY3MDJk/5628?single

https://t.me/EgUNSNP43FY3MDJk/5626?single

https://t.me/EgUNSNP43FY3MDJk/5624?single

https://t.me/EgUNSNP43FY3MDJk/5622?single

https://t.me/EgUNSNP43FY3MDJk/5620?single

https://t.me/EgUNSNP43FY3MDJk/5618?single

https://t.me/EgUNSNP43FY3MDJk/5616?single

https://t.me/EgUNSNP43FY3MDJk/5613?single

https://t.me/EgUNSNP43FY3MDJk/5611?single

https://t.me/EgUNSNP43FY3MDJk/5609?single

https://t.me/EgUNSNP43FY3MDJk/5607?single

https://t.me/EgUNSNP43FY3MDJk/5607?single

https://t.me/EgUNSNP43FY3MDJk/5852?single

Rute Santos

unread,
Aug 19, 2024, 12:55:21 PM8/19/24
to us...@opencast.org
This tells you that admin is trying to dispatch the job, but the worker is not accepting it:

2024-08-17T10:05:09,324 | DEBUG | (JobDispatcher$JobDispatcherRunner:537) - Service org.opencastproject.composer@https://opencast.worker1.node:8443 is currently refusing to accept jobs of type ParallelEncode

I would look at the worker logs when services are starting to see if there are any errors. Also, see if there are other job running on that worker that would cause it to reach the max capacity.

Thanks,

Rute


On Aug 17, 2024, at 4:24 AM, 's.la...@googlemail.com' via Opencast Users <us...@opencast.org> wrote:

Hi Rute,

thanks for your advice.
I checked again and the worker node is online (I set all other worker nodes to maintenance mode, to keep it simple) and all services are in normal condition.
<Bildschirmfoto 2024-08-17 um 10.00.43.png>

s.la...@googlemail.com

unread,
Aug 19, 2024, 2:22:49 PM8/19/24
to Opencast Users
Hi Rute,

the opencast cluster is currently mostly idle cause of summer break. So there were no jobs running.
The worker log is as follows:

2024-08-19T16:01:59,146 | INFO | (IngestDownloadServiceEndpoint:99) - starting ingest-download Service
2024-08-19T16:02:01,156 | INFO | (IngestDownloadServiceImpl:264) - Skipping already existing element https://opencast.admin.node/files/mediapackage/8e820e67-503e-49ad-8282-0730e4ad699c/48bb4cf3-b9fd-4d0a-b5d3-71da2951971e/block_opencast_video_uploadGhHeSP_Schmerzen__Leiden_Sch_den__humane_end.mp4
2024-08-19T16:02:01,156 | INFO | (IngestDownloadServiceImpl:264) - Skipping already existing element https://opencast.admin.node/files/mediapackage/8e820e67-503e-49ad-8282-0730e4ad699c/2d1999df-c214-4dbf-ae6a-0a188df1c842/dublincore.xml
2024-08-19T16:02:01,156 | INFO | (IngestDownloadServiceImpl:264) - Skipping already existing element https://opencast.admin.node/files/mediapackage/8e820e67-503e-49ad-8282-0730e4ad699c/afcd6d9e-ac0a-4de8-a0b2-a98ebfcd6e3d/dublincore.xml
2024-08-19T16:02:01,156 | INFO | (IngestDownloadServiceImpl:264) - Skipping already existing element https://opencast.admin.node/files/mediapackage/8e820e67-503e-49ad-8282-0730e4ad699c/security-policy-series/xacml.xml
2024-08-19T16:02:01,156 | INFO | (IngestDownloadServiceImpl:264) - Skipping already existing element https://opencast.admin.node/files/mediapackage/8e820e67-503e-49ad-8282-0730e4ad699c/security-policy-episode/xacml.xml
2024-08-19T16:02:10,973 | INFO | (MediaInspector:294) - Successfully inspected track https://opencast.admin.node/files/mediapackage/8e820e67-503e-49ad-8282-0730e4ad699c/48bb4cf3-b9fd-4d0a-b5d3-71da2951971e/block_opencast_video_uploadghhesp_schmerzen__leiden_sch_den__humane_end.mp4
2024-08-19T16:02:26,321 | INFO | (ComposerServiceImpl:572) - Starting parallel encode with profile mp4-preview.http with job load 1,5

How can I enable DEBUG logging for the composer part of opencast?
I tried 
log4j2.logger.composer.name = org.opencastproject.composer.impl.ComposerServiceImpl
log4j2.logger.composer.level = TRACE

but didn't succeed.

Rute Santos

unread,
Aug 19, 2024, 4:13:23 PM8/19/24
to us...@opencast.org
Hi Sven,

The composer logging directive looks ok to me so I don’t know why it didn’t work... Please also add DEBUG logging for org.opencastproject.job.api.AbstractJobProducer on the worker. This may give you more information on why the job is refused by the worker node.

Thanks,

Rute



s.la...@googlemail.com

unread,
Aug 23, 2024, 6:28:30 AM8/23/24
to Opencast Users
Hi Rute,

thanks for all your help.
I still don't know exactly what caused the problem, but after restarting the worker nodes everything is okay now.
All jobs are being processed correctly.

Regards
Sven
Reply all
Reply to author
Forward
0 new messages