BBB-2.5.11 | Specific meeting recording failed to process.

[Sergey MikSer]

Hello,

I have problem with record processing that reproduced only on specific events. The other events are processed and published as expected. Not sure if that problem is related with bigbluebutton or ffmpeg/gstreamer.  Everything was working normal until I updated from 2.5.6.

bbb-record ---debug/check and bbb-conf --check returns nothing.

When i try to republish the record i got the following output from bbb-rap-resque:

● bbb-rap-resque-worker.service - BigBlueButton resque worker for recordings
     Loaded: loaded (/lib/systemd/system/bbb-rap-resque-worker.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2023-01-20 19:00:48 UTC; 6 days ago
   Main PID: 599397 (ruby2.7)
      Tasks: 4 (limit: 18961)
     Memory: 2.4G
     CGroup: /system.slice/bbb-rap-resque-worker.service
             ├─599397 ruby2.7 /usr/local/bigbluebutton/core/vendor/bundle/ruby/2.7.0/bin/rake -f Rakefile resque:workers
             └─599666 resque-2.4.0: Waiting for rap:archive,rap:publish,rap:process,rap:sanity,rap:capti

Jan 27 15:46:13 ip-10-2-15-34 bbb-rap[1447519]: /usr/local/bigbluebutton/core/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block in top_level'
Jan 27 15:46:13 ip-10-2-15-34 bbb-rap[1447519]: /usr/local/bigbluebutton/core/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:125:in `run_with_threads'
Jan 27 15:46:13 ip-10-2-15-34 bbb-rap[1447519]: /usr/local/bigbluebutton/core/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:110:in `top_level'
Jan 27 15:46:13 ip-10-2-15-34 bbb-rap[1447519]: /usr/local/bigbluebutton/core/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:83:in `block in run'
Jan 27 15:46:13 ip-10-2-15-34 bbb-rap[1447519]: /usr/local/bigbluebutton/core/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:186:in `standard_exception_handling'
Jan 27 15:46:13 ip-10-2-15-34 bbb-rap[1447519]: /usr/local/bigbluebutton/core/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:80:in `run'
Jan 27 15:46:13 ip-10-2-15-34 bbb-rap[1447519]: /usr/local/bigbluebutton/core/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
Jan 27 15:46:13 ip-10-2-15-34 bbb-rap[1447519]: /usr/local/bigbluebutton/core/vendor/bundle/ruby/2.7.0/bin/rake:23:in `load'
Jan 27 15:46:13 ip-10-2-15-34 bbb-rap[1447519]: /usr/local/bigbluebutton/core/vendor/bundle/ruby/2.7.0/bin/rake:23:in `<main>'

Jan 27 15:46:13 ip-10-2-15-34 bbb-rap[1447519]: (Job{rap:publish} | BigBlueButton::Resque::PublishWorker | [{"single_step"=>false, "format_name"=>"presentation", "meeting_id"=>"c920a80d978c932059b440a30879eb097c11d029-1674811527565"}]) failed: #<RuntimeError: Worker publish for c920a80d978c932059b440a30879eb097c11d029-1674811527565 failed with result false>

The last fragment of process.log, the full log in attachment:
I, [2023-01-27T10:29:49.303853 #1389466]  INFO -- :     Metadata:
I, [2023-01-27T10:29:49.303861 #1389466]  INFO -- :       encoder         : Lavc58.54.100 rawvideo
I, [2023-01-27T10:29:49.303868 #1389466]  INFO -- : av_interleaved_write_frame(): Broken pipe
I, [2023-01-27T10:29:49.303876 #1389466]  INFO -- : Error writing trailer of /var/bigbluebutton/recording/process/presentation/c920a80d978c932059b440a30879eb097c11d029-1674811527565/webcams_57.mp4.webcam_x2_y2.nut: Broken pipe
I, [2023-01-27T10:29:49.303883 #1389466]  INFO -- : frame=   23 fps=0.0 q=-0.0 Lsize=    1094kB time=00:05:18.95 bitrate=  28.1kbits/s speed= 353x
I, [2023-01-27T10:29:49.303891 #1389466]  INFO -- : video:1143kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
I, [2023-01-27T10:29:49.303898 #1389466]  INFO -- : Conversion failed!
E, [2023-01-27T10:29:49.303951 #1389466] ERROR -- : At least one auxiliary ffmpeg process failed
E, [2023-01-27T10:29:49.303970 #1389466] ERROR -- : /usr/local/bigbluebutton/core/lib/recordandplayback/edl/video.rb:611:in `composite_cut'
E, [2023-01-27T10:29:49.303980 #1389466] ERROR -- : /usr/local/bigbluebutton/core/lib/recordandplayback/edl/video.rb:262:in `block in render'
E, [2023-01-27T10:29:49.303988 #1389466] ERROR -- : /usr/local/bigbluebutton/core/lib/recordandplayback/edl/video.rb:256:in `each'
E, [2023-01-27T10:29:49.303996 #1389466] ERROR -- : /usr/local/bigbluebutton/core/lib/recordandplayback/edl/video.rb:256:in `render'
E, [2023-01-27T10:29:49.304004 #1389466] ERROR -- : /usr/local/bigbluebutton/core/lib/recordandplayback/generators/video.rb:50:in `process_webcam_videos'
E, [2023-01-27T10:29:49.304011 #1389466] ERROR -- : /usr/local/bigbluebutton/core/scripts/process/presentation.rb:228:in `<main>'


The full processing and publish logs is in attachment.

For info, I use с5a.2xlarge server(8vCPU x86, 16Gb RAM) running Ubuntu 20.04.

Any help appreciated . The problem does not appear on all events and I did not change any parameters from the previous installation before the update.

[sd...@distancelearning.cloud]

First thing I would do is run sudo bbb-record –rebuild c920a80d978c932059b440a30879eb097c11d029-1674811527565, and see if ffmpeg has the same issue again

 

Some others on 2.5 have had repeatable ffmpeg crashes on some meetings recently,  when I ran the conversion on a 2.6 beta server the screenshare/video conversions seemed to work.

 

If you happen to have a 2.5.6 server still,  would be interested if moving the archive to that server may get it converted

 

Regards,

Stephen

--
You received this message because you are subscribed to the Google Groups "BigBlueButton-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bigbluebutton-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-dev/a00f1b96-e0ca-4225-b595-0fab08c6dfefn%40googlegroups.com.

[Sergey MikSer]

Thank you. I forgot to mention, i tried to rebuild several times. The behaviour remains the same. 

[Sergey MikSer]

I had deployed 2.5.6 and transferred raw folder. Rebuild/republish finished successful without any error.

[Anton Georgiev]

Hi @mik...,

Not sure if this is the same issue but this thread/suggestion might be helpful: https://github.com/bigbluebutton/bigbluebutton/issues/16613#issuecomment-1411883735 

-Anton