Scalelite recordings transfer not working, manually or automatically

[jhartl...@gmail.com]

We have setup nfs as per the docs, followed everything and then we try and run the scalelite_batch_import.sh script to move existing recordingsm we get:

/usr/lib/ruby/vendor_ruby/rubygems/defaults/operating_system.rb:10: warning: constant Gem::ConfigMap is deprecated
/usr/lib/ruby/vendor_ruby/rubygems/defaults/operating_system.rb:29: warning: constant Gem::ConfigMap is deprecated
/usr/lib/ruby/vendor_ruby/rubygems/defaults/operating_system.rb:30: warning: constant Gem::ConfigMap is deprecated
./post_publish/scalelite_post_publish.rb:91: syntax error, unexpected end-of-input, expecting keyword_end
  FileUtils.rm_f(archive_file)

The system also does not move new recordings over to the directory

Is there a log for this?

Any help appreciated.

Thanks

James

[jhartl...@gmail.com]

Anyone come across this? It is the only thing stopping us from moving to scalelite

[jhartl...@gmail.com]

doesnt look like it runs the script at all from post_publish, scalelite_post_publish.rb Am i missing a step for this  .rb to run, I thought it was a drop in, and they run in alphbetical order?

I, [2021-12-13T12:32:51.974652 #8302]  INFO -- : Recording Ready notify ends
I, [2021-12-13T12:43:07.229091 #529]  INFO -- : Recording Ready Notify for [1665b62af66ea20a29f13a97e2cd48dc5b02ec52-1639399289977] starts
I, [2021-12-13T12:43:07.231085 #529]  INFO -- : Task: Getting meeting metadata
I, [2021-12-13T12:43:07.232392 #529]  INFO -- : Recording Ready notify ends
I, [2021-12-13T13:11:48.024693 #8344]  INFO -- : Recording Ready Notify for [1665b62af66ea20a29f13a97e2cd48dc5b02ec52-1639401007219] starts
I, [2021-12-13T13:11:48.024768 #8344]  INFO -- : Task: Getting meeting metadata
I, [2021-12-13T13:11:48.025914 #8344]  INFO -- : Recording Ready notify ends
I, [2021-12-14T10:09:15.940745 #26119]  INFO -- : Recording Ready Notify for [13b4b9e8a6a6b5d3ac5778d05437d0ce9ce11b29-1639476421067] starts
I, [2021-12-14T10:09:15.940802 #26119]  INFO -- : Task: Getting meeting metadata
I, [2021-12-14T10:09:15.941161 #26119]  INFO -- : Recording Ready notify ends
I, [2021-12-14T10:29:05.995013 #9575]  INFO -- : Recording Ready Notify for [1665b62af66ea20a29f13a97e2cd48dc5b02ec52-1639477699922] starts
I, [2021-12-14T10:29:05.995230 #9575]  INFO -- : Task: Getting meeting metadata
I, [2021-12-14T10:29:05.996153 #9575]  INFO -- : Recording Ready notify ends

[Arun Kumar Bhat]

could be a permissions issue.

--
You received this message because you are subscribed to the Google Groups "BigBlueButton-Setup" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bigbluebutton-s...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-setup/8f82caa0-3c8c-4f9f-9975-ce25b751cb5an%40googlegroups.com.

[jhartl...@gmail.com]

but according to the post_publish log it doesnt even execute the script. I have done chmod +x to the script

root@bbbm1:/usr/local/bigbluebutton/core/scripts/post_publish# ls -l
total 12
-rw-r--r-- 1 root root 1492 Oct 21 15:50 post_publish.rb.example
-rw-r--r-- 1 root root 3938 Oct 21 15:50 post_publish_recording_ready_callback.rb
-rwxr-xr-x 1 root root 3208 Dec  3 15:50 scalelite_post_publish.rb
root@bbbm1:/usr/local/bigbluebutton/core/scripts/post_publish#

[jhartl...@gmail.com]

I think I am going to start from scratch again and write off the many hours and days I have spent on this already

[Arun Kumar Bhat]

We also faced a similar issue,
we replaced scalelite_post_publish.rb from another working BBB server and the issue was resolved.
Here is the  scalelite_post_publish.rb  that we are currently using: 
``` 

#!/usr/bin/ruby
# frozen_string_literal: true

# Scalelite recording transfer script
# Copyright © 2020 Blindside Networks
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU Affero General Public License as
# published by the Free Software Foundation, either version 3 of the
# License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU Affero General Public License for more details.
#
# You should have received a copy of the GNU Affero General Public License
# along with this program.  If not, see <https://www.gnu.org/licenses/>.

require 'optparse'
require 'psych'
require 'fileutils'
require File.expand_path('../../lib/recordandplayback', __dir__)

puts('Recording transferring to Scalelite starts')

meeting_id = nil
OptionParser.new do |opts|
  opts.on('-m', '--meeting-id MEETING_ID', 'Internal Meeting ID') do |v|
    meeting_id = v
  end
  opts.on('-f', '--format FORMAT', 'Recording Format') do |v|
  end
end.parse!

unless meeting_id
  msg = 'Meeting ID was not provided'
  puts(msg) && raise(msg)
end

props = Psych.load_file(File.join(__dir__, '../bigbluebutton.yml'))
published_dir = props['published_dir'] || raise('Unable to determine published_dir from bigbluebutton.yml')
recording_dir = props['recording_dir'] || raise('Unable to determine recording_dir from bigbluebutton.yml')

scalelite_props = Psych.load_file(File.join(__dir__, '../scalelite.yml'))
work_dir = scalelite_props['work_dir'] || raise('Unable to determine work_dir from scalelite.yml')
spool_dir = scalelite_props['spool_dir'] || raise('Unable to determine spool_dir from scalelite.yml')
extra_rsync_opts = scalelite_props['extra_rsync_opts'] || []

puts("Transferring recording for #{meeting_id} to Scalelite")
format_dirs = []
FileUtils.cd(published_dir) do
  format_dirs = Dir.glob("*/#{meeting_id}")
end
if format_dirs.empty?
  puts('No published recording formats found')
  exit
end

format_dirs.each do |format_dir|
  puts("Found recording format: #{format_dir}")
end

archive_file = "#{work_dir}/#{meeting_id}.tar"
begin
  puts('Creating recording archive')
  FileUtils.mkdir_p(work_dir)
  FileUtils.cd(published_dir) do
    system('tar', '--create', '--file', archive_file, *format_dirs) \
      || raise('Failed to create recording archive')
  end

  puts("Transferring recording archive to #{spool_dir}")
  system('rsync', '--verbose', '--protect-args', *extra_rsync_opts, archive_file, spool_dir) \
    || raise('Failed to transfer recording archive')

  puts('Create sender.done file')
  File.open("#{recording_dir}/status/published/#{meeting_id}-sender.done", 'w') do |f|
    f.write("Published #{meeting_id}")
  end

  puts('Recording transferring to Scalelite ends')
ensure
  FileUtils.rm_f(archive_file)
end

```

To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-setup/209558b4-a932-4f71-bb6d-3d0bfbfa7448n%40googlegroups.com.

[jhartl...@gmail.com]

wow amazing, now the manual script works, but it only seems to have moved things into the spool directory as tar files, can't find the rest in the infrastructure. how does scalelite process the files, or doesnt it?

[Arun Kumar Bhat]

Once the files are in the spool directory scalelite-recording-importer will process that tar file and then put them into the published/presentation directory in scalelite.
Can you provide the logs of scalelite-recording-importer?

To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-setup/79ff6b97-c0c1-40f1-bed4-4897c4ad94e1n%40googlegroups.com.

[jhartl...@gmail.com]

hmmm,

I, [2021-12-14T17:16:25.183680 #8]  INFO -- : Importing recording from file: /var/bigbluebutton/spool/0d452a66e6c9ec19e4b358638611521b0ddd4e3c-1638965035413.tar
E, [2021-12-14T17:16:25.184085 #8] ERROR -- : Failed to import recording: Permission denied @ dir_s_mkdir - /var/bigbluebutton/recording/scalelite/d20211214-8-aaorod
W, [2021-12-14T17:16:25.184241 #8]  WARN -- : /usr/lib/ruby/2.7.0/tmpdir.rb:85:in `mkdir': Permission denied @ dir_s_mkdir - /var/bigbluebutton/recording/scalelite/d20211214-8-aaorod (Errno::EACCES)

but my spool directory is not at  /var/bigbluebutton/spool/ it is at /mnt/scalelite-recordings/var/bigbluebutton/spool/, or is that a red herring, and that is just where it is mounted in the image and it is just a permissions thing after all

[jhartl...@gmail.com]

yep. think i've got it, needed 1000:2001 and chowned the directories :) (2001 as using 2000 already)

Now to test if its all automatic

[jhartl...@gmail.com]

ok, so it works manually with the batch import script and is processed and shows, which is great, but the post_publish script does not run:

I, [2021-12-15T08:36:32.019735 #3477]  INFO -- : Recording Ready Notify for [1665b62af66ea20a29f13a97e2cd48dc5b02ec52-1639557344374] starts
I, [2021-12-15T08:36:32.019828 #3477]  INFO -- : Task: Getting meeting metadata
I, [2021-12-15T08:36:32.020646 #3477]  INFO -- : Recording Ready notify ends

script is in the directory:

root@bbbm1:/usr/local/bigbluebutton/core/scripts/post_publish# ls -l
total 12
-rw-r--r-- 1 root root 1492 Oct 21 15:50 post_publish.rb.example
-rw-r--r-- 1 root root 3938 Oct 21 15:50 post_publish_recording_ready_callback.rb

-rwxr-xr-x 1 root root 2961 Dec 14 17:15 scalelite_post_publish.rb

with x permissions, is there anything else i need to do to get it to actually run?

Thanks

James

[jhartl...@gmail.com]

right, it is running, but more permissions problems:

Executing: ruby /usr/local/bigbluebutton/core/scripts/post_publish/scalelite_post_publish.rb -m 1665b62af66ea20a29f13a97e2cd48dc5b02ec52-1639558936535 -f presentation
Dec 15 09:02:59 bbbm1 bbb-rap[23923]: Recording transferring to Scalelite starts
Dec 15 09:02:59 bbbm1 bbb-rap[23923]: Transferring recording for 1665b62af66ea20a29f13a97e2cd48dc5b02ec52-1639558936535 to Scalelite
Dec 15 09:02:59 bbbm1 bbb-rap[23923]: Found recording format: presentation/1665b62af66ea20a29f13a97e2cd48dc5b02ec52-1639558936535
Dec 15 09:02:59 bbbm1 bbb-rap[23923]: Creating recording archive
Dec 15 09:02:59 bbbm1 bbb-rap[23923]: Transferring recording archive to /mnt/scalelite-recordings/var/bigbluebutton/spool
Dec 15 09:02:59 bbbm1 bbb-rap[23923]: 1665b62af66ea20a29f13a97e2cd48dc5b02ec52-1639558936535.tar
Dec 15 09:02:59 bbbm1 bbb-rap[23923]: rsync: mkstemp "/mnt/scalelite-recordings/var/bigbluebutton/spool/.1665b62af66ea20a29f13a97e2cd48dc5b02ec52-1639558936535.tar.POMAMq" failed: Permission denied (13)
Dec 15 09:02:59 bbbm1 bbb-rap[23923]:
Dec 15 09:02:59 bbbm1 bbb-rap[23923]: sent 358,612 bytes  received 204 bytes  717,632.00 bytes/sec
Dec 15 09:02:59 bbbm1 bbb-rap[23923]: total size is 358,400  speedup is 1.00
Dec 15 09:02:59 bbbm1 bbb-rap[23923]: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1196) [sender=3.1.2]
Dec 15 09:02:59 bbbm1 bbb-rap[23923]: /usr/local/bigbluebutton/core/scripts/post_publish/scalelite_post_publish.rb:75:in `<main>': Failed to transfer recording archive (RuntimeError)
Dec 15 09:02:59 bbbm1 bbb-rap[23923]: Success?: false
Dec 15 09:02:59 bbbm1 bbb-rap[23923]: Process exited? true
Dec 15 09:02:59 bbbm1 bbb-rap[23923]: Exit status: 1
Dec 15 09:02:59 bbbm1 bbb-rap[23923]: Post publish script /usr/local/bigbluebutton/core/scripts/post_publish/scalelite_post_publish.rb/scalelite_post_publish failed

[jhartl...@gmail.com]

got it :) bit of group adding and working out, had to add a bigbluebutton user on uid 997 on the nfs server and add it to scalelite-spool group and it looks like it is working

Thanks for all the help, onto the next blocker...