Looking to upgrade my custom version of Greenlight

[Carl Gosselin]

Hello,

I've got a custom version of Greenlight running in a dockerized container (I perform a docker stop.., docker rm.., docker build.., and docker run.. for all changes).  

I'd like to perform an upgrade (grab the latest version of Greenlight w/ super admin functionality, etc...).  

I'm a little nervous doing this so I'm wondering if I can post the steps here for review:

Steps:

1. Perform backup of my BBB/greenlight server on DigitalOcean

On the command line...

2. git remote add upstream https://github.com/bigbluebutton/greenlight/tree/v2

3. git fetch upstream

4. git checkout master

5. git rebase upstream/master

6. (Do I do this one too?) git push -f origin master

7. the, perform the usual docker stop.., docker rm.., docker build.., and docker run.. commands

Do the above steps seem right?

A bit of context:  I previously forked Greenlight into my Github repository (it is now an older/ outdated version of greenlight):  https://github.com/carldgosselin/greenlight.git

Questions:  

- Do I need to fork an updated version of Greenlight in my own github repository? (not sure if I need to do this)

- Also, is there a way to keep my customizations during this upgrade?  Or should I just upgrade and then add my customizations back to the upgraded Greenlight version?

[Chad Pilkey]

If your customizations are sitting uncommitted then I would recommend committing and pushing them to a new branch on your fork to preserve them. If you've already committed locally, but just haven't pushed then you should definitely push first. If you don't commit and push then you could get in to a state that you can't recover from and have to reapply from scratch.

Once you have a branch with your changes, you can then do "git merge upstream/v2" instead of a rebase. You'll likely have to resolve conflicts. Once everything is resolved, you build and test it and if the merge looks like it worked then you commit and push the update to preserve it.

I much prefer to create my own branches with names rather than messing with origin/master, but you could just work in origin/master if you wanted to.

There's no need to fork again because the up to date code can be fetched and merged from the "upstream" remote.

Whether you go with the merge or manually add your customizations back depends on what exactly you've done. If you try the merge and there are a lot more conflicts than you want to deal with then reapplying might be easier.

One last thing is that I would recommend backing up your Greenlight database. It should be under ~/greenlight. Backing that up will save you if you decide you need to completely roll back.

[Carl Gosselin]

Thanks Chad.  I'll make sure to perform the appropriate back ups.  It may take me a few tries to get it right.  thanks again.

[Carl Gosselin]

Hello - I'm prepping to merge my custom greenlight code with the latest version of greenlight.  Do the following commands make sense?

context:  unfortunately, I'm starting off with a new github repository since I didn't fork the code properly from the original greenlight repository

$ git branch upgrade1       # move away from the master branch

$ git checkout upgrade1   # switch to my experimental branch

$ git show-branch             # quick checkpoint

$ git fetch upstream https://github.com/bigbluebutton/greenlight/tree/v2    # Does this line seem right?

I'm expecting the resolve a bunch of conflicts due to the differences in updated code + all of my customizations.  

When all conflicts are resolved, pull the updated code to my test server to see if it worked.  Does this make sense?

[Ahmad Farhat]

The process you should follow really depends on how many changes you've made. It might be alot easier just to follow the "Customize" instructions from scratch and manually add your changes back in

[Carl Gosselin]

ok.  sounds good.  

1. In looking at the documentation to upgrade Greenlight, it says to pull the latest image from DockerHub (as opposed to GitHub):

command:  docker pull bigbluebutton/greenlight:v2

Is this correct?

2. Also, the documentation mentions using docker-compose.  I'm currently using Docker Run as the 4 commands I use to stop, rm, build, and restart are saved on my command line.  I can still use Docker Run correct?

3. My last question, what happens when another release of Greenlight comes along?  Will I need to manually re-implement my customizations again?

thanks for all of your help.

[Chad Pilkey]

The Greenlight install docs are for people looking to run the default containers. You're building your own containers so you would want the development/customization docs.

The documentation also used to list two options for running the containers "docker run" and "docker-compose". We streamlined it and got rid of the "docker run" steps to make things simpler.

Every time you want to update to the latest upstream changes you'll need to do a merge and resolve any conflicts.

Also in regards to the commands to update, instead of the last "git fetch" command I would instead do the following:

git remote add upstream https://github.com/bigbluebutton/greenlight # Only needs to be run once per git environment

git fetch upstream

# Check that you're in the correct branch

git merge upstream/v2

[Carl Gosselin]

Great.  Thanks.  I ran the following commands and looks like it's working:

$ git remote set-url upstream https://github.com/bigbluebutton/greenlight.git

$ git fetch upstream

$ git merge upstream/v2

I've got about 12 conflicts or so to resolve and then I should be good to go.  

[Carl Gosselin]

Hello - I resolved the conflicts after merging with upstream/v2 for the latest version of Greenlight (all while keeping my customizations)

on my new branch, I performed the following commands to rebuild greenlight:

1. docker stop custom

2. docker rm custom

3. docker build -t custom_greenlight .

4. docker run --restart unless-stopped -d -p 5000:80 -v $(pwd)/db/production:/usr/src/app/db/production --env-file .env --name custom custom_greenlight

result  

I'm now getting a permanent 404 error:  

404 not found - nginx/ 1.10.3 (Ubuntu)

Can you help me troubleshoot?

[Ahmad Farhat]

Can you send the output of "docker logs greenlight-v2" 

[Carl Gosselin]

I have a customized container called 'custom' so I ran 'docker logs greenlight-v2'

The output is really long (attached).

[Carl Gosselin]

typo - 'docker logs custom'

Looks like there may be an issue with my existing sqlite3 database?

[Ahmad Farhat]

Try adding this line here:

gem 'sqlite3', '~> 1.3.6'

In this spot here:

https://github.com/bigbluebutton/greenlight/blob/master/Gemfile#L78

And then doing a rebuild and run again

[Carl Gosselin]

No luck.  I also noticed the following line during the rebuild:

Your Gemfile lists the gem sqlite3 (~> 1.3.6) more than once.

You should probably keep only one of them.

While it's not a problem now, it could cause errors if you change the version of one of them later.

I've attached the following txt files:

1. Output on the command line (during rebuild)

2. Output of 'docker logs custom'

[Carl Gosselin]

In reviewing the logs, looks like the new roles.provider column is triggering the error.  There is no roles.provider in my current version of the database.  Would I be able to simply add this new column?

[Carl Gosselin]

As I'm digging deeper into the migration, it looks like the migration is bothered that the database already exists:

Database 'db/production/production.sqlite3' already exists

>>> Database migration

rake aborted!

I took a look at the differences between my existing schema.rb file and the new schema.rb for migration.  Most tables are new but 2 tables only require modifications.

new tables:

- features

- invitations

- role_permissions

- roles

- settings

- shared_accesses

- users_roles

modified tables:

- rooms

- users

Instead of using ruby's 'create_table' function, would it be better suited to use 'change_table' to add the additional columns?  Therefore...

change_table(:rooms) do |t|

  t.string "access code"

  t.boolean "deleted", default: false, null: false

  t.index ["deleted"], name: "index_rooms_on_deleted"

end

change_table(:users) do |t|

  t.boolean "deleted", default: false, null: false

  t.index ["created_at"], name: "index_users_on_created_at"

  t.index ["deleted"], name: "index_users_on_deleted"

  t.index ["email"], name: "index_users_on_email"

  t.index ["provider"], name: "index_users_on_provider"

end 

Does this seem right?

[Carl Gosselin]

Would it be a better idea to remove my existing sqlite3 database from the Production folder for the greenlight upgrade?

Afterwards, I'll figure out a way to migrate my existing database to the new, and empty, database?  

I'm open to any approach.  I'm just looking forward to completing my upgrade to the new greenlight.

[Carl Gosselin]

Hello - I attempted a few more things to get greenlight updated.

1st attempt

- I modified the schema.rb file to only modify the rooms and users table as these tables already exist.  Modified code below:  

change_table(:rooms) do |t|

  t.string "access code"

  t.boolean "deleted", default: false, null: false

  t.index ["deleted"], name: "index_rooms_on_deleted"

end

change_table(:users) do |t|

  t.boolean "deleted", default: false, null: false

  t.index ["created_at"], name: "index_users_on_created_at"

  t.index ["deleted"], name: "index_users_on_deleted"

  t.index ["email"], name: "index_users_on_email"

  t.index ["provider"], name: "index_users_on_provider"

end 

1s attempt - result:  rebuilt the docker container... same error as before.  logs are complaining that the database already exists:

Database 'db/production/production.sqlite3' already exists

>>> Database migration

rake aborted!  

2nd attempt

- In a new git branch, I tried removing the database before the upgrade but I can't seem to be able to delete it

- command:  git rm production.sqlite3

- result:  fatal: pathspec 'production.sqlite3' did not match any files

Can anyone help? I'm just trying to upgrade to the latest Greenlight from a the previous version with a few html customizations

[Ahmad Farhat]

> Database 'db/production/production.sqlite3' already exists=

> >>> Database migration

> rake aborted!

The above errors are expected behaviour. The issues you're having are with the migrations themselves. There are some known issues that occur when updating from a really old version of Greenlight

At this point you might be better off moving your database to another folder, starting from scratch, and then manually importing your data.

If you want to do that, make a copy of your database first, cp ~/greenlight/db/production.sqlite3 place-you-want to copy to

Then remove the database and start again sudo rm -f ~/greenlight/db/production.sqlite3

[Carl Gosselin]

ok.  Can you provide some guidance on importing the data after the migration?  Looks like I'll need to import the users and rooms tables.

Also, will the recordings still be linked properly to the user accounts?

[Ahmad Farhat]

I guess the question I should've asked was how many users and rooms do you have?

You might need to add them back one by one because of some of the migrations that you changed. It's probably going to be impossible to do it in one shot

The recordings will still be linked properly as long as you set the bbb_id field for the new rooms to match the old room

[Carl Gosselin]

We currently have about 400 users and growing rapidly.  

So there is no other option than to add them manually one by one in the new database?

[Ahmad Farhat]

Let's try something else first.

It'll be a back and forth process between me and you, so it will take a bit of time.

First step:

Remove 3 lines starting at:

https://github.com/bigbluebutton/greenlight/blob/master/db/migrate/20190314152108_rolify_create_roles.rb#L21

Then rebuild the image, rerun (it will probably fail again) and send me the next logs that appear

[Carl Gosselin]

Thanks for the help.  I appreciate it.  

Here is the latest Docker log (attached)

[Ahmad Farhat]

Oh wow it looked like it actually worked completely. You just need to add the "user" role to all users now.

1- docker exec -it CONTAINER_NAME bash

2- bundle exec rails c

3- Paste the following lines, then hit enter

User.all.each do |user|

user.add_role(:user) if user.roles.blank?

end

4- It should take some time to run, and you should hopefully be good to go

[Carl Gosselin]

Looks like we're making progress.  I can see the new tables in the database (see screenshot).

However, browser is still showing the following...

404 Not Found

---

nginx/1.10.3 (Ubuntu)

[Carl Gosselin]

Something ran after I entered: bundle exec rails c (coudn't paste the other 3 lines).  Should I try again from docker exec -it custom bash ?

[Carl Gosselin]

Here is the output (attached)

Again... wasn't able to paste the following lines as it executed too quicly:

[Ahmad Farhat]

That error happened after doing the bundle exec rails c? or after running the User.all.each....?

Also, can you try starting up the image, waiting until everything compiles (~1 min) and seeing if you can access Greenlight?

If not, what are the errors in the docker container?

[Carl Gosselin]

It happened after doing the 'bundle exec rails c'

Didn't get the chance to add the User.all.each...

Should I try again?

[Ahmad Farhat]

Nono no need. I just saw the last line

> /usr/src/app/app/controllers/application_controller.rb:269: syntax error, unexpected end-of-input, expecting keyword_end

You have a syntax error in your merged code. Most likely means you have and extra "end" that doesn't correspond to a "def" block.

Try fixing that and then rebuilding and following the steps I can above

[Carl Gosselin]

Good catch.  I fixed the syntax error.

I tried re-running the following steps but I can't seem to type them fast enough on the command prompt.  Basically the 'bash' displays for half a second before coming back to the normal prompt.  I can't seem to execute the following on the command line...

1- docker exec -it custom bash

2- bundle exec rails c

3- Paste the following lines, then hit enter

[Ahmad Farhat]

Seems like your docker container keeps restarting. Have to go back to the docker logs now. What's the output of "docker logs IMAGENAME"

[Carl Gosselin]

latest docker log attached.

I also noticed a new error in the browser (instead of the '404 not found'):

500 Internal Server Error
If you are the administrator of this website, then please read this web application's log file and/or the web server's log file to find out what went wrong.

[Ahmad Farhat]

> Error during failsafe response: can not load translations from /usr/src/app/config/locales/en.yml: #<Psych::SyntaxError: (<unknown>): mapping values are not allowed in this context at line 267 column 12>

Theres another syntax error in your config/locales/en.yml

[Carl Gosselin]

Shoot!  the line wasn't indented properly.  fixed.

Good news.  The site is up!  thank you so much.

Net step:  I still require to run the following correct?

[Carl Gosselin]

I ran the following and got the following error message....

$ docker exec -it custom bash

bash-4.4# bundle exec rails c

User.all.each do |user|

  user.add_role(:user) if user.roles.blank?

end

Loading production environment (Rails 5.2.3)

irb(main):001:0> User.all.each do |user|

irb(main):002:1*   user.add_role(:user) if user.roles.blank?

irb(main):003:1> end

Traceback (most recent call last):

        3: from (irb):1

        2: from (irb):2:in `block in irb_binding'

        1: from app/models/user.rb:200:in `add_role'

NoMethodError (undefined method `strip' for :user:Symbol)

irb(main):004:0> 

[Ahmad Farhat]

Try

User.all.each do |user|

  user.add_role("user") if user.roles.blank?

end

[Carl Gosselin]

Here's the output...

$ docker exec -it custom bash

bash-4.4# bundle exec rails c

Loading production environment (Rails 5.2.3)

irb(main):001:0> User.all.each do |user|

irb(main):002:1*   user.add_role("user") if user.roles.blank?

irb(main):003:1> end

Traceback (most recent call last):

        2: from (irb):1

        1: from (irb):2:in `block in irb_binding'

NameError (undefined local variable or method `  user' for main:Object)

irb(main):004:0> 

[Ahmad Farhat]

It doesn't look like it copied over to the console properly. Can you manually type it out?

[Carl Gosselin]

ok thanks.  Looks like it processed the query now.  Here is an abbreviated output (removed real email, etc.....)

$ docker exec -it custom bash

bash-4.4# bundle exec rails c

Loading production environment (Rails 5.2.3)

irb(main):001:0> User.all.each do |user|

irb(main):002:1* user.add_role("user") if user.roles.blank?

irb(main):003:1> end

=> [#<

User id: 1, room_id: 1, provider: "greenlight", uid: "gl-abvmcxjuftfz", name: "carldgosselin", username: nil, email: "rem...@email.com", social_uid: nil, image: "", password_digest: "sdfkjsdlkjdslfjdlskfjlksdjflsdjsdlfj...", accepted_terms: false, created_at: "2019-04-02 20:07:37", updated_at: "2020-03-18 00:18:38", email_verified: true, language: "default", reset_digest: "dsfjlsdjfldskjflkdsjflkdsjfldsf...", reset_sent_at: "2020-03-18 00:18:38", activation_digest: "lskdjflkdsjfldsjfldskfsd...", activated_at: "2019-04-02 20:07:37", deleted: false>, #<User id: 2, ...

I took a look at the database.  Looks like the new tables are still empty though.  e.g. user roles table is still empty.  Is this right?

[pheenil chauhan]

how to connect database sqlite3 with bigbluebutton server 

i am finding database connection but i am not getting my database 

[Matias Silva]

Hi pheenil chauhan,

BigBlueButton doesn't require a database as per the documentation. May I
suggest you seek professional help as you seem to have difficulty
grasping a few technical concepts? We're happy to help but since this is
a repeat occurrence perhaps you should consider this.

---
Thanks, Matias

On 2020-09-24 07:59, pheenil chauhan wrote:
> how to connect database sqlite3 with bigbluebutton server
> i am finding database connection but i am not getting my database
>

> --
> You received this message because you are subscribed to the Google
> Groups "BigBlueButton-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to bigbluebutton-...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/bigbluebutton-dev/49ef07a9-e6e7-470a-9956-794c40d8ed42n%40googlegroups.com
> [1].
>
>
> Links:
> ------
> [1]
> https://groups.google.com/d/msgid/bigbluebutton-dev/49ef07a9-e6e7-470a-9956-794c40d8ed42n%40googlegroups.com?utm_medium=email&utm_source=footer