Docker, devops, easier install, and whatever else seems relevant to discuss

[Michael Greene]

Hey folks, We had the discussion below on the sakai-marketing list, I've anonymized it and moved it over here to get more input. My primary question is, would you be interested in a community meeting around devops, docker, dev workflow, easier install, or whatever else might come up along those lines? The core call has too much agenda to fit these topics in there and I was wondering if it warranted a call vs google group/slack channel. I'm person F in the thread below and am of the opinion that we can make it easier to install, kick the tires, or onboard new developers if more active developers used a container based workflow than the current workflows. What would it take for you to reconfigure your machine to use something other than manually installed java/maven/tomcat?

Thanks!

MG

Begin thread from sakai-marketing. new entries separated by ======

Hi,

For next time's agenda, the following point:

As discussed briefly during the marketing group's meeting, I think that truly from marketing perspective, Sakai needs to provide a fool-proof installation guide for a vanilla installation for (the preferred) linux distribution, Windows (10?) and MacOS.

Reason I'm stating is that it took me ca. 2 hours to get Moodle up and running on a totally formatted, no operating system running laptop, following the instruction they have on their 'Step-by-step' installation guide for Ubuntu (in my case).

https://docs.moodle.org/38/en/Step-by-step_Installation_Guide_for_Ubuntu#Step_1:_Install_Ubuntu

Looking at the Sakai Developers group and sakai Users group I saw from very recent posts that I tend not to be the only person that has this experience. It was while I tried to showcase a colleague how easy it would be...

It is equally simple to have something like this, with a 100% guarantee success rate* (*if you follow the letter) created for Sakai. NOT starting somewhere 'halfway' down the road, assuming that some steps have been performed already, but truly from 'nothing' to an up-and-running instance of the latest version of Sakai. It didn't take long to convince my colleague to show why Sakai was better, but I'm not sure if I got her buy-in on the ease of installation.

I'm more than willing to run any 'setup description' pas my utter unknowledgeability to test it. If even I can do it, anyone can!

Im pretty sure you can turn this is a 'to the point' agenda point for discussion! 😊🙏

Kind regards,

Person A

======

+100

I am so tired of nagging about this while our community is shrinking. No one can install this damn thing.

For QA to control the properties and configuration easily.

For DEV to begin contributing at even the tiniest level.

For Demonstrations to peers

For LTI developers

For anyone who wants just to kick the tires - the more familiar people are with it, the better our chances of adoption

Person B

======

+1

Person C

======

Hi all,

It's my plan to get this on the agenda for the next marketing meeting in 2 weeks, so that we can get a full airing of the arguments on both sides of this issue. So, please come and share your views.

In the meantime, here's my $0.02. I take the point that installation of Sakai is needlessly hard. But as one who deals with lots of schools making an LMS transition, schools aren't leaving Sakai, or choosing a competitor when they switch to a new LMS, because of this issue. 

Person D

======

And new schools aren't choosing Sakai either. But maybe they would if we made it easier.

Person B

======

I have been thinking a lot about this lately.   README files for projects like Sakai are not well set up for adopters - they are set up for those that will be involved with an ongoing project.

Moodle is designed from the beginning to be relatively easy to use and 90% of the people use it and never submit code.  Also many hosting environment pre-install 80% of the infrastructure to install Moodle (PHP+MySQL) - Getting a PHP install is no small task on its own.  Installing MAMP is easy and making Moodle work on MAMP is easy.   Or making it work on a hosting provider like DreamHost is easy.

If you look at the Canvas documentation, Blackboard(internal), or D2L (internal) documentation - they are almost as obtuse as Sakai’s because they are for insiders and developers - frankly this is why I cannot use the HAX web components stuff or PatternFly … - The getting started is crazy difficult.

Here is the problem - it is actually pretty complex to do it from a naked operating system and there is nothing like MAMP that speeds it up.

What Blackboard did for developers is that they made an AMI and directions on how to configure it.  I am going to do that for Tsugi.   So if you want to run production - just use my AMI.  One stop shop - two pages of instruction max.

For developers it is Docker - and we have some of that working.

I am working on a “getting started” for Tsugi that demonstrates this approach.  Once I figure out all the logistics - I can do a Sakai AMI and perhaps even a Docker.

By the way - I got so tired of reading our own documentation that I built a set of scripts to automate the process:

https://github.com/csev/sakai-scripts/

You could try following the README on that on a bare Ubuntu box - and it should be up and running pretty quickly.   But is is still not as good as an AMI :)

I think this is the way forward.

Person E

======

I think Docker is the answer for more than developers. Especially people using mac/pc, the app has a desktop installer that resembles the MAMP experience and we already have a robust set of images thanks to David Hutchins' contribution and there's a smattering of docker images around that aren't even in the project repo. 

I think there are some workflow improvements that could be done for people that aren't looking to become core contributors and just want to kick the tires. It seems like moving what is currently in sakaiproject/sakai/docker to its own repo sakaiproject/docker is a pattern I've seen elsewhere, odoo is one example I found but I'm not deep enough in this space to know who the exemplars are that we should emulate. It seems like we need to push and maintain the docker image at https://hub.docker.com/r/sakaiproject/sakai as well or remove that if it's not applicable anymore?

The current workflow for using sakaiproject/sakai/docker confused me because I had to essentially clone the source code twice, moving the docker folder to it's own repo would resolve that problem and there may be additional changes that could make our docker install even more approachable to people. Asking people to do a git clone and a docker-compose to get an enterprise scale web app up and running is perfectly acceptable but it has to be rock solid and just work. 

I wonder if more of our core developers used the docker stack for their personal setup if we'd reach more alignment/progress here? I assume not many are currently using it because they are experts at the traditional install or virtual machine install or have created their own docker images. I've not asked them, so that may be a faulty assumption.

An AMI is also a good idea but I don't have any experience creating/maintaining those, so I don't know how difficult it would be to maintain them for more than 1 cloud provider. Here's a link to what they were referring to https://aws.amazon.com/marketplace/pp/Blackboard-Inc-Blackboard-Learn-for-REST-and-LTI-D/B077T4SX2B

I find the irony of searching the most popular java repos on github for good examples only to find this a reminder that even large communities struggle with this issue and it's just not an easy thing to solve.

Thanks!

Person F

======

In my work with Tsugi, docker is necessary but not sufficient if folks want to install and run the product.  For Tsugi, two years ago I went through a docker and swarm phase in Tsugi on the Google Infrastructure (I think I borrowed my first Dockerfiles from you actually).  I made it all work smoothly for test / demo servers - but was very far from anything that I would call production grade.  Docker is good at stamping out linux images with a minimum of fuss.  But if you think about backup / restore / scaling space or performance - docker is no where near sufficient.

If you look at the current Sakai docker stuff - I am sure if is very elegant making the images - but putting those images into production for a place like Duke is almost more work than reading the Sakai install documentation.

Once I realized that docker was not easy to mint production boxes - I gave up on Docker completely and started building AMIs that I could instance and configure and then back with EFS and Aurora and a load balancer.   I can now make a production grade autoscaling Tsugi in abut 10 minutes to handle 100,000 users in production with about a page of steps.

But then I had to tweak my build process to handle more complexity where Tsugi was just one piece of an overall production environment.  The server to support my Postgres course includes Tsugi, MySQL, PostgreSQL and ElasticSearch.  As I set out to build and test this environment, I realized that AMI without docker was great for production but terrible for dev / test / demo purposes.

So what I have now built is a set of shell scripts that construct my instances with 1-page of AMI instructions and several docker files that are super short and only run those setup scripts.

With that I could test all my build processes as I developed them on my laptop with nothing but docker and then when I was all done - I used the same build process slightly differently to make the “golden AMI” to run production.  It worked really well.

What I am doing now is making it so that instead of making my own AMI and using it - I make a community AMI that others can just grab and configure to have production Tsugi - the Blackboard image you mention below is my inspiration - I ran a dev blackboard server with built-in Postgres on Amazon for 2 years so I could test Tsugi with Blackboard during the development of LTI Advantage.  I found that process amazing.  I had an AMI vendor and about once per quarter I could get the new version and upgrade.

I want to do this hybrid docker (upload to docker hub) and community AMI in Tsugi and then use that experience to design a real solution that will benefit both devs and smaller adopters for Sakai.

Person E

======

Very interesting point, thank you so much.

As you may know I've been helping many users in the DEV list in the past years, helping them to get their installations up and running, Windows and Linux.

Not a problem for me, I enjoy doing it because I was helped in the past too, however, most of them get stuck on the same problems again and again. The most common problems are not related to the documentation which works fine for a developer with experience in Java, Maven and Tomcat, they are more related to common mistakes of people not familiar with that stack (Version mismatch, typos in connection strings, missing steps in maven or tomcat, etc) it's normal and frustrating when you deal with a pile of new concepts.

Using a technology that deals with all those steps and that stack looks very good to me, most users don't want to deal with Maven, Java, Tomcat, MySQL, etc, they just want to use Sakai in the browser and play with it. 

And you're right, I did more than 50 Sakai deployments since 2008 and I never worked with Docker or similar solutions, I use linux/windows scripts, but having a wizard or a double click bundle to install Sakai sounds the best option to me for most users.

Person G

======

It sounds like a continuum to me, maybe something like this?

Manual install - - docker - - ami - - kubernetes? (I know almost nothing about kube, there’s a guy on my team that’s really into it and it’s the direction our infrastructure is going, so i don’t know if it’s relevant)

I definitely hear you on certain aspects like databases having real issues with docker in production. I wasn’t thinking about that use case in my previous message, only the kick the tires or onboarding a new developer/tester use cases. In theory, it could be baked into our automated testing workflow too, right?

One of the things I like about our current set of docker images is they are already separated by service, it shouldn’t take (last words of every noob) too much effort to pull the DB portion out to a standalone DB service. That isn’t currently covered in the docker documentation we have.

Person F

======

We should move this out of the marketing list which is (a) private and (b) missing a lot of expertise in devops.

We might even want to start a small Sakai subgroup like UI, etc - called "Devops" to progress this better.

Person E

[Matthew Jones]

Hi Michael, 

This comes up often. I remember when I was an early Java developer and if you "know what to do" now installing Sakai is so much easier than it was back then. However knowing "what to do" isn't well documented or very easy to find

1) There's (at least) 3 different audiences that want to deploy Sakai.

  a) People who want to run a local demo instance. 

      Moodle has this and Sakai USED to have this but Sakai doesn't release this "all-in-one" distribution anymore. There are reasons for this but if this still was important we could potentially make this much easier. The closest you get is a binary that you have to drop into Tomcat. But even this step alone can be confusing. And it requires MySQL (but so does Moodle).

  b) People that want to build from source and deploy easily locally. 

      I feel like the biggest problem for this is that it's realitively easy on Linux but many people don't run Linux. It's somewhat easy on OSX but  a lot of people still have Windows where this can be a pain and varies depending on if you have Windows Home or anything else.. However if you have Windows 10 Enterprise/Pro/Education and, use a Docker based install with WSL Windows it's a now. I don't feel like Moodle is that much easier here.

  c) People that want to install and run it on a cloud service provider. We don't really make this very easy. 

For 1b I wrote this script a few years ago and have been using it exclusively to build, run and install Sakai with Docker. All you need is Docker. I just tested it with WSL2 (on Windows, recently released) and Docker Desktop. If you follow these steps to do that first it's really easy to run this. If you're on Linux or OSX just install Docker, clone the repo and you're 90% of the way toward running Sakai locally. 

I was interested in WSL2 this morning and set it all up, built/installed and started Sakai in under an hour WHILE watching Open Apereo. After it's setup it's much faster to update and reinstall.  I'd be interested in if anyone uses this and has an problems with this. I'd be interested in if anyone has advice for improving this. I believe this is "similar" to the docker code in master but that looked to be more for the 1c case than 1b. 

If we use Docker we can control everything, from the java version, to the maven version to the mysql version. We can hardcode all of the startup commands someone needs and provide all of the configuration. I feel because Sakai still requires older specific versions of things this is what causes people the most trouble trying to run it outside this environment. 

I don't have much time to work on Sakai anymore and I need it easy/fast. I previously used Virtualbox but after seeing the startup of WSL2 I expect I'll switch over. I feel like we really should embrace docker-compose, docker-swarm or some type of container-first solution as trying to pack everything into one think Sakai container really doesn't seem like it makes things easier. 

--
You received this message because you are subscribed to the Google Groups "Sakai Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sakai-dev+...@apereo.org.
To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/sakai-dev/27a1808f-c256-4379-92de-dbe0403d8843o%40apereo.org.

[P Ngyen]

It's about time for someone to bring up the "elephant-in-the-room".   I belong to the audience in 1. (as listed below).  Once Sakai has been dockerized into an image (with an efficient Dockerfile), a lot of people will be start to look at Sakai in a new light.  Audience in 1a.  can whip up a demo in shorter time.   Audience in 1b.  will be able to tweak the Dockerfile and customize according to his/her environment.

Just my $.02

P Nguyn

On Tuesday, June 16, 2020 at 11:26:19 AM UTC-5, Matthew Jones wrote:

Hi Michael, 

This comes up often. I remember when I was an early Java developer and if you "know what to do" now installing Sakai is so much easier than it was back then. However knowing "what to do" isn't well documented or very easy to find

1) There's (at least) 3 different audiences that want to deploy Sakai.

  a) People who want to run a local demo instance. 

      Moodle has this and Sakai USED to have this but Sakai doesn't release this "all-in-one" distribution anymore. There are reasons for this but if this still was important we could potentially make this much easier. The closest you get is a binary that you have to drop into Tomcat. But even this step alone can be confusing. And it requires MySQL (but so does Moodle).

  b) People that want to build from source and deploy easily locally. 

      I feel like the biggest problem for this is that it's realitively easy on Linux but many people don't run Linux. It's somewhat easy on OSX but  a lot of people still have Windows where this can be a pain and varies depending on if you have Windows Home or anything else.. However if you have Windows 10 Enterprise/Pro/Education and, use a Docker based install with WSL Windows it's a now. I don't feel like Moodle is that much easier here.

  c) People that want to install and run it on a cloud service provider. We don't really make this very easy. 

For 1b I wrote this script a few years ago and have been using it exclusively to build, run and install Sakai with Docker. All you need is Docker. I just tested it with WSL2 (on Windows, recently released) and Docker Desktop. If you follow these steps to do that first it's really easy to run this. If you're on Linux or OSX just install Docker, clone the repo and you're 90% of the way toward running Sakai locally. 

I was interested in WSL2 this morning and set it all up, built/installed and started Sakai in under an hour WHILE watching Open Apereo. After it's setup it's much faster to update and reinstall.  I'd be interested in if anyone uses this and has an problems with this. I'd be interested in if anyone has advice for improving this. I believe this is "similar" to the docker code in master but that looked to be more for the 1c case than 1b. 

If we use Docker we can control everything, from the java version, to the maven version to the mysql version. We can hardcode all of the startup commands someone needs and provide all of the configuration. I feel because Sakai still requires older specific versions of things this is what causes people the most trouble trying to run it outside this environment. 

I don't have much time to work on Sakai anymore and I need it easy/fast. I previously used Virtualbox but after seeing the startup of WSL2 I expect I'll switch over. I feel like we really should embrace docker-compose, docker-swarm or some type of container-first solution as trying to pack everything into one think Sakai container really doesn't seem like it makes things easier. 

To unsubscribe from this group and stop receiving emails from it, send an email to saka...@apereo.org.