Proposal: automating setup and upgrade of development environment

[Federico Capoano]

I think it's time to invest in the automation of the initial setup of the development environment.

Here's what I do.

I have a directory in which I clone every OpenWISP module, I install each module using:

pip install -e .

In my editor I open the main openwisp directory as a project so I can edit all the repositories at once.

Each time there are new changes I pull from git, reinstalling via pip if needed:

- pip install -U -e .

Some modules, require extra dependencies, eg:

- django-freeradius / openwisp-radius require cairo

- openwisp-monitoring requires influxdb

We could automate this. Automating the upgrade is useful and would give us a reason to keep using the tool so we can keep it up to date always.

The procedure should be:

- create a virtualenv using python 3.6 or higher, return error if no python 3.6 available on the system, if virtualenv already exists skip

- clone each openwisp module, starting from the base dependencies, openwisp-utils, netjsonconfig, netdiff, django-x509, etc, if repo is already cloned, pull from the current branch (eg: git pull, this would allow us to change branches in development)

- install any required system package (eg: cairo, influxdb, sqlite), we may as well add these system packages in the repositories somehow

- install the python package with pip install -U -e .
- if it's a django package, cd into tests/ and run ./manage migrate, then check whether there's a superuser in the DB, and if not, create one with username/password admin/admin

I'd initially aim to support only linux distributions like Debian, Ubuntu, Fedora.
We may implement this as a bash script in openwisp-utils, or something else like ansible.

Having it in openwisp-utils would have the advantage of being integrated with the rest of the best practices we're introducing, so one would have to clone and install only openwisp-utils manually, then openwisp-utils does the rest.

The disadvantage is that it may be cumbersome to support both Debian and Fedora, but I think the differences are just in the package manager and package names, that can be easily resolved.

Then we'd just have to document this feature in https://github.com/openwisp/openwisp2-docs/

Providing suggestions regarding forks and things like that.
For forks, I'd suggest to add it as a remote named "fork", and suggest to always set development branches to track the fork branch, so that "git pull" will work without problems when upgrading.

This should help new contributors getting up to speed very quickly.

Let me know what you think.

Federico

[Federico Capoano]

I already thought about an improvement of the idea.

We mainly have 3 default roles right now:

- superadmin (can do everything)

- administrator (can manage most of the objects of their org)

- operator (limited permissions, usually a collaborator of a company or volunteer of a no-profit who help out with specific operations)

I mostly test with superadmin and lately while testing with administrator and operator I've found many issues.

We could have the procedure automatically create these 3 roles for every openwisp module which is also a django project and makes use of openwisp-users, eg:

- find out if it's a django project (has the file tests/manage.py or tests/openwisp2/manage.py), if not, stop this operation

- create superuser, eg: superadmin/superadmin (if it doesn't exist yet)

- if there's a group (as django.contrib.auth.models.Group), create a staff user which is named after the group, all lowercase, so if the group is named "Operator", the user will be called "operator" (a small exception that in case the result is "administrator", abbreviate that to "admin", once the user is created, assign it to the default organization. The operation should be skipped if user already exists

This way we'll have the 3 roles created automatically for each project which we work on and we'll be able to test much faster.

[Ajay Tripathi]

Hi,

This sounds good to me, I think having ansible script would be better as it would give us idempotence which would be useful in development environment where a developer can manually, say delete db or virtualenv and re-create without worrying about the state of other modules.

Also, if feasible, we can create some other basic fixtures that are almost always used during testing as well, like when working with controller, I always create a dummy device, with network-topology, I create a graph and so on, this could potentially speed up the manual testing process as well. :-)

Best,

Ajay Tripathi

[Federico Capoano]

I wonder if with ansible we can use virtualenvwrapper instead of plain virtualenv.

Does anybody know?

I think we should discuss further the details of this idea and then start to decompose in smaller issues somewhere (either a new github repo if we go for ansible or openwisp-utils).

We can then implement this gradually over the next months.

[Oliver Kraitschy]

On Mon, Apr 06, 2020 at 10:56:09AM -0500, Federico Capoano wrote:
> I think it's time to invest in the automation of the initial setup of the
> development environment.
>
> Here's what I do.

Hi Federico,

this is a good idea. To continue the brainstorming, I'll describe what we
use here.

Our OWM (OpenWISP Manager) consists of the following pieces:

- a file 'modules':
contains a list of the names of all OpenWISP repositories and, for
each repository, the following shell variables
- URL of our fork of the repository
- URL of the upstream repository, for pull requests
- branch to use, e.g. master
- sha of the commit to use

- a Pipfile which installs local copies of the repositories in the modules
file, plus development packages and any additional packages

- a directory 'django' which contains:
- manage.py
- a directory 'openwisp2' with the files settings.py, urls.py, wsgi.py
and __init__.py

- a makefile with the following targets (commands):
- prepare: create a build directory, clone all repositories specified
in the modules file into build/sources - using the given URL, branch
and commit; copy Pipfile, manage.py and the directory 'openwisp2'
into build/
- build: with pipenv, create a virtual environment in the build/
directory, install the packages from the Pipfile and thus all deps;
with manage.py, do makemigrations, migrate and createsuperuser
- run: with manage.py, launch runserver using the given settings.py
and urls.py
- clean: remove the virtual environment, delete the build directory

- a directory 'scripts' with scripts which do the work:
- create directories, copy the files
- clone the repositories and add remotes "origin" and "pullrequest"
- create and install the virtual environment, create and fill the
database
- run the application
- remove virtual environment and build directory

To be able to easily handle customizations and pending pull requests, we
have another directory 'patches' with subdirectories for the various
OpenWISP repositories. It contains patches which get applied by
"make prepare", after the repositories have been cloned.

I'm looking forward to hear your thoughts about our dev environment.

Greetings,
Oliver

[Federico Capoano]

Sounds cool. Glad to see you have a well structured way of working with OpenWISP.

The only thing not convincing to me was Pipfile, my experience with it was not great (I didn't feel it simplified my work) and I did not adopt it in my everyday workflow.

Regarding the system dependencies to run all the modules, do you manage those by hand or have you automated those as well?

Federico 

--
You received this message because you are subscribed to the Google Groups "OpenWISP" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openwisp+u...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/openwisp/20200414110611.GA3434%40okWS.

[Oliver Kraitschy]

On Tue, Apr 14, 2020 at 09:33:48AM -0500, Federico Capoano wrote:

> The only thing not convincing to me was Pipfile, my experience with it was
> not great (I didn't feel it simplified my work) and I did not adopt it in
> my everyday workflow.

We chose pipenv because it sounded like the new standard way of managing
dependencies and virtual environment. I guess that's good marketing by
Kennetz Reitz :-)
In reality, it has its flaws. The dependency resolution is sometimes
fragile and it downloads a lot of stuff although it could look it up on
PyPi.
I don't know if we would choose pipenv again for the next project. Maybe
we would just work with requirements files, pip and venv/virtualenv/etc -
all the included tools.

> Regarding the system dependencies to run all the modules, do you manage
> those by hand or have you automated those as well?

For the development machines, we didn't automate the installation of those
packages. It's mainly the dependencies of openwisp-controller which need
to be installed manually once.

I'm not sure if we should put such a dev environment into openwisp-utils.
It could also be handled as a separate repository so that openwisp-utils
doesn't come with stuff some people don't want. And we could maintain the
environment independently.

Greetings,
Oliver

[Federico Capoano]

On Wed, Apr 15, 2020 at 12:38 AM Oliver Kraitschy <oli...@okraits.de> wrote:

On Tue, Apr 14, 2020 at 09:33:48AM -0500, Federico Capoano wrote:

> The only thing not convincing to me was Pipfile, my experience with it was
> not great (I didn't feel it simplified my work) and I did not adopt it in
> my everyday workflow.

We chose pipenv because it sounded like the new standard way of managing
dependencies and virtual environment. I guess that's good marketing by
Kennetz Reitz :-)
In reality, it has its flaws. The dependency resolution is sometimes
fragile and it downloads a lot of stuff although it could look it up on
PyPi.
I don't know if we would choose pipenv again for the next project. Maybe
we would just work with requirements files, pip and venv/virtualenv/etc -
all the included tools.

I see, thanks for sharing the feedback. 

> Regarding the system dependencies to run all the modules, do you manage
> those by hand or have you automated those as well?

For the development machines, we didn't automate the installation of those
packages. It's mainly the dependencies of openwisp-controller which need
to be installed manually once.

You've been focused on the controller functionality.

The other modules bring other set of dependencies (radius, openwisp-monitoring).

It would be really great to have a procedure that installs everything (with flags to disable what is not needed) and a firmware image as well which has everything (so one can also try the complete feature set on OpenWRT, being a physical device or a VM).

It will take a while to get there but we can start gradually.

The feature set of OpenWISP is quite big and the problem is that most users are not aware of it. Automating the development environment could also push us and new contributors to document everything. 

 

I'm not sure if we should put such a dev environment into openwisp-utils.
It could also be handled as a separate repository so that openwisp-utils
doesn't come with stuff some people don't want. And we could maintain the
environment independently.

I think you're right.

Federico 

[Oliver Kraitschy]

On Wed, Apr 15, 2020 at 09:20:23AM -0500, Federico Capoano wrote:
>
> You've been focused on the controller functionality.
>
> The other modules bring other set of dependencies (radius,
> openwisp-monitoring).

That's true. There are some modules we don't have in our environment.

>
> It would be really great to have a procedure that installs everything (with
> flags to disable what is not needed) and a firmware image as well which has
> everything (so one can also try the complete feature set on OpenWRT, being
> a physical device or a VM).
> It will take a while to get there but we can start gradually.

That would be really handy. I would vote for an image for virtualbox to be
independent of a physical device and to be able to cover the whole feature
set. If somebody has a physical device, he can install openwisp-config on
it anyway (if it's OpenWrt based of course).

> The feature set of OpenWISP is quite big and the problem is that most users
> are not aware of it. Automating the development environment could also push
> us and new contributors to document everything.

The reproducibility will be better too because there won't be such a big
diversity in dev environments anymore, hopefully.

Greetings,
Oliver

[Federico Capoano]

Hi everyone,

so just to wrap this up, these are the points I got from this discussion:

- there's some consensus on using ansible to set up the development environment

- having an OpenWRT image ready to install on a VM or a device with all which is needed to work is OpenWISP would be very helpful (this overlaps with the plan of the next major release, which is good)

- the goals are:

  1. run the tool to get a full development env working out of the box and start contributing to openwisp

  2. run the tool periodically to update the openwisp modules according to one's configuration (if somebody is working on a fork & branch of some module, it may have to specify this in the configuration).

If anyone has anything else more to add, please do.

I want to start converting these ideas into an actionable plan on github.

Federico

[Federico Capoano]

Quick update!

- I created a repo to track the plan on this effort: https://github.com/openwisp/openwisp-dev-env

- I also created a related project board: https://github.com/orgs/openwisp/projects/15

Continuing the discussion, before getting started, I believe it's a good idea to look at how other successful open source projects implemented a similar tool: https://github.com/openwisp/openwisp-dev-env/issues/2

I propose continuing to discuss on github and start prototyping.

It would be great to have a first draft of the tool by the time the Google Code-In begins, so that we can help new contributors getting up to speed quickly, it would also be a great chance to stress test it and make it mature faster.

Federico

[Oliver Kraitschy]

Am Montag, 18. Mai 2020 20:44:50 UTC+2 schrieb Federico Capoano:

Hi everyone,

so just to wrap this up, these are the points I got from this discussion:

- there's some consensus on using ansible to set up the development environment

 

I doubt that ansible is the right tool. It's mostly for administration and configuration management. But we want to set up and manage a local development environment. Most people will run the tool either locally on their machine or in a virtual machine.I think there are faster tools for that job. I would prefer a shellscript or a makefile in combination with a shellscript. Idempotence can be easily achieved through the logic in the shellscript.

Just my two cents :-)

Greetings,

Oliver

[Oliver Kraitschy]

Additionally, an ansible playbook is harder to read and to maintain than a shellscript. It's also harder to implement a certain logic.

[Federico Capoano]

Oliver, I get your point and I agree with it.

I think the best thing is to look at how other projects did this.
I started a discussion here: https://github.com/openwisp/openwisp-dev-env/issues/2

Gitlab has a tool called Gitlab Development Kit, which is called with a command line tool named gdk, so commands look like "gdk init", "gdk update" and so on.
I like this approach, but I would like to find out more examples before deciding.

If I'm not wrong, if we use a Makefile we necessarily have to call "make" from the directory in which the Makefile is saved, right? Or not?

F.

--

You received this message because you are subscribed to the Google Groups "OpenWISP" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openwisp+u...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/openwisp/f79a4200-d750-4fb9-b393-a8b767d11857%40googlegroups.com.