Django Box - A vagrant virtual machine for testing Django

[Josh Smeaton]

Hi All,

At the PyCON AU sprints I began working on a project for building a vagrant virtual machine that would facilitate testing django, itself, against most of the databases that django supports. I forked https://github.com/jphalip/djangocore-box which was last updated about 3 years ago and made a number of improvements such as provisioning with ansible roles that django uses to configure the jenkins environment, utilising the new tox.ini file that django provides, and adding commands for running flake8, isort, and docs spelling tests.

You can find the new project here: https://github.com/django/django-box

Note that there's currently no support for just running `vagrant up` because we haven't actually deployed the compiled box image yet. That's the reason for my post today. I've got a few questions that we should talk through.

1. Should we host the django-box project under the django organisation?

I pushed the project under the django organisation so I could have some review to make sure that no private credentials were exposed. That doesn't mean the project needs to remain here though. Where does the community think this project should live? What about the current core devs? Should this project be proposed for inclusion within the django organisation under DEP7 https://github.com/django/deps/blob/master/final/0007-official-projects.rst if consensus leads us to leaving it within the django organisation?

2. Where should we host the final box image?

The original vagrant image is hosted under djangoproject.com at https://www.djangoproject.com/m/vms/djangocore-box-1.1.box but that doesn't mean the new image should be hosted there too. We could host the image on atlas.hashicorp which is the official place to share box images. I think that'll make it more discoverable and easier to update. If we do want to host on atlas.hashicorp, should it be under a django account or a private (mine?) account? I guess this last question depends on the answer to where the repo is hosted.

Finally, I'd just like to try and gauge the interest in this project. I found the original djangocore-box incredibly useful for developing against the ORM. I've also spoken to a number of people at sprints that have had quite a bit of trouble getting their machine set up with all of the dependencies required to run the full test suite. There was a very short lived project about 12 months ago to get a docker environment going with all of the dependencies but it fizzled out rather quickly. This vagrant image supports just about everything except Oracle and Windows testing. Would you use this?

Thanks,

Josh

[Shai Berger]

Hi Josh,

This looks very promising. Thanks for making it.

On Sunday 28 August 2016 08:55:05 Josh Smeaton wrote:
>
> *1. Should we host the django-box project under the django organisation?*
>
> [...] Should this project be proposed for inclusion within the django

> organisation under DEP7
> https://github.com/django/deps/blob/master/final/0007-official-projects.rst
> if consensus leads us to leaving it within the django organisation?
>

I think yes on both counts (host under Django and DEP7), but see below.

> *2. Where should we host the final box image?*
>

Sorry, I am not experienced enough with vagrant and its ecosystem to form an
opinion.

>
> Finally, I'd just like to try and gauge the interest in this project.

> [...] Would you use this?
>

I have my environment set up already, and I suspect this would hold for a lot
of the more experienced developers here; do you think the new box will improve
things for me?

On a different angle -- is the intention to have this box set up to ease
testing of django apps?

Even if the answer to both above questions is "no", I suppose it will make it
easier to make the first contributions; this is enough, IMO, to justify DEP7
inclusion.

Have fun,
Shai.

[Josh Smeaton]

I have my environment set up already, and I suspect this would hold for a lot
of the more experienced developers here; do you think the new box will improve
things for me?

It probably won't make a big difference for those that already have environments setup for django development, but it'll definitely ease the process for newer contributors. Where it will benefit those of us who already have environments setup is when django deprecates older databases or libraries, so that you'll always be able to have a newer environment without manually going through and upgrading existing components. The commands for running flake8/isort/spelling would help with discovery unless you've already transitioned to running them with the new tox configuration. 

 

On a different angle -- is the intention to have this box set up to ease
testing of django apps?

Not really, no, but that doesn't mean it can't be used for that purpose. All that's required is a new synced_folder declaration within the Vagrantfile that'd mount a users project within the virtual machine, and then they can link to the existing settings modules to run their tests against the various databases. I guess this would be more useful for library developers rather than regular projects. For more advanced projects that want to use vagrant for development, they could include their own Vagrantfile that uses the django-box as a base from which to build upon.

Thanks for the feedback.

 

[Josh Smeaton]

An interesting idea put forward by Ed Morley on the issues tracker (https://github.com/django/django-box/issues/12) is to merge the django-box project into the django main repository directly. The meat of the suggestion is copied below:

The ideal end state I see is that this project lives in the main django repo. That way (regardless of compiled box vs build locally) people just have to:

• git clone django repo

• install virtualbox/vagrant

• vagrant up

...with no added complication of having to clone a second repo and put it in the correct location.

I like the idea in principle, but I realise it'd add a non trivial burden to the existing django project for what is an unproven project. Still, if I'm already soliciting feedback it's worth asking about this too :)

Cheers 

[Tim Allen]

I'd be happy to contribute configuration for connecting Django to SQL Server. I've had to deploy this for some projects at work that don't (yet) use PostgreSQL. This sounds like a fantastic tool for testing, as well as examples for people just getting started; wiring up a database for the first time can be a challenge, with many moving parts in several layers.

[Tim Graham]

I don't think a formal approval/proposal is needed to keep django box under the Django org. For example, tools to help with Django development don't require things like fixing security issues (I hope). As long as it's working as semi-maintained, we can add a link to it from the contributing docs. If it's neglected and becomes an embarrassment then we could remove it.

Hosting on atlas.hashicorp seems fine. If you create an account with some credentials you can share in case you stop maintaining it, that sounds good.

I don't want to force any particular development workflow on anyone which is why I'm not thrilled with the idea of including this in the Django repo itself. It might give the impression that all new development work requires you to fix the box as well or if not, then we'll see later commits just fixing the box which sort of clutters Django's commit log.

[Josh Smeaton]

Hi Tim,

Doesn't SQL Server still require windows? I know there are plans from Microsoft to allow SQLServer running on Linux and drivers for linux to connect to SQL server, but I wasn't aware that those projects had materialized yet. This specific Vagrant image is an Ubuntu 16.04 virtual machine. If we need Windows as well that'd probably involve a new Vagrant file capable of spinning up a windows machine, and refactoring the ansible roles to handle both. Perhaps you can open up an issue on the django-box repo so we can discuss it properly there?

Thanks

[Josh Smeaton]

I don't think a formal approval/proposal is needed to keep django box under the Django org. For example, tools to help with Django development don't require things like fixing security issues (I hope). As long as it's working as semi-maintained, we can add a link to it from the contributing docs. If it's neglected and becomes an embarrassment then we could remove it.

I was kind of thinking along the same lines. We aren't deploying packages and it won't be a critical piece of infrastructure to any users projects.

 

Hosting on atlas.hashicorp seems fine. If you create an account with some credentials you can share in case you stop maintaining it, that sounds good.

Seems reasonable to me.

 

I don't want to force any particular development workflow on anyone which is why I'm not thrilled with the idea of including this in the Django repo itself. It might give the impression that all new development work requires you to fix the box as well or if not, then we'll see later commits just fixing the box which sort of clutters Django's commit log.

Yeah I feel the same here. Although it'd be nice to be able to do a simple `vagrant up` and get working on Django itself, it comes with quite a bit of overhead for everybody else. Having it hosted outside of the core repo means we can move at a different pace which is a good thing for a "newer" project. 

[Josh Smeaton]

Hi all, a quick update.

https://github.com/django/django-box now has a live compiled box available from https://atlas.hashicorp.com/djangoproject/boxes/django-box-1.11

What this means is that a minimal Vagrantfile can now be:

Vagrant.configure("2") do |config|

  config.ssh.forward_agent = true

  config.vm.synced_folder "../django", "/django", nfs: true

  config.vm.network "private_network", ip: "1.2.3.4"

end

If we choose to compile the box with a Vagrantfile approximating the above, then getting a vagrant image would be as simple as:

mkdir djangobox

cd djangobox

vagrant init djangoproject/django-box-1.11

vagrant up

No need for cloning the djangobox repository at all, but does still require the image to be installed in a folder beside the django directory.

There's a further step we could go to, which would be bundling only the Vagrantfile within the django repo. It'd look something like this (provided we bundle the previous Vagrantfile within the image):

Vagrant.configure("2") do |config|

  config.vm.synced_folder ".", "/django", nfs: true

end

We might want to play with that a bit more to be solid on linux/windows/osx, but it removes the concept of compiling a system and places it within the django-box repo, but still allows a user to checkout django, and bring up a VM immediately to do testing. I'm liking this avenue a lot more than I had previously, because it only involves bundling a Vagrantfile and not the entire ansible roles needed to compile it.

I should say that I'm OK with waiting and proving that this project has some following before moving down that road. No point bundling a Vagrantfile if no one is going to use it. But I can see the value and simplicity now. In the mean time, I'll try and get around to updating the Wiki to point people to the django-box image, and the various docs around the place.

Thanks