Paver start of lms failing with Build failed: Unknown task: devstack

[Damion Young]

I'm now wondering whether there's something more fundamentally wrong with my system as, having failed to get a vagrant connection to my virtual machine working (see https://groups.google.com/forum/#!topic/edx-code/-PeyPS_nejQ), I've just tried seeing if I can get the lms started in the VirtualBox GUI.

I ran

sudo su edexapp

to switch users and the prompt now reads

edxapp@precise64:~/edx-platform$

as I think it should(?)

But when I try to run

paver devstack lms

, I get:

Build failed: Unknown task: devstack

Very frustrating - any ideas anyone?

Many thanks

Damion

[Yarko Tymciurak]

On Jul 22, 2014 9:00 AM, "Damion Young" <damion...@googlemail.com> wrote:
>
> I'm now wondering whether there's something more fundamentally wrong with my system as, having failed to get a vagrant connection to my virtual machine working (see https://groups.google.com/forum/#!topic/edx-code/-PeyPS_nejQ), I've just tried seeing if I can get the lms started in the VirtualBox GUI.
>
> I ran
>
> sudo su edexapp
>
> to switch users and the prompt now reads
>
> edxapp@precise64:~/edx-platform$
>
> as I think it should(?)

After you do this, if you list the directory ("ls") you should see essentially a listing which looks like

https://github.com/edx/edx-platform

Including the pavelib directory.

That's a good next check.

[Damion Young]

Thanks very much @Yarko Tymciurak - should have thought of that! It's empty (ie 'ls' returns nothing) which explains a lot. I'm a bit confused though as I did follow the instructions here: http://edx.readthedocs.org/projects/edx-installing-configuring-and-running/en/latest/devstack/install_devstack.html very carefully and didn't see any problems until I tried:

vagrant up

And while the virtual machine does appear to be running, some parts of it obviously haven't completed properly. Any ideas at which stage this directory is supposed to be filled and what might be causing it not to be? I'm wondering whether this is related to my inability to run:

vagrant ssh

Interestingly, the log for the virtual machine as it booted does log a number of port forwarding errors:

00:00:01.537342 NAT: set redirect TCP host port 9200 => guest port 9200 @ 10.0.2.15
00:00:01.537542 NAT: failed to redirect TCP 9200 => 9200
00:00:01.537547 NAT: set redirect TCP host port 8765 => guest port 8765 @ 10.0.2.15
00:00:01.537624 NAT: failed to redirect TCP 8765 => 8765
00:00:01.537629 NAT: set redirect TCP host port 8001 => guest port 8001 @ 10.0.2.15
00:00:01.537700 NAT: failed to redirect TCP 8001 => 8001
00:00:01.537704 NAT: set redirect TCP host port 8000 => guest port 8000 @ 10.0.2.15
00:00:01.537774 NAT: failed to redirect TCP 8000 => 8000
00:00:01.537778 NAT: set redirect TCP host port 18080 => guest port 18080 @ 10.0.2.15
00:00:01.537846 NAT: failed to redirect TCP 18080 => 18080
00:00:01.537850 NAT: set redirect TCP host port 2222 => guest port 22 @ 10.0.2.15
00:00:01.537918 NAT: failed to redirect TCP 2222 => 22

I wonder if this is the problem?

Thanks again

[Damion Young]

Don't know whether this might be the problem but have just noticed that my devstack\edx-platform directory on my host (Windows 8) machine is also empty - should vagrant have populated this when I ran:

vagrant plugin install vagrant-vbguest

? Could there be something wrong with my vagrantfile which I curled according to the instructions:

curl -L https://raw.github.com/edx/configuration/master/vagrant/release/devstack/Vagrantfile > Vagrantfile

?

[Nacho P]

Hi all,

You're running it under Windows, is it right? Maybe you haven't set the environment variable  VAGRANT_USE_VBOXFS = true   and it could explain your problems with NFS folders.

Hope it helps. Best regards,

Nacho P

[Damion Young]

Thanks @Nacho P. You're right, it is Windows but, although I've checked it's set, still no joy.

Looking at the output from 

vagrant up

I think I'm stumbling at a point before the shared folders stage - see end of output below, despite the fact that the machine has definitely booted and is showing me a login prompt in the virtualbox gui:

    default: Warning: Connection refused. Retrying...
    default: Warning: Connection refused. Retrying...
Timed out while waiting for the machine to boot. This means that
Vagrant was unable to communicate with the guest machine within
the configured ("config.vm.boot_timeout" value) time period.


If you look above, you should be able to see the error(s) that
Vagrant had when attempting to connect to the machine. These errors
are usually good hints as to what may be wrong.


If you're using a custom box, make sure that networking is properly
working and you're able to connect to the machine. It is a common
problem that networking isn't setup properly in these boxes.
Verify that authentication configurations are also setup properly,
as well.


If the box appears to be booting properly, you may want to increase
the timeout ("config.vm.boot_timeout") value.

Thanks again

[Yarko Tymciurak]

On Tue, Jul 22, 2014 at 10:03 AM, Damion Young <damion...@googlemail.com> wrote:

Thanks very much @Yarko Tymciurak - should have thought of that! It's empty (ie 'ls' returns nothing) which explains a lot. I'm a bit confused though as I did follow the instructions here: http://edx.readthedocs.org/projects/edx-installing-configuring-and-running/en/latest/devstack/install_devstack.html very carefully and didn't see any problems until I tried:

vagrant up

If you had problems during   vagrant up   then you need to sort that out first, obviously.

During the first "up" on a base machine (where the starting image is downloaded from edx's aws store - its rather large) timeouts and incompletions are not too uncommon - just persist (run vagrant halt; vagrant up until all completes).

If you have other errors than timeouts, then you need to route those out as appropriate (e.g. the windows hint someone provided to you in this thread).

Also at this point if you want useful help, you need to be specific enough that someone can _try_ to reproduce your problem in order to help - so what you're running on, which version of virtualbox, which version of vagrant, etc.   "I followed the steps exactly" is not as useful as "after this command, I got this error msg."

And while the virtual machine does appear to be running, some parts of it obviously haven't completed properly.

How do you determine it "appears to be running"?

What makes you say "it didn't complete properly" - the virtual machine most likely wouldn't run if _it_ didn't complete properly.   And if you mean "the edx devstack installation didn't complete properly" then specifics about that should be pointed out.

Any ideas at which stage this directory is supposed to be filled and what might be causing it not to be? I'm wondering whether this is related to my inability to run:

vagrant ssh

If you can't run vagrant ssh  then how are you getting into the VM, and to a command prompt?

 

Interestingly, the log for the virtual machine as it booted does log a number of port forwarding errors:

00:00:01.537342 NAT: set redirect TCP host port 9200 => guest port 9200 @ 10.0.2.15
00:00:01.537542 NAT: failed to redirect TCP 9200 => 9200
00:00:01.537547 NAT: set redirect TCP host port 8765 => guest port 8765 @ 10.0.2.15
00:00:01.537624 NAT: failed to redirect TCP 8765 => 8765
00:00:01.537629 NAT: set redirect TCP host port 8001 => guest port 8001 @ 10.0.2.15
00:00:01.537700 NAT: failed to redirect TCP 8001 => 8001
00:00:01.537704 NAT: set redirect TCP host port 8000 => guest port 8000 @ 10.0.2.15
00:00:01.537774 NAT: failed to redirect TCP 8000 => 8000
00:00:01.537778 NAT: set redirect TCP host port 18080 => guest port 18080 @ 10.0.2.15
00:00:01.537846 NAT: failed to redirect TCP 18080 => 18080
00:00:01.537850 NAT: set redirect TCP host port 2222 => guest port 22 @ 10.0.2.15
00:00:01.537918 NAT: failed to redirect TCP 2222 => 22

I wonder if this is the problem?

This is a symptom of a problem you are having.

For devstack, the repositories are checked out into your host machine.

Windows - since it doesn't support symbolic links - poses special problems.  You can search the various wikis (configuration and edx-platform) for some notes about trouble-shooting windows VM bring-ups.

You'll find some troubleshooting tips and links in https://github.com/edx/configuration/wiki/edX-Developer-Stack  (be sure to look through the whole wiki page).

The directory where you copied Vagrantfile    - if the ansible playbooks ran for installing a devstack - would now have a few subdirectories, such as edx-platform, etc.   Those are NFS mounted into your vm, so when you login to the VM and run "mount" to list the VM's filesystem mounts, you should see NFS mounted filesystems listed corresponding nto the edx repositories checked out into your host machine.

[Damion Young]

Thanks @Yarko Tymciurak for the very helpful reply. Sorry to be so ignorant of this sort of automated provisioning/virtualisation approach - I'm mainly a CakePHP/JS developer but have an academic colleague here at Oxford who is very keen to find out if we can get EdX to do some of the more interesting things he would like to be able to do.  I'll try and be a bit more precise about what's going on.

I'm using Vagrant 1.6.3, VirtualBox 4.3.14, have activated Client for NFS on my Windows 8 Enterprise host machine and am running the commands using Windows Command Prompt run as Administrator.

In answer to your questions/comments:

1. I have tried vagrant reload (which I thinks is the same as halt/up) numerous times to no avail. Interestingly vagrant halt says:

==> default: Attempting graceful shutdown of VM...
    default: Guest communication could not be established! This is usually because
    default: SSH is not running, the authentication information was changed,
    default: or some other networking issue. Vagrant will force halt, if default: capable.
==> default: Forcing shutdown of VM...

2. I can tell the machine is running because I have added vb.gui = true inside

config.vm.provider :virtualbox do |vb|
    ...

    vb.gui = true

end

in the vagrantfile so the Oracle VM VirtualBox console fires up with a login prompt but the host Command Prompt continues to say:

default: Warning: Connection refused. Retrying...

I can run commands inside the VirtualBox console (logging in with vagrant | vagrant and/or sudo suing to edxapp)

3. I think, as you say, that the EdX devstack installtion is what has failed to complete properly. as running the 

mount

command you suggested, either as vagrant or as edxapp on the virtual machine doesn't list any of the edx-platform etc. directories. However a 'ls' on /edx/app/edxapp on the virtual machine does list subdirectories edxapp_env and edx-platform...but they are empty. On my host machine, the devstack directory contains: cs_comments_service; edx-platform; ora and themes subdirectories.

4. I've checked all the wiki pages I can find as well as the various forums and have set and confirmed that 'VAGRANT_USE_VBOXFS = true'

I think I must be missing something basic - and I thin k it may all come down to why I'm getting this 'Connection refused' error on vagrant up. What sort of connection is it trying to make? Is there anything else that could be causing a problem? I've ruled out the firewall by switching it off with no success. I don't have my Apache server running so it shouldn't be interfering...

Thanks again

On Tuesday, 22 July 2014 17:06:10 UTC+1, Yarko Tymciurak wrote:

On Tue, Jul 22, 2014 at 10:03 AM, Damion Young <damion...@googlemail.com> wrote:

Thanks very much @Yarko Tymciurak - should have thought of that! It's empty (ie 'ls' returns nothing) which explains a lot. I'm a bit confused though as I did follow the instructions here: http://edx.readthedocs.org/projects/edx-installing-configuring-and-running/en/latest/devstack/install_devstack.html very carefully and didn't see any problems until I tried:

vagrant up

If you had problems during   vagrant up   then you need to sort that out first, obviously.

During the first "up" on a base machine (where the starting image is downloaded from edx's aws store - its rather large) timeouts and incompletions are not too uncommon - just persist (run vagrant halt; vagrant up until all completes).

If you have other errors than timeouts, then you need to route those out as appropriate (e.g. the windows hint someone provided to you in this thread).

Also at this point if you want useful help, you need to be specific enough that someone can _try_ to reproduce your problem in order to help - so what you're running on, which version of virtualbox, which version of vagrant, etc.   "I followed the steps exactly" is not as useful as "after this command, I got this error msg."

And while the virtual machine does appear to be running, some parts of it obviously haven't completed properly.

How do you determine it "appears to be running"?

What makes you say "it didn't complete properly" - the virtual machine most likely wouldn't run if _it_ didn't complete properly.   And if you mean "the

 

installation didn't complete properly" then specifics about that should be pointed out.

[Damion Young]

Just tried removing everything, installing cygwin and running everything through the cygwin terminal but the problems are the same. Is there any way to set this up without vagrant, virtual machines, etc?

[Yarko Tymciurak]

On Wed, Jul 23, 2014 at 11:07 AM, Damion Young <damion...@googlemail.com> wrote:

Just tried removing everything, installing cygwin and running everything through the cygwin terminal but the problems are the same. Is there any way to set this up without vagrant, virtual machines, etc?

Others have done this successfully on windows;

This is a vagrant + ssh + windows issue.

Read through the various subsections on https://docs.vagrantup.com/v2/synced-folders/index.html

You'll note you have a couple of options.   Note that vagrant explicitly says NFS mounts don't work under windows.

If you have trouble with virtualbox provider mounts also, then:

A drop dead simple option is rsync'd folders.   It takes a little manual intervention in the long run, but this is _only_ for development purposes anyway, and it just works, works, works.

Also, see:

https://github.com/edx/configuration/wiki/Vagrant-troubleshooting#dealing-with-line-endings-and-symlinks-under-windows

[Yarko Tymciurak]

One other thing, Damion -

The devstack is so you can easily make modifications and use your "native" (local) development tools...

If you just want to check out running and "feeling" your own edx, you can simply use fullstack on vagrant+virtualbox:

https://github.com/edx/configuration/wiki/edx-Production-stack--installation-using-Vagrant-Virtualbox

You will have production servers, and making changes won't be as easy, but its a place to start and get your feet wet.

[Damion Young]

Thanks again @Yarko but production stack failed with the same problem. I had a look at the various links you suggested but still no joy - as you say - a vagrant +ssh+ windows issue which, after nearly 3 days' time wasted, I've now given up on. However, I've just VNC'd into a Mac Mini across the office and, in about half an hour, I'm now looking at a fully working edX LMS - what a difference on a Mac! It's not quite as good as having it on my local machine, but certainly workable.

Thanks again for all your help

Damion

[Yarko Tymciurak]

Sorry to hear you had so much trouble.  Another option is to simply try a standalone installation, but from within a VM.