appscale up gives could not authenticate error

[Jeff O'Neill]

The first time I used "appscale up" it succeeded.  That EC2 instance is now deleted and I'm using "appscale up" a second time and I get this error:

$ appscale up
Machine image ami-0d1e4f65c0675c7a5 does exist
Availability zone None does exist
Starting AppScale without specifying persistent disks means your data will not be saved when your cloud is destroyed.
Are you sure you want to do this? (Y/N) y
Starting AppScale 3.6
...
Using on-demand instances
Waiting for your instances to start...
Waiting for your instances to start...
Started 1 on-demand instances in 22 seconds


Please wait for AppScale to prepare your machines for use. This can take few minutes.
Enabling root ssh on 54.209.XXX.XXX
Root login not enabled for 54.209.XXX.XXX - enabling it now.
Log in to your head node: ssh -i /Users/.../.appscale/appscaleXXX.key root@54.209.XXX.XXX
Starting AppController at 54.209.XXX.XXX
Please wait for the AppController to finish pre-processing tasks.
Head node successfully initialized at 54.209.XXX.XXX.
Waiting for head node to initialize...
Could not authenticate successfully to the AppController. You may need to change the keyname in use.


A log with more information is available at
/Users/joneill/.appscale/log-XXX.

In running this command, my the bottom of my AppScalefile file gets updated with a keyname and group that matches what I see on the EC2 console.

One difference from the previous run was that I was previously asked for admin info:

Enter your desired admin e-mail address: me@example.com
Enter new password:
Confirm password:
Creating new user account me@example.com
Creating new user account me@18.205.XXX.XXX

Your XMPP username is me@18.205.XXX.XXX
Granting admin privileges to me@example.com

But it could be that the error occurs before this request for admin info.

Could you help me figure out how to diagnose this?

[Jeff O'Neill]

I was able to get appscale up to work by adding this to my AppScalefile file:

admin_user: 'm...@example.com'
admin_pass: 'pw'

Where I used the values I entered in response to prompts the first time I used appscale up.

Two follow up questions:

(1) Where is the admin user and password stored so I can delete them?  I don't like having passwords stored on my computer.

(2) How can I get appscale up to prompt me each time instead of using previously provided values?

thanks for your help!

Jeff

[Chris Donati]

Hi Jeff,

The admin user and password are not stored on your local machine. It's only stored as a hashed value on the deployment for authentication. However, the deployment secret and private key are also sensitive. They are stored at ~/.appscale/<your-key-name>.key and ~/.appscale/<your-key-name>.secret.

I wouldn't recommend deleting them since the tools need to use them for operations like `appscale deploy`, `appscale status`, `appscale down`, etc. You could encrypt those files when not in use and decrypt them before running appscale commands. To maintain security for our production customers, we store these credentials and run appscale commands from a bastion host.

The error you saw in your first email is complaining about the deployment secret not being the same between the tools and what is stored on the deployment. Predefining the admin user and password in your AppScalefile should not influence that, and omitting them is the standard way of forcing AppScale to ask you. There may be another issue going on. I'll do some digging and try to reproduce the problem you saw.

--
You received this message because you are subscribed to the Google Groups "AppScale Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to appscale_commun...@googlegroups.com.
To post to this group, send email to appscale_...@googlegroups.com.
Visit this group at https://groups.google.com/group/appscale_community.
For more options, visit https://groups.google.com/d/optout.

[Chris Donati]

It looks like the AMI you are using is a faststart image, which is configured to start AppScale automatically when the machine boots. When it failed with the authentication error, it's because the tools on the machine started appscale before your tools did. It probably succeeds occasionally depending on which client writes the secret key first.

I would recommend using the AMI from http://s3.amazonaws.com/appscale/releases/latest/ami.txt instead for a normal AppScale image.

[Jeff O'Neill]

Thanks, that got "appscale up" working.  If you Google AWS and appscale you end up here:

    https://aws.amazon.com/marketplace/pp/B01IAJDYJG

It would be great if someone on your team could update that page to indicate when that ami should be used.  I spent hours trying to debug the wrong ami.

I'm now working through errors for the deploy process, and it seems like the GAE supported third-party libraries are really out of date.

First, I got lxml 3.7.3 isn't supported so I removed it and now I get pytz (any version) not supported.

What is the recommended solution for this? I'd rather not install them in my vendored lib directory because that would be confusing with the GAE version that I still use.

To unsubscribe from this group and stop receiving emails from it, send an email to appscale_community+unsub...@googlegroups.com.

[Graziano Obertelli]

Hi Jeff,

it turns out that we do use a different image for marketplace in order
to make it easier for people to start AppScale the first time around.
Since it's effectively a dev-test environment, we start everything in a
single instance so we can configure AppScale directly.

If you look at out website, under Get AppScale (
https://www.appscale.com/try-appscale/) you will see the instruction to
get the generic AMI.

Do you think that mentioning on the marketplace image that the image is
meant to be dev-test or something similar like easy-to-start would be a
good step to avoid confusion?

cheers
graziano

> > > > /Users/.../.appscale/appscaleXXX.key ro...@54.209.XXX.XXX

> > > > Starting AppController at 54.209.XXX.XXX
> > > > Please wait for the AppController to finish pre-processing
> > > > tasks.
> > > > Head node successfully initialized at 54.209.XXX.XXX.
> > > > Waiting for head node to initialize...
> > > > Could not authenticate successfully to the AppController. You
> > > > may need to change the keyname in use.
> > > >
> > > >
> > > > A log with more information is available at
> > > > /Users/joneill/.appscale/log-XXX.
> > > >
> > > >
> > > > In running this command, my the bottom of my AppScalefile file
> > > > gets updated with a keyname and group that matches what I see
> > > > on the EC2 console.
> > > >
> > > > One difference from the previous run was that I was previously
> > > > asked for admin info:
> > > >

> > > > Enter your desired admin e-mail address: m...@example.com

> > > > Enter new password:
> > > > Confirm password:

> > > > Creating new user account m...@example.com
> > > > Creating new user account m...@18.205.XXX.XXX
> > > > Your XMPP username is m...@18.205.XXX.XXX
> > > > Granting admin privileges to m...@example.com

> > > >
> > > >
> > > > But it could be that the error occurs before this request for
> > > > admin info.
> > > >
> > > > Could you help me figure out how to diagnose this?
> > > >
--

Graziano Obertelli | VP Operations
805.570.1647 graz...@appscale.com
www.appscale.com

[Jeff O'Neill]

Hi Graziano,

My 2 cents is that faststart image should be removed.  It isn't meaningfully faster than the normal image, and it creates confusion for developers. 

If I understand it right, for faststart you do this:

    bash appscale/scripts/fast-start.sh --no-demo-app

    appscale deploy .

and for normal you do this:

    appscale up

    appscale deploy .

How is that any faster?  I suppose the fast start script is faster than appscale up, but who cares.  There is a big learning curve to getting started with AppScale so I'd rather have a slightly slower process and have clear instructions.  Anyone using AppScale seriously will need to quickly migrate to the normal API anyway.

If you keep faststart, then I recommend making it clear here(https://github.com/AppScale/appscale/wiki/AppScale-on-Amazon-EC2) and here (https://aws.amazon.com/marketplace/pp/B01IAJDYJG) so people like me don't accidentally mix the two.

Jeff

[Jeff O'Neill]

So just removing the "libraries" section from my app.yaml got me a little further since those Python packages are already on the image.  Though I'd appreciate your advice for the more general case.  Is the best solution to just ssh to the instance and manually install needed Python packages?

I followed the instructions here to allow my libraries (lxml and pytz):

    https://github.com/AppScale/appscale/wiki/FAQs#can-i-run-libraries-that-are-restricted-in-app-engine
I removed "and submodule not in self._white_list_c_modules" from the if statement.

I also used "appscale relocate <appname> 80 443" to use the default ports.

When I request the home page, I now get the traceback below.  I'd appreciate your advice for fixing this.

Nov  6 20:58:38 appscale-image0 Traceback (most recent call last):
Nov  6 20:58:38 appscale-image0  File "/root/appscale/AppServer/_python_runtime.py", line 183, in <module>
Nov  6 20:58:38 appscale-image0    _run_file(__file__, globals())
Nov  6 20:58:38 appscale-image0  File "/root/appscale/AppServer/_python_runtime.py", line 178, in _run_file
Nov  6 20:58:38 appscale-image0    execfile(script_path, globals_)
Nov  6 20:58:38 appscale-image0  File "/root/appscale/AppServer/google/appengine/tools/devappserver2/python/runtime.py", line 189, in <module>
Nov  6 20:58:38 appscale-image0    main()
Nov  6 20:58:38 appscale-image0  File "/root/appscale/AppServer/google/appengine/tools/devappserver2/python/runtime.py", line 163, in main
Nov  6 20:58:38 appscale-image0    sandbox.enable_sandbox(config)
Nov  6 20:58:38 appscale-image0  File "/root/appscale/AppServer/google/appengine/tools/devappserver2/python/sandbox.py", line 158, in enable_sandbox
Nov  6 20:58:38 appscale-image0    thread = __import__('thread')
Nov  6 20:58:38 appscale-image0  File "/root/appscale/AppServer/google/appengine/tools/devappserver2/python/sandbox.py", line 829, in load_module
Nov  6 20:58:38 appscale-image0    raise ImportError('No module named %s' % fullname)
Nov  6 20:58:38 appscale-image0 ImportError: No module named thread
Nov  6 20:58:38 appscale-image0 ERROR    2018-11-06 20:58:38,885 module.py:699] Request to '/' failed
Nov  6 20:58:38 appscale-image0 Traceback (most recent call last):
Nov  6 20:58:38 appscale-image0  File "/root/appscale/AppServer/google/appengine/tools/devappserver2/module.py", line 693, in _handle_request_impl
Nov  6 20:58:38 appscale-image0    environ, wrapped_start_response)
Nov  6 20:58:38 appscale-image0  File "/root/appscale/AppServer/google/appengine/tools/devappserver2/request_rewriter.py", line 311, in _rewriter_middleware
Nov  6 20:58:38 appscale-image0    response_body = iter(application(environ, wrapped_start_response))
Nov  6 20:58:38 appscale-image0  File "/root/appscale/AppServer/google/appengine/tools/devappserver2/module.py", line 1112, in _handle_script_request
Nov  6 20:58:38 appscale-image0    inst = self._add_instance(permit_warmup=False)
Nov  6 20:58:38 appscale-image0  File "/root/appscale/AppServer/google/appengine/tools/devappserver2/module.py", line 1164, in _add_instance
Nov  6 20:58:38 appscale-image0    if not inst.start():
Nov  6 20:58:38 appscale-image0  File "/root/appscale/AppServer/google/appengine/tools/devappserver2/instance.py", line 271, in start
Nov  6 20:58:38 appscale-image0    self._runtime_proxy.start()
Nov  6 20:58:38 appscale-image0  File "/root/appscale/AppServer/google/appengine/tools/devappserver2/http_runtime.py", line 217, in start
Nov  6 20:58:38 appscale-image0    self._port = int(line.split()[-1])
Nov  6 20:58:38 appscale-image0 IndexError: list index out of range
Nov  6 20:58:38 appscale-image0 INFO     2018-11-06 20:58:38,886 module.py:651] default: "GET / HTTP/1.0" 500 -

[Jeff O'Neill]

The code highlighting messed up the last part the of traceback.  Replace the last line with this:

Nov  6 20:58:38 appscale-image0    self._port = int(line.split()[-1])
Nov  6 20:58:38 appscale-image0 IndexError: list index out of range
Nov  6 20:58:38 appscale-image0 INFO     2018-11-06 20:58:38,886 module.py:651] default: "GET / HTTP/1.0" 500 -

Nov  6 20:58:38 appscale-image0    self._port =<span style="color: #000;" class="styled-by-prettif

[Jeffrey O'Neill]

Hi Chris,

I'd appreciate your assistance with this when you have a moment.

best,

Jeff

--

You received this message because you are subscribed to the Google Groups "AppScale Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to appscale_commun...@googlegroups.com.

[Chris Donati]

Sorry about the delay. It's been a hectic week.

For the built-in libraries, specifying "latest" usually meets my needs. I do agree it would be an improvement for AppScale to support all of the versions of the bundled libraries that GAE does, though.

For libraries that aren't built-in, I vendor them into the app before deploying (as described here: https://cloud.google.com/appengine/docs/standard/python/tools/using-libraries-python-27).

For libraries that aren't built-in and have C extensions, installing them on the machine and whitelisting them in the sandbox is the usual procedure.

Just removing the "and" clause from that check will make the check more restrictive. I believe the note in the FAQ meant that the entire block should be removed. However, I think that FAQ section is a bit out of date anyways.

Without access to your app, I don't know exactly why the import error is being thrown in that spot in the runtime. I would recommend reverting whatever changes you've made to the AppServer code and trying the following diff: https://github.com/AppScale/appscale/compare/AppScale:25331e2...cdonati:e69ea90

I tried it with a simple test app that included the following in the app.yaml:

 - name: lxml
   version: latest
 - name: pytz
   version: latest

[Jeff O'Neill]

Chris, thanks for the help, but I am going to put this on hold for now and check out some alternatives.

The main appeal of AppScale was that it was supposed to very easy to deploy your GAE apps on other clouds, but I'm finding that it is not the case.