Weird error excecuting appscale up

Jose Angel Espinoza Portillo

unread,

Jul 30, 2019, 1:51:23 PM7/30/19

to AppScale Community

Hi I have the following error:

Executing ssh-copy-id for host: 35.184.180.255The authenticity of host '35.184.180.255 (35.184.180.255)' can't be established.ECDSA key fingerprint is SHA256:T1En0mYPNUfVNE/+V2FySkK97hDE3lW0p44wlrrN0+s.Are you sure you want to continue connecting (yes/no)? yesExecuting command 'ssh-copy-id -i /root/.appscale/appscale86aff62c53654c00a5c53ce4489633fb ro...@35.184.180.255' failed:/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.appscale/appscale86aff62c53654c00a5c53ce4489633fb.pub"/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keysPermission denied (publickey).

The log is:

stacktrace : Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/appscale/tools/scripts/appscale.py", line 45, in main appscale.up() File "/usr/local/lib/python2.7/dist-packages/appscale/tools/appscale.py", line 301, in up AppScaleTools.add_keypair(add_keypair_opts) File "/usr/local/lib/python2.7/dist-packages/appscale/tools/appscale_tools.py", line 218, in add_keypair options.verbose) File "/usr/local/lib/python2.7/dist-packages/appscale/tools/local_state.py", line 1004, in shell .format(command, output))ShellException: Executing command 'ssh-copy-id -i /root/.appscale/appscale86aff62c53654c00a5c53ce4489633fb ro...@35.184.180.255' failed:/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.appscale/appscale86aff62c53654c00a5c53ce4489633fb.pub"/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keysPermission denied (publickey).exception : ShellExceptiontools_version : 3.7.1platform : Linux-4.15.0-1036-gcp-x86_64-with-Ubuntu-16.04-xenialmessage : Executing command 'ssh-copy-id -i /root/.appscale/appscale86aff62c53654c00a5c53ce4489633fb ro...@35.184.180.255' failed:/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.appscale/appscale86aff62c53654c00a5c53ce4489633fb.pub"/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keysPermission denied (publickey).runtime : CPython

The AppScalefile is:

ips_layout: - roles: - master - compute - database - zookeeper nodes: [35.184.180.255]

And I am running this from 35.184.180.255

Chris Donati

unread,

Jul 30, 2019, 8:21:12 PM7/30/19

to appscale_...@googlegroups.com

I would recommend using the machine's private IP address in the ips_layout and the public IP address under the "login" directive. In other words, something like

ips_layout:

- roles: [master, compute, database, zookeeper]

nodes: <private-ip>

login: 35.184.180.255 (or a dns name if you already have one set up)

Do you have password-less SSH already set up? That is, can you `ssh root@<private-ip>` from the location you are running `appscale up` from?

--
You received this message because you are subscribed to the Google Groups "AppScale Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to appscale_commun...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/appscale_community/d183b3e5-8e0b-4fa1-a87a-1f81a83b3de7%40googlegroups.com.

Jose Angel Espinoza Portillo

unread,

Aug 1, 2019, 5:13:31 PM8/1/19

to AppScale Community

I am trying to run this in one GCP instance.

And by now I am re generate a rsa key and the error changed

The response in the console is:

root@instance-1:~/configuraciones/deploy# appscale up

Starting AppScale 3.7

Log in to your head node: ssh -i /root/.appscale/appscale86aff62c53654c00a5c53ce4489633fb.key ro...@35.184.180.255

Starting AppController at 35.184.180.255

Please wait for the AppController to finish pre-processing tasks.

Head node successfully initialized at 35.184.180.255.

Waiting for head node to initialize...

The AppScalefile:

ips_layout: - roles: [master, compute, database, zookeeper] nodes: 10.128.0.5login: 35.184.180.255

the Log of the oputput is:

stacktrace : Traceback (most recent call last):

File "/usr/local/lib/python2.7/dist-packages/appscale/tools/scripts/appscale.py", line 45, in main

appscale.up()

File "/usr/local/lib/python2.7/dist-packages/appscale/tools/appscale.py", line 303, in up

AppScaleTools.run_instances(run_instances_opts)

File "/usr/local/lib/python2.7/dist-packages/appscale/tools/appscale_tools.py", line 852, in run_instances

while not acc.is_initialized():

File "/usr/local/lib/python2.7/dist-packages/appscale/tools/appcontroller_client.py", line 255, in is_initialized

self.server.is_done_initializing, self.secret)

File "/usr/local/lib/python2.7/dist-packages/appscale/tools/appcontroller_client.py", line 119, in run_with_timeout

*args)

File "/usr/local/lib/python2.7/dist-packages/appscale/tools/appcontroller_client.py", line 119, in run_with_timeout

*args)

File "/usr/local/lib/python2.7/dist-packages/appscale/tools/appcontroller_client.py", line 119, in run_with_timeout

*args)

File "/usr/local/lib/python2.7/dist-packages/appscale/tools/appcontroller_client.py", line 119, in run_with_timeout

*args)

File "/usr/local/lib/python2.7/dist-packages/appscale/tools/appcontroller_client.py", line 119, in run_with_timeout

Chris Donati

unread,

Aug 2, 2019, 6:23:23 PM8/2/19

to appscale_...@googlegroups.com

It sounds like the controller may have run into an issue early on in the startup process. Are there any error messages in /var/log/appscale/controller.log?

--
You received this message because you are subscribed to the Google Groups "AppScale Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to appscale_commun...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/appscale_community/3e80ebfa-e7f6-403c-9613-d746fdab84b2%40googlegroups.com.

Jose Angel Espinoza Portillo

unread,

Aug 2, 2019, 11:56:27 PM8/2/19

to AppScale Community

HI I have attached all the logs

controller.log

Graziano Obertelli

unread,

Aug 3, 2019, 5:22:55 PM8/3/19

to appscale_...@googlegroups.com

Hi there,

it looks like cassandra was not able to prime the database, hence the
system is not able to come up. What system are you using? # of cores,
amount of memory etc ...?

cheers
graziano

--
You received this message because you are subscribed to the Google
Groups "AppScale Community" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to appscale_commun...@googlegroups.com.
To view this discussion on the web visit

https://groups.google.com/d/msgid/appscale_community/3bead7a5-56a8-4cab-b636-922f38436ab4%40googlegroups.com
.
--
Graziano Obertelli | VP Operations
805.570.1647 graz...@appscale.com
www.appscale.com

Jose Angel Espinoza Portillo

unread,

Aug 3, 2019, 6:52:10 PM8/3/19

to appscale_...@googlegroups.com

Sure the virtual machine is a 2 core 7.55 Gb of Ram with 20 Gb of hard drive from GCE.

You received this message because you are subscribed to a topic in the Google Groups "AppScale Community" group.

To unsubscribe from this topic, visit https://groups.google.com/d/topic/appscale_community/FwObQ77pV1s/unsubscribe.

To unsubscribe from this group and all its topics, send an email to appscale_commun...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/appscale_community/c5d7598816d8fb7927a3b59d4583e277731f22db.camel%40appscale.com.

Graziano Obertelli

unread,

Aug 5, 2019, 2:09:33 AM8/5/19

to appscale_...@googlegroups.com

Thanks: that should be enough. Do you see this consistently? In our QA
environment we see the error when some nodes is overloaded and the disk
performance are poor enough to cause issues to cassandra initialization
...

https://groups.google.com/d/msgid/appscale_community/5d461018.1c69fb81.ed051.12d3%40mx.google.com

Jose Angel Espinoza Portillo

unread,

Aug 5, 2019, 12:42:40 PM8/5/19

to appscale_...@googlegroups.com

Mmm that consistent when I try the “run from source” steps from the web page. When I try in the same instance (from your image) with the fast-start it. It works like a sharm. But I will be installing in a local cluster so I cant make it work like that.

To view this discussion on the web visit https://groups.google.com/d/msgid/appscale_community/9ce8a079323f590b045fdca90f84aef584607d5d.camel%40appscale.com.

Chris Donati

unread,

Aug 5, 2019, 10:07:38 PM8/5/19

to appscale_...@googlegroups.com

I was able to reproduce the error you are seeing. It turns out that the cassandra-driver package was updated after the 3.7.1 release in a way that was incompatible with the way we use it. We fixed the issue in master, but we haven't made a release since doing so.

You should be able to fix the issue with the following steps:

- `ruby /root/appscale/AppController/terminate.rb` # This stops any running appscale-related services that might be lingering from a failed start. You can verify they are stopped with `ps ax | grep appscale`.

- `pip install "cassandra-driver<3.18.0"`

- If /etc/appscale/zookeeper_locations exists, remove it. The error message "Unable to get state from zookeeper: trying again." indicates that the controller thinks the deployment is already up due to the presence of this file. The terminate.rb script I mentioned above should remove the file, but I'm mentioning it here just in case you see that error again.

After the above, try `appscale up` again.

To view this discussion on the web visit https://groups.google.com/d/msgid/appscale_community/5d485c7e.1c69fb81.899ca.ae6b%40mx.google.com.

Jose Angel Espinoza Portillo

unread,

Aug 7, 2019, 1:38:26 PM8/7/19

to AppScale Community

Waooo Ok I will retry this instructions latter today. Any ETA for this to be released, I will leave this instalation to be supported by the client and if I lieave this instructions in the manual they will get crazy.

Chris Donati

unread,

Aug 7, 2019, 8:22:46 PM8/7/19

to appscale_...@googlegroups.com

We've pushed a new 3.7.2 tag with the cassandra-driver version constraint, so building from source should work as expected now.

On Wed, Aug 7, 2019 at 10:38 AM Jose Angel Espinoza Portillo <onlyan...@gmail.com> wrote:

Waooo Ok I will retry this instructions latter today. Any ETA for this to be released, I will leave this instalation to be supported by the client and if I lieave this instructions in the manual they will get crazy.

--

You received this message because you are subscribed to the Google Groups "AppScale Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to appscale_commun...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/appscale_community/affc560a-a19a-4865-bdda-cf24ade59b0a%40googlegroups.com.

Jose Angel Espinoza Portillo

unread,

Aug 7, 2019, 11:59:06 PM8/7/19

to AppScale Community

This is great!!!!!!!! Thanks!!!!

Jose Angel Espinoza Portillo

unread,

Aug 9, 2019, 11:36:20 AM8/9/19

to AppScale Community

I have successfully started an instance from source modifing the bootstrap to gather the 3.7.2 tag. Thanks every one.

On Wednesday, August 7, 2019 at 10:59:06 PM UTC-5, Jose Angel Espinoza Portillo wrote:

This is great!!!!!!!! Thanks!!!!

Jose Angel Espinoza Portillo

unread,

Aug 10, 2019, 11:33:16 AM8/10/19

to AppScale Community

Now One on every 4 instances I try to run

bash bootstrap.sh

I run with the error:

Failed to install AppScale Agents

Look this is the screen:

status = self.run(options, args) File "/usr/lib/python2.7/dist-packages/pip/commands/install.py", line 328, in run wb.build(autobuilding=True) File "/usr/lib/python2.7/dist-packages/pip/wheel.py", line 748, in build self.requirement_set.prepare_files(self.finder) File "/usr/lib/python2.7/dist-packages/pip/req/req_set.py", line 360, in prepare_files ignore_dependencies=self.ignore_dependencies)) File "/usr/lib/python2.7/dist-packages/pip/req/req_set.py", line 512, in _prepare_file finder, self.upgrade, require_hashes) File "/usr/lib/python2.7/dist-packages/pip/req/req_install.py", line 273, in populate_link self.link = finder.find_requirement(self, upgrade) File "/usr/lib/python2.7/dist-packages/pip/index.py", line 442, in find_requirement all_candidates = self.find_all_candidates(req.name) File "/usr/lib/python2.7/dist-packages/pip/index.py", line 400, in find_all_candidates for page in self._get_pages(url_locations, project_name): File "/usr/lib/python2.7/dist-packages/pip/index.py", line 545, in _get_pages page = self._get_page(location) File "/usr/lib/python2.7/dist-packages/pip/index.py", line 648, in _get_page return HTMLPage.get_page(link, session=self.session) File "/usr/lib/python2.7/dist-packages/pip/index.py", line 757, in get_page "Cache-Control": "max-age=600", File "/usr/share/python-wheels/requests-2.9.1-py2.py3-none-any.whl/requests/sessions.py", line 480, in get return self.request('GET', url, **kwargs) File "/usr/lib/python2.7/dist-packages/pip/download.py", line 378, in request return super(PipSession, self).request(method, url, *args, **kwargs) File "/usr/share/python-wheels/requests-2.9.1-py2.py3-none-any.whl/requests/sessions.py", line 468, in request resp = self.send(prep, **send_kwargs) File "/usr/share/python-wheels/requests-2.9.1-py2.py3-none-any.whl/requests/sessions.py", line 576, in send r = adapter.send(request, **kwargs) File "/usr/share/python-wheels/CacheControl-0.11.5-py2.py3-none-any.whl/cachecontrol/adapter.py", line 46, in send resp = super(CacheControlAdapter, self).send(request, **kw) File "/usr/share/python-wheels/requests-2.9.1-py2.py3-none-any.whl/requests/adapters.py", line 376, in send timeout=timeout File "/usr/share/python-wheels/urllib3-1.13.1-py2.py3-none-any.whl/urllib3/connectionpool.py", line 610, in urlopen _stacktrace=sys.exc_info()[2]) File "/usr/share/python-wheels/urllib3-1.13.1-py2.py3-none-any.whl/urllib3/util/retry.py", line 228, in increment total -= 1TypeError: unsupported operand type(s) for -=: 'Retry' and 'int'You are using pip version 8.1.1, however version 19.2.1 is available.You should consider upgrading via the 'pip install --upgrade pip' command.Makefile:31: recipe for target 'install-no-venv' failedmake: *** [install-no-venv] Error 2Failed to install AppScale Agents

Chris Donati

unread,

Aug 10, 2019, 12:05:03 PM8/10/19

to appscale_...@googlegroups.com

The behavior of the bootstrap script depends on which version of the bootstrap script you run. This is because appscale-agents was added recently as a dependency in master.

Assuming you are trying to install the latest _released_ version of AppScale (instead of the bleeding edge master), you'll want the bootstrap version from 3.7.2 which does not try to install appscale-agents.

A few days ago, we changed the behavior of bootstrap.appscale.com to retrieve the latest released version of the script. Before that, it would retrieve the latest version from master. I'm guessing the machines that are failing to complete the build have a bootstrap file that was downloaded more than a few days ago.

Alternatively, if you are using the bootstrap file from the git repo, make sure the git repo is checked out to 3.7.2 before running it.

--

You received this message because you are subscribed to the Google Groups "AppScale Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to appscale_commun...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/appscale_community/8b91d217-dff2-4aa9-b50a-c618770fe4db%40googlegroups.com.

Jose Angel Espinoza Portillo

unread,

Aug 10, 2019, 6:58:40 PM8/10/19

to AppScale Community

Jose Angel Espinoza Portillo

unread,

Aug 10, 2019, 6:58:55 PM8/10/19

to AppScale Community

Well this is weird because to start the work in te automation phase of what I am doing I created a repsitory with what I need And I modify your bootstrap to support ONLY 3.7.2 as you can see in the lines 12 and 13 of this file.

https://github.com/crypt0cloud/appscale/blob/master/bootstrap.sh

Pleace let me know what I was doing wrong? As far as I know I take that file from the v3.7.2 of the appscale repository.

Chris Donati

unread,

Aug 10, 2019, 8:01:31 PM8/10/19

to appscale_...@googlegroups.com

The file in the repo you linked is based off master. It contains references to appscale-agents. You can access the 3.7.2 version of the bootstrap script either via bootstrap.appscale.com or https://github.com/AppScale/appscale/blob/3.7.2/bootstrap.sh.

--
You received this message because you are subscribed to the Google Groups "AppScale Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to appscale_commun...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/appscale_community/3c59d763-131e-4392-8e19-0b4d63e1faaa%40googlegroups.com.

Reply all

Reply to author

Forward