Re: [E] PNDA Installation issues

Donald Hunter (donaldh)

unread,

Feb 4, 2019, 6:05:26 AM2/4/19

to NEKRASSOV, ALEXEI, Thiruveedula, Bharath, VENKATESH KUMAR, VIJAY, Rajesh R, PNDA Users

Hi Alex,

The “Checking connectivity” log message is emitted here:

https://github.com/pndaproject/pnda-cli/blob/develop/cli/backend_base.py#L392

I’m assuming it is checking for the gateway (bastion) host first, which uses prepare_bastion as the check_func here:

https://github.com/pndaproject/pnda-cli/blob/develop/cli/backend_base.py#L462-L490

It is doing a bit more than just checking connectivity and it is using ssh. I would expect to see error details in cli/logs/pnda-cli.*.log – is there any more information in the log?

Thanks,

Donald.

On 3 Feb 2019, at 01:44, NEKRASSOV, ALEXEI <an4...@att.com> wrote:

Hello Donald,

2019-01-31 22:26:11 INFO     Checking connectivity to 10.12.5.138

How is the connectivity checked, is it by ICMP ping?

I can ssh to PNDA gateway, but cannot ping, because

            ALLOW IPv4 icmp from dev1-onap-pnda-pnda_securityGroupPnda

but I’m installing from machine that’s not in dev1-onap-pnda-pnda_securityGroupPnda

Thanks,

Alex

From: NEKRASSOV, ALEXEI
Sent: Thursday, January 31, 2019 5:59 PM
To: 'Donald Hunter (donaldh)' <don...@cisco.com>; Thiruveedula, Bharath <bharath.th...@verizon.com>
Cc: VENKATESH KUMAR, VIJAY <vv7...@att.com>; Rajesh R <raje...@verizon.com>
Subject: RE: [E] PNDA Installation issues

Thanks, I now found where to delete the stacks in the Dashboard.

The VM’s seem to be created OK, but I got timeout (log below). Looks like networking is not set up correctly in the new VM’s

Vijay, are these differences in Network and IP expected? Or should PNDA VM’s be on oam_network_sAsv instead?

dev1-onap-pnda-pnda-gateway

<image001.png>

k8s3-rancher

<image002.png>

Donald,

How do I control what VM flavors are being requested? The flavors that are used by default are not available in WindRiver.

For now I modified bootstrap.sh to overwrite defaults as

mgr1Flavor: m1.xlarge

edgeFlavor: m2.xlarge

kafkaFlavor: m1.large

datanodeFlavor: m1.large

gatewayFlavor: m1.medium

I couldn’t find good matches for two of the flavors:

m4.xlarge        => m1.xlarge (8 CPU instead of 4 required, more disk)

m4.large        => m1.large (4 CPU instead of 2 required)

Thanks,

Alex

+ ./cli/pnda-cli.py create -e dev1-onap-pnda-pnda -f pico -n 2 -k 1 -b release/5.0 -s pnda-key --set 'mirrors.PNDA_MIRROR=http://10.0.0.11:80'

2019-01-31 22:25:19 INFO         ____ _   ______ ___

2019-01-31 22:25:19 INFO        / __ \/ | / / __ \/   |

2019-01-31 22:25:19 INFO       / /_/ / |/ / / / / /| |

2019-01-31 22:25:19 INFO      / ____/ /| / /_/ / ___ |

2019-01-31 22:25:19 INFO     /_/   /_/ |_/_____/_/ |_|

2019-01-31 22:25:19 INFO

2019-01-31 22:25:19 INFO     Saving debug log to logs/pnda-cli.1548973517.33.log

2019-01-31 22:25:19 INFO     Auth version provided is v3

2019-01-31 22:25:21 INFO     Auth version provided is v3

2019-01-31 22:25:21 INFO     Private Key...... OK

2019-01-31 22:25:21 INFO     PNDA mirror...... OK

2019-01-31 22:25:21 INFO     Auth version provided is v3

2019-01-31 22:25:21 INFO     Openstack connection... OK

2019-01-31 22:25:21 INFO     Auth version provided is v3

2019-01-31 22:25:31 INFO     Stack is: CREATING

2019-01-31 22:25:36 INFO     Stack is: CREATE_IN_PROGRESS

2019-01-31 22:26:07 INFO     Stack is: CREATE_COMPLETE

2019-01-31 22:26:07 INFO     Auth version provided is v3

2019-01-31 22:26:09 INFO     Auth version provided is v3

2019-01-31 22:26:11 INFO     Checking connectivity to 10.12.5.138

2019-01-31 22:26:13 INFO     Checking connectivity to 10.12.5.138

2019-01-31 22:26:16 INFO     Checking connectivity to 10.12.5.138

2019-01-31 22:26:18 INFO     Checking connectivity to 10.12.5.138

2019-01-31 22:26:20 INFO     Checking connectivity to 10.12.5.138

2019-01-31 22:26:22 INFO     Checking connectivity to 10.12.5.138

2019-01-31 22:26:24 INFO     Checking connectivity to 10.12.5.138

2019-01-31 22:26:26 INFO     Checking connectivity to 10.12.5.138

2019-01-31 22:26:28 INFO     Checking connectivity to 10.12.5.138

2019-01-31 22:26:30 INFO     Checking connectivity to 10.12.5.138

2019-01-31 22:26:32 INFO     Checking connectivity to 10.12.5.138

2019-01-31 22:26:34 INFO     Checking connectivity to 10.12.5.138

2019-01-31 22:26:36 INFO     Checking connectivity to 10.12.5.138

2019-01-31 22:26:38 INFO     Checking connectivity to 10.12.5.138

2019-01-31 22:26:40 INFO     Checking connectivity to 10.12.5.138

2019-01-31 22:26:42 INFO     Checking connectivity to 10.12.5.138

2019-01-31 22:26:44 INFO     Checking connectivity to 10.12.5.138

2019-01-31 22:26:46 INFO     Checking connectivity to 10.12.5.138

2019-01-31 22:26:49 INFO     Checking connectivity to 10.12.5.138

2019-01-31 22:26:51 INFO     Checking connectivity to 10.12.5.138

2019-01-31 22:26:53 INFO     Checking connectivity to 10.12.5.138

2019-01-31 22:27:36 INFO     Checking connectivity to 10.12.5.138

2019-01-31 22:28:19 INFO     Checking connectivity to 10.12.5.138

2019-01-31 22:29:02 INFO     Checking connectivity to 10.12.5.138

2019-01-31 22:29:45 INFO     Checking connectivity to 10.12.5.138

2019-01-31 22:30:28 INFO     Checking connectivity to 10.12.5.138

2019-01-31 22:31:11 INFO     Checking connectivity to 10.12.5.138

2019-01-31 22:31:54 INFO     Checking connectivity to 10.12.5.138

2019-01-31 22:32:37 INFO     Checking connectivity to 10.12.5.138

2019-01-31 22:33:31 INFO     Checking connectivity to 10.12.5.138

2019-01-31 22:34:14 INFO    Checking connectivity to 10.12.5.138

2019-01-31 22:34:57 INFO     Checking connectivity to 10.12.5.138

2019-01-31 22:35:40 INFO     Checking connectivity to 10.12.5.138

2019-01-31 22:36:11 ERROR    Error waiting for host connectivity, timeout after 10 minutes. See debug log (logs/pnda-cli.1548973517.33.log) for details.

Traceback (most recent call last):

File "./cli/pnda-cli.py", line 225, in <module>

    main()

File "./cli/pnda-cli.py", line 213, in main

    console_dns = deployment_target.create(fields)

File "/pnda-cli/cli/backend_base.py", line 81, in create

    self._install_pnda()

File "/pnda-cli/cli/backend_base.py", line 493, in _install_pnda

    self._wait_for_host_connectivity([bastion_ip], False, prepare_bastion)

File "/pnda-cli/cli/backend_base.py", line 408, in _wait_for_host_connectivity

    self._wait_on_host_operations('waiting for host connectivity', wait_threads, bastion_used, None)

File "/pnda-cli/cli/backend_base.py", line 381, in _wait_on_host_operations

    raise Exception("Error %s, timeout after %s minutes. See debug log (%s) for details." % (action, generic_timeout_minutes, LOG_FILE_NAME))

Exception: Error waiting for host connectivity, timeout after 10 minutes. See debug log (logs/pnda-cli.1548973517.33.log) for details.

alexein...@gmail.com

unread,

Feb 5, 2019, 5:18:57 PM2/5/19

to PNDA Users

Hi Donald,

Please see logs below. The gateway is accessible by IP address, but not by short host name.

Any suggestions?

Also:

- A recurring theme is we need to check logs or update files in container that’s completed, and thus we cannot exec into it. We need to find a way to make this more usable. Perhaps programmatically check that install failed, and run long sleep to allow login for troubleshooting?..
error: cannot exec into a container in a completed pod; current phase is Failed
- Is there a way to control what VM flavors are being requested?

Thanks,

Alex

2019-02-05 21:50:02 - INFO - ____ _ ______ ___

2019-02-05 21:50:02 - INFO - / __ \/ | / / __ \/ |

2019-02-05 21:50:02 - INFO - / /_/ / |/ / / / / /| |

2019-02-05 21:50:02 - INFO - / ____/ /| / /_/ / ___ |

2019-02-05 21:50:02 - INFO - /_/ /_/ |_/_____/_/ |_|

2019-02-05 21:50:02 - INFO -

2019-02-05 21:50:02 - INFO - Saving debug log to logs/pnda-cli.1549403397.8.log

2019-02-05 21:50:02 - INFO - Auth version provided is v3

2019-02-05 21:50:04 - INFO - Auth version provided is v3

2019-02-05 21:50:04 - INFO - Private Key...... OK

2019-02-05 21:50:05 - INFO - PNDA mirror...... OK

2019-02-05 21:50:05 - INFO - Auth version provided is v3

2019-02-05 21:50:05 - INFO - Openstack connection... OK

2019-02-05 21:50:07 - INFO - Auth version provided is v3

2019-02-05 21:50:17 - INFO - Stack is: CREATING

2019-02-05 21:50:22 - INFO - Stack is: CREATE_IN_PROGRESS

2019-02-05 21:50:48 - INFO - Stack is: CREATE_COMPLETE

2019-02-05 21:50:49 - INFO - Auth version provided is v3

2019-02-05 21:50:50 - INFO - Auth version provided is v3

2019-02-05 21:50:53 - INFO - Checking connectivity to 10.12.5.8

2019-02-05 21:50:53 - INFO - 10.12.5.8 Warning: Permanently added '10.12.5.8' (ECDSA) to the list of known hosts.

2019-02-05 21:50:54 - INFO - 10.12.5.8

2019-02-05 21:50:54 - INFO - 10.12.5.8 Warning: Permanently added '10.12.5.8' (ECDSA) to the list of known hosts.

2019-02-05 21:50:54 - INFO - 10.12.5.8 sudo: unable to resolve host dev1-onap-pnda-pnda-gateway

2019-02-05 21:50:54 - INFO - 10.12.5.8

2019-02-05 21:50:54 - INFO - 10.12.5.8 #!/bin/bash -v

2019-02-05 21:50:54 - INFO - 10.12.5.8

2019-02-05 21:50:54 - INFO - Traceback (most recent call last):

File "/pnda-cli/cli/backend_base.py", line 394, in do_wait

check_func()

File "/pnda-cli/cli/backend_base.py", line 490, in prepare_bastion

raise Exception("Error running ssh commands on host %s. See debug log (%s) for details." % (bastion_ip, LOG_FILE_NAME))

Exception: Error running ssh commands on host 10.12.5.8. See debug log (logs/pnda-cli.1549403397.8.log) for details.

ubuntu@k8s-rancher:~$ ssh -i ~/pnda/key 10.12.5.8

Welcome to Ubuntu 16.04.2 LTS (GNU/Linux 4.4.0-64-generic x86_64)

* Documentation: https://help.ubuntu.com

* Management: https://landscape.canonical.com

* Support: https://ubuntu.com/advantage

Get cloud support with Ubuntu Advantage Cloud Guest:

http://www.ubuntu.com/business/services/cloud

0 packages can be updated.

0 updates are security updates.

New release '18.04.1 LTS' available.

Run 'do-release-upgrade' to upgrade to it.

ubuntu@dev1-onap-pnda-pnda-gateway:~$ logout

Connection to 10.12.5.8 closed.

ubuntu@k8s-rancher:~$ ssh -i ~/pnda/key dev1-onap-pnda-pnda-gateway

ssh: Could not resolve hostname dev1-onap-pnda-pnda-gateway: Name or service not known