cbtool Xenial Alpha 2 install issues

[James Scollard]

Since Xenial is one of the few OS releases that can satisfy the bleeding edge dependencies here, we have moved to it as a last ditch effort to make cbtool usable with Openstack clouds.  Here is what we get on a fresh install:

There are 1 dependencies missing: gcloud

Please add the missing dependency(ies) and re-run install again.

You have new mail in /var/mail/root

root@cloudbench-xenial:/opt/cbtool# pip install gcloud

Requirement already satisfied (use --upgrade to upgrade): gcloud in /usr/local/lib/python2.7/dist-packages

Cleaning up...

root@cloudbench-xenial:/opt/cbtool# pip install --upgrade gcloud

Requirement already up-to-date: gcloud in /usr/local/lib/python2.7/dist-packages

Cleaning up...

When we run the executable it errors out for no apparently reason:

root@cloudbench-xenial:/opt/cbtool# cd cb

-bash: cd: cb: Not a directory

root@cloudbench-xenial:/opt/cbtool# ./cb

Cbtool version is "0545f81"

Parsing "cloud definitions" file..... "/opt/cbtool/lib/auxiliary//../..//configs/cloud_definitions.txt" opened and parsed successfully.

Checking "Object Store".....An Object Store of the kind "Redis" (shared) on node 192.168.0.10, TCP port 6379, database id "0" seems to be running.

Checking "Log Store"..... Error while executing the command line "sudo pkill -9 -f /opt/cbtool/lib/auxiliary//../../stores/root_rsyslog.conf" (returncode = 27680) :sudo: unable to resolve host cloudbench-xenial

9

This is pretty much our last day of trying to make cloudbench work after having 2 full time personnel spend every day for over a week trying to get it to run and connect to our clouds.  SInce this isnt usable we will probably have to move to Rally again.

Thanks.

[Joe Talerico]

Hey James - Not sure what dependencies you are referring to, with
CBTool the only weird dep I have found is the specific version for
netcat, other than that, I get CBTool installs on CentOS7 and RHEL7
pretty regularly.

Also, I don't find Rally a good tool for workload management -- it is
great for control-plane, but not so great for workload management,
which CBTool does well at.

Joe

[James Scollard]

We have failed to make in install that connect to an openstack cloud correctly on rhel 7, centos 7, ubuntu 14.04, ubuntu 15.04, and ubuntu 16.04 alpha 2.

We also hate rally for managing workloads which is why we created odin in the mythos repo to manage it and chain load test plans. We were really hoping to get this working but it just doesnt.

Bear in mind that our rhel and derivative images are using outdated icehouse era software repos which is not a fault of cbtool (literally redhouse official versions though) but even the generic ubuntu images with global repos dont install it correctly due to missing deps. Mainly with gcloud which we dont care about and arent even using.

We did manage to get a dashboard up and add apache security to it sinve there is no auth mechanism provided, but it couldnt connect to our openstack deployments or do anything useful.

Our prime directive is to make some kind of progress, and we simply haven with cloudbench.

Thanks.

[Michael R. Hines]

James,

You're almost there. Startup issues with rsyslogd are a known problem
---- an issue posted on github with your experience would be helpful.
The tool makes attempts to start local copies of all the daemons that it
needs to isolate itself from the host ---- that's why it's trying to
startup it's own rsyslog daemon.

Until it's fixed, first shutdown your local rsyslogd daemon and kill any
remaining daemons and try it again.

Really, we need to get proper RPM and DEB packaging for CloudBench, but
just haven't yet had the time. Maybe 2016 will be the year =)

- Michael

[Joe Talerico]

On Mon, Feb 1, 2016 at 2:38 PM, Michael R. Hines <mic...@hinespot.com> wrote:
> James,
>
> You're almost there. Startup issues with rsyslogd are a known problem ----
> an issue posted on github with your experience would be helpful. The tool
> makes attempts to start local copies of all the daemons that it needs to
> isolate itself from the host ---- that's why it's trying to startup it's own
> rsyslog daemon.
>
> Until it's fixed, first shutdown your local rsyslogd daemon and kill any
> remaining daemons and try it again.
>
> Really, we need to get proper RPM and DEB packaging for CloudBench, but just
> haven't yet had the time. Maybe 2016 will be the year =)
>
> - Michael

+1!

Eventually I was going to build a set of ansible scripts to automate
the install, and validation of CBTool.


Eventually......

[Marcio Silva]

Hello James,

4 comments:

1) This dependency error that you see is actually caused by the fact the pip install gcloud does not configure the executable to be on the PATH. Login out (or re-running .bashrc) should fix it.

2) Can you please run the command "sudo pkill -9 -f /opt/cbtool/lib/auxiliary//../../stores/root_rsyslog.conf" and check the result? It is not clear why is it failing.

3) Please don't forget to add an entry in /etc/hosts for the orchestrator (cloudbench-xenial). CBTOOL will try to resolve this name in multiple occasions, and having it there in /etc/hosts should avoid any trouble.

4) Please be aware that we have not tested CBTOOL with Xenial yet.

Regards,

Marcio

[Michael R. Hines]

Ansible + docker image would probably be a perfect bandaid solution.

- Michael

[Salman Baset]

There are typically few issues in OpenStack cloud and CBTOOL when setting it up:

(1) Are instances in VM reachable from CBTOOL machine? Is the network name configured correctly? It is best to start with flat networking.

(2) Does your cloud use HTTP or HTTPS?

(3) Did you add image names correctly?

CBTOOL is used in the upcoming SPEC Cloud IaaS 2016 Benchmark, and has been tested by multiple companies.

Here is an example of CBTOOL configuration.

 
[USER-DEFINED : CLOUDOPTION_MYOPENSTACK]
OSK_ACCESS = http://PUBLICIP:5000/v2.0/                   # Address of controlled node (where nova-api runs)
OSK_CREDENTIALS =  admin-admin-admin                      # user-password-tenant
OSK_SECURITY_GROUPS = default                             # Make sure that this group exists first
OSK_INITIAL_VMCS = RegionOne                              # Change "RegionOne" accordingly
OSK_LOGIN = cbuser                                        # The username that logins on the VMs
OSK_KEY_NAME = spec_key                                   # SSH key for logging into workload VMs
OSK_SSH_KEY_NAME = spec_key                               # SSH key for logging into workload VMs
OSK_NETNAME = public

ADD BELOW

[VM_TEMPLATES : OSK_CLOUDCONFIG]                            # setting various CBTOOL roles and images
# the images have to exist in OpenStack glance.
#choose required images (centos or ubuntu) and comment other images.
# centos images

CASSANDRA = size:m1.medium, imageid1:cb_speccloud_cassandra_centos
YCSB = size:m1.medium, imageid1:cb_speccloud_cassandra_centos
SEED = size:m1.medium, imageid1:cb_speccloud_cassandra_centos
HADOOPMASTER = size:m1.medium, imageid1:cb_speccloud_kmeans_centos
HADOOPSLAVE = size:m1.medium, imageid1:cb_speccloud_kmeans_centos

Salman 

[Joe Talerico]

On Mon, Feb 1, 2016 at 2:42 PM, Michael R. Hines <mic...@hinespot.com> wrote:
> Ansible + docker image would probably be a perfect bandaid solution.
>
> - Michael
>

Agreed... I am hoping to have CBTool integrated with Browbeat at some
point in 2016... Right now we have Rally/Shaker integration...
CBTool... On deck!

[Salman Baset]

For this specific issue, please do the following:

vi /etc/hosts
IPADDR cloudbench-xenial

If this machine has multiple network interfaces, please be sure to specific the IP address that is reachable from VMs in OpenStack.

Also, if the machine has multiple network interfaces, please specify in the CloudBench configuration file the IP address of the interface reachable from VMs in OpenStack.

MANAGER_IP = IPADDRESS_OF_INTERFACE_FOR_CLOUDAPI

Salman

On Mon, Feb 1, 2016 at 2:17 PM, James Scollard <spyde...@gmail.com> wrote:

[Salman Baset]

Also, to make it more interactive, I have started an IRC channel. Please join there:

http://webchat.freenode.net/?channels=cbtool

salman

On Mon, Feb 1, 2016 at 2:17 PM, James Scollard <spyde...@gmail.com> wrote:

[James Scollard]

OK.  Its alive after manually killing the system rsyslog daemon and manually editing /etc/hosts to map the node name to the local IP address.  It is able to start and attach an experiment now:

root@cloudbench-xenial:/opt/cbtool# ./cb

Cbtool version is "0545f81"

Parsing "cloud definitions" file..... "/opt/cbtool/lib/auxiliary//../..//configs/cloud_definitions.txt" opened and parsed successfully.

Checking "Object Store".....An Object Store of the kind "Redis" (shared) on node 192.168.0.10, TCP port 6379, database id "0" seems to be running.

Checking "Log Store".....A Log Store of the kind "rsyslog" (private) on node 192.168.0.10, UDP port 5114 seems to be running.

Checking "Metric Store".....A Metric Store of the kind "MongoDB" (shared) on node 192.168.0.10, TCP port 27017, database id "metrics" seems to be running.

Checking "File Store".....A File Store of the kind "rsync" (private) on node 192.168.0.10, TCP port 873 seems to be running.

Checking for a running API service daemon.....API Service daemon was successfully started. The process id is ['2243'] (http://192.168.0.10:7070).

Checking for a running GUI service daemon.....GUI Service daemon was successfully started. The process id is ['2268', '2269'], listening on port 8080. Full url is "http://192.168.0.10:8080".

 status: Creating "OpenVPN" VPN server unified CB configuration: /opt/cbtool/lib/operations//../../util/openvpn/make_keys.sh 192.168.0.0 255.255.0.0 MYSIMCLOUD 192.168.0.10 1194, please wait ...

 status: VPN configuration success: (192.168.0.0 255.255.0.0)

The "sim" cloud named "MYSIMCLOUD" was successfully attached to this experiment.

The experiment identifier is EXP-02-02-2016-04-21-11-PM-UTC

 status: VMC ACC51D29-6EEA-54F0-A881-33C9C54E16E2 was successfully registered on SimCloud "MYSIMCLOUD".

 status: VMC 6A8DB9B6-4203-52EF-B10F-AAEC58A89BE6 was successfully registered on SimCloud "MYSIMCLOUD".

 status: VMC BDB17D71-0AE7-555F-A1CE-7DBA9A6A8B15 was successfully registered on SimCloud "MYSIMCLOUD".

 status: VMC 0A8ACAE8-0CFC-57F6-8352-4886A079723E was successfully registered on SimCloud "MYSIMCLOUD".

 status: Attribute "collect_from_host" was set to "false". Skipping Host OS performance monitor daemon startup

All VMCs successfully attached to this experiment.

(MYSIMCLOUD)

Moving on to setting up the Openstack connection now.

[Michael R. Hines]

Don't be a stranger. We may all be at different companies, but we want to help. =)

/*
 * Michael R. Hines
 * Platform Engineer, DigitalOcean.
 */

- Michael

[James Scollard]

Much appreciated.

:)

[James Scollard]

So we appear to b e back to the issue we posted a few days ago where the Openstack connector does not appear to be interacting with the API correctly:

root@cloudbench-xenial:/opt/cbtool# ./cb -x

Cbtool version is "0545f81"

Parsing "cloud definitions" file..... "/opt/cbtool/lib/auxiliary//../..//configs/cloud_definitions.txt" opened and parsed successfully.

Checking "Object Store".....An Object Store of the kind "Redis" (shared) on node 192.168.0.10, TCP port 6379, database id "0" seems to be running.

Checking "Log Store".....A Log Store of the kind "rsyslog" (private) on node 192.168.0.10, UDP port 5114 seems to be running.

Checking "Metric Store".....A Metric Store of the kind "MongoDB" (shared) on node 192.168.0.10, TCP port 27017, database id "metrics" seems to be running.

Checking "File Store".....A File Store of the kind "rsync" (private) on node 192.168.0.10, TCP port 873 seems to be running.

Executing "hard" reset: (killing all running toolkit processes and flushing stores) before starting the experiment......

Killing all processes... done

Flushing Object Store... done

Flushing Log Store... done

Flushing Metric Store... done

Checking for a running API service daemon.....API Service daemon was successfully started. The process id is ['4097'] (http://192.168.0.10:7070).

Checking for a running GUI service daemon.....GUI Service daemon was successfully started. The process id is ['4121', '4122'], listening on port 8080. Full url is "http://192.168.0.10:8080".

 status: Creating "OpenVPN" VPN server unified CB configuration: /opt/cbtool/lib/operations//../../util/openvpn/make_keys.sh 192.168.0.0 255.255.0.0 MYOPENSTACK 192.168.0.10 1194, please wait ...

 status: VPN configuration success: (192.168.0.0 255.255.0.0)

 status: OpenStack connection parameters: username=admin, password=<omitted>, tenant=admin, cacert=None, insecure=False, region_name=ap-tokyo-1, access_url=https://int.ap, endpoint_type=publicURL

/usr/local/lib/python2.7/dist-packages/novaclient/v2/client.py:109: UserWarning: 'novaclient.v2.client.Client' is not designed to be initialized directly. It is inner class of novaclient. Please, use 'novaclient.client.Client' instead. Related lp bug-report: 1493576

  _LW("'novaclient.v2.client.Client' is not designed to be "

 status: VMC "ap-tokyo-1" did not pass the connection test." : OpenStack connection failure: HTTPSConnectionPool(host='int.ap', port=443): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7fb79ea6fc10>: Failed to establish a new connection: [Errno -2] Name or service not known',))

The "osk" cloud named "MYOPENSTACK" could not be attached to this experiment: VMC "ap-tokyo-1" did not pass the connection test." : OpenStack connection failure: HTTPSConnectionPool(host='int.ap', port=443): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7fb79ea6fc10>: Failed to establish a new connection: [Errno -2] Name or service not known',))

Usage: vmcattach <cloud_name> <identifier> [temp_attr_list = empty=empty] [mode] 

() 

[James Scollard]

Changing the url encoding to the endpoint does not fix the issue:

###CONFIG

[USER-DEFINED : CLOUDOPTION_MYOPENSTACK]

OSK_ACCESS = https://int.country_dash_place_dash_zone.cloud.somebody.com:5000/v2.0/

###RESULT

root@cloudbench-xenial:/opt/cbtool# ./cb -x

Cbtool version is "0545f81"

Parsing "cloud definitions" file..... "/opt/cbtool/lib/auxiliary//../..//configs/cloud_definitions.txt" opened and parsed successfully.

Checking "Object Store".....An Object Store of the kind "Redis" (shared) on node 192.168.0.10, TCP port 6379, database id "0" seems to be running.

Checking "Log Store".....A Log Store of the kind "rsyslog" (private) on node 192.168.0.10, UDP port 5114 seems to be running.

Checking "Metric Store".....A Metric Store of the kind "MongoDB" (shared) on node 192.168.0.10, TCP port 27017, database id "metrics" seems to be running.

Checking "File Store".....A File Store of the kind "rsync" (private) on node 192.168.0.10, TCP port 873 seems to be running.

Executing "hard" reset: (killing all running toolkit processes and flushing stores) before starting the experiment......

Killing all processes... done

Flushing Object Store... done

Flushing Log Store... done

Flushing Metric Store... done

Checking for a running API service daemon.....API Service daemon was successfully started. The process id is ['4802'] (http://192.168.0.10:7070).

Checking for a running GUI service daemon.....GUI Service daemon was successfully started. The process id is ['4827', '4828'], listening on port 8080. Full url is "http://192.168.0.10:8080".

 status: VPN configuration for this cloud already generated: /opt/cbtool/lib/auxiliary//../../configs/generated/MYOPENSTACK_server-cb-openvpn.conf

 status: OpenStack connection parameters: username=admin, password=<omitted>, tenant=admin, cacert=None, insecure=False, region_name=country-place-zone, access_url=https://int.country-place-zone.cloud.somebody.com:5000/v2.0/, endpoint_type=publicURL

/usr/local/lib/python2.7/dist-packages/novaclient/v2/client.py:109: UserWarning: 'novaclient.v2.client.Client' is not designed to be initialized directly. It is inner class of novaclient. Please, use 'novaclient.client.Client' instead. Related lp bug-report: 1493576

  _LW("'novaclient.v2.client.Client' is not designed to be "

[Marcio Silva]

Hello James,

Not clear about the error there. Can you please try the following from a python prompt?

>>> from novaclient.v2 import client; oskconn=client.Client("admin", YOURPW, "admin", "https://int.country-place-zone.cloud.somebody.com:5000/v2.0/", region_name = "ap-tokyo-1", service_type="compute")
>>> oskconn.flavors.list()

[James Scollard]

Installed a new host (Ubuntu 16.04 Xenial 64 bit Alpha 2) on the control plane with full access to internal APIs and admin access allowed.  The cloud admin account for tenant space is still not able to authenticate and take control of the cloud using the APIs.  These are the same credentials we use to run Rally and Tempest testing, form the same netowrk where those services take over, with the same priveges as those services and it still does not connect.

Config:

OSK_ACCESS = http://int.country_dash_place_dash_zone.cloud.cisco.com:5000/v2.0/                    # Address of controlled node (where nova-api runs)

OSK_CREDENTIALS =  admin-password-admin'

OSK_SECURITY_GROUPS = default                              # Make sure that this group exists first

OSK_INITIAL_VMCS = country-place-zone:sut                           # Change "RegionOne" accordingly

OSK_LOGIN = cbuser                                         # The username that logins on the VMs

OSK_NETNAME = public-net-name

Output:

root@cloudbench-xenial:/opt/cbtool# ./cb -x

Cbtool version is "0545f81"

Parsing "cloud definitions" file..... "/opt/cbtool/lib/auxiliary//../..//configs/root_cloud_definitions.txt" opened and parsed successfully.

Checking "Object Store".....An Object Store of the kind "Redis" (shared) on node 10.202.145.105, TCP port 6379, database id "0" seems to be running.

Checking "Log Store".....A Log Store of the kind "rsyslog" (private) on node 10.202.145.105, UDP port 5114 seems to be running.

Checking "Metric Store".....A Metric Store of the kind "MongoDB" (shared) on node 10.202.145.105, TCP port 27017, database id "metrics" seems to be running.

Checking "File Store".....A File Store of the kind "rsync" (private) on node 10.202.145.105, TCP port 873 seems to be running.

Executing "hard" reset: (killing all running toolkit processes and flushing stores) before starting the experiment......

Killing all processes... done

Flushing Object Store... done

Flushing Log Store... done

Flushing Metric Store... done

Checking for a running API service daemon.....API Service daemon was successfully started. The process id is ['28539'] (http://10.202.145.105:7070).

Checking for a running GUI service daemon.....GUI Service daemon was successfully started. The process id is ['28564', '28565'], listening on port 8080. Full url is "http://10.202.145.105:8080".

 status: VPN configuration for this cloud already generated: /opt/cbtool/lib/auxiliary//../../configs/generated/MYOPENSTACK_server-cb-openvpn.conf

 status: OpenStack connection parameters: username=admin, password=<omitted>, tenant=admin', cacert=None, insecure=False, region_name=country-place-zone, access_url=http://int.country-place-zone.cloud.cisco.com:5000/v2.0/, endpoint_type=publicURL

/usr/local/lib/python2.7/dist-packages/novaclient/v2/client.py:109: UserWarning: 'novaclient.v2.client.Client' is not designed to be initialized directly. It is inner class of novaclient. Please, use 'novaclient.client.Client' instead. Related lp bug-report: 1493576

  _LW("'novaclient.v2.client.Client' is not designed to be "

 status: VMC "country-place-zone" did not pass the connection test." : OpenStack connection failure: ('Connection aborted.', BadStatusLine("''",))

The "osk" cloud named "MYOPENSTACK" could not be attached to this experiment: VMC "country-place-zone" did not pass the connection test." : OpenStack connection failure: ('Connection aborted.', BadStatusLine("''",))

Usage: vmcattach <cloud_name> <identifier> [temp_attr_list = empty=empty] [mode] 

()  cldattach osk MYOPENSTACK

status: OpenStack connection parameters: username=admin, password=<omitted>, tenant=admin', cacert=None, insecure=False, region_name=country-place-zone, access_url=http://int.country-place-zone.cloud.cisco.com:5000/v2.0/, endpoint_type=publicURL

 status: VMC "country-place-zone" did not pass the connection test." : OpenStack connection failure: ('Connection aborted.', BadStatusLine("''",))

The "osk" cloud named "MYOPENSTACK" could not be attached to this experiment: VMC "country-place-zone" did not pass the connection test." : OpenStack connection failure: ('Connection aborted.', BadStatusLine("''",))

[James Scollard]

OSK_CREDENTIALS =  admin-password-admin'

should be:

OSK_CREDENTIALS =  admin-password-admin

One typo to rule them all, and in the darkness bind them...

[Michael R. Hines]

James,

Marcio maintains the OpenStack adapter, and he needs to see directly in python where the API call is breaking down.

Can you run his snippet on the python command line ---- there's probably an API compatibility breakage somewhere
and he needs to see the difference between your installed novaclient and his.

/*
 * Michael R. Hines
 * Platform Engineer, DigitalOcean.
 */

[Marcio Silva]

Hello Michael,

I had a quick google hangout with James, and that particular problem was fixed. It turns out that there was an extra single-quote (') at the end of the password string.

I am thinking about including a small test script in case of Cloud connection failures (for instance, outputting a series of nova commands to be run outside CBTOOL just to isolate cloud-specific problems). Still pondering the best of way of implementing something like that (it should be highly cloud adapter-specific).

Also, I still need to fix the now deprecated use  of "novaclient.v2.client.Client" in the OpenStack cloud adapter, which is resulting in a deeply annoying Warning message. Should have something ready to push soon.

...