Openstack Nova+Neutron support

[Rahul Mohan Rekha]

Hi all,

Here's a quick overview of the current state of Openstack Nova+Neutron support:
https://bitbucket.org/thisisrmr/thisisrmr.bitbucket.org/wiki/state-of-things

Feedback welcome! :-)

Regards,
Rahul

[Luke Gorrie]

On 21 January 2014 13:50, Rahul Mohan Rekha <rahu...@icloud.com> wrote:
> Here's a quick overview of the current state of Openstack Nova+Neutron support:
> https://bitbucket.org/thisisrmr/thisisrmr.bitbucket.org/wiki/state-of-things

Nice write-up :-)

> Feedback welcome! :-)

LIBVIRT:

I can report that the vhost-user QEMU extension work is going forward well. The code is now fairly mature and the hackers at VirtualOpenSystems are working closely with the maintainer Michael Tsirkin to do everything that's needed to bring it upstream as a standard QEMU feature.

NOVA:

Just want to mention some Nova development that can be interesting/useful for us.

Ian Wells, Irena Berezovsky, and others are extending the Nova scheduler to be aware of provider networks when deciding which compute host to place a VM on [1]. So it should be possible to assign a Port to a provider physical_network (called e.g. "DataCenter7VoiceNetwork42") and for Nova to figure out which compute host(s) are able to connect to this network.

This is useful for networks like TeraStream. The network designers should be able to use meaningful (to them) names to talk about which physical network in the data center a VM should be connected to. So they could configure that DataCenter7VoiceNetwork42 maps to (say) a particular PCI address on a particular server.

This work seems to be already going in the right direction for our needs, many thanks to Ian & all.

[1] http://lists.openstack.org/pipermail/openstack-dev/2014-January/024927.html

NEUTRON:

I like the idea of continuously polling all relevant database tables for changes. Very "Snabb" :-). The alternative, to extend the Mechanism Driver API to support a create/update/delete method for every Neutron extension we will ever want to use, makes my head hurt a bit.

I have a crazy idea. Bear with me for a moment :-). Could this potentially be done by the ML2 driver starting a small shell script instead of a Python/SQLchemy thread?

In wild pseudo code:

mysqldump --tables network,port,... > old.csv

while true; do

  mysqldump --tables network,port,... > new.csv

  # check for config change

  if diff old.csv new.csv; then

    # create new global conf from database

    snabbswitch csv2conf new.csv snabbswitch.conf

    # sync config to all compute nodes

    for node in $compute_nodes; do

      rsync snabbswitch.conf $node:/etc/snabb.conf

    done

    # keep new config for future reference

    cp new.csv old.csv

  fi

  # wait before next check

  sleep 0.2

done

This way each Neutron configuration change is pushed to all compute nodes with ~ 200ms latency. The traffic process on the compute node could be poll (or inotify) to automatically load new versions of /etc/snabb.conf.

Then we would have a tiny amount of code and no RPCs. Heaven? (Or hell...?)

The usual mechanism driver API (network_created, etc) could potentially be used separately to call snabbswitch on the network node to check if a new config is OK or not -- without having to worry about actually pushing changes out to compute nodes.

I suspect I have missed some really important stuff with that idea. I would love to know what, for the purposes of my own education :)

[Rahul M R]

Hi Luke,

On 21-Jan-2014, at 8:33 PM, Luke Gorrie <lu...@snabb.co> wrote:
>
> LIBVIRT:
>
> I can report that the vhost-user QEMU extension work is going forward well. The code is now fairly mature and the hackers at VirtualOpenSystems are working closely with the maintainer Michael Tsirkin to do everything that's needed to bring it upstream as a standard QEMU feature.
>

I guess then its time to start discussing about vhost-user support with libvirt maintainers.

> NOVA:
>
> Just want to mention some Nova development that can be interesting/useful for us.
>
> Ian Wells, Irena Berezovsky, and others are extending the Nova scheduler to be aware of provider networks when deciding which compute host to place a VM on [1]. So it should be possible to assign a Port to a provider physical_network (called e.g. "DataCenter7VoiceNetwork42") and for Nova to figure out which compute host(s) are able to connect to this network.
>
> This is useful for networks like TeraStream. The network designers should be able to use meaningful (to them) names to talk about which physical network in the data center a VM should be connected to. So they could configure that DataCenter7VoiceNetwork42 maps to (say) a particular PCI address on a particular server.
>
> This work seems to be already going in the right direction for our needs, many thanks to Ian & all.
>
> [1] http://lists.openstack.org/pipermail/openstack-dev/2014-January/024927.html

Thanks for bringing this up. I agree that PCI flavor based device assignment will be useful once it gets implemented. In the meantime, how about using the Neutron/Agent config file to provide physical_network to PCI address mapping ?

>
> NEUTRON:
>
> I like the idea of continuously polling all relevant database tables for changes. Very "Snabb" :-). The alternative, to extend the Mechanism Driver API to support a create/update/delete method for every Neutron extension we will ever want to use, makes my head hurt a bit.
>
> I have a crazy idea. Bear with me for a moment :-). Could this potentially be done by the ML2 driver starting a small shell script instead of a Python/SQLchemy thread?
>
> In wild pseudo code:
>
> mysqldump --tables network,port,... > old.csv
> while true; do
> mysqldump --tables network,port,... > new.csv
> # check for config change
> if diff old.csv new.csv; then
> # create new global conf from database
> snabbswitch csv2conf new.csv snabbswitch.conf
> # sync config to all compute nodes
> for node in $compute_nodes; do
> rsync snabbswitch.conf $node:/etc/snabb.conf
> done
> # keep new config for future reference
> cp new.csv old.csv
> fi
> # wait before next check
> sleep 0.2
> done
>
> This way each Neutron configuration change is pushed to all compute nodes with ~ 200ms latency. The traffic process on the compute node could be poll (or inotify) to automatically load new versions of /etc/snabb.conf.
>

> Then we would have a tiny amount of code and no RPCs. Heaven? (Or hell…?)
>

I've considered the idea of using an external script to do the config sync.

some observations:

1. gives us some flexibility regarding the sync implementation (rsync, git, some other pub-sub mechanism)
2. less code on Neutron end and more code on SnS end (BTW processing the data is slightly easier on python end ;)
3. introduces external dependencies - have to run/maintain additional script (on Controller node) and additional daemons on Compute nodes.
4. requires additional config info (for e.g.: DB access credentials) - can be assigned to ENV variables, before launching the script
5. doesn't use Openstack's message queue based RPC mechanism (which *probably* limits max msg size - can be an issue if config is huge) ;)

Overall, I think it is a reasonable approach and worth a try :)

> The usual mechanism driver API (network_created, etc) could potentially be used separately to call snabbswitch on the network node to check if a new config is OK or not -- without having to worry about actually pushing changes out to compute nodes.
>

I don't know if this is necessary. :)

> I suspect I have missed some really important stuff with that idea. I would love to know what, for the purposes of my own education :)
>

I guess we'll try it out and see if it works!

Thanks for the great feedback! :-)

Regards,
Rahul

[Luke Gorrie]

Fascinating discussion :)

On 22 January 2014 18:58, Rahul M R <this...@gmail.com> wrote:

I guess then its time to start discussing about vhost-user support with libvirt maintainers.

Cool. I can ask the guys doing the vhost-user qemu implementation for advice on this.

Thanks for bringing this up. I agree that PCI flavor based device assignment will be useful once it gets implemented. In the meantime, how about using the Neutron/Agent config file to provide physical_network to PCI address mapping ?

Yes. Sounds simple and like it will work.

I've considered the idea of using an external script to do the config sync.

some observations:

1. gives us some flexibility regarding the sync implementation (rsync, git, some other pub-sub mechanism)
2. less code on Neutron end and more code on SnS end (BTW processing the data is slightly easier on python end ;)

Could we eliminate data processing somehow?

(Could we dump the database tables as CSV and have snabbswitch read them in with a fast CSV parser? That's what R hackers do.)

 

3. introduces external dependencies - have to run/maintain additional script (on Controller node) and additional daemons on Compute nodes.

Controller node... I suppose we could also write the script in Python too. ("I can write shell scripts in any language!" :))

On the compute node... can we eliminate daemons? for example to run only N x snabbswitch traffic processes that pick up new configuration files without needing assistance of a daemon?

 

4. requires additional config info (for e.g.: DB access credentials) - can be assigned to ENV variables, before launching the script
5. doesn't use Openstack's message queue based RPC mechanism (which *probably* limits max msg size - can be an issue if config is huge) ;)

if we ship the config with rsync then we won't have a limit?

I'm not really sure yet what the performance bottleneck would be. dumping the database? pushing updates to compute nodes? parsing configurations in traffic processes? these all seem a bit more predictable to me than Python based RPC/ORM frameworks, but that's my limited experience talking.

> The usual mechanism driver API (network_created, etc) could potentially be used separately to call snabbswitch on the network node to check if a new config is OK or not -- without having to worry about actually pushing changes out to compute nodes.
>

I don't know if this is necessary. :)

Perhaps not. Are there any configurations we would want to reject? (I had been thinking e.g. a config that creates more ports than what we have hardware TX/RX queues for -- but probably it's better to just make the traffic process support that.)

Thanks for the great feedback! :-)

Any time! (No guarantees on quality :-))

[Rahul M R]

On 24-Jan-2014, at 12:44 AM, Luke Gorrie <lu...@snabb.co> wrote:

> Cool. I can ask the guys doing the vhost-user qemu implementation for advice on this.

Thanks! :)

> Could we eliminate data processing somehow?
>
> (Could we dump the database tables as CSV and have snabbswitch read them in with a fast CSV parser? That's what R hackers do.)
>

> I'm not really sure yet what the performance bottleneck would be. dumping the database? pushing updates to compute nodes? parsing configurations in traffic processes? these all seem a bit more predictable to me than Python based RPC/ORM frameworks, but that's my limited experience talking.
>

Yeah, I think the SQLAlchemy + in-memory data processing approach will only work for small DB tables and isn't scalable. Same for pushing config through Openstack RPC.

So, we'll use the DB dump (csv) + csv2conf setup and see how it works.

> 3. introduces external dependencies - have to run/maintain additional script (on Controller node) and additional daemons on Compute nodes.
>
> Controller node... I suppose we could also write the script in Python too. ("I can write shell scripts in any language!" :))
>

True :)

> On the compute node... can we eliminate daemons? for example to run only N x snabbswitch traffic processes that pick up new configuration files without needing assistance of a daemon?

Pardon my ignorance, but how will the N x traffic processes get launched initially? spawned by the agent? init script?

>
> 5. doesn't use Openstack's message queue based RPC mechanism (which *probably* limits max msg size - can be an issue if config is huge) ;)
>
> if we ship the config with rsync then we won't have a limit?

Sorry, I worded this incorrectly - my intention was to write about the max msg size limit in the RPC mechanism, not rsync. :)

>
> > I don't know if this is necessary. :)
>
> Perhaps not. Are there any configurations we would want to reject? (I had been thinking e.g. a config that creates more ports than what we have hardware TX/RX queues for -- but probably it's better to just make the traffic process support that.)
>

I suppose the validation can happen in csv2conf ? For e.g.: if we use the "Router {X} / Port {Y} ( / VF {Z} )" format for the port, then check: Z < Max TX/RX queues per port (defined in intel10g?). will that work?

Regards,
Rahul

[Luke Gorrie]

On 24 January 2014 22:33, Rahul M R <this...@gmail.com> wrote:

So, we'll use the DB dump (csv) + csv2conf setup and see how it works.

Next idea... :-)

Can we use the raw CSV files from the Neutron database as config to the traffic processes? (i.e. skip the csv2conf conversion step.)

Then we could have a collection of files, like:

  /etc/snabb/neutron/Network.csv

  /etc/snabb/neutron/Port.csv

  /etc/snabb/neutron/SecurityGroup.csv

  ...

and the traffic processes would quickly slurp these up and filter/cherry-pick the parts that they care about.

This feels simple and future proof... we could rewrite the whole snabbswitch without having to make any changes on the network node, which is only blindly dumping the neutron database to the compute nodes.

Stop me if I'm being crazy..

I suppose the validation can happen in csv2conf ? For e.g.: if we use the "Router {X} / Port {Y} ( / VF {Z} )" format for the port, then check: Z < Max TX/RX queues per port (defined in intel10g?). will that work?

I'm not sure. I suspect we're better off revisiting this later when it's more clear what restrictions we really have on the Neutron config. e.g. do we support only ~128 Virtio-net ports per physical port or do we find a way to make this unlimited.

[Rahul M R]

On 27-Jan-2014, at 1:33 PM, Luke Gorrie <lu...@snabb.co> wrote:

>
> Can we use the raw CSV files from the Neutron database as config to the traffic processes? (i.e. skip the csv2conf conversion step.)
>

> This feels simple and future proof... we could rewrite the whole snabbswitch without having to make any changes on the network node, which is only blindly dumping the neutron database to the compute nodes.
>
> Stop me if I'm being crazy..

sounds reasonable to me :)

>
> I suppose the validation can happen in csv2conf ? For e.g.: if we use the "Router {X} / Port {Y} ( / VF {Z} )" format for the port, then check: Z < Max TX/RX queues per port (defined in intel10g?). will that work?
>
> I'm not sure. I suspect we're better off revisiting this later when it's more clear what restrictions we really have on the Neutron config. e.g. do we support only ~128 Virtio-net ports per physical port or do we find a way to make this unlimited.
>
>

OK understood.

One thing I forgot to mention before: For each compute node, the allocation of hugepages and mounting the hugetlbfs must be handled by some provisioning script (currently, I'm using vagrant's shell provisioning script to do this). Also, note that <memoryBacking> support is enabled using Nova's InstanceResourceQuota < https://wiki.openstack.org/wiki/InstanceResourceQuota >.

for e.g.:

nova flavor-key m1.small set quota:mem_hugepages=True
nova flavor-key m1.small set quota:mem_nosharepages =True
nova flavor-key m1.small set quota:mem_locked=True


Regards,
Rahul

[Luke Gorrie]

Howdy!

This does not affect us immediately but interesting to be aware of:

Neutron's predecessor, nova-network, has been unfrozen for development and is now a competitor to Neutron. http://lists.openstack.org/pipermail/openstack-dev/2014-January/025824.html

Hopefully our back-to-basics design will contribute to simplicity & robustness in Neutron. It would be nice to make a positive contribution to the design.

Regards,
Rahul

--
You received this message because you are subscribed to the Google Groups "Snabb Switch development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to snabb-devel...@googlegroups.com.
To post to this group, send an email to snabb...@googlegroups.com.
Visit this group at http://groups.google.com/group/snabb-devel.

[Rahul M R]

On 29-Jan-2014, at 11:58 PM, Luke Gorrie <lu...@snabb.co> wrote:

> This does not affect us immediately but interesting to be aware of:
>
> Neutron's predecessor, nova-network, has been unfrozen for development and is now a competitor to Neutron. http://lists.openstack.org/pipermail/openstack-dev/2014-January/025824.html
>
> Hopefully our back-to-basics design will contribute to simplicity & robustness in Neutron. It would be nice to make a positive contribution to the design.

That's interesting!

I hope things will be bit more stable in Neutron after the Icehouse release.

Regards,
Rahul

[Rahul M R]

small update:

Please check the repo: db-sync-conf-gen < https://bitbucket.org/thisisrmr/db-sync-conf-gen/overview >

The repo contains:
1. db-conf-sync.py: sample script to sync DB dump
2. conf-gen.lua: sample script to generate SnS conf (per-process) from DB dump

Please note:

* Based on the NFV draft (with slight modifications)
* All DB dumps are Tab separated (easier to parse)
* A dummy "pci_mappings.txt" file is currently required (this will be generated from SnS Agent config by the SnS AgentMechanismDriver prior to calling db-conf-sync.py):

````
root@precise64:~/db-sync-conf-gen# cat /etc/snabbswitch/neutron/pci_mappings.txt
Router1_Port1 00:05.0
Router1_Port2 00:06.0
````

conf-gen.lua:
1. assumes 1 SnS process per physical NIC port
2. takes PCI address and NIC driver as CLI args
3. currently excludes QoS and Tunnel apps (DB not available in neutron yet).
4. sample output:

````
root@precise64:~/db-sync-conf-gen# ./conf-gen.lua 00:05.0 intel10g

apps.Router1_Port1 = intel10g:new{pciaddr="00:05.0", ...}
apps.Router1_Port1_VF2 = multiqueue:new{vlan_id=112, ...}
apps.Router1_Port1_VF1 = multiqueue:new{vlan_id=111, ...}

apps.SecGroupA = SecGroup:new{rules={
{
ethertype="IPv6", port_range_max="80", direction="ingress", protocol="tcp", port_range_min="80",
},
{
ethertype="IPv6", port_range_max="65535", direction="egress", protocol="tcp", port_range_min="1",
},
}}
apps.SecGroupB = SecGroup:new{rules={
{
ethertype="IPv6", port_range_max="65535", direction="egress", protocol="tcp", port_range_min="1",
},
{
ethertype="IPv6", port_range_max="443", direction="ingress", protocol="tcp", port_range_min="443",
},
}}


apps.VirtioB = VhostUser:new{socket="/var/snabb/vhost/VirtioB.socket", color=apps.Router1_Port1_VF2.get_color()}
apps.VirtioA = VhostUser:new{socket="/var/snabb/vhost/VirtioA.socket", color=apps.Router1_Port1_VF1.get_color()}


app.connect(apps.Router1_Port1, "top", apps.Router1_Port1_VF2, "bottom")
app.connect(apps.Router1_Port1, "top", apps.Router1_Port1_VF1, "bottom")
app.connect(apps.Router1_Port1_VF2, "top", apps.SecGroupB, "bottom")
app.connect(apps.Router1_Port1_VF1, "top", apps.SecGroupA, "bottom")
app.connect(apps.SecGroupB, "top", apps.VirtioB, "bottom")
app.connect(apps.SecGroupA, "top", apps.VirtioA, "bottom")

--------------------------------------------------------

root@precise64:~/db-sync-conf-gen# ./conf-gen.lua 00:06.0 intel10g

apps.Router1_Port2 = intel10g:new{pciaddr="00:06.0", ...}

apps.SecGroupC = SecGroup:new{rules={
{
ethertype="IPv6", port_range_max="65535", direction="egress", protocol="tcp", port_range_min="1",
},
}}


apps.VirtioC = VhostUser:new{socket="/var/snabb/vhost/VirtioC.socket", color=apps.Router1_Port2.get_color()}


app.connect(apps.Router1_Port2, "top", apps.SecGroupC, "bottom")
app.connect(apps.SecGroupC, "top", apps.VirtioC, "bottom")


````

[Luke Gorrie]

On 6 February 2014 02:17, Rahul M R <this...@gmail.com> wrote:
> Please check the repo: db-sync-conf-gen < https://bitbucket.org/thisisrmr/db-sync-conf-gen/overview >
>
> The repo contains:
> 1. db-conf-sync.py: sample script to sync DB dump
> 2. conf-gen.lua: sample script to generate SnS conf (per-process) from DB dump

Cool!

I find it really exciting to think that the integration can be this
simple. <500 lines to push the configuration to all compute hosts and
<500 lines to load it into the snabbswitch process. No RPCs and so no
race conditions, lost messages, lost synchronization, etc. #bliss?

Braindump..

Do we need --lock-tables arg to mysqldump to ensure the tables are
consistent with each other? (is our usage of mysqldump likely to cause
problems for other users of the neutrondb somehow? I am not good with
mysql so I don't know...)

Can we make the rsync to each compute host run independently? So that
if one runs slowly it (a) doesn't slow down the others and (b) is
given a chance to finish without being killed (except by e.g. rsync's
own timeout feature) and (c) not accidentally sync inconsistent files
due to a race condition. Maybe rsync has this built in? (daemon mode?)

Or maybe Git is a reasonable alternative to rsync? Could e.g. use a
branch/tag to track the latest version pushed to each compute hosts,
and only initiate a new push when it knows there are changes (compare
with rsync ssh-and-diff algorithm).

Scalability... going forward if people ask "How is the scalability?"
I'd like to have a quantitative answer such as "with 1,000 VMs on 50
compute nodes and 10 configuration changes per second, it costs around
140% CPU and 200MB RAM on the network node to push out changes at
least once per second." Then people can decide if that's adequate for
them or not. (The answer I want to avoid is unquantitative "it should
work with a million VMs but we haven't tested that" which is a red
flag for service provider people.)

I wonder how we will handle smooth app config changes in
db-sync-conf-gen. for example, new security group rules. any ideas?

regarding "VIFs" and "plugging" in general... is it right that qemu
and snabbswitch can auto-detect each other via the vhost_user socket,
and don't really need a callback from Nova to know when a VM is
started?

db-conf-sync feels to me like it wants to be written in bash rather
than python :).

btw the workflow of hosting on Github and sending pull requests to
snabbswitch master seems to be working well for code review, in case
you want to get feedback directly on the code instead of email.

very very very cool :)
-Luke

[Justin Cormack]

On Thu, Feb 6, 2014 at 9:20 AM, Luke Gorrie <lu...@snabb.co> wrote:
> On 6 February 2014 02:17, Rahul M R <this...@gmail.com> wrote:
>> Please check the repo: db-sync-conf-gen < https://bitbucket.org/thisisrmr/db-sync-conf-gen/overview >
>>
>> The repo contains:
>> 1. db-conf-sync.py: sample script to sync DB dump
>> 2. conf-gen.lua: sample script to generate SnS conf (per-process) from DB dump
>
> Cool!
>
> I find it really exciting to think that the integration can be this
> simple. <500 lines to push the configuration to all compute hosts and
> <500 lines to load it into the snabbswitch process. No RPCs and so no
> race conditions, lost messages, lost synchronization, etc. #bliss?
>
> Braindump..
>
> Do we need --lock-tables arg to mysqldump to ensure the tables are
> consistent with each other? (is our usage of mysqldump likely to cause
> problems for other users of the neutrondb somehow? I am not good with
> mysql so I don't know...)

You can use --single-transaction if you have a sane mysql setup,
--lock-tables is guaranteed to work even if you have an insane setup
(ie non transactional backend).

> Can we make the rsync to each compute host run independently? So that
> if one runs slowly it (a) doesn't slow down the others and (b) is
> given a chance to finish without being killed (except by e.g. rsync's
> own timeout feature) and (c) not accidentally sync inconsistent files
> due to a race condition. Maybe rsync has this built in? (daemon mode?)

You could just spawn them in parallel, but there would be a large load
on the host. At least the config should fit in buffer cache, but there
is no way that I know of not to recompute the checksums for each host.
Might be interesting to make an optimised one to many rsync.

Bittorrent has been used as an efficient way of moving configs around
a cluster eg see
http://www.ebaytechblog.com/2012/01/31/bittorrent-for-package-distribution-in-the-enterprise/#.UvN2kHWHWk0

> Or maybe Git is a reasonable alternative to rsync? Could e.g. use a
> branch/tag to track the latest version pushed to each compute hosts,
> and only initiate a new push when it knows there are changes (compare
> with rsync ssh-and-diff algorithm).

Unlikely to perform well.

> Scalability... going forward if people ask "How is the scalability?"
> I'd like to have a quantitative answer such as "with 1,000 VMs on 50
> compute nodes and 10 configuration changes per second, it costs around
> 140% CPU and 200MB RAM on the network node to push out changes at
> least once per second." Then people can decide if that's adequate for
> them or not. (The answer I want to avoid is unquantitative "it should
> work with a million VMs but we haven't tested that" which is a red
> flag for service provider people.)

Certainly puppet had really bad scaling issues one pushing out
configs, which they never really solved (they say just use rsync or
something else now).

Ideally you want a nice model like Serf has
http://www.serfdom.io/docs/internals/simulator.html
(alas I dont think Serf will solve the problem as the states are too
big but it is a nice tool in line with the philosophy, and very small,
and you might want to look at it anyway as it might come in handy for
propagating the status of updates if you need to see if all hosts have
received the update).

Justin

[Luke Gorrie]

On 6 February 2014 13:05, Justin Cormack <jus...@specialbusservice.com> wrote:
>> Or maybe Git is a reasonable alternative to rsync? Could e.g. use a
>> branch/tag to track the latest version pushed to each compute hosts,
>> and only initiate a new push when it knows there are changes (compare
>> with rsync ssh-and-diff algorithm).
>
> Unlikely to perform well.

I'd guesstimate that we're talking ~ 1MB of configuration and ~ 1
second of acceptable propagation delay. To me it seems plausible that
this can be achieved with simple and robust tools and no fancy
concepts like peer-to-peer.

where do you see the problem(s) with git? the commit/tag/diff part on
the network host or the "push" to compute hosts?

[Luke Gorrie]

On 6 February 2014 13:05, Justin Cormack <jus...@specialbusservice.com> wrote:

>> Or maybe Git is a reasonable alternative to rsync? Could e.g. use a
>> branch/tag to track the latest version pushed to each compute hosts,
>> and only initiate a new push when it knows there are changes (compare
>> with rsync ssh-and-diff algorithm).
>
> Unlikely to perform well.

The last time we discussed this idea I had some feedback from a friend
who did a sanity-check on a 300-node cluster.

Quoted:

Using git is an interesting idea. I set up a little simulation with a

sudo git daemon --reuseaddr --base-path=`pwd` --max-connections=0 `pwd`

running on one machine and multiple copies of

#!/bin/bash
git clone git://api1.ostk.sc1.corp.gq1.yahoo.com/config-repo $$
cd $$
while true; do git pull >> $$.out; sleep 0.3; done

running on 300 other machines.

I decided to pull every third of a second because if you want to do
live migration then you probably need to update the network config at
least that fast or you'll get hiccups.

The config-repo contains one file called "db" with 200000 lines of
"98:4B:E1:73:80:CB 10.89.32.52 blah".

The CPU on the box with the git daemon (dual quad core E5620 with
hyperthreading enabled) started to saturate at 2100 clients (7 per
client machine). Network traffic and disk IO were negligible. When I
edited the db to remove two lines in the middle and committed it, the
network traffic briefly hopped up to 320K/s, but not for more than a
second or two.

Each git pull takes about 40ms.

So, I think using git is feasible. Moving the database in this way
does imply that every node sees the same DB, so they all need to
download the whole ACL set. Also, if you want the 300ms latency
bound, a polling model is obviously expensive.

[Luke Gorrie]

On 6 February 2014 10:20, Luke Gorrie <lu...@snabb.co> wrote:
> Can we make the rsync to each compute host run independently?

I see a simple way to do this now, from the git experiment in the mail
I quoted. If each compute node "pulls" the latest config (with
rsync/git/whatever) that's probably much simpler than the network node
"pushing" to every compute node. especially when it comes to (say)
network problems that slow down one compute host's connectivity etc.

[Luke Gorrie]

next wild idea...

Could we run a read-only MySQL slave on each compute node, and let
MySQL's replication algorithm move changes around for us?

Then we don't need any code on the network node at all...?

[Rahul M R]

Great discussion! :-)

I'll try to address the questions below:

1. I think the name db-conf-sync.py is misleading - It should be changed to db-dump-sync.py (somehow it escaped my attention, sorry!). As you can see, it currently just syncs the DB dumps. When the script was initially written, the DB dump -> SnS config generation was supposed to happen at the network node (hence I chose to write the script in Python :). But as it stands now, it is probably better to rewrite it in Bash.

2. I feel the git "pull" model may be better suited for the config distribution (network-node: config generation + update master repo; compute-node: pull from master repo). In this case, maybe it is better to have an SnS "app.config-master" (to run at network-node) and "app.config-slave" (to run at compute-node)?
The pull model will avoid the need to run multiple rsync processes to push config simultaneously to all compute nodes.

3. Currently, the setup involves having rsyncd (rsync daemon) at each compute node.

4. Using Bittorrent for config distribution is an interesting idea :) (but probably suited for larger payloads?)

5. MySQL replication across compute nodes sounds interesting since it avoids the need for db-conf-sync.py :-).

6. For testing, I've db-conf-sync.py running in terminal 1, issuing neutron commands in terminal 2 and running conf-gen.lua (repeatedly) in terminal 3. I didn't see any DB table corruption during my limited testing.

7. During my limited testing, conf-gen.lua handled CRUD security-group-rules without issues.

8. Regarding VIF and plugging, my understanding is that Snabbswitch creates the Unix socket and acts as the server and QEMU will be accessing the socket as client. I haven't looked at the communication protocol for vhost_user, but can the server assume vm is up when a client connects and down when client disconnects/timeout?

9. I thought Email/ML was good for initial feedback :). will send pull requests for code review in the future.

Thanks for the feedback! :-)

Regards,
Rahul

[Rahul M R]

Here's an interesting issue:

the current NFV model uses provider networks for the virtual function names:

````
$ neutron net-create NetworkA --provider:network_type vlan --provider:physical_network "Router1_Port1_VF1" --provider:segmentation_id 111 --router:external=True

$ neutron port-create --name VirtioA --security-group <SecGroupA_uuid> --qos <QosA_uuid> NetworkA
````

Please note that the provider networks are configured via ml2_conf:

````
$ cat /etc/neutron/plugins/ml2/ml2_conf.ini
…

[ml2_type_flat]
flat_networks = Router1_Port2

[ml2_type_vlan]
network_vlan_ranges = Router1_Port1_VF1,Router1_Port1_VF2

…
````

This means a create/update/delete operation of a virtual function name requires a restart of the Neutron server.

Also the current NFV model maps each Neutron port to a virtual function. This means that to avoid restarting the Neutron server, we have to define all VF names that we'll ever use in advance:

```
[ml2_type_vlan]
network_vlan_ranges = Router1_Port1_VF1,Router1_Port1_VF2,…, Router1_Port1_VF128, \
Router1_Port3_VF1,…,Router1_Port3_VF128,...
```

How about we restrict the provider network names to match "physical" NIC ports:

````
$ cat /etc/neutron/plugins/ml2/ml2_conf.ini
…

[ml2_type_flat]
flat_networks = Router1_Port2

[ml2_type_vlan]
network_vlan_ranges = Router1_Port1,Router1_Port3

…
````

and use Neutron networks to map to virtual functions:

````
$ neutron net-create Router1_Port1_VF1 --provider:network_type vlan --provider:physical_network "Router1_Port1" --provider:segmentation_id 111 --router:external=True

$ neutron port-create --name VirtioA --security-group <SecGroupA_uuid> --qos <QosA_uuid> Router1_Port1_VF1
````

This way a create/update/delete operation of virtual function names (and associated ports) doesn't require a Neutron server restart.

Regards,
Rahul

[Rahul M R]

Re: NFV model

Another option will be to completely avoid adding VF names in Neutron:

````
$ cat /etc/neutron/plugins/ml2/ml2_conf.ini
…

[ml2_type_flat]
flat_networks = Router1_Port2

[ml2_type_vlan]

network_vlan_ranges = Router1_Port1,Router1_Port3 # Note: vlan ranges optional

…

$ neutron net-create NetworkA --provider:network_type vlan --provider:physical_network "Router1_Port1" --provider:segmentation_id 111 --router:external=True

$ neutron port-create --name VirtioA --security-group <SecGroupA_uuid> --qos <QosA_uuid> NetworkA
````

The conf-gen.lua should then generate something like this:

````
root@precise64:~/db-sync-conf-gen# ./conf-gen.lua 00:05.0 intel10g

apps.Router1_Port1 = intel10g:new{pciaddr="00:05.0", ...}

apps.NetworkA = multiqueue:new{vlan_id=111, ...}

apps.SecGroupA = SecGroup:new{...}

apps.VirtioA = VhostUser:new{socket="/var/snabb/vhost/VirtioA.socket", color=apps.NetworkA.get_color()}

app.connect(apps.Router1_Port1, "top", apps.NetworkA, "bottom")
app.connect(apps.NetworkA, "top", apps.SecGroupA, "bottom")

app.connect(apps.SecGroupA, "top", apps.VirtioA, "bottom")

````

Regards,
Rahul

[Luke Gorrie]

On 6 February 2014 22:12, Rahul M R <this...@gmail.com> wrote:
> Great discussion! :-)
>
> I'll try to address the questions below:
>
> 1. I think the name db-conf-sync.py is misleading - It should be changed to db-dump-sync.py (somehow it escaped my attention, sorry!). As you can see, it currently just syncs the DB dumps. When the script was initially written, the DB dump -> SnS config generation was supposed to happen at the network node (hence I chose to write the script in Python :). But as it stands now, it is probably better to rewrite it in Bash.

Cool. I really like this process of trying a bunch of wildly different
implementations and looking for the simplest one with the least code
and fewest moving parts. I think it will lead us to something that
works robustly in practice.

> 2. I feel the git "pull" model may be better suited for the config distribution (network-node: config generation + update master repo; compute-node: pull from master repo). In this case, maybe it is better to have an SnS "app.config-master" (to run at network-node) and "app.config-slave" (to run at compute-node)?

So, how do we divide up the responsibilities between the network node
and the compute node?

I see value in keeping the interface between compute node and network
node simple and stable. When we change that protocol then we need to
make an upgrade of the network node and all the compute nodes at the
same time (ugh). but when we don't change the protocol we can make
major upgrades to individual nodes and even run different software
versions on them if we want to (e.g. for staged upgrades).

That makes me lean towards somehow transferring the whole Neutron
database contents between nodes (by mysql/csv/rsync/git/whatever). If
we do fancy preprocessing on the network node then likely we will want
to change how that works and that will make the protocol churn.

> 5. MySQL replication across compute nodes sounds interesting since it avoids the need for db-conf-sync.py :-).

I googled this a bunch yesterday but didn't discover whether an answer
to whether or not this is a practical idea. Like, can you have
hundreds of read-only slaves? How is the performance likely to be
compared with mysqldump+rsync? I have now asked this of the center of
all knowledge in the universe (twitter).

> 7. During my limited testing, conf-gen.lua handled CRUD security-group-rules without issues.

The question on my mind is: how does the snabbswitch traffic process
load a new configuration? e.g. the administrator modifies the rules
for a security group, how does the individual app within snabbswitch
pick up this change? what's a good and simple solution?

> 8. Regarding VIF and plugging, my understanding is that Snabbswitch creates the Unix socket and acts as the server and QEMU will be accessing the socket as client. I haven't looked at the communication protocol for vhost_user, but can the server assume vm is up when a client connects and down when client disconnects/timeout?

Yep that's right. (There is now a config option so QEMU can be either
client or server.)

One idea I had was that snabbswitch could actually create a Unix
socket for every Neutron Port in the entire cluster i.e. for all
compute nodes. Then it is always "ready" for qemu to spin up any VM on
any host and connect to the socket. The app network within snabbswitch
would need to be either created dynamically (when vhost socket is
connected) or prepared for all ports in advance.

Otherwise.. perhaps we can filter the Port list to only the ones with
a "physical_network" that exists on this compute host.

> 9. I thought Email/ML was good for initial feedback :). will send pull requests for code review in the future.

Mailing list is fine too and seems to be working really well. Please
continue to do whatever you prefer :). I am just full of curiosity
about Github workflows (this will probably pass.. :)).

[Luke Gorrie]

On 7 February 2014 01:22, Rahul M R <this...@gmail.com> wrote:
> Another option will be to completely avoid adding VF names in Neutron:

[...]

> $ neutron net-create NetworkA --provider:network_type vlan --provider:physical_network "Router1_Port1" --provider:segmentation_id 111 --router:external=True
>
> $ neutron port-create --name VirtioA --security-group <SecGroupA_uuid> --qos <QosA_uuid> NetworkA

This scheme looks good to me.

Interesting that the Neutron server has to be restarted in order to
define a new physical_network in ml2_conf.ini... yet another reason to
make sure that the compute nodes can operate indefinitely without
contact to the Neutron server, doing everything except receiving new
config updates.

[Rahul M R]

On 07-Feb-2014, at 1:40 PM, Luke Gorrie <lu...@snabb.co> wrote:
>
> Cool. I really like this process of trying a bunch of wildly different
> implementations and looking for the simplest one with the least code
> and fewest moving parts. I think it will lead us to something that
> works robustly in practice.
>

Agreed. :)

>
> So, how do we divide up the responsibilities between the network node
> and the compute node?
>
> I see value in keeping the interface between compute node and network
> node simple and stable. When we change that protocol then we need to
> make an upgrade of the network node and all the compute nodes at the
> same time (ugh). but when we don't change the protocol we can make
> major upgrades to individual nodes and even run different software
> versions on them if we want to (e.g. for staged upgrades).
>
> That makes me lean towards somehow transferring the whole Neutron
> database contents between nodes (by mysql/csv/rsync/git/whatever). If
> we do fancy preprocessing on the network node then likely we will want
> to change how that works and that will make the protocol churn.
>

OK, so in that case I think the Network node can simply add the DB dump to the master git repo (and wait for clients to "pull" changes from the master repo).

> I googled this a bunch yesterday but didn't discover whether an answer
> to whether or not this is a practical idea. Like, can you have
> hundreds of read-only slaves? How is the performance likely to be
> compared with mysqldump+rsync? I have now asked this of the center of
> all knowledge in the universe (twitter).
>

Not sure about scaling to hundreds of slaves :). I hope someone can come up with actual data...

If we are going for the "pull" model and we have clients pulling (using rsync) from the network node while mysqldump is running, I think there exists the possibility that we have clients pulling in files in an inconsistent state.
So, maybe its better to stick with git for the time being ?

> The question on my mind is: how does the snabbswitch traffic process
> load a new configuration? e.g. the administrator modifies the rules
> for a security group, how does the individual app within snabbswitch
> pick up this change? what's a good and simple solution?
>

I think I misread your question earlier, sorry!

how about having the git-poller script (running on each compute node) simply kill and relaunch the traffic processes when it detects changes caused by a git pull (check output/exit status of git pull)?

>
> Yep that's right. (There is now a config option so QEMU can be either
> client or server.)
>

OK.

> One idea I had was that snabbswitch could actually create a Unix
> socket for every Neutron Port in the entire cluster i.e. for all
> compute nodes. Then it is always "ready" for qemu to spin up any VM on
> any host and connect to the socket. The app network within snabbswitch
> would need to be either created dynamically (when vhost socket is
> connected) or prepared for all ports in advance.
>
> Otherwise.. perhaps we can filter the Port list to only the ones with
> a "physical_network" that exists on this compute host.

Filtering the Port list based on available physical_network(s) sounds like the better option :)

>
> Mailing list is fine too and seems to be working really well. Please
> continue to do whatever you prefer :). I am just full of curiosity
> about Github workflows (this will probably pass.. :)).
>

OK :)

Regards,
Rahul

[Luke Gorrie]

On 8 February 2014 18:35, Rahul M R <this...@gmail.com> wrote:
> If we are going for the "pull" model and we have clients pulling (using rsync) from the network node while mysqldump is running, I think there exists the possibility that we have clients pulling in files in an inconsistent state.
> So, maybe its better to stick with git for the time being ?

Yeah, "git pull" does seem like the best option on the table at the
moment to me.

There is a risk of hitting a performance dead-end with git and/or
mysqldump but I don't think that would be catastrophic. I think the
moving parts are simple enough that we can torture-test this pretty
well ourselves and won't be too surprised by behavior in the wild. I
also think that if it does break down under load at least by then
we'll have a better understanding of what we need when shopping for a
Plan B.

> how about having the git-poller script (running on each compute node) simply kill and relaunch the traffic processes when it detects changes caused by a git pull (check output/exit status of git pull)?

Mixed feelings...

This would be a very simple and clean "crash-only software" design.
http://en.wikipedia.org/wiki/Crash-only_software

The restart would have to be so smooth and fast that nobody knows it
has happened. That's a hard engineering goal. Connected switches and
VMs should not see linkDown/linkUp events, for example.

Fast and smooth restarts are valuable for other reasons too, e.g.
upgrades and troubleshooting. So I think we should optimize this in
any case.

How would Apps feel about restarts? On the one hand it's a luxury not
to have to worry about configuration changes. On the other it could be
hard to implement control protocols (e.g. VRRP) when you a being
continuously restarted. (On the third hard, maybe that's life and Apps
need to have graceful restart behavior in any case.)


Food for thought.

Supposing we did _not_ do this and instead did a live config change on
the traffic process. Does anybody have a good idea for how we should
do that?

>> Otherwise.. perhaps we can filter the Port list to only the ones with
>> a "physical_network" that exists on this compute host.
>
> Filtering the Port list based on available physical_network(s) sounds like the better option :)

So how will the software look on the compute node?

Will we still have a process that takes table-*.csv and creates
app-config.lua? if so, does this process filter away irrelevant
information belonging to other compute nodes?

[Rahul M R]

On 10-Feb-2014, at 3:10 PM, Luke Gorrie <lu...@snabb.co> wrote:

> Food for thought.
>
> Supposing we did _not_ do this and instead did a live config change on
> the traffic process. Does anybody have a good idea for how we should
> do that?

here's a crazy idea :) ...

How about the main event loop of core.app periodically check a global flag (let's say CONFIG_UPDATE_FLAG) and if a config update is required, it will read the app network configuration data (per-process; can be generated by git-poller) and create/update/delete the apps (except the driver app - which is fixed for a traffic process) that it is currently managing? The traffic process will have a signal handler (for say, SIGUSR1) which will simply set the CONFIG_UPDATE_FLAG. The git-poller script will check the output of git-pull and generate per-process app network config data (maybe serialized lua tables for fast reading?), diff the generated config and send SIGUSR1 signal to corresponding process( /var/snabb/process/<pci-address>/<process-id>.pid ?).

Hope others come up with far better ideas :) …

[Luke Gorrie]

On 10 February 2014 20:57, Rahul M R <this...@gmail.com> wrote:
> How about the main event loop of core.app periodically check a global flag (let's say CONFIG_UPDATE_FLAG) and if a config update is required, it will read the app network configuration data (per-process; can be generated by git-poller) and create/update/delete the apps (except the driver app - which is fixed for a traffic process) that it is currently managing?

This sounds like a reasonable idea.

I have a rough idea for an algorithm to update the app config. (caution: this might not be a good idea):

Suppose we have a data structure called an "app network" that is a set of apps (nodes) and links (edges). For example, it could look like:

{ nodes = { nic={"Intel82599", "{pciaddr='0000:00:01.0', ...}}, ...},

  links = { "nic.tx -> vm.rx", "vm.tx -> nic.rx", ...} }

Here each node is defined by two strings: app name and app config.

Then we have a function app.setup(myconfig) that creates this app network. Then first time you call app.setup() it creates the apps from scratch, but on later calls it updates the existing app network by creating/deleting/reconfiguring apps and links on demand.

The update algorithm would look something like:

1. Delete all apps and links that are not in the new configuration.

2. Create all apps and links that were not in the old configuration.

3. Update all apps that have a new configuration i.e. config string is not the same as it was before:

3a. If the app has a reconfigure() method then call that with the new config.

3b. If the app does not have a reconfigure() method then delete it and recreate it.

End of idea.

[Rahul M R]

Hi all,

Just a small update: Got some nova + neutron + neutron-snabbswitch-agent tests working today.

Currently, creating a neutron port and launching a nova instance (with the created port) works (Note: limited testing only - have an instance running for the past hour :).

There is still some issue with QEMU giving the following warning:

2014-02-18 01:01:43.723+0000: 10777: warning : qemuOpenVhostNet:521 : Unable to open vhost-net. Opened so far 0, requested 1

Also noticed that the Tap device is not being created (by default, the /etc/qemu-ifup should be called, but looks like its not being called in this case). The test "vapp" server (listening on <port-id>.socket) is not showing any traffic yet. Tried manually creating the tap device and pinging the interface, but the vapp server doesn't show any traffic. Needs further investigation...

[Please note that I'm using the first patchset from VOS that used the TAP interface. libvirt support for the newer patchsets can be added only after the QEMU CLI for vhost-user and mem-path is finalized.]

OK, here's some relevant output from the test:

````
### Terminal 1 ###

$ nova flavor-key m1.nano set quota:mem_hugepages=True

$ neutron security-group-create SecGroupC --description "security group for VirtioC"

$ neutron net-create NetworkC --provider:network_type flat --provider:physical_network Router1_Port2 --router:external=True

$ neutron subnet-create --name SubnetC --ip-version 6 --no-gateway --disable-dhcp NetworkC a::0/64

$ neutron port-create --name VirtioC --fixed-ip subnet_id=YYYYYYYY,ip_address=a::1 --security-group SecGroupC NetworkC

$ nova boot --image debian_squeeze_amd64_standard --flavor m1.nano --nic port-id=XXXXXXXX ServerC


### Terminal 2 ###

$ sudo /tmp/vapp -s /var/snabbswitch/vhost/XXXXXXXX.socket # run after port-create above

### Terminal 1 ###

$ sudo chmod 777 /var/snabbswitch/vhost/XXXXXXXX.socket # Not sure if this is necessary...

$ nova boot --image debian_squeeze_amd64_standard --flavor m1.nano --nic port-id=XXXXXXXX ServerC

$ nova list
+--------------------------------------+---------+--------+------------+-------------+---------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+---------+--------+------------+-------------+---------------+
| 853b6d6b-d5c1-49e5-989b-8af55feb2e1d | ServerC | ACTIVE | - | Running | NetworkC=a::1 |
+--------------------------------------+---------+--------+------------+-------------+---------------+

$ sudo virsh list
Id Name State
----------------------------------------------------
6 instance-00000007 running


$ neutron port-show VirtioC
+-----------------------+-----------------------------------------------------------------------------+
| Field | Value |
+-----------------------+-----------------------------------------------------------------------------+
| admin_state_up | True |
| allowed_address_pairs | |
| binding:capabilities | {"port_filter": true} |
| binding:host_id | precise64 |
| binding:vif_type | snabbswitch |
| device_id | 853b6d6b-d5c1-49e5-989b-8af55feb2e1d |
| device_owner | compute:None |
| extra_dhcp_opts | |
| fixed_ips | {"subnet_id": "6db7f9ac-04a5-444f-85cb-d9a0a91fc0ea", "ip_address": "a::1"} |
| id | 9a96c1a9-9893-4d5c-a9c9-f3d4eee030a4 |
| mac_address | fa:16:3e:5d:e0:91 |
| name | VirtioC |
| network_id | d3afbd4c-56c3-48a5-a865-d103abda8b09 |
| security_groups | a16dff71-122e-48df-9d8e-513e37a431b6 |
| status | DOWN |
| tenant_id | c2d9440210cd4134ad094fee3dc975f8 |
+-----------------------+-----------------------------------------------------------------------------+
````

### LOGS ###

From /opt/stack/logs/screen/screen-n-cpu.log:

````
[...]
2014-02-18 01:01:43.456 DEBUG nova.virt.libvirt.config [req-f8b4eebc-5695-49fc-ad8e-4c9bf59ef7a1 admin demo] Generated XML
<domain type="kvm">
<uuid>853b6d6b-d5c1-49e5-989b-8af55feb2e1d</uuid>
<name>instance-00000007</name>
<memory>65536</memory>
<vcpu>1</vcpu>
<sysinfo type="smbios">
<system>
<entry name="manufacturer">OpenStack Foundation</entry>
<entry name="product">OpenStack Nova</entry>
<entry name="version">2013.2.1</entry>
<entry name="serial">f682103d-19a7-4ef6-9ea7-80ae61910197</entry>
<entry name="uuid">853b6d6b-d5c1-49e5-989b-8af55feb2e1d</entry>
</system>
</sysinfo>
<os>
<type>hvm</type>
<boot dev="hd"/>
<smbios mode="sysinfo"/>
</os>
<features>
<acpi/>
<apic/>
</features>
<memoryBacking>
<hugepages/>
</memoryBacking>
<clock offset="utc">
<timer name="pit" tickpolicy="delay"/>
<timer name="rtc" tickpolicy="catchup"/>
</clock>
<devices>
<disk type="file" device="disk">
<driver name="qemu" type="qcow2" cache="none"/>
<source file="/opt/stack/data/nova/instances/853b6d6b-d5c1-49e5-989b-8af55feb2e1d/disk"/>
<target bus="virtio" dev="vda"/>
</disk>
<interface type="ethernet">
<mac address="fa:16:3e:5d:e0:91"/>
<model type="virtio"/>
<script path=""/>
<vhostsock path="/var/snabbswitch/vhost/9a96c1a9-9893-4d5c-a9c9-f3d4eee030a4.socket"/>
<target dev="tap9a96c1a9-98"/>
</interface>
<serial type="file">
<source path="/opt/stack/data/nova/instances/853b6d6b-d5c1-49e5-989b-8af55feb2e1d/console.log"/>
</serial>
<serial type="pty"/>
<graphics type="vnc" autoport="yes" keymap="en-us" listen="127.0.0.1"/>
</devices>
</domain>
[...]
````

From /opt/stack/logs/screen/screen-q-agt.log:

````
2014-02-18 00:57:58.858 12018 INFO neutron.plugins.snabbswitch.agent.snabbswitch_neutron_agent [-] Update ports: added=['9a96c1a9-9893-4d5c-a9c9-f3d4eee030a4'], removed=[]
````

From /opt/stack/logs/screen/screen-q-svc.log:

````
2014-02-18 00:57:04.087 12077 DEBUG neutron.db.db_base_plugin_v2 [-] Allocated IP a::1 (d3afbd4c-56c3-48a5-a865-d103abda8b09/6db7f9ac-04a5-444f-85cb-d9a0a91fc0ea/9a96c1a9-9893-4d5c-a9c9-f3d4eee030a4) create_port /opt/stack/neutron/neutron/db/db_base_plugin_v2.py:1360
[...]
2014-02-18 00:57:58.862 12077 DEBUG neutron.plugins.ml2.rpc [-] Device 9a96c1a9-9893-4d5c-a9c9-f3d4eee030a4 up at agent snabbswitch-q-agent.precise64 update_device_up /opt/stack/neutron/neutron/plugins/ml2/rpc.py:182
2014-02-18 00:57:58.868 12077 DEBUG neutron.plugins.ml2.rpc [-] Device 9a96c1a9-9893-4d5c-a9c9-f3d4eee030a4 not bound to the agent host precise64 update_device_up /opt/stack/neutron/neutron/plugins/ml2/rpc.py:188
[...]
2014-02-18 01:01:43.083 12077 DEBUG neutron.plugins.ml2.managers [-] Attempting to bind port 9a96c1a9-9893-4d5c-a9c9-f3d4eee030a4 on host precise64 bind_port /opt/stack/neutron/neutron/plugins/ml2/managers.py:440
2014-02-18 01:01:43.083 12077 DEBUG neutron.plugins.ml2.drivers.mech_agent [-] Attempting to bind port 9a96c1a9-9893-4d5c-a9c9-f3d4eee030a4 on network d3afbd4c-56c3-48a5-a865-d103abda8b09 bind_port /opt/stack/neutron/neutron/plugins/ml2/drivers/mech_agent.py:57
2014-02-18 01:01:43.086 12077 DEBUG neutron.plugins.ml2.drivers.mech_agent [-] Checking agent: {'binary': u'neutron-snabbswitch-agent', 'description': None, 'admin_state_up': True, 'heartbeat_timestamp': datetime.datetime(2014, 2, 18, 1, 1, 41), 'alive': True, 'topic': u'N/A', 'host': u'precise64', 'agent_type': u'Snabbswitch agent', 'created_at': datetime.datetime(2014, 2, 17, 23, 34, 55), 'started_at': datetime.datetime(2014, 2, 17, 23, 34, 55), 'id': u'8911b0a4-f47f-4c82-a300-465bd3333e5f', 'configurations': {u'devices': 1}} bind_port /opt/stack/neutron/neutron/plugins/ml2/drivers/mech_agent.py:59
2014-02-18 01:01:43.086 12077 DEBUG neutron.plugins.ml2.drivers.mech_snabbswitch [-] Checking segment: {'segmentation_id': None, 'physical_network': u'Router1_Port2', 'id': u'73c775e8-43f7-4d6e-af84-c5c13a6f9c93', 'network_type': u'flat'} check_segment_for_agent /opt/stack/neutron/neutron/plugins/ml2/drivers/mech_snabbswitch.py:43
2014-02-18 01:01:43.087 12077 DEBUG neutron.plugins.ml2.drivers.mech_agent [-] Bound using segment: {'segmentation_id': None, 'physical_network': u'Router1_Port2', 'id': u'73c775e8-43f7-4d6e-af84-c5c13a6f9c93', 'network_type': u'flat'} bind_port /opt/stack/neutron/neutron/plugins/ml2/drivers/mech_agent.py:66
2014-02-18 01:01:43.087 12077 DEBUG neutron.plugins.ml2.managers [-] Bound port: 9a96c1a9-9893-4d5c-a9c9-f3d4eee030a4, host: precise64, driver: snabbswitch, vif_type: snabbswitch, cap_port_filter: True, segment: 73c775e8-43f7-4d6e-af84-c5c13a6f9c93 bind_port /opt/stack/neutron/neutron/plugins/ml2/managers.py:455
[...]
````

Regards,
Rahul

[Luke Gorrie]

On 18 February 2014 03:32, Rahul M R <this...@gmail.com> wrote:

Just a small update: Got some nova + neutron + neutron-snabbswitch-agent tests working today.

Awesome!!

 

There is still some issue with QEMU giving the following warning:

2014-02-18 01:01:43.723+0000: 10777: warning : qemuOpenVhostNet:521 : Unable to open vhost-net. Opened so far 0, requested 1

This is probably only worth investigating with the latest version of the qemu vhost support (v8). The implementation has evolved quite a bit. 

The qemu vhost command-line arguments have changed recently too with chardev stuff.

I am doing some testing now with the latest version of vhost_user + vapp + snabbswitch.

### Terminal 1 ###

$ nova flavor-key m1.nano set quota:mem_hugepages=True

$ neutron security-group-create SecGroupC --description "security group for VirtioC"

$ neutron net-create NetworkC --provider:network_type flat --provider:physical_network Router1_Port2 --router:external=True

$ neutron subnet-create --name SubnetC --ip-version 6 --no-gateway --disable-dhcp NetworkC a::0/64

$ neutron port-create --name VirtioC --fixed-ip subnet_id=YYYYYYYY,ip_address=a::1 --security-group SecGroupC NetworkC

$ nova boot --image debian_squeeze_amd64_standard --flavor m1.nano --nic port-id=XXXXXXXX ServerC

Cool that you are working out all the arcane arguments that we really need to use in practice :-)

[Luke Gorrie]

On 12 February 2014 10:32, Luke Gorrie <lu...@snabb.co> wrote:

I have a rough idea for an algorithm to update the app config. (caution: this might not be a good idea):

Suppose we have a data structure called an "app network" that is a set of apps (nodes) and links (edges).

I couldn't resist and started hacking on this. First sketch here: https://github.com/SnabbCo/snabbswitch/pull/89

[Rahul M R]

I was planning to try the proposed algorithm before the end of the week (as I was focused on getting all the Nova + Neutron stuff working). Sorry I couldn't get the time to try it earlier…

The first sketch looks good btw :-)

Regards,
Rahul

[Rahul M R]

Hi Luke,

On 18-Feb-2014, at 12:54 PM, Luke Gorrie <lu...@snabb.co> wrote:

> Awesome!!

Thanks!

> This is probably only worth investigating with the latest version of the qemu vhost support (v8). The implementation has evolved quite a bit.
>
> The qemu vhost command-line arguments have changed recently too with chardev stuff.
>
> I am doing some testing now with the latest version of vhost_user + vapp + snabbswitch.
>

OK. So I'll target libvirt v1.2.1 (current stable) and vhost_user v8.

>
> Cool that you are working out all the arcane arguments that we really need to use in practice :-)
>

Its worth noting that the IPv6 address support only exists in neutron-client (L2 agent still doesn't support IPv6).

Regards,
Rahul

[Rahul M R]

Hi all,

just a small update:

1. new libvirt (v1.2.1) patch to support the CLI (-mem-path, -chardev, -netdev) for QEMU (vhost-user-v8): https://bitbucket.org/thisisrmr/libvirt-1.2.1

2. Nova updated to use the libvirt patch above.

I've been using the old "vapp" server from VOS for testing. The server quits immediately after receiving communication from the client:

````
vagrant@precise64:~/vapp$ ./vapp -s /var/snabbswitch/vhost/8c7cbdc1-9c20-4134-bd84-0ae87a7acef0.socket
Cmd: VHOST_USER_SET_OWNER (0x3)
Flags: 0x1
................................................................................
Processing message: VHOST_USER_SET_OWNER
_set_owner
Cmd: VHOST_USER_GET_FEATURES (0x1)
Flags: 0x1
u64: 0x0
................................................................................
Processing message: VHOST_USER_GET_FEATURES
_get_features
Cmd: VHOST_USER_SET_VRING_CALL (0xd)
Flags: 0x1
file: 8 0x0
................................................................................
Processing message: VHOST_USER_SET_VRING_CALL
_set_vring_call
vapp: vapp_server.c:312: _set_vring_call: Assertion `idx<2' failed.
Aborted
````

BTW this means we can boot an instance using Nova and the instance is successfully communicating with the vhost_user "server" :-)

Is there an updated server app that I can use for testing?

Regards,
Rahul

[Luke Gorrie]

Great to have VMs booted successfully via Nova!

On 26 February 2014 07:29, Rahul M R <this...@gmail.com> wrote:

I've been using the old "vapp" server from VOS for testing.  The server quits immediately after receiving communication from the client:

Hm, the latest vapp and qemu from VOS are working together for me on chur.

but! now you can actually test with snabbswitch master :-) try this:

1. Edit vhost_user.selftest() to pick a new pci address and socket filename (ok should be configurable).

2. Run snabbswitch -t vhost_user

3. Run qemu with args like: -chardev socket,id=chr0,path=$HOME/qemu.sock,server -netdev type=vhost-user,id=net0,chardev=chr0

... and then qemu should create the socket, snabbswitch attaches, and all packets sent by the VM should be immediately received again - the VM's NIC will be in loopback mode via connection to an Intel NIC in loopback-mode.

Let me know if it works for you too! or if you want help to troubleshoot the issue with vapp.

Here is the exact qemu command line that I'm using btw:

sudo ~/hacking/qemu/x86_64-softmmu/qemu-system-x86_64 --enable-kvm -nographic -m 1024 -mem-path /hugetlbfs,prealloc=on,share=on -chardev socket,id=chr0,path=/home/luke/qemu.sock,server -netdev type=vhost-user,id=net0,chardev=chr0 -device virtio-net-pci,netdev=net0 -drive file=deb.qcow2,if=virtio

[Rahul M R]

On 26-Feb-2014, at 1:40 PM, Luke Gorrie <lu...@snabb.co> wrote:

>
> Hm, the latest vapp and qemu from VOS are working together for me on chur.
>

I'll try the latest vapp then.

> but! now you can actually test with snabbswitch master :-) try this:
>

Cool! I'll try this once the libvirt stuff gets sorted out. Currently, I'm looking at a bug where QEMU assigns the same PCI address for the video adapter and the netdev device preventing the instance from booting.

Regards,
Rahul

[Luke Gorrie]

On 27 February 2014 13:12, Rahul M R <this...@gmail.com> wrote:

> Hm, the latest vapp and qemu from VOS are working together for me on chur.

I'll try the latest vapp then.

Important note: --enable-kvm seems to be needed on the qemu command line with the latest version. otherwise the vapp doesn't receive traffic. (root cause not known yet.)

Can you do --enable-kvm inside a Vagrant VM though? :-\

[Rahul M R]

On 27-Feb-2014, at 6:43 PM, Luke Gorrie <lu...@snabb.co> wrote:

> Important note: --enable-kvm seems to be needed on the qemu command line with the latest version. otherwise the vapp doesn't receive traffic. (root cause not known yet.)
>

OK.

> Can you do --enable-kvm inside a Vagrant VM though? :-\

I've been using a combination of vagrant + libvirt provider (with kvm nested virtualization enabled on chur). So far,
kvm seems to be working fine inside the vm (just to be sure, i did a quick: `qemu-system-x86_64 -enable-kvm -cpu host` and it seems to be working :)).

Regards,
Rahul

[Luke Gorrie]

On 27 February 2014 15:13, Rahul M R <this...@gmail.com> wrote:

I've been using a combination of vagrant + libvirt provider (with kvm nested virtualization enabled on chur).

Cool that this is possible :-)

[Rahul M R]

update:

* patched libvirt vhost_user support code to fix the PCI address issue mentioned earlier
* started testing using the latest vapp server
* looks like stuff is working (see below) :-)

sample vapp output:

````
ff ff ff ff ff ff fa 16 3e d9 6e 13 08 00 45 10
01 48 00 00 00 00 80 11 39 96 00 00 00 00 ff ff
ff ff 00 44 00 43 01 34 9e 36 01 01 06 00 66 1d
77 56 00 38 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 fa 16 3e d9 6e 13 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 63 82 53 63 35 01 01 37 0d 01
1c 02 03 0f 06 77 0c 2c 2f 1a 79 2a ff 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00
````

(Note: the mac address above matches the corresponding neutron port's mac address.)

Regards,
Rahul

[Luke Gorrie]

On 27 February 2014 19:02, Rahul M R <this...@gmail.com> wrote:

update:

* patched libvirt vhost_user support code to fix the PCI address issue mentioned earlier
* started testing using the latest vapp server
* looks like stuff is working (see below) :-)

Nice !!!

There are always so many small things to fix during integration... wonderful that you are already down this path so far :)

[Rahul M R]

On 27-Feb-2014, at 11:34 PM, Luke Gorrie <lu...@snabb.co> wrote:

> Nice !!!
>
> There are always so many small things to fix during integration... wonderful that you are already down this path so far :)

Thanks! :)

I was thinking of trying a multi-node Openstack setup (controller node + compute node(s)).

there is an issue related to PCI mapping: currently, there is no way to specify that a provider network should be mapped to a particular compute node (this will change in the future when PCI flavour based device assignment becomes available in Nova).

For the time being, the current PCI mapping scheme can be altered to include the provider network <-> node relationship (many-to-many?). For e.g.:
````
[snabbswitch]
pci_mappings = Router...@node1.example.com=0000:00:01.0, Router...@node2.example.com=0000:00:01.0
````

The Snabbswitch mechanism driver will generate the "pci_mappings.txt" during initialization:
````
node1.example.com Router1_Port1 0000:00:01.0
node2.example.com Router1_Port2 0000:00:01.0
````

(Note: the pci_mappings.txt will be in the same directory as the DB dumps).

The compute_node_script (git poller) will only pick the lines in pci_mappings.txt that matches the machine's hostname (such as node1.example.com) to generate the app-network config for the Snabbswitch traffic processes.

Thoughts?

Regards,
Rahul

[Luke Gorrie]

On 28 February 2014 17:23, Rahul M R <this...@gmail.com> wrote:

there is an issue related to PCI mapping: currently, there is no way to specify that a provider network should be mapped to a particular compute node (this will change in the future when PCI flavour based device assignment becomes available in Nova).

For the time being, the current PCI mapping scheme can be altered to include the provider network <-> node relationship (many-to-many?). For e.g.:
````
[snabbswitch]
pci_mappings = Router...@node1.example.com=0000:00:01.0, Router...@node2.example.com=0000:00:01.0
````

The Snabbswitch mechanism driver will generate the "pci_mappings.txt" during initialization:
````
node1.example.com         Router1_Port1        0000:00:01.0
node2.example.com         Router1_Port2        0000:00:01.0
````

(Note: the pci_mappings.txt will be in the same directory as the DB dumps).

The compute_node_script (git poller) will only pick the lines in pci_mappings.txt that matches the machine's hostname (such as node1.example.com) to generate the app-network config for the Snabbswitch traffic processes.

Thoughts?

The mechanism sounds nice and simple: the compute node cherry-picks the relevant config items based on a well-defined global name (hostname of the compute node).

I didn't understand what high-level problem this solves though. Can you explain a bit more about the problem with provider networks not being mapped onto compute nodes, and how that's a problem for us, and what the solution is?

I have also another use case I'd like to discuss - important for TeraStream:

Can we somehow assign a port to a provider network based on available _bandwidth_? That is: instead of specifying the name of the provider network that the port should use, we would specify how much bandwidth is needed (e.g. 6 Gbps) in the QoS settings, and software would automatically pick the name of a provider network on a NIC that is not too "crowded" already.

This would be done at the time the port is created and available bandwidth would be calculated based on the configured QoS requirements in the Neutron database (not on e.g. measuring actual load on the server).

Have you got any ideas for how we might support that in a good way?

[Rahul M R]

On 03-Mar-2014, at 2:48 PM, Luke Gorrie <lu...@snabb.co> wrote:
>
> I didn't understand what high-level problem this solves though. Can you explain a bit more about the problem with provider networks not being mapped onto compute nodes, and how that's a problem for us, and what the solution is?
>

You've mentioned about provider network aware scheduling in a previous email:

````
Just want to mention some Nova development that can be interesting/useful for us.

Ian Wells, Irena Berezovsky, and others are extending the Nova scheduler to be aware of provider networks when deciding which compute host to place a VM on [1]. So it should be possible to assign a Port to a provider physical_network (called e.g. "DataCenter7VoiceNetwork42") and for Nova to figure out which compute host(s) are able to connect to this network.

This is useful for networks like TeraStream. The network designers should be able to use meaningful (to them) names to talk about which physical network in the data center a VM should be connected to. So they could configure that DataCenter7VoiceNetwork42 maps to (say) a particular PCI address on a particular server.

This work seems to be already going in the right direction for our needs, many thanks to Ian & all.

[1] http://lists.openstack.org/pipermail/openstack-dev/2014-January/024927.html
````

The proposed mechanism is an attempt to get this behaviour in Havana using the default Nova scheduler :)

The example I gave in the previous email shows Router1_Port1 connected to node1's NIC and Router1_Port2 connected to node2's NIC. i.e., only node1 can access Router1_Port1 and only node2 can access Router1_Port2.

Currently there is no provision to support such non-uniform connectivity to physical_networks. i.e., there is no way for a node to know what physical networks it has access to; each compute node makes the assumption that it is connected to all physical_networks given in the pci_mappings and Nova will choose any node for booting a vm.

OTOH if a node knew what physical_networks it has access to, it'll only create the ports that's connected to those physical networks. Once the port is created, the agent running in the node reports to the Neutron server that the port is up. IIUC Neutron stores the reporting node to the list of devices where the port is available. When Nova is requested to boot a vm using the particular port-id, Nova checks with Neutron about the given port and Neutron's reply contains the list of devices with the given port. I *assume* (haven't tried a multi-node setup yet, so cannot guarantee) that the Nova scheduler takes into consideration this list of available devices when it tries to boot the vm.

So in the example scenario, if we have a port (say, VirtioA) created using physical_network Router1_Port1, the Nova scheduler will attempt to boot the vm on node1 and not node2 (hopefully!).

If the above mentioned heterogeneous setup is not required (or not a priority), then it's not worth implementing.

> I have also another use case I'd like to discuss - important for TeraStream:
>
> Can we somehow assign a port to a provider network based on available _bandwidth_? That is: instead of specifying the name of the provider network that the port should use, we would specify how much bandwidth is needed (e.g. 6 Gbps) in the QoS settings, and software would automatically pick the name of a provider network on a NIC that is not too "crowded" already.
>
> This would be done at the time the port is created and available bandwidth would be calculated based on the configured QoS requirements in the Neutron database (not on e.g. measuring actual load on the server).
>
> Have you got any ideas for how we might support that in a good way?

IIUC the QoS limits are assigned only to ports and not network(s)/physical_network(s):
````

neutron port-create --name VirtioA --security-group <SecGroupA_uuid> --qos <QosA_uuid> NetworkA

````
The proposed port creation scenario is not supported in Neutron and will be difficult to get this functionality in Neutron (QoS API proposal for Physical networks, port-create API extension, Neutron extension supporting the described scenario etc.).

However, if the TeraStream project's provisioning script can make use of an external shell script (containing some helper functions) then maybe it isn't too hard...

Suppose we add support for specifying bandwidth limits for physical_networks in ML2/Snabbswitch config:
````
physical_network_qos = Router1_Port1=10Gbps,Router1_Port2=10Gbps
````
and a helper function is available in the external script:
````
function find_a_physical_network(target_bandwidth)
````
the function can get the max bandwidth for all physical_networks (from ML2 config), list of available ports for each physical_network (from neutron {port,net}-* commands) and qos limits for each port (from neutron qos commands); the function can calculate and then return the first available physical_network meeting the target bandwidth.

neutron net-create command can take the output of this function when specifying provider:network.

Regards,
Rahul

[Luke Gorrie]

On 3 March 2014 18:01, Rahul M R <this...@gmail.com> wrote:

So in the example scenario, if we have a port (say, VirtioA) created using physical_network Router1_Port1, the Nova scheduler will attempt to boot the vm on node1 and not node2 (hopefully!).

Thanks for the explanation!

Yes, this is important to support when running on Havana.

TeraStream's current design will have each provider network existing on only one port of one server. That provider network will have a subnet statically configured in the router. The VM will be assigned an address in the right subnet by a mechanism outside of OpenStack. Nova will need to place the VM on the machine where its provider network is available.

However, if the TeraStream project's provisioning script can make use of an external shell script (containing some helper functions) then maybe it isn't too hard...

[...]

 

neutron net-create command can take the output of this function when specifying provider:network.

Great. This sounds nice and simple to me and straightforward to integrate into the script that provisions new VMs for TeraStream. I'll confirm that.