robustness visualization for netjsongraph

[Leonardo Maccari]

Hi all,

I have been working on two things that I hope can be useful for those
that run mesh networks.
The first is a python module that converts a NetJSON into a block-cut tree
(see https://en.wikipedia.org/wiki/Biconnected_component). Basically, it
transforms a generic
graph into a graph that evidences the nodes that are cutpoints, that is,
if they fail, they partition
the network. The goal of this visualization is to give at a glance, the
importance of nodes in the
network. Source code is here
https://github.com/netCommonsEU/netjson-robustness-analyser

The second thing is just a small set of modifications to netjsongraph.js
to show the new network.
Source code is here https://github.com/netCommonsEU/netjsongraph.js in
the robustness_graph branch.

One example is here (sorry for the bad certs, I have to fix it) :
- original ninux network: https://opendata.netcommons.eu/examples/
- bloc-cut tree:
https://opendata.netcommons.eu/examples/condensed-ninux.html

in the second graph blue balls are blocks (cliques of nodes that resist
to one node failure) and
the other ones are cutpoints (if they break, network is partitioned),
size and color changes with
the size of the fracture they produce.

My goal would be to add this (and possibly more) visualization to the
openwisp topology viewer,
embedding the python into the topology module. This is all work in
progress, so any feedback is
largely welcome.

l.

--
Leonardo Maccari, Assistant Professor @DISI, University of Trento
Tel: +39 0461 285323, www.disi.unitn.it/~maccari, gpg ID: AABE2BD7

[Federico Capoano]

Hi Leonardo,

On Thursday, October 12, 2017 at 11:20:28 AM UTC+2, leonardo.maccari wrote:

Hi all,

I have been working on two things that I hope can be useful for those
that run mesh networks.
The first is a python module that converts a NetJSON into a block-cut tree
(see https://en.wikipedia.org/wiki/Biconnected_component). Basically, it
transforms a generic
graph into a graph that evidences the nodes that are cutpoints, that is,
if they fail, they partition
the network. The goal of this visualization is to give at a glance, the
importance of nodes in the
network. Source code is here
https://github.com/netCommonsEU/netjson-robustness-analyser

 

Great thanks!

The second thing is just a small set of modifications to netjsongraph.js
to show the new network.
Source code is here https://github.com/netCommonsEU/netjsongraph.js in
the robustness_graph branch.

I checked the commits, have you only changed the examples or also the source code of the library?

 

One example is here (sorry for the bad certs, I have to fix it) :
- original ninux network: https://opendata.netcommons.eu/examples/
- bloc-cut tree:
https://opendata.netcommons.eu/examples/condensed-ninux.html

in the second graph blue balls are blocks (cliques of nodes that resist
to one node failure) and
the other ones are cutpoints (if they break, network is partitioned),
size and color changes with
the size of the fracture they produce.

 

At first sight it seems that the bigger circles are more important, but after reading the description I understand that those circles are just group of nodes, not particularly critical in the network.

Would make the orange nodes bigger instead of the blue block nodes make sense?

My goal would be to add this (and possibly more) visualization to the
openwisp topology viewer,
embedding the python into the topology module. This is all work in
progress, so any feedback is
largely welcome.

Great news!

Could you add a few more lines in the README of netjson-robustness-analyser to explain how to use it?

Federico

[Leonardo Maccari]

Hi Federico

https://github.com/netCommonsEU/netjson-robustness-analyser

 

Great thanks!

glad you like it.

The second thing is just a small set of modifications to netjsongraph.js
to show the new network.
Source code is here https://github.com/netCommonsEU/netjsongraph.js in
the robustness_graph branch.

I checked the commits, have you only changed the examples or also the source code of the library?

Just the examples. The modifications I made are only to the CSS and to the way the d3 function is called from the HTML.

 

One example is here (sorry for the bad certs, I have to fix it) :
- original ninux network: https://opendata.netcommons.eu/examples/
- bloc-cut tree:
https://opendata.netcommons.eu/examples/condensed-ninux.html

in the second graph blue balls are blocks (cliques of nodes that resist
to one node failure) and
the other ones are cutpoints (if they break, network is partitioned),
size and color changes with
the size of the fracture they produce.

 

At first sight it seems that the bigger circles are more important, but after reading the description I understand that those circles are just group of nodes, not particularly critical in the network.

Would make the orange nodes bigger instead of the blue block nodes make sense?

Blue circles are important because they tell you visually how big is the part of the network that remains disconnected if you remove one cutpoint, without clicking on the cutpoint (which shows the number of potentially disconnected nodes). So they are not nodes, but their size is important to understand what is happening. maybe we can change the color, or the alpha, or we may change the shape of the cutpoint (from circles to diamonds).
I suppose for the latter thing we need to change the library though.

Note that when I use json.dumps() whatever property of the node ends in the "properties" section of the node, which means it is displayed. I hide them with CSS, but maybe there is a better way of putting them somewhere else in the node data.

embedding the python into the topology module. This is all work in
progress, so any feedback is
largely welcome.

Great news!

Could you add a few more lines in the README of netjson-robustness-analyser to explain how to use it?

done. do you think it is a good idea to add it the topology module, so when one visualizes the topology he can actually choose between several views? I can start digging into it if you think it makes sense.

l.

Federico

--
You received this message because you are subscribed to the Google Groups "OpenWISP" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openwisp+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Federico Capoano]

On Thu, Oct 19, 2017 at 9:11 AM Leonardo Maccari <leonardo...@unitn.it> wrote:

Hi Federico

https://github.com/netCommonsEU/netjson-robustness-analyser

 

Great thanks!

glad you like it.

The second thing is just a small set of modifications to netjsongraph.js
to show the new network.
Source code is here https://github.com/netCommonsEU/netjsongraph.js in
the robustness_graph branch.

I checked the commits, have you only changed the examples or also the source code of the library?

Just the examples. The modifications I made are only to the CSS and to the way the d3 function is called from the HTML.

 

Nice, I guess we can include the most significant examples in the main netjsongraph.js repository, I'd suggest to choose one or two, what do you think?
Have time to send a PR?

 

One example is here (sorry for the bad certs, I have to fix it) :
- original ninux network: https://opendata.netcommons.eu/examples/
- bloc-cut tree:
https://opendata.netcommons.eu/examples/condensed-ninux.html

in the second graph blue balls are blocks (cliques of nodes that resist
to one node failure) and
the other ones are cutpoints (if they break, network is partitioned),
size and color changes with
the size of the fracture they produce.

 

At first sight it seems that the bigger circles are more important, but after reading the description I understand that those circles are just group of nodes, not particularly critical in the network.

Would make the orange nodes bigger instead of the blue block nodes make sense?

Blue circles are important because they tell you visually how big is the part of the network that remains disconnected if you remove one cutpoint, without clicking on the cutpoint (which shows the number of potentially disconnected nodes). So they are not nodes, but their size is important to understand what is happening. maybe we can change the color, or the alpha, or we may change the shape of the cutpoint (from circles to diamonds).
I suppose for the latter thing we need to change the library though.

 

Ok we can play with the alpha.

Note that when I use json.dumps() whatever property of the node ends in the "properties" section of the node, which means it is displayed. I hide them with CSS, but maybe there is a better way of putting them somewhere else in the node data. 

 

We should look into add a way to do what you are doing now without using properties but other custom attributes that are in the node.

embedding the python into the topology module. This is all work in
progress, so any feedback is
largely welcome.

Great news!

Could you add a few more lines in the README of netjson-robustness-analyser to explain how to use it?

done. do you think it is a good idea to add it the topology module, so when one visualizes the topology he can actually choose between several views? I can start digging into it if you think it makes sense. 

Yes, I think it would be a nice feature, we could start thinking about a generic way to create different elaborations and visualisations.

To accomplish this we should think about either using functions or classes that take something in input (NetJSON NetworkGraph?), perform some work on it and return NetJSON NetworkGraph output.

Once we have this defined, we can create a django setting that lists the available elaborations/visualisations (let's find a good name for this) and we can have a menu item in the visualiser that when clicked shows the available elaborations/visualisations, if the user clicks on one of this is taken to that view.

Is your python module published in pypi yet? I don't see any setup.py file so I guess it's not published yet.

 

Federico

[Leonardo Maccari]

On 19/10/17 10:33, Federico Capoano wrote:

Just the examples. The modifications I made are only to the CSS and to the way the d3 function is called from the HTML.

 

Nice, I guess we can include the most significant examples in the main netjsongraph.js repository, I'd suggest to choose one or two, what do you think?

there are already two of them, ninux and AWMN, in the examples folder.

Have time to send a PR?

is it ok with the CSS trick for the properties or we wait to have it fixed?

 

Blue circles are important because they tell you visually how big is the part of the network that remains disconnected if you remove one cutpoint, without clicking on the cutpoint (which shows the number of potentially disconnected nodes). So they are not nodes, but their size is important to understand what is happening. maybe we can change the color, or the alpha, or we may change the shape of the cutpoint (from circles to diamonds).
I suppose for the latter thing we need to change the library though.

 

Ok we can play with the alpha.

ok, open an issue please?

Note that when I use json.dumps() whatever property of the node ends in the "properties" section of the node, which means it is displayed. I hide them with CSS, but maybe there is a better way of putting them somewhere else in the node data. 

 

We should look into add a way to do what you are doing now without using properties but other custom attributes that are in the node.

another issue?

done. do you think it is a good idea to add it the topology module, so when one visualizes the topology he can actually choose between several views? I can start digging into it if you think it makes sense. 

Yes, I think it would be a nice feature, we could start thinking about a generic way to create different elaborations and visualisations.

To accomplish this we should think about either using functions or classes that take something in input (NetJSON NetworkGraph?), perform some work on it and return NetJSON NetworkGraph output.

Once we have this defined, we can create a django setting that lists the available elaborations/visualisations (let's find a good name for this) and we can have a menu item in the visualiser that when clicked shows the available elaborations/visualisations, if the user clicks on one of this is taken to that view.

Exactly, my original plan is to process the NetJSON as soon as it is fetched/received  by openwisp from the source url, and store the modified NetJSON graph somewhere in the DB (elaborations on the graph may take some time, depending on the graph size, so it is better to store then and only serve them when you click on the view). Then add a URL to expose the corresponding NetJSON and finally as you say integrate the menu in the topology view.

What do you think is better:
 1 - integrate everything into openwisp-network-topology, modify the model for the topology adding another field that determines the type ("original", "compressed" and one for all the views we add)
 2 - keep the library separate (as it is now), do a separate django module with its own model?

I suppose we need to modify the topology view page, so some modifications in the topology module are needed even if we go for the second choice.

Is your python module published in pypi yet? I don't see any setup.py file so I guess it's not published yet.

not yet, need to take 1/2 decision before.

cheers,
leonardo.

[Federico Capoano]

On Thu, Oct 19, 2017 at 12:55 PM Leonardo Maccari <leonardo...@unitn.it> wrote:

On 19/10/17 10:33, Federico Capoano wrote:

Just the examples. The modifications I made are only to the CSS and to the way the d3 function is called from the HTML.

 

Nice, I guess we can include the most significant examples in the main netjsongraph.js repository, I'd suggest to choose one or two, what do you think?

there are already two of them, ninux and AWMN, in the examples folder.

Have time to send a PR?

is it ok with the CSS trick for the properties or we wait to have it fixed?

I'm super busy, so if we want this to be showcased sooner rather than later is better to do the PR now and fix the issue later.

 

 

Blue circles are important because they tell you visually how big is the part of the network that remains disconnected if you remove one cutpoint, without clicking on the cutpoint (which shows the number of potentially disconnected nodes). So they are not nodes, but their size is important to understand what is happening. maybe we can change the color, or the alpha, or we may change the shape of the cutpoint (from circles to diamonds).
I suppose for the latter thing we need to change the library though.

 

Ok we can play with the alpha.

ok, open an issue please?

which repo?

 

Note that when I use json.dumps() whatever property of the node ends in the "properties" section of the node, which means it is displayed. I hide them with CSS, but maybe there is a better way of putting them somewhere else in the node data. 

 

We should look into add a way to do what you are doing now without using properties but other custom attributes that are in the node.

another issue?

 

Yes please, open one if you can: https://github.com/netjson/netjsongraph.js/issues/new

done. do you think it is a good idea to add it the topology module, so when one visualizes the topology he can actually choose between several views? I can start digging into it if you think it makes sense. 

Yes, I think it would be a nice feature, we could start thinking about a generic way to create different elaborations and visualisations.

To accomplish this we should think about either using functions or classes that take something in input (NetJSON NetworkGraph?), perform some work on it and return NetJSON NetworkGraph output.

Once we have this defined, we can create a django setting that lists the available elaborations/visualisations (let's find a good name for this) and we can have a menu item in the visualiser that when clicked shows the available elaborations/visualisations, if the user clicks on one of this is taken to that view.

Exactly, my original plan is to process the NetJSON as soon as it is fetched/received  by openwisp from the source url, and store the modified NetJSON graph somewhere in the DB (elaborations on the graph may take some time, depending on the graph size, so it is better to store then and only serve them when you click on the view). Then add a URL to expose the corresponding NetJSON and finally as you say integrate the menu in the topology view.

What do you think is better:
 1 - integrate everything into openwisp-network-topology, modify the model for the topology adding another field that determines the type ("original", "compressed" and one for all the views we add)
 2 - keep the library separate (as it is now), do a separate django module with its own model?

 

We should probably do 1,  including such a feature in django-netjsongraph, in a similar fashion as we did for the Snapshot/History feature. Rohith knows what I'm talking about.

openwisp-network-topology is a glue layer that integrates django-netjsongraph into openwisp2 so we won't need to do much there apart from integrating the new feature into openwisp2.

I suppose we need to modify the topology view page, so some modifications in the topology module are needed even if we go for the second choice.

Yes we need to change the visualiser for sure. 

Is your python module published in pypi yet? I don't see any setup.py file so I guess it's not published yet.

not yet, need to take 1/2 decision before.

 

The module could become part of django-netjsongraph, or it could live outside and have its own life, immagine being able to build more visualizations over time without changing django-netjsongraph, that could be an advantage if you think it will likely happen.

So what name would you use in place of visualization/elaboration?

Federico

[Leonardo Maccari]

On 19/10/17 13:30, Federico Capoano wrote:

is it ok with the CSS trick for the properties or we wait to have it fixed?

I'm super busy, so if we want this to be showcased sooner rather than later is better to do the PR now and fix the issue later.

Ok, so I will just do the PR. After the merge please then open an issue so we keep track of it (I prefer you to do it rather than myself).

 

 

Blue circles are important because they tell you visually how big is the part of the network that remains disconnected if you remove one cutpoint, without clicking on the cutpoint (which shows the number of potentially disconnected nodes). So they are not nodes, but their size is important to understand what is happening. maybe we can change the color, or the alpha, or we may change the shape of the cutpoint (from circles to diamonds).
I suppose for the latter thing we need to change the library though.

 

Ok we can play with the alpha.

ok, open an issue please?

which repo?

https://github.com/netCommonsEU/netjsongraph.js/tree/robustness_graph

 

Exactly, my original plan is to process the NetJSON as soon as it is fetched/received  by openwisp from the source url, and store the modified NetJSON graph somewhere in the DB (elaborations on the graph may take some time, depending on the graph size, so it is better to store then and only serve them when you click on the view). Then add a URL to expose the corresponding NetJSON and finally as you say integrate the menu in the topology view.

What do you think is better:
 1 - integrate everything into openwisp-network-topology, modify the model for the topology adding another field that determines the type ("original", "compressed" and one for all the views we add)
 2 - keep the library separate (as it is now), do a separate django module with its own model?

 

We should probably do 1,  including such a feature in django-netjsongraph, in a similar fashion as we did for the Snapshot/History feature. Rohith knows what I'm talking about.

openwisp-network-topology is a glue layer that integrates django-netjsongraph into openwisp2 so we won't need to do much there apart from integrating the new feature into openwisp2.

but openwisp-network-topology contains the model for the NetJSON topology in the DB, right? where do you want the new topologies models be stored?

I suppose we need to modify the topology view page, so some modifications in the topology module are needed even if we go for the second choice.

Yes we need to change the visualiser for sure.

ok, this will come after.

Is your python module published in pypi yet? I don't see any setup.py file so I guess it's not published yet.

not yet, need to take 1/2 decision before.

 

The module could become part of django-netjsongraph, or it could live outside and have its own life, immagine being able to build more visualizations over time without changing django-netjsongraph, that could be an advantage if you think it will likely happen.

it will probably be developed step by step. So yes, I will do a separate module.

So what name would you use in place of visualization/elaboration?

visualization fits better I think.

So the pseudo-roadmap is:
 - I do a PR for netjsongraph.js, as is
 - you please open two tickets for the alpha and for the properties
 - i do the pipy package for the analyser module
 - i start making up my mind on what modifications are needed on django-netjsongraph and openwisp-network-topology to integrate the view. I will need some help from you on this again, but then I will make you a proposal.

On a different level, I'd like to start matching, wherever possible, the owners of the nodes with the nodes in the topology.  I don't have clear ideas now on how to do it, so we will need more discussion.

thanks,

[Federico Capoano]

On Fri, Oct 20, 2017 at 11:12 AM Leonardo Maccari <leonardo...@unitn.it> wrote:

On 19/10/17 13:30, Federico Capoano wrote:

is it ok with the CSS trick for the properties or we wait to have it fixed?

I'm super busy, so if we want this to be showcased sooner rather than later is better to do the PR now and fix the issue later.

Ok, so I will just do the PR. After the merge please then open an issue so we keep track of it (I prefer you to do it rather than myself).

 

 

Blue circles are important because they tell you visually how big is the part of the network that remains disconnected if you remove one cutpoint, without clicking on the cutpoint (which shows the number of potentially disconnected nodes). So they are not nodes, but their size is important to understand what is happening. maybe we can change the color, or the alpha, or we may change the shape of the cutpoint (from circles to diamonds).
I suppose for the latter thing we need to change the library though.

 

Ok we can play with the alpha.

ok, open an issue please?

which repo?

https://github.com/netCommonsEU/netjsongraph.js/tree/robustness_graph

 

It seems I can't open issues there, so I opened it here: https://github.com/netjson/netjsongraph.js/issues/51

 

Exactly, my original plan is to process the NetJSON as soon as it is fetched/received  by openwisp from the source url, and store the modified NetJSON graph somewhere in the DB (elaborations on the graph may take some time, depending on the graph size, so it is better to store then and only serve them when you click on the view). Then add a URL to expose the corresponding NetJSON and finally as you say integrate the menu in the topology view.

What do you think is better:
 1 - integrate everything into openwisp-network-topology, modify the model for the topology adding another field that determines the type ("original", "compressed" and one for all the views we add)
 2 - keep the library separate (as it is now), do a separate django module with its own model?

 

We should probably do 1,  including such a feature in django-netjsongraph, in a similar fashion as we did for the Snapshot/History feature. Rohith knows what I'm talking about.

openwisp-network-topology is a glue layer that integrates django-netjsongraph into openwisp2 so we won't need to do much there apart from integrating the new feature into openwisp2.

but openwisp-network-topology contains the model for the NetJSON topology in the DB, right? where do you want the new topologies models be stored? 

 

Main logic is in django-netjsongraph, openwisp-network-topology is a wrapper that integrates it in openwisp2 adding multitenancy to it.

Maybe if you take a look ata mode

I suppose we need to modify the topology view page, so some modifications in the topology module are needed even if we go for the second choice.

Yes we need to change the visualiser for sure.

ok, this will come after.

Is your python module published in pypi yet? I don't see any setup.py file so I guess it's not published yet.

not yet, need to take 1/2 decision before.

 

The module could become part of django-netjsongraph, or it could live outside and have its own life, immagine being able to build more visualizations over time without changing django-netjsongraph, that could be an advantage if you think it will likely happen.

it will probably be developed step by step. So yes, I will do a separate module.

So what name would you use in place of visualization/elaboration?

visualization fits better I think.

So the pseudo-roadmap is:
 - I do a PR for netjsongraph.js, as is
 - you please open two tickets for the alpha and for the properties

 

Could you remind me the problem regarding CSS hack please?

 - i do the pipy package for the analyser module
 - i start making up my mind on what modifications are needed on django-netjsongraph and openwisp-network-topology to integrate the view. I will need some help from you on this again, but then I will make you a proposal.

On a different level, I'd like to start matching, wherever possible, the owners of the nodes with the nodes in the topology.  I don't have clear ideas now on how to do it, so we will need more discussion.

Indeed this needs more discussion because the way to match depends on different variables and we need to find a way that won't make us go crazy.

Federico

[Leonardo Maccari]

On 29/10/17 16:56, Federico Capoano wrote:

On Fri, Oct 20, 2017 at 11:12 AM Leonardo Maccari <leonardo...@unitn.it> wrote:

On 19/10/17 13:30, Federico Capoano wrote:

is it ok with the CSS trick for the properties or we wait to have it fixed?

I'm super busy, so if we want this to be showcased sooner rather than later is better to do the PR now and fix the issue later.

Ok, so I will just do the PR. After the merge please then open an issue so we keep track of it (I prefer you to do it rather than myself).

 

 

Blue circles are important because they tell you visually how big is the part of the network that remains disconnected if you remove one cutpoint, without clicking on the cutpoint (which shows the number of potentially disconnected nodes). So they are not nodes, but their size is important to understand what is happening. maybe we can change the color, or the alpha, or we may change the shape of the cutpoint (from circles to diamonds).
I suppose for the latter thing we need to change the library though.

 

Ok we can play with the alpha.

ok, open an issue please?

which repo?

https://github.com/netCommonsEU/netjsongraph.js/tree/robustness_graph

 

It seems I can't open issues there, so I opened it here: https://github.com/netjson/netjsongraph.js/issues/51

Ok, I enabled issues in the original project. Will check it asap, or, we can deal with it with Google Code-in.

but openwisp-network-topology contains the model for the NetJSON topology in the DB, right? where do you want the new topologies models be stored? 

 

Main logic is in django-netjsongraph, openwisp-network-topology is a wrapper that integrates it in openwisp2 adding multitenancy to it.

Maybe if you take a look ata mode

Ok will do.

visualization fits better I think.

So the pseudo-roadmap is:
 - I do a PR for netjsongraph.js, as is
 - you please open two tickets for the alpha and for the properties

 

Could you remind me the problem regarding CSS hack please?

my understanding is that you use _netjson_networkgraph from netdiff.utils to convert a networkx graph to netjson. this returns an OrderedDict which you dump to NetJSON using json.dums(). whatever property of the node ends in the "properties" section of the NetJSON node, which means it is displayed. I hide them with CSS, but maybe there is a better way of putting them somewhere else in the node data.

I am talking about the code in the example main.py, here:

https://github.com/netCommonsEU/netjson-robustness-analyser/blob/master/main.py

 - i do the pipy package for the analyser module
 - i start making up my mind on what modifications are needed on django-netjsongraph and openwisp-network-topology to integrate the view. I will need some help from you on this again, but then I will make you a proposal.

On a different level, I'd like to start matching, wherever possible, the owners of the nodes with the nodes in the topology.  I don't have clear ideas now on how to do it, so we will need more discussion.

Indeed this needs more discussion because the way to match depends on different variables and we need to find a way that won't make us go crazy.

Ok, one thing at a time.

l.