Configuation files

[tim.in...@gmail.com]

Hi Tobias,

I have my evaluation of crossbar.io working, to deliver dynamic information from a database to a webpage,
largely by building on your examples and very little understanding on my part.

One area that I need to understand better is the crossbar.io configuration file. Is there a 'user manual' for this
that details everything that can go in this file, and what each keyword means?
I have found many examples of different config. files and I recognise what a lot of the entries might mean,
but there is ambiguity. It might help if each line were commented, explaining exactly what crossbar.io does
with that line of information.

For example, is there a list of "types", and for each "type" a list of associated entries ?

Regards,
Tim

[Tobias Oberstein]

Hi Tim,

Am 14.08.2014 15:47, schrieb tim.in...@gmail.com:
> Hi Tobias,
>
> I have my evaluation of crossbar.io working, to deliver dynamic
> information from a database to a webpage,
> largely by building on your examples and very little understanding on my
> part.

So the examples do already help;) Good to know ..

>
> One area that I need to understand better is the crossbar.io
> configuration file. Is there a 'user manual' for this
> that details everything that can go in this file, and what each keyword
> means?

Yes. It's not in a single page, but each option is discussed under the
respective documentation page.

http://crossbar.io/docs/Configuration-Overview/
http://crossbar.io/docs/WebSocket-Transports/

and so on.

> I have found many examples of different config. files and I recognise
> what a lot of the entries might mean,
> but there is ambiguity. It might help if each line were commented,
> explaining exactly what crossbar.io does
> with that line of information.

Yes, I agree, it would help if the examples configs would have comment
lines.

Unfortunately, JSON does not allow embedded comments;(

Which is the main reason we want to add YAML as an alternative config
format:

https://github.com/crossbario/crossbar/issues/99
http://en.wikipedia.org/wiki/Yaml

Let me see if I can add this in <1 hour;) Needs to be done anyway ..

Crossbar.io codebase is already prepared to support arbitrary config
formats (as long as they are as "capable" as JSON)

Cheers,
/Tobias

>
> For example, is there a list of "types", and for each "type" a list of
> associated entries ?
>
> Regards,
> Tim
>

> --
> You received this message because you are subscribed to the Google
> Groups "Crossbar" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to crossbario+...@googlegroups.com
> <mailto:crossbario+...@googlegroups.com>.
> To post to this group, send email to cross...@googlegroups.com
> <mailto:cross...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/crossbario/b9a43bc1-b48b-4f35-bbd7-66de9d6b9548%40googlegroups.com
> <https://groups.google.com/d/msgid/crossbario/b9a43bc1-b48b-4f35-bbd7-66de9d6b9548%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

[Tobias Oberstein]

Hi Tim,

alright, Crossbar.io trunk now support YAML as an alternative
configuration format.

$ crossbar start
2014-08-14 17:41:43+0200 [Controller 4180] Log opened.
...
2014-08-14 17:41:43+0200 [Controller 4180] Starting from node
directory c:\Temp\.crossbar
2014-08-14 17:41:44+0200 [Controller 4180] Starting from local
configuration 'c:\Temp\.crossbar\config.yaml'

where "config.yaml" can contain comments


controller: {}
workers:
- realms:
- name: realm1
roles:
- name: anonymous
permissions:

# allow just anyone to do anything ..
- {call: true, publish: true, register: true, subscribe: true,
uri: '*'}
transports:
- endpoint: {port: 8080, type: tcp}
paths:
/: {directory: .., type: static}
ws: {type: websocket}
type: web

# we are just running a single worker of type "router"
type: router


===

Crossbar.io will do the following:

1. If no "--config" option is given, first check if there is a
CBDIR/config.json. If so, use that.
2. If not, check if there is CBDIR/config.yaml. If so use that.
3. If not, bail out

If "--config" is given, use that, and determine file type by file extension.

Means: it should behave "with least suprise".

===

Now I only need to learn YAML;) No clue, other than it's used widely.

Cheers,
/Tobias

Am 14.08.2014 15:47, schrieb tim.in...@gmail.com:

[tim.in...@gmail.com]

Hello Tobias,

Wow, that was fast !
I think supporting a config file that can contain comments (YAML) is a big step forward
in helping people to understand the configuration.

Here is some feedback on the documentation. This is going to sound negative but I mean
it to be constructive. If you can understand what I am finding difficult I hope you will be able
to add enough to the documentation to make it much easier for lots more people to understand
and use. I am beginning to fall in love with Crossbar.io, but not the documentation...

One of the biggest hurdles for me in using Crossbar.io is that I don't know what
it is capable of at the coding level, and I think that might be answered from the configuration documentation.
I've already looked at the documentation and often found more questions than answers.

For example, I look in a configuration file and see "type": "container"...
so I look at the documentation here http://crossbar.io/docs/,
click on the container configuration link and find all the documentation - great.
I look back at the configuration file and see an entry "components": [
inside the container, but when I try and look up "components" under the
"container configuration" I don't find anything.
There is a link to "router components" under "router configuration" but its
unclear if that is valid also for container components.

In summary, its great to have demos and example configuration files giving a tantalising
glimpse of what might be possible and then frustrating when you can't find documentation
 that covers the items in those examples/config.

BR, Tim

[Tobias Oberstein]

Hi Tim,

Am 15.08.2014 16:26, schrieb tim.in...@gmail.com:
> Hello Tobias,
>
> Wow, that was fast !
> I think supporting a config file that can contain comments (YAML) is a
> big step forward
> in helping people to understand the configuration.

Do you think it would make sense to rewrite the examples (app templates)
and docs to YAML format while adding comments?

I mean, there might be a downside: JSON can be read probably by anyone.
YAML might not be readable to all.

>
> Here is some feedback on the documentation. This is going to sound
> negative but I mean
> it to be constructive. If you can understand what I am finding difficult

No, this is great! Don't hold back with critics .. it's not about "being
inpolite" .. we need to improve on this.

> I hope you will be able
> to add enough to the documentation to make it much easier for lots more
> people to understand
> and use. I am beginning to fall in love with Crossbar.io, but not the
> documentation...

I see. Lot's of feedback we got is like this: "this is awesome. but you
guys can't explain things, and the docs suck". I totally agree.

We don't have marketing pros or technical writers on the team. No
excuse, but that's a fact. Guess we need help with that.

>
> One of the biggest hurdles for me in using Crossbar.io is that I don't
> know what
> it is capable of at the coding level, and I think that might be answered
> from the configuration documentation.
> I've already looked at the documentation and often found more questions
> than answers.

Not good.

>
> For example, I look in a configuration file and see "type": "container"...
> so I look at the documentation here http://crossbar.io/docs/,
> click on the container configuration link and find all the documentation
> - great.
> I look back at the configuration file and see an entry "components": [
> inside the container, but when I try and look up "components" under the
> "container configuration" I don't find anything.
> There is a link to "router components" under "router configuration" but its
> unclear if that is valid also for container components.

Yes. Confusing. Incomplete. I've opened

https://github.com/crossbario/crossbar/issues/106

to track that specific issue.

>
> In summary, its great to have demos and example configuration files
> giving a tantalising
> glimpse of what might be possible and then frustrating when you can't
> find documentation
> that covers the items in those examples/config.

That sounds like you would welcome having YAML configs with extensive
comments inside? Probably even backlinks into the docs?

Thanks for feedback!
/Tobias

>
> BR, Tim

>
>
> --
> You received this message because you are subscribed to the Google

> Groups "Autobahn" group.

> To unsubscribe from this group and stop receiving emails from it, send

> an email to autobahnws+...@googlegroups.com
> <mailto:autobahnws+...@googlegroups.com>.
> To post to this group, send email to autob...@googlegroups.com
> <mailto:autob...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/autobahnws/70de0b1e-4618-4176-9da9-4068413cba4a%40googlegroups.com
> <https://groups.google.com/d/msgid/autobahnws/70de0b1e-4618-4176-9da9-4068413cba4a%40googlegroups.com?utm_medium=email&utm_source=footer>.

[tim.in...@gmail.com]

Hi Tobias,

Re-writing all the config files in YAML sounds a lot of work, and I'm not sure that is necessary.
Sure it would be great to have verbose comments in the config. files, explaining all the details, and
links back to the documentation, but that sounds a monumental job.
Another way would be to make sure that all the demos have multiple references to the documentation
so its easy to get to the documentation no matter where you find the demo (github, tavendo, http://crossbar.io, etc),
and then make sure that everything in every demo has documentation to explain it.

I would say that as you add a new demo, much more important than the demo itself is having the documentation
in place first explaining how the demo works and any new configuration elements the demo introduces.

Also, in trying to understand Crossbar.io I am having to learn or understand a lot of other support stuff that Crossbar.io
builds on (Autobahn, WAMP, WebSocket, Twisted(whatever that is), etc). It would be good to have a picture of how all this fits together,
and perhaps, in particular, explain what each of these adds to the equation.
The problem for people like me is that we can see from the demos that Crossbar.io is incredible, but the learning curve
with all these new modules at times seems steeper than vertical !

For quite a long time I couldn't figure this out.
WebSocket I understood right way because it adds native support for the bi-direction comms that has been sadly missing from browsers
for so long.
I think I now have a clear picture that WAMP is an implementation of WebSocket, and that Autobahn is an implementation of WAMP,
and that Crossbar.io adds to Autobahn features like polyglot and a built in static web server(brilliant idea!).

Cheers,
Tim

[tim.in...@gmail.com]

Tobias,

Have you thought about an FAQ for Autobahn and Crossbar.io? That could be another resource
that helps explain what both of these are all about as well as how to configure them.

Tim

[Tobias Oberstein]

FAQ: good idea. We are collection suggestions for improving docs here: https://github.com/crossbario/crossbar/issues/107

[tim.in...@gmail.com]

Hello Tobias and Alexander,

I will try and add questions here: https://github.com/crossbario/crossbar/issues/107

It strikes me now that perhaps my biggest challenge was trying to deal with many new things all at once
(Websocket, WAMP, Crossbar.io, Autobahn, Twisted, Python, HTML, CSS, javascript, etc). I have many years experience with C and C++, but have
never looked at Python(my head has been yanked out of the sand and my eyes are now wide open!) and
the differences from C make a lot of new things very challenging, even though there is a lot of similarity too; The devil
is in the detail.
Of course, this is my problem not yours, but if your examples had lots of explanation it would help people like me to try out
Crossbar.io quicker and with less pain and to persevere with it. There were times when I thought the time investment
was getting too much for such slow progress, but I'm glad now I carried on. Things are becoming more clear, and I have
a great, but simple, application thanks to Crossbar.io.

Perhaps one of the things to cover in your tutorial is which things a newcomer needs to learn a bit more about and what
they can safely ignore for now. When I first started, I looked at the Crossbar.io demos, liked what I saw, and realised
this was exactly what I needed to transfer data out of a Firebird database, dynamically and triggered by database activity, and
across to a remote web browser for display, so I immediately started about modifying the crossbar.io pubsub demo
to perform some tests. This is where the learning curve went near vertical. I didn't know where to start ! I didn't know
which modules I needed to consider editing first, WAMP, Twisted, Autobahn, or Crossbar.io.

I know now, for example, that I didn't need to initially worry about WAMP, or Twisted but I did need to understand
the difference between Autobahn and Crossbar.io or, at least, what each of these does.
I know now (I think) that Autobahn, built on top of Websockets, WAMP and Twisted, basically implements
the websockets protocol (and much more) and allows me to pass data from the database to the web browser
without worrying about the transport etc.
Crossbar.io allows multiple applications to run on potentially any number of different
machines and talk to each other using the Autobahn support.
Crossbar.io also supports a simple web-server for static pages, so I don't have to have a separate web-server.
But the boundaries between Autobahn and Crossbar.io are still a very grey area to me.
If you asked me what exactly Crossbar.io does that Autobahn does not I would struggle to answer.
I think the answer is that my Python applications would be written on top of AutobahnPython, and that
Crossbar.io is responsible for starting those applications (as referred to by the Crossbar configuration file),
supports applications to be written in a number of other languages, is responsible for running a web-server,
is responsible for passing configuration information from the configuration file to the Autobahn derived
application (eg name of communications realm).
Is any of that right ?

I think the tutorial could add a lot of help if it explains everything in detail. For example, in the Python file
hello.py for the pubsub demo it might be helpful to add comments at each of the import statements.
Some examples...
1.
## Import just the sleep class from the Python module Autobahn
## Autobahn itself is derived from the 'util' class in the Twisted module
from autobahn.twisted.util import sleep

2.
## AppSession is the main application class and is derived from the Autobahn class ApplicationSession
## This class provides access to the communication mechanisms that can be used by your applications
## to communicate over transport mechanisms such as websockets.
class AppSession(ApplicationSession)

3.
Give detailed information about the call to subscribe, saying what each of the parameters is, what type
they are and what information they can take. eg what can the topic string be or consist of.
Why is the demo topic 'com.example.onhello' and why are there periods in this string. What is there
significance?

4.
yield self.publish('com.example.oncounter', counter)
yield sleep(1)
Why is the yield keyword used in these statements. Is that critical or optional and what is its significance ?

5.
What can the application do in the ApplicationSession derived class, are there any limitations
or special requirements ?

A lot of the answers to the questions posed above are probably documented elsewhere but having all
the information in one place is of great benefit to the newcomer, otherwise finding the correct documentation
from many different sources and pulling it all together can be daunting, possibly overwhelming for some.

Hope this helps !

Cheers,
Tim

[tim.in...@gmail.com]

BTW, Your answers to this thread..
crossbar.io: reasons why onJoin() might not be called?
Answer quite a lot of questions. By going through the sample config file that Scott provided
and adding comments about specific parts together with your other answers you have disclosed
a lot more helpful information.

Why not take all the questions posed in this googlegroup and add them to an FAQ on the crossbar.io
documentation site ?

Tim

[tim.in...@gmail.com]

Hi Alex,

Firstly let me say thanks so much to both of you  for your great support, Autobahn and crossbar.io
have huge potential. My imagination is running wild with thoughts on different application areas that they could benefit.

I am mainly working with Python, but I am also working a little bit, out of necessity, with javascript, but only
to subscribe to the pubsub messages, pickup a string of data and display it to a few elements in a web page.

For me the Javascript side of things was not the difficult part. I could clearly see the statements there to subscribe
to messages and was able to pickup new data and forward it without much fuss.

The more challenging part was trying to separate all the components in the backend, and in particular the Python/
Autobahn/Crossbar mix.

Let me know when you have the tutorial ready and I'll let you have any comments I can. Of course
I am not on the same part of the learning curve that I was once, thankfully !

Thanks
Tim

[Roger Erens]

I'd like to thank Tim for bringing this up!

I'm still in the phase of having difficulty to create a mental map of crossbar.io and autobahn|python, and deciding which examples/demos to look at next.

I see the documentation improving, great to see where it's going, Alexander and Tobias. I can't believe you only use 24 hours in a day...

This list pointed me to KnockoutJS, which has a very nice interactive tutorial, I'd recommend it to others. I'm curious to see what Alexander comes up with...

And a suggestion for web-presenting the comments that Tim wants sprinkled spuriously in code: separated in such a way that both the code and the comments look less daunting by means of pycco:

http://fitzgen.github.io/pycco/

Finally (this could be only me of course): even though the hello template is very simple, it's still 4 concepts being introduced/shown at once. With all the other new stuff Tim already mentioned that newbies need to wrap their head around, I'd say that 1 babystep at a time is enough. Maybe have even simpler hello1, hello2, hello3, and hello4 templates available before bringing up the 'grand finale' of combining PubSub, and RPC both ways, all together :-). This way you'll give a newbie 5 opportunities to get a feeling of accomplishment when he get's the 'Aha'-erlebnis and starts feeling confident about the code and the output rolling over the console.

Cheers,

Roger

Op donderdag 4 september 2014 15:39:04 UTC+2 schreef tim.in...@gmail.com:

[tim.in...@gmail.com]

The more I think about the Crossbar.io/Autobahn/Python mix the more clear the

picture is becoming. The fog is slowly clearing as I think about and understand a little better what is involved in

achieving a simple pubsub demo where, essentially and conceptually, data is pushed from

a simple Web Server to an HTML page displayed in a Web Browser; This happens automatically, as new

data becomes available at the Web Server, and does not require intervention by a person(eg pressing the browser refresh button)

sat in front of the Web Browser. The problem in trying to understand

the Crossbar.io/Autobahn/Python mix is that to achieve the above goal requires a staggering amount

of complexity for something that sounds so simple. Complexity

that is hidden away inside Crossbar.io/Autobahn, together with many other software layers,

and us newbies need to identify what we can safely ignore and where to focus.

Writing this entry is helping me understand Autobahn and Crossbar.io and hopefully others will

correct my mistakes and thereby help clarify many things for other newbies.

First there is the concept of pubsub, pushing data into a pipe at the server end and that data dropping

out the other end of the pipe into our lap, our browser lap. This simple concept is immediately complicated

by one of the amazing autobahn features that lets any number of clients subscribe to the data published

by the server; Autobahn effectively adds a new ‘pipe’ between the server and each subscribed client and

copies the data out to each client. So in the simple pubsub demo, the data at the server end can be displayed

in any number of web browsers, without the server knowing how many browsers are watching or how

to route the information to them. The data routing is all taken care of by Autobahn.

At Coastform we are experimenting with Autobahn/Crossbar.io to offer access control features.

In a simple example the data is the name of a user who has just walked through a security door, together with

the time of access and a pathname to a photograph of the person. This data is extracted from an

event-driven firebird database by a simple Python script and pushed across to the waiting web browsers

by Autobahn, in near real time. Somebody scans their tag at a door reader and within a fraction of a second

their name and picture are displayed in a remote web browser(s) perhaps located at the entry door or

perhaps at a security desk elsewhere in the building. What a fantastic result ! And all with a few lines

of Python code plus a little more HTML and Javascript. All thanks to the supporting framework provided

by Autobahn, Crossbar.io, and the many other layers they are built on.

One critical development in all of this, that slipped in under my radar until just recently, is webSockets.

WebSockets essentially add native HTML support for bi-directional communication between web server and

browser, so the server can deliver data over to the browser as it becomes available rather than waiting

for a user to refresh their web page. Although many other solutions have been created to allow

similar bi-directional comms, these are typically convoluted. WebSockets are a much, much cleaner solution.

It is something that has been sadly lacking in HTML for way too long. Web browsers have become very

sophisticated graphics rendering engines over recent years, capable of formatting data and graphics presentation

any which way desired, and remote from the data source, but until the introduction of webSockets it has

been difficult to tap into that powerful display. With Autobahn and Crossbar.io making it easy to use

webSockets, they have opened up a new universe of opportunity and will change the way many applications

are written to display data and graphics. Finally there is an easy way to exploit the powerful features of web browsers.

Hardware products no longer need to have a dedicated user interface;

The user interface can run on a smartphone, tablet, or any other device capable of running a modern Web browser.

Many applications no longer need to include complicated code

to format data and display graphics, they can use the sophisticated rendering engine built-in to modern

browsers. Networked device and apps do not need to solve the problem of offering a remote user interface or transmitting data

long distances to be displayed on remote terminals; Autobahn and Crossbar.io handle all of that, freeing products

and applications to focus much more on things specific to them. Wow! What flexibility and power!

What does Crossbar.io do? Hmmm, a lot! I think that’s part of the problem trying to understand where

it fits in. I think another hurdle in the way of understanding is that the authors of Crossbar.io have added

a very powerful feature that enables a single command to start any number of client or server application

components, as well as crossbar itself, and have used this feature in many of the examples/demos, so a newbie

has to dig deeper and understand a lot more to try and figure out what is going on.

All the necessary information is held neatly in a single (JSON)configuration file, but that introduces another hurdle

in the learning process. A newbie tries to dig a little to see what is going on and finds a JSON file, whatever that is,

and then encounters a list of apparently disjoint string pairs...

I think a better place for a newbie to start would be Autobahn because it is the thing offering an easy

way to use webSockets (via another layer called WAMP). It also includes a basic WAMP router, something

that is necessary to actually get the data to the right destination. ie Autobahn includes everything needed

to see a simple PubSub demo in operation. You can clearly see an example broken down into three components:

router, backend(or server), and frontend(or user interface). You can start these components separately too,

so there is less black magic to worry about.

Once you can see these components, the delightful simplicity of the system becomes apparent. The communication

is limited and directed using a couple of strings, which can be anything you like(I think),

to define a realm and topics. The realm is simply a way of ring fencing those speakers and listeners that want to talk about the

same things (topics). A topic is just a way of tagging or identifying a specific bit of information, such as a person’s name.

The backend and frontend components can be just a few lines of code, communicating using nothing more than

a few string identifiers to couple them together, with the coupling performed by Autobahn and its router. Amazing.

Maybe the Crossbar.io demos page should point newbies at the Autobahn demos first ?
Maybe the demos should all start out as Autobahn demos that are tweaked to show where

Crossbar adds more?
 ie each demo could have an Autobahn variant, so the boundaries between

components is clear and easily identified, and a crossbar.io variant which allows the advanced features of Crossbar.io

to be equally easily identified and tested. As I said

earlier Crossbar.io uses a single (JSON) configuration file to pull together all the configuration information

for a communicating application components, no matter how many of them there are. Application

components can be referenced in the configuration file, in which case Crossbar will automatically start them.

Furthermore, if an application component is written as a Python script, the main class (or entry point) can be referenced in the

configuration file, in which case Crossbar will start the Python script at your specified entry point.

Crossbar.io does much more than offer this convenience though. It includes a more advanced WAMP router than

Autobahn, one that supports authorisation and authentication, so the components you write can communicate

securely but without the need to write security code.

It includes a Web Server for static pages. We use this with our simple access control example to deliver the HTML, CSS

and javascript files across to the browser for formatting and displaying the name and photo of the person walking

through the security door. This feature alone is fantastic, it means we don’t have to install and run yet another application

(web server).

And, much more that I don't understand yet...

WAMP, Autobahn, Crossbar.io, etc are all open-source, BTW. I can't find enough superlatives for that !

Tim

[Roger Erens]

Maybe it could be helpful if there could be some additional templates. I don't find the hello template minimal, as it has three peers (WAMP router, front-end HTML5 client, back-end in <your-favorite-language>), of which the front-end and back-end clients play each four possible (non-router) roles: Callee, Caller, Publisher, and Subscriber.

I think there's room in the 'Crossbar.io Template Universum' to have next to the

crossbar init

default that creates just a WAMP router, a set of

crossbar init callee:<language>

crossbar init caller:<language>

crossbar init publisher:<language>

crossbar init subscriber:<language>

templates. I think it would help newcomers to focus on one concept at a time. For those who think these baby steps are too tiny, there's the hello:<language> template to get started with.

If you'd like me to create a Pull Request for those templates, /me ('worrelsik') would be happy to get some help on #autobahn to get started on them.

Op vrijdag 5 september 2014 16:40:52 UTC+2 schreef tim.in...@gmail.com: