New Crossbar.io / AutobahnPython releases

[Tobias Oberstein]

Hi,

just released new versions of Crossbar.io and Autobahn|Python

https://pypi.python.org/pypi/crossbar/0.12.1
https://pypi.python.org/pypi/autobahn/0.12.1

Of course, feedback is welcome! And if you like, please spread the word
about Crossbar.io and Autobahn;)

Cheers,
/Tobias


The releases aggregate the last (hopefully) bunch of Python 3 (unicode
vs bytes), and other smaller fixes. But there is more.

A completely new feature is "event history". Often, a subscriber wants
to catch up with the latest event _history_, not only receive new
events. Here is an example:

https://github.com/crossbario/crossbarexamples/tree/master/event-history

Then the releases also contain fixes for WAMP authentication methods.
I've tested all methods on both Python 2 and 3, and we now have full
examples of all auth. methods:

https://github.com/crossbario/crossbarexamples/tree/master/authentication

There are also 2 completely new WAMP authentication methods:

- WAMP-TLS - this is using TLS client certificates for authentication
- WAMP-cryptosign - this is using state-of-the-art, public-private key
based mechanism using Curve25519

Curve25519 is an elliptic curve designed by Dan Bernstein ("djb"). djb
is a crypto guru, and, a trusted person in the crypto community.

Crossbar.io and WAMP-cryptosign are listed here

https://ianix.com/pub/ed25519-deployment.html

under "upcoming" already;)

Looking forward, WAMP-cryptosign will be our preferred authentication
method for our own apps.

The fact that it's public-private key means there are no secrets
whatsoever on the server side, and the fact that it's based on
Curve25519 and NaCl

http://nacl.cr.yp.to/
http://pynacl.readthedocs.org

means it is very secure.

---

Then, there is another crypto thing in the cooking (alpha code is in
above releases already!): WAMP-cryptobox

This, for me, is _the_ most interesting feature in a while!

It's adding an end-to-end encryption option to Crossbar.io and
AutobahnPython (for now, but JS and others is possible too).

Yes, that's right: using e2e, not even Crossbar.io is able to read the
application payload of your calls and events!

This very much preview .. but here is an example that should give you a
better idea of how it looks

https://github.com/crossbario/crossbarexamples/blob/master/encryption/cryptobox/client1.py#L45

The nice thing: once you've added a "keyring" (optionally, configured
per-URI keypairs), it is completely transparent from app code. No code
changes there at all!

---

Crossbar.io 0.12.1 identifies as:

(python351_2) oberstet@thinkpad-t430s:~$ crossbar version
Automatically choosing optimal Twisted reactor
Running on Linux and optimal reactor (epoll) was installed.
__ __ __ __ __ __ __ __
/ `|__)/ \/__`/__`|__) /\ |__) |/ \
\__,| \\__/.__/.__/|__)/~~\| \. |\__/

Crossbar.io : 0.12.1
Autobahn : 0.12.1 (with JSON, MessagePack, CBOR)
Twisted : 15.5.0-EPollReactor
LMDB : 0.88/lmdb-0.9.17
Python : 3.5.1/CPython
OS : Linux-3.13.0-76-generic-x86_64-with-debian-jessie-sid
Machine : x86_64

at the command line. Of course we are running the release now on the
demo instance (on latest PyPy / Linux) as well:

https://demo.crossbar.io/ws

The demo instance now also use Let's Encrypt certificates, and we've
further hardened TLS

https://www.ssllabs.com/ssltest/analyze.html?d=demo.crossbar.io

Also: the demo instance are actually 2 instances behind geolocation
based DNS. Depending on your location, you'll be connected to one of these:

https://cbdemo-us-west-1.crossbar.io/
https://cbdemo-eu-central-1.crossbar.io/

When you open a demo, you should see a statusline at the bottom like

Status: Connected to node cbdemo-eu-central-1 at
wss://cbdemo-eu-central-1.crossbar.io/ws

that tells you to which instance you are connected.

Note that the instances are not yet clustered (eg when you are connected
to the Votes demo on instance A, you won't get votes on instance B).

---

[Michel Desmoulin]

Fantastic. Event history gives you the hability to build services like firebase while crypto auth will make prod much more secured.

I wrote an article introducing the new features, and will probably write a tutorial en auth now that we got the last bug fixed.

I also made some publication on the usual media (twitter, reddit, etc).

Let's spread the love :)

[Greg Keys]

Tobias,

Excellent work, 

I see the minor version bumped from 11 to 12 does this mean there are breaking changes we need to be aware of?

[pe...@koodaamo.fi]

Good news. Would it be difficult to add support for fetching just a subset of event history based on a timestamp? Any downsides to that?

 Petri

[Tobias Oberstein]

Hi Greg,

> Excellent work,

Thanks;)

>
> I see the minor version bumped from 11 to 12 does this mean there are
> breaking changes we need to be aware of?

Anonymous auth configuration needs to have "type" now in the config

https://github.com/crossbario/crossbarexamples/blob/master/authentication/anonymous/static/.crossbar/config.json#L40

Defining cert chains has changed in config:

https://github.com/crossbario/crossbarexamples/blob/master/encryption/tls/.crossbar/config.json#L52

Dynamic authenticators now all (regardless of specific authmethod) need
to have the same signature

(authid|string, realm|string, details|dict) -> auth|dict

and must return a dict with at least authrole, but can override any of
realm, authid, authextra too

https://github.com/crossbario/crossbarexamples/blob/master/authentication/advanced/authenticator.py

There are likely more nitty gritty details I forgot right now. Nothing
big though.

Beginning from 1.0.0, we'll follow a strict semantic versioning
approach, with a documented, well-defined set of APIs (rgd. CLI, config
files, WAMP meta API and CDC management API), proper changelogs etc etc.

We are not quite there. Things are still in flux .. yeah, I know, app
developers need (and deserve) stability especially in infrastructure
stuff. No question. Thing is: at this stage, we still need some freedom
to experiment and see what works in practice.

Our overall approach of getting something out into real world use
quickly, and then do some iterations to hash things out has worked
pretty well, I'd say. Both for WAMP, and for Autobahn and Crossbar.io.

IMO, you can't design at the drawing table, in thin air only. There need
to be iterations. But rest assured: I am very well aware that we need to
raise the bar with dev processes also. My fav. benchmark: PostgreSQL.
It's one of the most dependable pieces of software out there. And the PG
project is rightfully proud of that! The "processes" that make that
happen (similar to Twisted) can be a pain in the ass sometimes - but;)

Cheers,
/Tobias

>
> On Saturday, January 30, 2016 at 10:35:28 AM UTC-8, Tobias Oberstein wrote:
>
> Hi,
>
> just released new versions of Crossbar.io and Autobahn|Python
>
> https://pypi.python.org/pypi/crossbar/0.12.1

> <https://pypi.python.org/pypi/crossbar/0.12.1>
> https://pypi.python.org/pypi/autobahn/0.12.1

> <https://cbdemo-us-west-1.crossbar.io/>
> https://cbdemo-eu-central-1.crossbar.io/

> <https://cbdemo-eu-central-1.crossbar.io/>
>
> When you open a demo, you should see a statusline at the bottom like
>
> Status: Connected to node cbdemo-eu-central-1 at
> wss://cbdemo-eu-central-1.crossbar.io/ws

> <http://cbdemo-eu-central-1.crossbar.io/ws>

>
> that tells you to which instance you are connected.
>
> Note that the instances are not yet clustered (eg when you are
> connected
> to the Votes demo on instance A, you won't get votes on instance B).
>
> ---
>

> --
> You received this message because you are subscribed to the Google
> Groups "Crossbar" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to crossbario+...@googlegroups.com
> <mailto:crossbario+...@googlegroups.com>.
> To post to this group, send email to cross...@googlegroups.com
> <mailto:cross...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/crossbario/c64d821c-ef76-4ae1-b9bb-11e1e554e960%40googlegroups.com
> <https://groups.google.com/d/msgid/crossbario/c64d821c-ef76-4ae1-b9bb-11e1e554e960%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

[Tobias Oberstein]

Am 01.02.2016 um 13:01 schrieb pe...@koodaamo.fi:
> Good news. Would it be difficult to add support for fetching just a
> subset of event history based on a timestamp? Any downsides to that?

Retrieving history based on timestamp, instead of "limit to last N" is
definitely something we want to add.

The plan is this:

- add LMDB based persistence (currently, history won't survive router
restarts)
- see how it interacts with clustering (we have a private alpha of that now)

And then expand on the query side of history.

Traps: clustering and history will be interesting;)

Cheers,
/Tobias

>
> Petri
>
> lauantai 30. tammikuuta 2016 20.35.28 UTC+2 Tobias Oberstein kirjoitti:
>
> Hi,
>
> just released new versions of Crossbar.io and Autobahn|Python
>
> https://pypi.python.org/pypi/crossbar/0.12.1

> <https://pypi.python.org/pypi/crossbar/0.12.1>
> https://pypi.python.org/pypi/autobahn/0.12.1

> <https://cbdemo-us-west-1.crossbar.io/>
> https://cbdemo-eu-central-1.crossbar.io/

> <https://cbdemo-eu-central-1.crossbar.io/>
>
> When you open a demo, you should see a statusline at the bottom like
>
> Status: Connected to node cbdemo-eu-central-1 at
> wss://cbdemo-eu-central-1.crossbar.io/ws

> <http://cbdemo-eu-central-1.crossbar.io/ws>

>
> that tells you to which instance you are connected.
>
> Note that the instances are not yet clustered (eg when you are
> connected
> to the Votes demo on instance A, you won't get votes on instance B).
>
> ---
>

> --
> You received this message because you are subscribed to the Google
> Groups "Crossbar" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to crossbario+...@googlegroups.com
> <mailto:crossbario+...@googlegroups.com>.
> To post to this group, send email to cross...@googlegroups.com
> <mailto:cross...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/crossbario/b801d884-5d71-4d5f-8e03-bfe0fd1b9681%40googlegroups.com
> <https://groups.google.com/d/msgid/crossbario/b801d884-5d71-4d5f-8e03-bfe0fd1b9681%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Steve Waterbury]

I notice that crossbar development is set up to get coverage
reports -- what do you use for testing? I am currently using
Twisted's trial as my test runner and am looking into using
coverage.py with it, but would like to get advice from the
crossbar developers team on how to do it, since I will be
integrating autobahn and crossbar into my app very soon.

TIA for any insights ... :)

Cheers,
Steve

[Elvis Stansvik]

2016-02-11 18:53 GMT+01:00 Steve Waterbury <wate...@pangalactic.us>:
> I notice that crossbar development is set up to get coverage
> reports -- what do you use for testing? I am currently using
> Twisted's trial as my test runner and am looking into using
> coverage.py with it, but would like to get advice from the
> crossbar developers team on how to do it, since I will be
> integrating autobahn and crossbar into my app very soon.

I also used Trial and coverage.py for a project, but it was some time
ago. The successResultOf and failureResultOf from Trial were
particularly useful. I also used the mock package to mock isolate the
functionality under test as much as possible, and to test that
appropriate things were called.

If you also write custom Twisted protocol classes (e.g. for
interfacing with hardware or doing other communication), I can also
recommend twisted.test.proto_helpers.StringTransport and
twisted.internet.task.Clock. The former for mocking the underlying
transport, and the latter to mock time (e.g. to test timeouts).

That's all I can remember for now :)

One nice thing with coverage.py is that PyCharm supports it, so you
can have it highlight uncovered lines of code right in the editor
margin when running tests.

Elvis

>
> TIA for any insights ... :)
>
> Cheers,
> Steve
>

> --
> You received this message because you are subscribed to the Google Groups

> "Autobahn" group.

> To unsubscribe from this group and stop receiving emails from it, send an

> email to autobahnws+...@googlegroups.com.
> To post to this group, send email to autob...@googlegroups.com.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/autobahnws/56BCCA95.8020309%40pangalactic.us.

[Elvis Stansvik]

2016-02-11 19:05 GMT+01:00 Elvis Stansvik <elvs...@gmail.com>:
> 2016-02-11 18:53 GMT+01:00 Steve Waterbury <wate...@pangalactic.us>:
>> I notice that crossbar development is set up to get coverage
>> reports -- what do you use for testing? I am currently using
>> Twisted's trial as my test runner and am looking into using
>> coverage.py with it, but would like to get advice from the
>> crossbar developers team on how to do it, since I will be
>> integrating autobahn and crossbar into my app very soon.
>
> I also used Trial and coverage.py for a project, but it was some time
> ago. The successResultOf and failureResultOf from Trial were
> particularly useful. I also used the mock package to mock isolate the
> functionality under test as much as possible, and to test that
> appropriate things were called.
>
> If you also write custom Twisted protocol classes (e.g. for
> interfacing with hardware or doing other communication), I can also
> recommend twisted.test.proto_helpers.StringTransport and
> twisted.internet.task.Clock. The former for mocking the underlying
> transport, and the latter to mock time (e.g. to test timeouts).
>
> That's all I can remember for now :)
>
> One nice thing with coverage.py is that PyCharm supports it, so you
> can have it highlight uncovered lines of code right in the editor
> margin when running tests.

To clarify: I never wrote any integration tests that fire up a full
WAMP router while running the tests. I only tested that the correct
registrations, publications et.c. were made (by mocking), and then
unit tested the handlers. In effect I treated Autobahn and Crossbar as
"friendlies" that should work (tm).

Elvis

[Tobias Oberstein]

Am 11.02.2016 um 18:53 schrieb Steve Waterbury:
> I notice that crossbar development is set up to get coverage
> reports -- what do you use for testing? I am currently using
> Twisted's trial as my test runner and am looking into using
> coverage.py with it, but would like to get advice from the
> crossbar developers team on how to do it, since I will be
> integrating autobahn and crossbar into my app very soon.

We are using Tox, Travis and Coverage

http://coverage.readthedocs.org/

Unit testing is done using Twisted trial

https://github.com/crossbario/crossbar/blob/master/tox.ini#L35

The stuff above is rendered into some nice reports:

https://travis-ci.org/crossbario/crossbar
https://travis-ci.org/crossbario/autobahn-python
https://travis-ci.org/crossbario/txaio

https://codecov.io/github/crossbario/crossbar
https://codecov.io/github/crossbario/autobahn-python
https://codecov.io/github/crossbario/txaio

Then, Autobahn is tested at the WebSocket level using a fuzzing test
approach: http://autobahn.ws/testsuite/

Then, we have an internal test system that is able to do
integration/functional testing of Crossbar.io in distributed setups.

Testing of _applications_ built on WAMP and Crossbar.io, and with
multiple, distributed components: this is yet another area. One approach
is to test each component individually, but via a router. That is, test
cases call into the tested component, and observe results and events,
and check versus "known good". Testing the interplay between different
components (integration testing) is the next level.

Cheers,
/Tobias