WT Publish / Subscribe Mechanism

61 views
Skip to first unread message

Hynek Urban

unread,
Sep 6, 2018, 11:00:07 AM9/6/18
to windi...@googlegroups.com
Hello,

"real-time availability updates" is one of the prominent items on the Winding Tree roadmap. I have started to do some thinking and research about it recently, as well as discussed some options with Jirka and Robin. Let me use this opportunity to propose an approach to real time updates. Ideally, if we can come to a common agreement, we can start with the implemention to get this out of the way.

Problem formulation

We can expect that some hotel information will be very short-lived in nature. That includes availability data but it might also include other information, such as price data. How can a WT data consumer keep track of all the changes, without resorting to polling all resources all the time?

Solution proposal

I propose the following:

1. A web interface for a service will be specified that enables:

    a. Data publishers to publish a notification about an update.
    b. Data consumers to subscribe for such notifications.

2. A reference implementation of a publish / subscribe system in node.js will be created.

3. The hotel data model will be enriched by an optional reference to an instance of this service. It is the hotel's responsibility to push update notifications there.

4. Consumers can subscribe for notifications on the given address.

A few additional remarks:

- The initial interface specification proposal is here: https://raw.githubusercontent.com/windingtree/wt-updates-api/master/docs/swagger.yaml

- The solution is decentralized in nature as it allows multiple independent publish / subscribe channel providers to coexist and be easily discovered via the WT index.

- Since only notifications about updates are published, we probably do not need to implement publisher authentication or authorization of any kind, at least initially, as the possibilities of abuse are fairly limited. If we neeed it, though, it would probably be possible to re-use the same keys that are used for Ethereum transactions.

Open questions

I. Service architecture

A natural way of implementing such a service would work with four subcomponents:

1. The API server itself that accepts connections.
2. A shared database where data about webhooks is stored.
3. A message queue where the API server relays notifications to be broadcast further.
4. Workers that take notifications from the message queue and distribute them to consumers based on webhooks in the shared database.

We should think about whether this is the right way with respect to our existing deployment approaches as well as ease of deployment and operation for future community members. Perhaps there is a way to build this monolithically?

II. Management of subscriptions

We should consider whether there is a danger of malevolent actors (over-)utilizing subscriptions to make requests "on their behalf" and if so, how to prevent it.

Also, we should come up with a way of cancelling subscriptions, even if the subscriber loses his or her credentials.

Alternatives

Considering the objective of "decentralization", we have also thought about having a decentralized network of mutually connected nodes in one big publish / subscribe system. Such an approach would have a number of advantages:

- Easier to use for both consumers and producers (no need to handle a multitude of publish / subscribe providers).
- Better redundancy.
- Possibly better latency (under certain assumptions).

There are two basic approaches to the implementation:

1. Utilize an existing distributed message passing network, such as Ethereum Whisper.

2. Build our own network from the ground up, perhaps using tools like libp2p.

While the main problem of the former option is immaturity of the networks available, the latter one would be significantly more laborious than the proposed solution. Therefore, I think that the proposed solution offers the best tradeoff between implementation difficulty, development risks and desirable features.

Finishing note

Let me know what you think - be it high-level or specific to the proposad API interface. Every and all feedback is welcome.

Thanks,
Hynek

Jirka Chadima

unread,
Sep 10, 2018, 5:55:02 AM9/10/18
to Winding Tree
Hello, great write up, Hynek!

I have just a couple of points that crossed my mind when I was reading your proposal:

1) Authentication - I can easily see a DDOS-like or flood attack vector if the network is completely free of authentication. That's probably something to worry about for the future.
2) Authorization - I think we have to propose a way of verifying the incoming data. I can imagine signatures or some model of "ownership" based on the data file tree spanning from the hotel on-chain record. Although this adds complexity and coupling and potentially slows things down a lot.
3) I like the idea of re-using an existing communication protocol such as whisper. The overall design should enable easy replacement of its parts.
4) The API design is sound, let's keep it as simple as possible

I can see a great potential in a service like this if we can keep it independent on Winding Tree.

Thanks for working on this topic,
Jirka Chadima

Hynek Urban

unread,
Sep 10, 2018, 6:30:27 AM9/10/18
to Winding Tree
Hi Jirka,

thanks for reading the proposal; let me answer your individual points.

1) This could probably be implemented in the next step as outlined above ("(...) it would probably be possible to re-use the same keys that are used for Ethereum transactions (...)").
2) The idea is that the incoming data is pretty much "empty" - it's just "a notification about an update", not an update itself. In other words, the consumer will just be notified that he / she should download the updated data. The data itself will not be propagated through this channel. It seems to me that this alleviates the need for data verification.
3) In my view, whisper would pretty much be an alternative to this, not an implementation detail. Once we go the API route, there's no place for Whisper - and vice versa - if we go the Whisper route, we don't need a dedicated service (we just decide about the protocol on the Whisper level).

Also, I'm afraid the interface semantics it is very much dependent on Winding Tree, at least for now. Building a "generic decentralized publish / subscribe system" seems like a needlessly gargantuan task at this stage.

Thanks for the feedback!

Hynek

--
You received this message because you are subscribed to the Google Groups "Winding Tree" group.
To unsubscribe from this group and stop receiving emails from it, send an email to windingtree+unsubscribe@googlegroups.com.
To post to this group, send email to windi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/windingtree/3a5299b5-df0f-49ed-b91a-c09832f45fa5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Jirka Chadima

unread,
Sep 11, 2018, 4:03:46 AM9/11/18
to Hynek Urban, Winding Tree
Hello,

1) I agree with that approach.
2) Yes, we don't need that for data integrity. I meant it as a possible way of mitigation flooding attacks. By allowing only proven owners to publish the notifications.
3) I believe there might be a place for it even with this architecture, it's just a matter of replacing one component (communication channel) with another. Or maybe I'm wrong.

Maybe I am missing something, but I think that notifying parties about URL changes is not really tied to WT's use case. Is it? Also, if we want to re-use this solution for other parts of the platform (airlines, shuttles, anything) - which we want, it shouldn't be too hotel-specific.




To unsubscribe from this group and stop receiving emails from it, send an email to windingtree...@googlegroups.com.

To post to this group, send email to windi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/windingtree/3a5299b5-df0f-49ed-b91a-c09832f45fa5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Winding Tree" group.
To unsubscribe from this group and stop receiving emails from it, send an email to windingtree...@googlegroups.com.

To post to this group, send email to windi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/windingtree/CAFEpYkSyopeYj1dOucBr6OTH983vbS-nt_YE3Kmn7qsAC%3DyFVw%40mail.gmail.com.

Hynek Urban

unread,
Sep 11, 2018, 4:20:39 AM9/11/18
to Jirka Chadima, Winding Tree
Hi,

as far as Whisper goes, my understanding is that you can use it via web3 (web3.shh), similarly to Ethereum itself (or Swarm). You configure a "provider" and then communicate with it (publish messages, subscribe to messages). As such, it replaces the need for a dedicated publish / subscribe backend (i.e. the newly created service). That's why it seems to me these two are alternives rather than complementaries.

On the other hand, it would *probably* be possible to base the communication between nodes on Whisper in the future, if we wanted to somehow organize all instances of the API servers to form a single network with shared publish / subscribe channels. So maybe that's what you meant from the start :)

Regarding WT-specificity, there's a tradeoff between flexibility and usability, as usual. In this specific example, theoretically, getting rid of all references to hotels or hotel data structure would be possible. However, as a result, in my view, it would be hard to configure it properly on both publisher / consumer end with no way of being sure if the configuration is right. In effect, making this an unreliable channel from the point of view of consumers.

That's why I prefer to keep it a bit tighter by validating things being consumed / subscribed to with WT specifics in mind. I think it should be rather easy to organically extend it with other use cases (airlines, ...) once the need arises. I'll try to change the API a little bit to reflect that.

Regards,
Hynek



To unsubscribe from this group and stop receiving emails from it, send an email to windingtree+unsubscribe@googlegroups.com.

To post to this group, send email to windi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/windingtree/3a5299b5-df0f-49ed-b91a-c09832f45fa5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Winding Tree" group.
To unsubscribe from this group and stop receiving emails from it, send an email to windingtree+unsubscribe@googlegroups.com.

To post to this group, send email to windi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/windingtree/CAFEpYkSyopeYj1dOucBr6OTH983vbS-nt_YE3Kmn7qsAC%3DyFVw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Jakub Vysoky

unread,
Sep 11, 2018, 4:36:50 AM9/11/18
to Hynek Urban, Jirka Chadima, Winding Tree
It would be awesome if we can figure out how any of the already built solutions can be utilised either temporarily of as a final solution, eg Kafka.

I would love to understand the architecture of the servise more in depth :)) Great conversation here!

To unsubscribe from this group and stop receiving emails from it, send an email to windingtree...@googlegroups.com.

To post to this group, send email to windi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/windingtree/3a5299b5-df0f-49ed-b91a-c09832f45fa5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Winding Tree" group.
To unsubscribe from this group and stop receiving emails from it, send an email to windingtree...@googlegroups.com.

To post to this group, send email to windi...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Winding Tree" group.
To unsubscribe from this group and stop receiving emails from it, send an email to windingtree...@googlegroups.com.

To post to this group, send email to windi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/windingtree/CAFEpYkSC6PSs249i_1ktAg9ObBxFo-4jG10M48FAVa76Ta4TMw%40mail.gmail.com.

Hynek Urban

unread,
Sep 11, 2018, 4:47:43 AM9/11/18
to Jakub Vysoky, Jirka Chadima, Winding Tree
Hi Jakub,

I personally do not see a place for Kafka or similar technologies at the moment at all. It could theoretically serve as a message queue to connect the API server with workers in a single instance of our service but, at least initially, I think that would be overkill and it would significantly higher the bar for anyone trying to deploy and operate this service.

If your thoughts were about using Kafka "standalone", as an existing publish / subscribe solution, then I also think that would be a little too flexible (sounds like a paradox, but I believe it has a merit :) ) and harder to operate.

However, if you disagree, have any specific questions or remarks about some aspects of the planned service, let me know for sure.

Thanks,
Hynek


To unsubscribe from this group and stop receiving emails from it, send an email to windingtree+unsubscribe@googlegroups.com.

To post to this group, send email to windi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/windingtree/3a5299b5-df0f-49ed-b91a-c09832f45fa5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Winding Tree" group.
To unsubscribe from this group and stop receiving emails from it, send an email to windingtree+unsubscribe@googlegroups.com.

To post to this group, send email to windi...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Winding Tree" group.
To unsubscribe from this group and stop receiving emails from it, send an email to windingtree+unsubscribe@googlegroups.com.

To post to this group, send email to windi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/windingtree/CAFEpYkSC6PSs249i_1ktAg9ObBxFo-4jG10M48FAVa76Ta4TMw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

Jakub Vysoky

unread,
Sep 11, 2018, 4:55:30 AM9/11/18
to Hynek Urban, Jirka Chadima, Winding Tree
Oh, no disagreements here! Great ideas over there! ;))

It sounds it could be a very useful generic service. That is why I am looking if there are no ready made solutions. Or people who could help us design or build it!

Cheers!

To unsubscribe from this group and stop receiving emails from it, send an email to windingtree...@googlegroups.com.

To post to this group, send email to windi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/windingtree/3a5299b5-df0f-49ed-b91a-c09832f45fa5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Winding Tree" group.
To unsubscribe from this group and stop receiving emails from it, send an email to windingtree...@googlegroups.com.

To post to this group, send email to windi...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Winding Tree" group.
To unsubscribe from this group and stop receiving emails from it, send an email to windingtree...@googlegroups.com.

To post to this group, send email to windi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/windingtree/CAFEpYkSC6PSs249i_1ktAg9ObBxFo-4jG10M48FAVa76Ta4TMw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

Hynek Urban

unread,
Sep 11, 2018, 4:56:19 AM9/11/18
to Jakub Vysoky, Jirka Chadima, Winding Tree
Good :) One more thing - I've realized some (temporary?) decisions about the service architecture that were made in the meantime have not been mentioned in this thread.

So, to clarify - for now we assume the reference implementation will be a "monolith", i.e. a single node.js process handling all the three aspects: API server, in-memory message queue as well as asynchronous propagation of updates to subscribed consumers. This should make things easy to grasp and deploy and even though it has it's warts (scalability, mostly), I believe the properties of the node.js runtime make it actually a reasonably good fit.

In the second iteration, based on what's most pressing and what has a priority, we could of course make this more scalable by separating these different concerns further.

Hope that answers your original question, Jakub.

Regards,
Hynek

To unsubscribe from this group and stop receiving emails from it, send an email to windingtree+unsubscribe@googlegroups.com.

To post to this group, send email to windi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/windingtree/3a5299b5-df0f-49ed-b91a-c09832f45fa5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Winding Tree" group.
To unsubscribe from this group and stop receiving emails from it, send an email to windingtree+unsubscribe@googlegroups.com.

To post to this group, send email to windi...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Winding Tree" group.
To unsubscribe from this group and stop receiving emails from it, send an email to windingtree+unsubscribe@googlegroups.com.

To post to this group, send email to windi...@googlegroups.com.

Jirka Chadima

unread,
Sep 11, 2018, 4:59:11 AM9/11/18
to Hynek Urban, Jakub Vysoky, Winding Tree
Ad Whisper - You are right, I meant it as a means of communication between nodes (or in extention as a one of possible means of communication between the service and its consumers - on par with http callbacks, websockets etc.). It didn't cross my mind that we can utilize it for the whole service. Interesting thought, but I wouldn't go that far.

WT-specifity - let's do it in the easiest and most secure way possible in the first iteration. We can either relax the conditions or make them configurable (for example) later. I can see your point.

Kafka - or any other existing queuing solution for that matter - We have talked about it and there is a lot to consider (for example replication in Kafka was not really designed to work well over generic networks, i. e. internet). For the internal use in the first iteration (server with workers), I also think it's an overkill.

Maybe we can draw some schema of the intended design to make this more clear to a wider audience. There is obviously a lot of moving parts and some parts might even be sharing the same names. It would be also really awesome if we were able to describe which parts could get replaced in the future (for example because of scaling or improved performance).

As a matter of fact, it kind of reminds me of a hosted RSS reader (which somehow gets notified from content owners) with push notifications for every user. Or maybe it's even a little bit like Twitter :-)
To unsubscribe from this group and stop receiving emails from it, send an email to windingtree...@googlegroups.com.

To post to this group, send email to windi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/windingtree/3a5299b5-df0f-49ed-b91a-c09832f45fa5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Winding Tree" group.
To unsubscribe from this group and stop receiving emails from it, send an email to windingtree...@googlegroups.com.

To post to this group, send email to windi...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Winding Tree" group.
To unsubscribe from this group and stop receiving emails from it, send an email to windingtree...@googlegroups.com.

To post to this group, send email to windi...@googlegroups.com.

Jakub Vysoky

unread,
Sep 11, 2018, 5:11:39 AM9/11/18
to Jirka Chadima, Hynek Urban, Winding Tree
Totally agree with a diagram that could help us figuring out the parts that can be replaced with other technologies, etc...
--

Hynek Urban

unread,
Sep 11, 2018, 6:57:44 AM9/11/18
to Jakub Vysoky, Jirka Chadima, Winding Tree
Hello again,

for purposes of discussion, see the attached informal diagram with assumed data flow attached to this e-mail. The general idea is as follows.

The producer (i.e. hotel manager):

1. Selects a WT Update service they would like to use (remember, this is supposed to be decentralized so we expect multiple instances of WT Update service to exist).
2. Publishes the URL of the selected WT Update service to blockchain.
3. Starts sending update notifications there.

The consumer (e.g. an online travel agency):

1. Finds out the service URL of interest from the WT Index on the blockchain.
2. Subscribes for notifications there.
3. From then on, retrieves notifications on a push basis.

Initially, the three principal components of the WT Update Service - API Server, Message Queue and Workers - will be implemented as a monolitihical node.js program, as outlined in the previous e-mails. This should make distribution and operation of the service very easy (for example via a single Dockerfile).

If we want to, though, we will be able to split the whole thing into individual parts for scalability reasons later on.

Another idea that floats in the air is to somehow connect all possible instances of the WT Updates service into a single network of nodes that would relay published notifications where needed. That step would lie in a rather distant future, though, *if* we deem it worthwhile.

Hope this makes the assumed solution a bit clearer,
Hynek



To unsubscribe from this group and stop receiving emails from it, send an email to windingtree+unsubscribe@googlegroups.com.

To post to this group, send email to windi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/windingtree/3a5299b5-df0f-49ed-b91a-c09832f45fa5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Winding Tree" group.
To unsubscribe from this group and stop receiving emails from it, send an email to windingtree+unsubscribe@googlegroups.com.

To post to this group, send email to windi...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Winding Tree" group.
To unsubscribe from this group and stop receiving emails from it, send an email to windingtree+unsubscribe@googlegroups.com.

To post to this group, send email to windi...@googlegroups.com.
--
wt-updates.png

hynek...@fragaria.cz

unread,
Sep 27, 2018, 10:07:38 AM9/27/18
to Winding Tree
Hello,

to gracefully finish this thread, let me state that the publish / subscribe mechanism has been fully specified, a reference impleentation was created and was deployed in the Playground environment. Further details as well as source code of the reference implementation are at https://github.com/windingtree/wt-update-api.

All the best,
Hynek

Jakub Vysoky

unread,
Sep 28, 2018, 12:35:09 AM9/28/18
to Hynek Urban, Winding Tree
:clap: :clap: :clap: @hynek - that sounds pretty amazing!

To unsubscribe from this group and stop receiving emails from it, send an email to windingtree...@googlegroups.com.

To post to this group, send email to windi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/windingtree/3a5299b5-df0f-49ed-b91a-c09832f45fa5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Winding Tree" group.
To unsubscribe from this group and stop receiving emails from it, send an email to windingtree...@googlegroups.com.

To post to this group, send email to windi...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Winding Tree" group.
To unsubscribe from this group and stop receiving emails from it, send an email to windingtree...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Winding Tree" group.
To unsubscribe from this group and stop receiving emails from it, send an email to windingtree...@googlegroups.com.

To post to this group, send email to windi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/windingtree/8937703b-4b29-427c-b8a1-bf1db980be62%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.


--
Reply all
Reply to author
Forward
0 new messages