need some design decision

21 views
Skip to first unread message

suma...@gmail.com

unread,
Mar 4, 2016, 5:08:29 AM3/4/16
to actionHero.js
Hi all,

I would like to create one high performance API server for sensor data.
There are a need for some functionality which needs multiple data store hits and combine the results from all.

I do understand, it can be written all functions inside action/initializer but have some question?

is AH block the connection until results comes or it just work like use connection for other request to save server resources?


Is it correct to write diff function in TASK and return task response to action, combine and return back to user? I was thinking with 'Map - reduce'
style.

or

please suggest me a good approach of such case

Thanks
Suman

suma...@gmail.com

unread,
Mar 4, 2016, 6:07:25 AM3/4/16
to actionHero.js, suma...@gmail.com
Hi,

I was thinking if "q" module can be implemented inside action.

ref: https://github.com/kriskowal/q

please help me to find out the right implementation strategy for this kind of use case?


Thanks
Suman

Evan Tahler

unread,
Mar 4, 2016, 12:18:23 PM3/4/16
to actionHero.js, suma...@gmail.com
I like async, which is much like Q.  
Either way, you can have long-lasting actions which do a lot of things if your client can stand to wait for it.  Node/actionhero can handle many requests at once, so as long as your actions are not CPU-bound (and are just waiting for data sources), it seems overly complex to offload the logic to a task.   

Clay Himmelberger

unread,
Mar 4, 2016, 12:21:56 PM3/4/16
to actionHero.js, suma...@gmail.com
Hey Suman,

Actionhero will work well for your multiple concurrent connections, and in particular sensor connections that could be HTTP requests or TCP. 

An action in actionhero is processed independently of other actions, and will return when it's processing is complete. Since this is NodeJS, lots of things could happen in that action (including a q-promise chain), and the action is "resolved" when it's final callback is called. For a HTTP request, that action will be held up until that processing completes, and another HTTP request could be requested by the client at the same time (they would process independently). Same is true for a TCP connection, though that TCP connection could have multiple concurrent actions come in at the same time. They would be processed independently as well.

It is not a good idea to try to have a Task process as a dependency of Action processing. They are two separate things, and the coordination would be awkward. Actions should be independent that have a fixed response. Using a persistent connection server (WebSockets or TCP), you could have Tasks push data/results to connections as they become available. Those messages to a client would not be kicked off by an Action, but strictly server-side.

I like to keep shared logic in Initalizers, and then can have Actions or Tasks use that logic. Your promises could live there (in Initializers), and an Action or Task could kick them off and use their responses. 

Hope that was helpful to you,
Clay

Michael Jensen

unread,
Mar 4, 2016, 2:14:38 PM3/4/16
to Clay Himmelberger, actionHero.js, suma...@gmail.com
Hi Suman,

If you're familiar with promises, you might like to look into Reactive Programming, RxJS. Reactive "Observables" behave similarly to promises, except that they represent a stream of data (perhaps a stream of updated measurements from sensors?) rather than a single promised value.

Netflix is a big proponent of Reactive Programming, and they use it for composing responses and workflows from multiple data sources. You can probably find a bunch of talks by Netflix guys on RX.

Good luck with your project!

--
You received this message because you are subscribed to the Google Groups "actionHero.js" group.
To unsubscribe from this group and stop receiving emails from it, send an email to actionhero-j...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

suma...@gmail.com

unread,
Mar 4, 2016, 10:54:42 PM3/4/16
to actionHero.js, suma...@gmail.com
Hi Evan,

Thanks for pointing out. I am very new to AH and excited to propose this for our new API server. Is it possible to return back task response to action?

Thanks

suma...@gmail.com

unread,
Mar 5, 2016, 2:22:18 AM3/5/16
to actionHero.js, suma...@gmail.com
Hi Clay,

Thanks for details discussion. Let me tell about my project.

We use elasticsearch for data store and for some basic analytics.So every sensor and associated data would have GET/POST/PUT/DELETE API, search API, analytics API. I also want to expose API for group sensor, prediction etc.

Now my main analytics server are in python and R. python and R fetches data from es and does analytics and return results to user. Sometimes it is needed to chain analytics algo to get desired result.

Example like do feature engg, create model and train and test. there are some other services which is also consumes data and analytics results too.

Data insertion could be happen using http/tcp/websocket/csv upload/mqtt protocol. So large number of devices is expected to connect all time. Also many concurrent sensor data posting and retrieving can be happen. We have some working system written with spring and runs over tomcat. It lacks performance and capability and many area. I was trying to think a new architecture for API server which can handle all scale factor and gives best performance.


We do have visualization tool which is based on node.js and angularJS.

Hope you understood my usecase. I am considering AH as once data comes or analytics results comes, I need to send same to many different channel.


So what you have pointed out, i will think on same line. If anything could be done, please let me know.

thanks
suman

suma...@gmail.com

unread,
Mar 5, 2016, 2:24:33 AM3/5/16
to actionHero.js, clay.him...@gmail.com, suma...@gmail.com
Dear Michael,

thanks, I am looking on this.I have given my project req. Please have a look on this.

Thanks
suman
Reply all
Reply to author
Forward
0 new messages