Asynchronous approach for processing message/object queue
251 views
Skip to first unread message
Psycho Punch
Jan 16, 2017, 12:30:34 PM1/16/17
to vert.x
Hello everyone.
So far, my experience with Vert.x is mostly limited to intercommunication among separate embedded Vert.x applications communicating together through the clustered event bus. Most of the code are still pretty much synchronous in nature, which I know wastes the potential of Vert.x.
Right now however, I'm trying to figure out how to design a message queue-like component using asynchronous processing. The problem: I have a list of objects/data I need to retrieve remotely. Once I have all of them, I want to process every one of them based on some given rules/requirements. Before, to address this problem using concurrency, I created a component that fetches the list of objects and put them into a thread-safe queue. At the same time, I have several worker components, each running on their own thread, that polls the queue and applies the rules to whatever data they retrieve until there aren't any object in the queue left.
In terms of a single-threaded Vert.x application, I'm not really sure how this can be done. I'm thinking of 2 approaches: 1) a similar approach employing shared queue, 2) message passing through the event bus.
Approach 1: shared queue
In this approach, what I'm thinking of doing is to define 2 Verticles (FetchVerticle, ProcessVerticle). Both Verticles will have access to a shared (unbounded?) queue to where FetchVerticle adds a collection of data/object, and from where ProcessVerticle gets the data to process. Both fetching and processing will most likely take a long time to do, so I will have to mark them as blocking (via executeBlocking).
Approach 2: message passing
In this approach, I'll also have 2 Verticles (FetchVerticle, ProcessVerticle) but the queue will only be handled internally by FetchVerticle. For this I will probably have several instances of ProcessVerticle either running as workers, or again running blocking code. However instead of directly getting their data from shared queue, they wait for the FetchVerticle to pass a message to any one of them (via EventBus' send method). One by one, FetchVerticle will send the contents of its queue to any of the ProcessVerticle instances.
Which of the 2 do you think fits the Vert.x paradigm better? Are there more sensible way in the realm of Vert.x to do what I want to do here?
So far, my experience with Vert.x is mostly limited to intercommunication among separate embedded Vert.x applications communicating together through the clustered event bus. Most of the code are still pretty much synchronous in nature, which I know wastes the potential of Vert.x.
Right now however, I'm trying to figure out how to design a message queue-like component using asynchronous processing. The problem: I have a list of objects/data I need to retrieve remotely. Once I have all of them, I want to process every one of them based on some given rules/requirements. Before, to address this problem using concurrency, I created a component that fetches the list of objects and put them into a thread-safe queue. At the same time, I have several worker components, each running on their own thread, that polls the queue and applies the rules to whatever data they retrieve until there aren't any object in the queue left.
In terms of a single-threaded Vert.x application, I'm not really sure how this can be done. I'm thinking of 2 approaches: 1) a similar approach employing shared queue, 2) message passing through the event bus.
Approach 1: shared queue
In this approach, what I'm thinking of doing is to define 2 Verticles (FetchVerticle, ProcessVerticle). Both Verticles will have access to a shared (unbounded?) queue to where FetchVerticle adds a collection of data/object, and from where ProcessVerticle gets the data to process. Both fetching and processing will most likely take a long time to do, so I will have to mark them as blocking (via executeBlocking).
Approach 2: message passing
In this approach, I'll also have 2 Verticles (FetchVerticle, ProcessVerticle) but the queue will only be handled internally by FetchVerticle. For this I will probably have several instances of ProcessVerticle either running as workers, or again running blocking code. However instead of directly getting their data from shared queue, they wait for the FetchVerticle to pass a message to any one of them (via EventBus' send method). One by one, FetchVerticle will send the contents of its queue to any of the ProcessVerticle instances.
Which of the 2 do you think fits the Vert.x paradigm better? Are there more sensible way in the realm of Vert.x to do what I want to do here?
Jochen Mader
Jan 16, 2017, 2:31:09 PM1/16/17
to ve...@googlegroups.com
Approach 1: Don't do this. Shared state is the ONE thing we want to avoid. The whole of Vert.x centers around the idea of not sharing state.
Approach 2: That's the one you want to use. It scales easily, no manual syncing involved. You can spread the processing out across multiple machine. All the Vert.x goodness :)--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/ac14d416-851d-4861-b571-11a6e25ef9db%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
Jochen Mader | Lead IT Consultant
tel: +49 89 215486633 | fax: +49 89 215486699 | mobil: +49 152 51862390
www.codecentric.de | blog.codecentric.de | www.meettheexperts.de | www.more4fi.de
Sitz der Gesellschaft: Düsseldorf | HRB 63043 | Amtsgericht Düsseldorf
Vorstand: Michael Hochgürtel . Rainer Vehns
Aufsichtsrat: Patric Fedlmeier (Vorsitzender) . Klaus Jäger . Jürgen Schütz
Julien Viet
Jan 17, 2017, 2:44:11 AM1/17/17
to ve...@googlegroups.com
As Jochen said, the 2/ option is more Vert.x-y
Julien
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/CAAn-prVtbsRtv4TsbWwxSa2KV77jx86DFWqRD%3DOxQ-xqsufZ5Q%40mail.gmail.com.
Psycho Punch
Jan 17, 2017, 7:16:25 AM1/17/17
to vert.x
Jochen, Julien,
Thanks for your inputs. I'll go with approach 2 then.
Thanks for your inputs. I'll go with approach 2 then.
javadevmtl
Jan 18, 2017, 4:46:56 PM1/18/17
to vert.x
Speaking of blueprints? Julien can blue print links be put as part of the official documentation on the site?
I don't see them...???
I don't see them...???
jklingsporn
Jan 20, 2017, 8:02:34 AM1/20/17
to vert.x
+1
I found them recently 'by accident' and found them pretty useful as well (found them one the awesome-page).
Julien Viet
Jan 20, 2017, 8:58:18 AM1/20/17
to ve...@googlegroups.com
can you open an issue on the web-site project ? https://github.com/vert-x3/vertx-web-site/issues
--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/3ee9a653-8e01-4311-b0ed-26d00f816dbc%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages