getting results back
Marius Seritan
A1. rails application executes through passenger
A2. rails application hosts some nanite mappers
A3. rails application generates a work request that goes to another
server and takes 10 minutes
A4. when the workunit is done there are some changes to be made to the
database
I currently pass a block to Nanite.request() to handle the tasks in
item A4. I am a bit unclear of what happens if the original process
died and a new one was started. My guess is that there will be no
block/closure associated with that job and my cleanup won't be
processed. Is that true?
Assuming I am right above I am thinking about using a different
approach:
B1. create a new agent
B2. when work is completed in step A3 have the agent create a new work
request for 4
I am wondering if I should try to host this agent into a Passenger
process or start a new process. Running into a passenger process may
interfere with the regular website. Running into a different process
requires a good part of the existing rail bits (database, email
templates) so one approach is to just run the rail app as a mongrel.
Thank your for sharing any thoughts you may have on this...
Marius
Mathias Meyer
Spam folder and saw it was unanswered.
On Sun, Sep 27, 2009 at 11:28 PM, Marius Seritan <lov...@gmail.com> wrote:
>
> I have the following scenario:
>
> A1. rails application executes through passenger
> A2. rails application hosts some nanite mappers
> A3. rails application generates a work request that goes to another
> server and takes 10 minutes
> A4. when the workunit is done there are some changes to be made to the
> database
>
> I currently pass a block to Nanite.request() to handle the tasks in
> item A4. I am a bit unclear of what happens if the original process
> died and a new one was started. My guess is that there will be no
> block/closure associated with that job and my cleanup won't be
> processed. Is that true?
>
That is correct. So far, jobs sent out via request() are stored
locally (in the mapper process), i.e. the block associated with the
request is stored, and called when the agent is calling back to tell
that it's done. I'm not too happy with that situation, but I also try
to use push() instead of request() as much as possible. I've also
thought about moving that state into Redis, so that there is some way
to restore. The problem is that an agent's answer will always go
directly to the queue of the mapper that originally sent the request,
so it needs to be ensure that a mapper process replacing the process
picks up its identifier and continues where it left off. This scenario
would also work in a Passenger process when it's ensured that the
identity stays the same.
> Assuming I am right above I am thinking about using a different
> approach:
>
> B1. create a new agent
> B2. when work is completed in step A3 have the agent create a new work
> request for 4
>
> I am wondering if I should try to host this agent into a Passenger
> process or start a new process. Running into a passenger process may
> interfere with the regular website. Running into a different process
> requires a good part of the existing rail bits (database, email
> templates) so one approach is to just run the rail app as a mongrel.
>
A separate process would be the way to go for me. It also gets
problematic when you have a mapper running inside of that Passenger
process already, the two might interfere in some way. We have our
agents running separately, but using the Rails environment. That's
rather simple to achieve, you just fire up an agent by starting it
manually in a script, and before that requiring Rails' environment.rb
file.
Let me know, if I misunderstood something, or if you need some more
information on anything.
Cheers, Mathias
--
http://paperplanes.de
http://twitter.com/roidrage