Rhino ETL - Updating a row after processing
215 views
Skip to first unread message
Alex Brown
Jan 12, 2011, 12:29:47 PM1/12/11
to Rhino Tools Dev
Hi,
I'm currently writing a few etl processes to move rows from one table
to another.
Firstly, I have an ImportCommandOperation that is basically doing
"SELECT * FROM table WHERE Transferred = 0"
This is working fine.
Then, I have a write operation to insert the rows
Finally, I need an operation that Updates the source table, sets
Transferred to 1
How should I go about doing this?
I'm currently writing a few etl processes to move rows from one table
to another.
Firstly, I have an ImportCommandOperation that is basically doing
"SELECT * FROM table WHERE Transferred = 0"
This is working fine.
Then, I have a write operation to insert the rows
Finally, I need an operation that Updates the source table, sets
Transferred to 1
How should I go about doing this?
Jason Meckley
Jan 12, 2011, 3:14:21 PM1/12/11
to Rhino Tools Dev
get data and branch to insert and update
register("select ...");
register(new BranchingOperation()
.Add("insert into...")
.Add("update ..."));
register("select ...");
register(new BranchingOperation()
.Add("insert into...")
.Add("update ..."));
Nathan Palmer
Jan 12, 2011, 4:40:41 PM1/12/11
to rhino-t...@googlegroups.com
Why would you choose a branching operations instead of just 3 operations one after another?
register("select from original..");
register("insert into destination..");
register("update from original...");
--
You received this message because you are subscribed to the Google Groups "Rhino Tools Dev" group.
To post to this group, send email to rhino-t...@googlegroups.com.
To unsubscribe from this group, send email to rhino-tools-d...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/rhino-tools-dev?hl=en.
Alex Brown
Jan 12, 2011, 5:20:53 PM1/12/11
to Rhino Tools Dev
but what should the "update from original..." operation be?
An AbstractDatabaseOperation? InputCommandOperation?
On Jan 12, 9:40 pm, Nathan Palmer <snowpal...@gmail.com> wrote:
> Why would you choose a branching operations instead of just 3 operations one
> after another?
>
> register("select from original..");
> register("insert into destination..");
> register("update from original...");
>
> > .
An AbstractDatabaseOperation? InputCommandOperation?
On Jan 12, 9:40 pm, Nathan Palmer <snowpal...@gmail.com> wrote:
> Why would you choose a branching operations instead of just 3 operations one
> after another?
>
> register("select from original..");
> register("insert into destination..");
> register("update from original...");
>
> On Wed, Jan 12, 2011 at 1:14 PM, Jason Meckley <jasonmeck...@gmail.com>wrote:
>
>
>
>
>
>
>
> > get data and branch to insert and update
>
> > register("select ...");
> > register(new BranchingOperation()
> > .Add("insert into...")
> > .Add("update ..."));
>
> > On Jan 12, 12:29 pm, Alex Brown <a...@alexjamesbrown.com> wrote:
> > > Hi,
> > > I'm currently writing a few etl processes to move rows from one table
> > > to another.
>
> > > Firstly, I have an ImportCommandOperation that is basically doing
> > > "SELECT * FROM table WHERE Transferred = 0"
> > > This is working fine.
>
> > > Then, I have a write operation to insert the rows
>
> > > Finally, I need an operation that Updates the source table, sets
> > > Transferred to 1
>
> > > How should I go about doing this?
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Rhino Tools Dev" group.
> > To post to this group, send email to rhino-t...@googlegroups.com.
> > To unsubscribe from this group, send email to
> > rhino-tools-d...@googlegroups.com<rhino-tools-dev%2Bunsubscribe@ googlegroups.com>
>
>
>
>
>
>
>
> > get data and branch to insert and update
>
> > register("select ...");
> > register(new BranchingOperation()
> > .Add("insert into...")
> > .Add("update ..."));
>
> > On Jan 12, 12:29 pm, Alex Brown <a...@alexjamesbrown.com> wrote:
> > > Hi,
> > > I'm currently writing a few etl processes to move rows from one table
> > > to another.
>
> > > Firstly, I have an ImportCommandOperation that is basically doing
> > > "SELECT * FROM table WHERE Transferred = 0"
> > > This is working fine.
>
> > > Then, I have a write operation to insert the rows
>
> > > Finally, I need an operation that Updates the source table, sets
> > > Transferred to 1
>
> > > How should I go about doing this?
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Rhino Tools Dev" group.
> > To post to this group, send email to rhino-t...@googlegroups.com.
> > To unsubscribe from this group, send email to
> > .
Jason Meckley
Jan 12, 2011, 8:45:29 PM1/12/11
to Rhino Tools Dev
@nathan
because database operations terminate the return of rows.
@alex
I would use any one of the inputcommandoperations. I try to use the
conventionbulksqlcommandoperation as much as possible. but anyone will
do.
because database operations terminate the return of rows.
@alex
I would use any one of the inputcommandoperations. I try to use the
conventionbulksqlcommandoperation as much as possible. but anyone will
do.
Jason Meckley
Jan 12, 2011, 8:48:34 PM1/12/11
to Rhino Tools Dev
to be more precise output commands terminate the return of rows:
https://github.com/ayende/rhino-etl/blob/master/Rhino.Etl.Core/Operations/OutputCommandOperation.cs
https://github.com/ayende/rhino-etl/blob/master/Rhino.Etl.Core/Operations/OutputCommandOperation.cs
Nathan Palmer
Jan 13, 2011, 11:39:41 AM1/13/11
to rhino-t...@googlegroups.com
Yeah, I realized that after I sent the email. Although now that I'm thinking about it I wonder if it would be bad to yield the same row from the output command so you could keep chaining operations even after the output. Going back and updating a source row isn't too uncommon and for some reason the branching operation seems strange to me.
Nathan Palmer
To unsubscribe from this group, send email to rhino-tools-d...@googlegroups.com.
Jason Meckley
Jan 13, 2011, 12:26:38 PM1/13/11
to Rhino Tools Dev
the idea is rows get into the pipeline from source. rows are
processed. and rows leave the pipeline to a destination. It would be
more unnatural to have the row go to a destination, but remain in the
pipeline. this is where branching comes in.
On Jan 13, 11:39 am, Nathan Palmer <snowpal...@gmail.com> wrote:
> Yeah, I realized that after I sent the email. Although now that I'm thinking
> about it I wonder if it would be bad to yield the same row from the output
> command so you could keep chaining operations even after the output. Going
> back and updating a source row isn't too uncommon and for some reason the
> branching operation seems strange to me.
>
> Nathan Palmer
>
> > > > > > rhino-tools-d...@googlegroups.com<rhino-tools-dev%2Bunsu...@googlegroups.com>
processed. and rows leave the pipeline to a destination. It would be
more unnatural to have the row go to a destination, but remain in the
pipeline. this is where branching comes in.
On Jan 13, 11:39 am, Nathan Palmer <snowpal...@gmail.com> wrote:
> Yeah, I realized that after I sent the email. Although now that I'm thinking
> about it I wonder if it would be bad to yield the same row from the output
> command so you could keep chaining operations even after the output. Going
> back and updating a source row isn't too uncommon and for some reason the
> branching operation seems strange to me.
>
> Nathan Palmer
>
> On Wed, Jan 12, 2011 at 6:48 PM, Jason Meckley <jasonmeck...@gmail.com>wrote:
>
> > to be more precise output commands terminate the return of rows:
>
> >https://github.com/ayende/rhino-etl/blob/master/Rhino.Etl.Core/Operat...
>
> > to be more precise output commands terminate the return of rows:
>
> > <rhino-tools-dev%2Bunsubscribe@ googlegroups.com>
> > > > > > .
> > > > > > For more options, visit this group at
> > > > > >http://groups.google.com/group/rhino-tools-dev?hl=en.
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Rhino Tools Dev" group.
> > To post to this group, send email to rhino-t...@googlegroups.com.
> > To unsubscribe from this group, send email to
> > rhino-tools-d...@googlegroups.com<rhino-tools-dev%2Bunsu...@googlegroups.com>
> > > > > > .
> > > > > > For more options, visit this group at
> > > > > >http://groups.google.com/group/rhino-tools-dev?hl=en.
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Rhino Tools Dev" group.
> > To post to this group, send email to rhino-t...@googlegroups.com.
> > To unsubscribe from this group, send email to
Nathan Palmer
Jan 13, 2011, 2:14:45 PM1/13/11
to rhino-t...@googlegroups.com
I see what you are saying. At the same time the last step in the pipeline process is updating the original source to mark that it's been loaded. In this scenario our final step is updating the original, not loading it to the destination. It would make sense that we could chain these however would seem to fit the process that we're working on. The branching operation I visualize as this. Hopefully the ascii shows up somewhat correctly here.
Operation 1
|
/ \
Op2 Op3
But I never want Op3 to execute before Op2. Even though the whole operation might be isolated by a transaction. It would still make sense to see this.
Operation 1 -> Select from source
|
Operation 2 -> Insert into destination
|
Operation 3 -> Mark source as transferred
Nathan Palmer
To unsubscribe from this group, send email to rhino-tools-d...@googlegroups.com.
Jason Meckley
Jan 13, 2011, 3:39:33 PM1/13/11
to Rhino Tools Dev
op2 & op3 happen simultaneously because of the branching operation.
you may need to implement your own operation inheriting from database
operation and allow the records to flow through.
On Jan 13, 2:14 pm, Nathan Palmer <snowpal...@gmail.com> wrote:
> I see what you are saying. At the same time the last step in the pipeline
> process is updating the original source to mark that it's been loaded. In
> this scenario our final step is updating the original, not loading it to the
> destination. It would make sense that we could chain these however would
> seem to fit the process that we're working on. The branching operation I
> visualize as this. Hopefully the ascii shows up somewhat correctly here.
>
> Operation 1
> |
> / \
> Op2 Op3
>
> But I never want Op3 to execute before Op2. Even though the whole operation
> might be isolated by a transaction. It would still make sense to see this.
>
> Operation 1 -> Select from source
> |
> Operation 2 -> Insert into destination
> |
> Operation 3 -> Mark source as transferred
>
> Nathan Palmer
>
> > <rhino-tools-dev%2Bunsu...@googlegroups.com<rhino-tools-dev%252Buns...@googlegroups.com>
you may need to implement your own operation inheriting from database
operation and allow the records to flow through.
On Jan 13, 2:14 pm, Nathan Palmer <snowpal...@gmail.com> wrote:
> I see what you are saying. At the same time the last step in the pipeline
> process is updating the original source to mark that it's been loaded. In
> this scenario our final step is updating the original, not loading it to the
> destination. It would make sense that we could chain these however would
> seem to fit the process that we're working on. The branching operation I
> visualize as this. Hopefully the ascii shows up somewhat correctly here.
>
> Operation 1
> |
> / \
> Op2 Op3
>
> But I never want Op3 to execute before Op2. Even though the whole operation
> might be isolated by a transaction. It would still make sense to see this.
>
> Operation 1 -> Select from source
> |
> Operation 2 -> Insert into destination
> |
> Operation 3 -> Mark source as transferred
>
> Nathan Palmer
>
>
> > > > <rhino-tools-dev%2Bunsubscribe@ googlegroups.com>
> > > > > > > > .
> > > > > > > > For more options, visit this group at
> > > > > > > >http://groups.google.com/group/rhino-tools-dev?hl=en.
>
> > > > --
> > > > You received this message because you are subscribed to the Google
> > Groups
> > > > "Rhino Tools Dev" group.
> > > > To post to this group, send email to rhino-t...@googlegroups.com.
> > > > To unsubscribe from this group, send email to
> > > > rhino-tools-d...@googlegroups.com<rhino-tools-dev%2Bunsu...@googlegroups.com>
> > <rhino-tools-dev%2Bunsu...@googlegroups.com<rhino-tools-dev%252Buns...@googlegroups.com>
> > > > <rhino-tools-dev%2Bunsubscribe@ googlegroups.com>
> > > > > > > > .
> > > > > > > > For more options, visit this group at
> > > > > > > >http://groups.google.com/group/rhino-tools-dev?hl=en.
>
> > > > --
> > > > You received this message because you are subscribed to the Google
> > Groups
> > > > "Rhino Tools Dev" group.
> > > > To post to this group, send email to rhino-t...@googlegroups.com.
> > > > To unsubscribe from this group, send email to
> > > > rhino-tools-d...@googlegroups.com<rhino-tools-dev%2Bunsu...@googlegroups.com>
Reply all
Reply to author
Forward
0 new messages