Rhino.ETL Dropping Pipeline Thread?

[rbellamy]

If this message should go to a different list, let me know and I'll
hop over there...

I'm looking for some guidance in troubleshooting a problem with the my
load process. Maybe dropping a pipeline (or something like that).
Should I be using partial processes for each of the input/output
loads? I've tried debugging, and I need some help or a helping hand
with the best way to do so - that threadsafe enumerator with delegate
is kicking my a%%.

1. I register 10 processes - 5 input, 5 output, staggered like so:
protected override void Initialize()
{
Register(new StaticStatesFromDatabase
(this.options.SourceConnectionStringName));
Register(new StaticStatesToDatabase
(this.options.TargetConnectionStringName));
Register(new CinemaProjectAuditoriumFromDatabase
(this.options.SourceConnectionStringName));
Register(new CinemaProjectAuditoriumToDatabase
(this.options.TargetConnectionStringName));
Register(new TheatreAddressFromDatabase
(this.options.SourceConnectionStringName));
Register(new TheatreAddressToDatabase
(this.options.TargetConnectionStringName));
Register(new TheatreAuditoriumFromDatabase
(this.options.SourceConnectionStringName));
Register(new TheatreAuditoriumToDatabase
(this.options.TargetConnectionStringName));
Register(new TheatreFromDatabase
(this.options.SourceConnectionStringName));
Register(new TheatreToDatabase
(this.options.TargetConnectionStringName));
}
2. Everything performs as expected, until the end where the log shows
that the pipeline is dropping the last ~1100 rows of the
TheatreAuditoriumToDatabase process:
2009-12-08 20:25:16,863 [3] NOTICE Etl.Professional.LoadProcess -
Finished CertificationToDatabase: 2076 Rows in 00:03:32.9030000
2009-12-08 20:26:39,287 [16] INFO Etl.Cinema.LoadProcess - Processed
12000 rows in TheatreAuditoriumToDatabase
2009-12-08 20:26:55,541 [18] INFO Etl.Professional.LoadProcess -
Processed 3000 rows in DesignerToDatabase
2009-12-08 20:28:20,884 [16] INFO Etl.Cinema.LoadProcess - Processed
13000 rows in TheatreAuditoriumToDatabase
2009-12-08 20:28:37,629 [18] INFO Etl.Professional.LoadProcess -
Processed 4000 rows in DesignerToDatabase
2009-12-08 20:30:03,224 [16] INFO Etl.Cinema.LoadProcess - Processed
14000 rows in TheatreAuditoriumToDatabase
2009-12-08 20:30:22,391 [18] INFO Etl.Professional.LoadProcess -
Processed 5000 rows in DesignerToDatabase
2009-12-08 20:31:27,566 [18] NOTICE Etl.Professional.LoadProcess -
Finished DesignerToDatabase: 5627 Rows in 00:09:43.6060000
2009-12-08 20:31:27,569 [9] NOTICE
Rhino.Etl.Core.Pipelines.ThreadPoolPipelineExecuter - Completed
process LoadProcess in 00:09:43.6090000

[Ayende Rahien]

That easiest way to debug is to change to single thread pipeline, that is why it is there.

--

You received this message because you are subscribed to the Google Groups "Rhino Tools Dev" group.
To post to this group, send email to rhino-t...@googlegroups.com.
To unsubscribe from this group, send email to rhino-tools-d...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/rhino-tools-dev?hl=en.

[webpaul]

I'd be willing to bet that you have an operation that isn't either
iterating the input rows or returning data for the next operation to
take. Keep in mind each operation reads the input rows and that if it
doesn't, rows stop flowing through.

Here's an intro article
http://www.codeproject.com/KB/cs/ETLWithCSharp.aspx

It also seems like you have a bunch of unrelated operations in the
same process, i.e. the output rows from CinemaProject probably don't
have anything to do with the input for TheatreAddress. If that is the
case you may want a process per type, each with two operations. If
they are related, some form of branching and joining may be needed.