Rhino Etl : How to run non-operations

[Mike G]

If I want to do pre and post execution actions such as "truncate table", "drop indexes", "recreate indexes" etc...how do I do this using the Rhino Etl dsl? It seems that a Process can only be composed of Operations. And operations can only receive and return IEnumerable<Row>, and are put in the pipeline stream, so for example having an operation run as the last thing in the execution (and not before) seems impossible.

Should I just create fake Operations to house these "non-operation" actions?

[Jeffrey Becker]

That's what I've done in the past.

On Wed, Jul 25, 2012 at 9:24 AM, Mike G <miket...@gmail.com> wrote:

If I want to do pre and post execution actions such as "truncate table", "drop indexes", "recreate indexes" etc...how do I do this using the Rhino Etl dsl? It seems that a Process can only be composed of Operations. And operations can only receive and return IEnumerable<Row>, and are put in the pipeline stream, so for example having an operation run as the last thing in the execution (and not before) seems impossible.

Should I just create fake Operations to house these "non-operation" actions?

--
You received this message because you are subscribed to the Google Groups "Rhino Tools Dev" group.
To view this discussion on the web visit https://groups.google.com/d/msg/rhino-tools-dev/-/2zDD5ha4-5oJ.
To post to this group, send email to rhino-t...@googlegroups.com.
To unsubscribe from this group, send email to rhino-tools-d...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/rhino-tools-dev?hl=en.

[Nathan Palmer]

For larger orchestrated processes I will usually have more than 1 DSL file that gets executed. The truncate/drop/recreate's have been put in their own DSL file with one operation and just executed in the right order.

Nathan Palmer

[Mike G]

Ok, that would work.

It would be nice to have a pre/post delegate property for a Process for this purpose in the DSL. Something like:

def drop_indexes():
    ...

def recreate_indexes():
    ...

Process main preprocess = drop_indexes, postprocess = recreate_indexes :
    operation1
    operation2
    etc...

[Nathan Palmer]

I like the idea of pre/post operations for this purpose but not sure on a good uncluttered way of doing it. This adds some layers on that and I just want to be careful so we keep things simple.

Nathan Palmer

--
You received this message because you are subscribed to the Google Groups "Rhino Tools Dev" group.

To view this discussion on the web visit https://groups.google.com/d/msg/rhino-tools-dev/-/Z967Yw0D11QJ.

[Mike G]

Well, the way I would want to see it would require changing some of the Rhino Etl language, and therefore probably not what a lot of people would agree to. The suggestion I would have would be to rename "Process" to "Pipeline", and then use Process to mean the overall ETL process, and Pipeline would mean the connected operations. So you would see something like:

[Boo code]
def truncate_table():
    ...truncate table code

def drop_indexes():
    ...drop indexes code

operation read_from_csv:
    ..read from csv code, yield results

process main_process:
    truncate_table()
    drop_indexes()

    pipeline main_pipeline:
        read_from_csv()
        sqlBulkInsert "name", "table_name"
            map "Id", "Id", int
            map "somevalue", "somevalue", string

    recreate_indexes()
[\Boo code]
       
You could do this without changing "Process" to "Pipeline" but I think it's a good idea anyway ;)