Add Single Row to Beginning of pipeline.

[stellard]

Is there any way to insert a single row into the beginning of the pipeline?

For example. I have a parent child hierarchy and before I load it into the Datawarehouse I need to create a single root parent.

I was thinking it could be done with a processor (with some kind of flag to only insert it the first time) but this is very inefficient as it would run against all rows. Is there any other way?

[Thibaut Barrère]

Hello!

Is there any way to insert a single row into the beginning of the pipeline?

For example. I have a parent child hierarchy and before I load it into the Datawarehouse I need to create a single root parent.

I was thinking it could be done with a processor (with some kind of flag to only insert it the first time) but this is very inefficient as it would run against all rows. Is there any other way?

I usually add an "enumerable" source before the first source to create such records. A classic example is creating an "unknown" record as the first row of a dimension.

Here is an example:

default_record = { field: value, ... }

source :default, { type: :enumerable, enumerable: [default_record] }

source :your_true_source

Your default record will have to go through the whole pipeline in that case, so you may want to fill correct default values.

I usually rely on the following trick when applicable:

default_record = Hash[*required_fields.zip(['unknown']*(required_fields.size)).flatten]

This is just one possibility, you could also create the record in a preprocessor manually using ActiveRecord or similar, or in another control file.

Let me know if it helps, or not!

PS: I like to make such ETL processes "idempotent", ie: able to be re-run without creating issues. Such a script would automatically upsert the root record and following records, so you can run the whole thing without thinking too much if you are bootstrapping or not.

Thibaut

--

http://www.logeek.fr

[stellard]

This is perfect. I did not realise you could have multiple sources per control file. Thanks again Thibaut. 

[Thibaut Barrère]

> This is perfect. I did not realise you could have multiple sources per control file. Thanks again Thibaut. 

You welcome :)

Thibaut

--

http://www.logeek.fr