How to use Model definitions in ETL

[Kenny Meyer]

Is it possible to re-use models from your Rails application in your ETL process?

[Andrew Kuklewicz]

There used to be a db source way to use AR models - I don't suggest it.

When doing etl, I don't think you want the overhead of marshalling into AR objects for every row.

Andrew Kuklewicz

On Mon, Apr 23, 2012 at 2:13 PM, Kenny Meyer <knny...@gmail.com> wrote:

Is it possible to re-use models from your Rails application in your ETL process?

--
You received this message because you are subscribed to the Google Groups "ActiveWarehouse Discuss" group.
To view this discussion on the web visit https://groups.google.com/d/msg/activewarehouse-discuss/-/3LLPnVTt_i4J.
To post to this group, send email to activewareh...@googlegroups.com.
To unsubscribe from this group, send email to activewarehouse-d...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/activewarehouse-discuss?hl=en.

[Thibaut Barrère]

Hi!

 

Is it possible to re-use models from your Rails application in your ETL process?

Sure you can do that, by requiring the Rails environment in your .ctl files like Sinisa wrote a few days ago here.

What do you have in mind exactly?

I agree with Andrew on avoiding using an ActiveRecord source, unless you have very few rows, or time doesn't matter that much to you :)

In one datawarehouse where the output is a SQL database, I do use ActiveRecord models mapping the target dimensional model (although they are not in a Rails application), but only for the following things:

- foreign key lookups with ActiveRecordResolver.new(model, "find_by_#{foreign_key}")

- screens to verify on sample data:

assert_equal "Expected", Model.where(:business_key => 'thisone').description

- screens to verify on stat data:

assert_equal ["a", "b"], Model.select('category').uniq

assert_greater_than 98/100.0, average(xxx)

- checking relationships (I can use has_many and belongs_to for dimensions data checks)

Hope this helps, and let us know what you have in mind!

Thibaut

--

http://www.logeek.fr

[Kenny Meyer]

Hi Thibaut and Andrew,

I just had to get it done for the project, and including the Rails environment in my .ctl files worked quite as much as I expected.

I am aware of that it's not the best method in terms of performance, but that doesn't really matter at this point.

Thanks for your feedback.

Kenny

[Shakti Shankar]

Hi,

I am Shakti and have been using Activewarehouse Etl for last eight months in my company.

You can use model as this 

source  :in,

{

  :type => :model,

  :model => :institution,

  :order => :institution_id

},[

  :institution_name,

  :institution_id,

  :address

]

Thanks

Shakti Shankar

On Mon, Apr 23, 2012 at 2:13 PM, Kenny Meyer <knny...@gmail.com> wrote:

Is it possible to re-use models from your Rails application in your ETL process?

--

[Thibaut Barrère]

Hi folks,

(Shakti: welcome on the list! Thanks for your feedback, always appreciated)

in all cases: do whatever works for you :)

"It depends" always applies here.

For instance, if you have a job that takes one week to carry out, but you have to run it once, then it's probably still useful etc.

On a aw-etl deployment, I had something fairly slow which took 10 hours each night, but in the end was so convenient it was worth it :)

Thibaut

--

http://www.logeek.fr