How to use Model definitions in ETL

Showing 1-6 of 6 messages
How to use Model definitions in ETL Kenny Meyer 4/23/12 11:13 AM
Is it possible to re-use models from your Rails application in your ETL process?
Re: How to use Model definitions in ETL Andrew Kuklewicz 4/23/12 11:16 AM
There used to be a db source way to use AR models - I don't suggest it.
When doing etl, I don't think you want the overhead of marshalling into AR objects for every row.

Andrew Kuklewicz


On Mon, Apr 23, 2012 at 2:13 PM, Kenny Meyer <knny...@gmail.com> wrote:
Is it possible to re-use models from your Rails application in your ETL process?

--
You received this message because you are subscribed to the Google Groups "ActiveWarehouse Discuss" group.
To view this discussion on the web visit https://groups.google.com/d/msg/activewarehouse-discuss/-/3LLPnVTt_i4J.
To post to this group, send email to activewareh...@googlegroups.com.
To unsubscribe from this group, send email to activewarehouse-discuss+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/activewarehouse-discuss?hl=en.

Re: How to use Model definitions in ETL Thibaut Barrère 4/24/12 12:34 AM
Hi!
 
Is it possible to re-use models from your Rails application in your ETL process?

Sure you can do that, by requiring the Rails environment in your .ctl files like Sinisa wrote a few days ago here.

What do you have in mind exactly?

I agree with Andrew on avoiding using an ActiveRecord source, unless you have very few rows, or time doesn't matter that much to you :)

In one datawarehouse where the output is a SQL database, I do use ActiveRecord models mapping the target dimensional model (although they are not in a Rails application), but only for the following things:

- foreign key lookups with ActiveRecordResolver.new(model, "find_by_#{foreign_key}")

- screens to verify on sample data:

assert_equal "Expected", Model.where(:business_key => 'thisone').description

- screens to verify on stat data:

assert_equal ["a", "b"], Model.select('category').uniq
assert_greater_than 98/100.0, average(xxx)

- checking relationships (I can use has_many and belongs_to for dimensions data checks)

Hope this helps, and let us know what you have in mind!

Thibaut
--
Re: How to use Model definitions in ETL Kenny Meyer 4/24/12 12:57 PM
Hi Thibaut and Andrew,

I just had to get it done for the project, and including the Rails environment in my .ctl files worked quite as much as I expected.

I am aware of that it's not the best method in terms of performance, but that doesn't really matter at this point.

Thanks for your feedback.

Kenny
Re: How to use Model definitions in ETL Shakti Shankar 4/24/12 1:41 PM
Hi,

I am Shakti and have been using Activewarehouse Etl for last eight months in my company.

You can use model as this 

source  :in,
{
  :type => :model,
  :model => :institution,
  :order => :institution_id
},[
  :institution_name,
  :institution_id,
  :address
]


Thanks
Shakti Shankar

On Mon, Apr 23, 2012 at 2:13 PM, Kenny Meyer <knny...@gmail.com> wrote:
Is it possible to re-use models from your Rails application in your ETL process?

--
You received this message because you are subscribed to the Google Groups "ActiveWarehouse Discuss" group.
To view this discussion on the web visit https://groups.google.com/d/msg/activewarehouse-discuss/-/3LLPnVTt_i4J.
To post to this group, send email to activewareh...@googlegroups.com.
To unsubscribe from this group, send email to activewarehouse-discuss+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/activewarehouse-discuss?hl=en.

Re: How to use Model definitions in ETL Thibaut Barrère 4/25/12 1:02 AM
Hi folks,

(Shakti: welcome on the list! Thanks for your feedback, always appreciated)

in all cases: do whatever works for you :)

"It depends" always applies here.

For instance, if you have a job that takes one week to carry out, but you have to run it once, then it's probably still useful etc.

On a aw-etl deployment, I had something fairly slow which took 10 hours each night, but in the end was so convenient it was worth it :)

Thibaut