Re: Rails 3 NameError (uninitialized constant ETL::Engine)

[Chris Roberts]

I'm trying to get etl working in a rails 3 app and when I create a new ETL::Engine in my controller action I get the error below. What do I have to do to make the ETL::Engine available to work with in my controllers and models?

NameError (uninitialized constant ETL::Engine):
  app/controllers/parcels_controller.rb:86:in `import'


  Rendered /home/adamar/.rvm/gems/ruby-1.9.3-p125/gems/actionpack-3.2.9/lib/action_dispatch/middleware/templates/rescues/_trace.erb (0.8ms)
  Rendered /home/adamar/.rvm/gems/ruby-1.9.3-p125/gems/actionpack-3.2.9/lib/action_dispatch/middleware/templates/rescues/_request_and_response.erb (0.6ms)
  Rendered /home/adamar/.rvm/gems/ruby-1.9.3-p125/gems/actionpack-3.2.9/lib/action_dispatch/middleware/templates/rescues/diagnostics.erb within rescues/layout (6.8ms)

This is my controller action:

 def import
      file_to_process = File.join(Rails.root, "etl/parcels.ctl.rb")
      etl_engine = ETL::Engine.new({:rails_root => File.join(Rails.root)})
      etl_engine.process(file_to_process)
   
    respond_to do |format|
      format.html { redirect_to parcels_url, notice: 'ETL has successfully started!'  }
      format.json { head :no_content }
    end
  end

these are the gems that I have included and bundle installed:

gem 'activewarehouse-etl', '1.0.0.rc1'
gem 'adapter_extensions', '1.0.0.rc1'

I have added the etl folder to the autoload paths, so my ctl files should be available:

config.autoload_paths += %W(#{config.root}/etl)

these are the routes I have for that controller action

   match 'parcels/import' => 'parcels#import'

 

 

Is anyone else creating ETL::Engines in their rails apps? I have no trouble creating ETL jobs using the command line tool. But I want to work with activewarehouse-etl from my controller actions and model methods.


Chris

[Andrew Kuklewicz]

Why do you need to try and trigger ETL from a controller?

I have not triggered an etl process from a controller or model.

This is generally something that should be done asynchronously, such as via a cron calling the etl script, not generally a good idea to try and have the etl process  run in the same process as your rails app, as it will lock that 

That said, I would try including:

require 'etl'

To the top of the controller.

Cheers,

-Andrew

Andrew Kuklewicz

Chris

--
You received this message because you are subscribed to the Google Groups "ActiveWarehouse Discuss" group.
To post to this group, send email to activewareh...@googlegroups.com.
To unsubscribe from this group, send email to activewarehouse-d...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/activewarehouse-discuss?hl=en.

[Chris Roberts]

My plan is to use delayed_job to handle the actual etl process, which will spawn a separate ruby process to handle the background process. This was a first step in getting a clean rails integration. I've got a data quality and data cleansing app I want to use activewarehouse-etl as the processing engine for a data cleansing work flow. Kind of like a WYSIWYG editor for activewarehouse-etl which will allow my business analysts to create their own etl workflows.

When others are using activewarehouse-etl in rails apps how are you calling the etl job? Using methods like( Kernel#exec, Kernel#system, IO#popen, Open3#popen3, Open4#popen4)?

Thank you for the suggestion I will try it.

Chris

--
CR

[Chris Roberts]

Along with this topic: What strategy could be used to make ETL::Engine and the other necessary files in the activewarehouse-etl gem available to controllers and models simply by including the gem in the gemfile?


Chris

On Sun, Dec 23, 2012 at 3:29 PM, Andrew Kuklewicz <kook...@gmail.com> wrote:

--
CR

[Chris Roberts]

Andrew,

I was able to trigger the etl job using:

def import
     
      Kernel.system("etl " + "#{File.join(Rails.root, "etl/parcels.ctl.rb")}")

   
    respond_to do |format|
      format.html { redirect_to parcels_url, notice: 'ETL has successfully started!'  }
      format.json { head :no_content }
    end
  end

Although I'm not sure yet if i get the full use of the rails environment in my control files yet, such as models, or the database connections defined in the database.yml.


Chris

--
CR

[Thibaut Barrère]

> Although I'm not sure yet if i get the full use of the rails environment in my control files yet, such as models, or the database connections defined in the database.yml.

You can boot the Rails environment from your ctl files, using code like (verify the path)

require File.dirname(__FILE__) + '/../../config/environment'

or something similar.

I usually stick this + many useful helpers in a common.rb file, which I include in all the control files.

See (not Rails related but gives ideas):

- https://github.com/activewarehouse/activewarehouse-etl-sample/blob/master/advanced/etl/common.rb

hope this helps!

-- Thibaut