how to use etl in delayed_job

[sk]

Hai Thibaut Barrère,

          As you told to use etl process from front end, we have to create some background jobs.I worked on one of background process "delayed_job".

it works fine with some exmples like delivering mails.But, now my doubt is how can we use delayed_job process with etl process, How to write etl code 

"etl name.ctl" in the "delayed_job" class files.Please clarify this...

  

        Thanks in advance...

[sk]

hello anyone is  there can u help me please, is this delayed_job is works for "etl process" to use from front end,then how it works............ 

[Thibaut Barrère]

Hello,

two possibilities:

- use ruby "system" command to launch the etl (make sure to verify the exit code though!)

- use the API to launch the engine (I'll give an example later on)

best,

-- Thibaut

--
You received this message because you are subscribed to the Google Groups "ActiveWarehouse Discuss" group.
To view this discussion on the web visit https://groups.google.com/d/msg/activewarehouse-discuss/-/OvJLUp9p0DcJ.
To post to this group, send email to activewareh...@googlegroups.com.
To unsubscribe from this group, send email to activewarehouse-d...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/activewarehouse-discuss?hl=en.

[sk]

hai dude,by using system command i can able to execute etl commands from my controller,but is it possible to pass input file dynamically to  controll files.if it is,Hoe it will be...........

To unsubscribe from this group, send email to activewarehouse-discuss+unsub...@googlegroups.com.

[Thibaut Barrère]

> hai dude,by using system command i can able to execute etl commands from my
> controller,but is it possible to pass input file dynamically to controll
> files.if it is,Hoe it will be...........

Quick answer: you can try passing an environment variable on the
command line for that purpose, and then to grab it back from the
control file with ENV['YOUR_VARIABLE_NAME'].

-- Thibaut

[sk]

Thanks for the quick reply, But my requirement is n't pass the file from command prompt,suppose i upload a csv file in my view page and when i click on the upload button then that file need to be passed as a input file to the controlle file...I am using paperclip gem to upload files and my file is stored in the "csv" object...now, in my controller how can i pass this file as a input to the controll file. 

[sk]

i am being stuck here...can u give me the response....

[Thibaut Barrère]

Hello,

 

i am being stuck here...can u give me the response....

This is not an activewarehouse question anymore, so please understand I won't be able to reply longer on this one.

My advice is to look at the paperclip documentation to see how you can retrieve the attachement filename, and pass it as an environment variable to the launched script.

If you do for instance system("export TARGETATTACHMENT=thepaperclippath.csv && etl script.ctl"), then later use ENV['TARGETATTACHMENT'] from script.ctl, you should be able to retrieve the name of the file to be processed.

As I said, I won't be able to answer more on this topic, though.

Good luck!

-- Thibaut

[Dave]

http://stackoverflow.com/questions/2232/calling-bash-commands-from-ruby

Pick the option that best suits your needs, put the bash command with the appropriate etl option there for your ctl.

I'm guessing there is no ruby command within aw-etl  to do what you're looking for because of the project's origins.  I believe it was designed to be an etl tool for ROLAP databases, so it's probably just meant to create a snapshot of a business's data at a specific point in time and probably at regular intervals, which is most likely called from a cron job for when reporting needs to be done (at the end of a month, or a quarter, or a year etc.) I doubt calling etl manually fits into the project's scope.

[sk]

I am new to this technology so, i don't know  where to get all these detailsThank u for answering.

[sk]

Thanks dave i think it will be helpful..to my task

On Monday, October 1, 2012 8:46:12 PM UTC+5:30, Dave wrote:

[opensourcechris]

Thibaut

I'm addressing the same problem as sk -> https://groups.google.com/forum/?fromgroups=#!topic/activewarehouse-discuss/AfrOfIbYpLY

Could you show us how to use the API to start an Engine and execute a etl job from within ruby code (to be used in controller action or AR model method)? If you can do that I will write a small tutorial for the documentation.


Chris

To unsubscribe from this group, send email to activewarehouse-discuss+unsub...@googlegroups.com.

[Thibaut Barrère]

Hello Chris,

I'm addressing the same problem as sk -> https://groups.google.com/forum/?fromgroups=#!topic/activewarehouse-discuss/AfrOfIbYpLY
Could you show us how to use the API to start an Engine and execute a etl job from within ruby code (to be used in controller action or AR model method)? If you can do that I will write a small tutorial for the documentation.

Sorry about the delay - we have a newborn here :-)

Two important remarks:

- I can strongly advise against running any ETL processing from an HTTP request; since the processing usually always takes more than 20ms, it's advised to use a background processing gem like resque, sidekiq or delayed_job, and run the ETL from there

- the simplest way to achieve this is not to rely on the API (which I never tested for that yet, so there may be sideeffects etc), but instead to start a specific process using IO.popen, system, backticks or similar (be sure to verify the output code though).

Hope this helps!

Thibaut

--

http://www.logeek.fr

[Oren Mazor]

just for the record, I use Sidekiq to trigger my jobs now.

https://gist.github.com/orenmazor/4725725

note, : execution_dbname is only available on my fork. I needed to redirect where etl_execution goes based on environments.

[Oren Mazor]

forgot to mention, I dont think the ETL::Engine is safe to share across threads, but I could be wrong.

[Thibaut Barrère]

Hi Oren!

thanks for sharing!

On threading: I didn't check, but thread-safety is very unlikely; good point mentioning this.

-- Thibaut

--

You received this message because you are subscribed to the Google Groups "ActiveWarehouse Discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to activewarehouse-d...@googlegroups.com.

To post to this group, send email to activewareh...@googlegroups.com.

Visit this group at http://groups.google.com/group/activewarehouse-discuss?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.