Hello,

For my purposes, I would need to describe one job, or one batch in
various configuration files. And when all the files are loaded process
the job/batch.

It seems to me the functionality doesn't exist.

I would like to know if it is the best way to do it. I would also like
to know in case I do the functionality if you want it.

Thanks

- --
Fabien Carrion

()  Campagne du ruban ASCII -- Contre les mails en html
/\  contre les pieces-jointes Microsoft
Web: http://fabien.carrion.free.fr/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkxW+JAACgkQTe/bvZWuSM9pjwCgzSZasGODtqPCKLxkz3aE55g/
RiUAn2p87FaRd2lmGNWrS7ookmYvAMqt
=T70V
-----END PGP SIGNATURE-----

cheers,

-- Thibaut

Sure,

Right now we have:
ETL::Engine.process("configfile.ctl")

And it parse the file configfile.ctl and it execute it.

I would need something like:
engine = ETL::Engine.new
engine.load("configfile1.ctl")
engine.load("configfile2.ctl")
engine.load("configfile3.ctl")
engine.process

And it would parse all the sources/processors/destinations in
configfile1.ctl, configfile2.ctl, configfile3.ctl and process all of them

This would let me use various configuration files as bricks for a bigger
job.

Is it more understandable?

Thanks

On 08/02/2010 12:22 PM, Thibaut Barr re wrote:

()  Campagne du ruban ASCII -- Contre les mails en html
/\  contre les pieces-jointes Microsoft
Web: http://fabien.carrion.free.fr/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkxXBVsACgkQTe/bvZWuSM/m1ACfZRKs+YpaBnuIe3KeAIgy4JYN
NdcAn0M0YNInebceUoLjbDaPrxhJcpId
=aF+W
-----END PGP SIGNATURE-----

That's what .ebf files are for, although I can't recall if they support
delayed execution. If not, then I'd suggest that's where the feature needs
to be implemented.

-Anthony

On Mon, Aug 2, 2010 at 1:50 PM, Fabien Carrion <fabien.carr...@gmail.com>wrote:

http://anthony.mp

I second what Anthony said: you can use an .ebf file to gather the sequence
of your .ctl files, eg:

require 'common'

run 'first.ctl'
run 'second.ctl' unless some_condition
run 'third.ctl'

Here are a few additional tips:
- you can share common code (any bits of Ruby, really) using a regular
require from inside your .ctl
- you can load whatever is needed, including YAML configuration, ENV
variables etc
- if you have one .ctl with a large setup you'd like to factor our, you can
do this:

before (unique.ctl):

source ..., :table => xxx
transform ...
destination ..., :table => yyy

extract to my_processing.rb:

def do_my_processing(source_table, destination_table)
  source ..., :table => source_table
  transform ...
  destination ..., :table => destination_table
end

then reuse in first.ctl and second.ctl:

# first.ctl
require File.dirname(__FILE__) + '/my_processing'
do_my_processing('A','B')

# second.ctl
require File.dirname(__FILE__) + '/my_processing'
do_my_processing('C','D')

Just the same way, you can extract a group of transform like:

transform :xxx, some_params...
transform :yyy

to

def macro_transform(some_params)
  transform :xxx, some_params
  transform :yy
end

which makes it easy to reuse a large block (eg: cleaning + lookup operations
for instance) accross multiple .ctl files.

Keep in mind that the source/transform/destination etc calls really are
"registrations" rather than "executions". Once the .ctl file is fully loaded
(and all the .ctl file has been parsed), the execution will start row by
row, more or less.

I hope these few tips will help you cover your need - given that you can
fallback to Ruby by design (on purpose), it's fairly easy to accomplish all
the glue code needed here.

-- Thibaut

Thanks for all those tips.

Activewarehouse is working really well for me. I have achieved almost
all my goals.

Let me know if you want to push some commits from my github to the
official repository.

Thanks.

On 08/02/2010 02:53 PM, Thibaut Barr re wrote:

()  Campagne du ruban ASCII -- Contre les mails en html
/\  contre les pieces-jointes Microsoft
Web: http://fabien.carrion.free.fr/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkyIHq0ACgkQTe/bvZWuSM9w3gCgw56JYVvH2ZMV49APpG3Cg318
ba4AmwQkIcjoCfk5qU45oFEDnnIyAIFp
=PyXU
-----END PGP SIGNATURE-----