Configuration files

[Fabien Carrion]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello,

For my purposes, I would need to describe one job, or one batch in
various configuration files. And when all the files are loaded process
the job/batch.

It seems to me the functionality doesn't exist.

I would like to know if it is the best way to do it. I would also like
to know in case I do the functionality if you want it.

Thanks

- --
Fabien Carrion

() Campagne du ruban ASCII -- Contre les mails en html
/\ contre les pieces-jointes Microsoft
Web: http://fabien.carrion.free.fr/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkxW+JAACgkQTe/bvZWuSM9pjwCgzSZasGODtqPCKLxkz3aE55g/
RiUAn2p87FaRd2lmGNWrS7ookmYvAMqt
=T70V
-----END PGP SIGNATURE-----

[Thibaut Barrère]

Hello Fabien,

> For my purposes, I would need to describe one job, or one batch in
> various configuration files. And when all the files are loaded process the job/batch.
> It seems to me the functionality doesn't exist.

I'm not sure to understand your needs here - can you describe your use
case a little bit more so we can try to help ?

cheers,

-- Thibaut

[Fabien Carrion]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Sure,

Right now we have:
ETL::Engine.process("configfile.ctl")

And it parse the file configfile.ctl and it execute it.

I would need something like:
engine = ETL::Engine.new
engine.load("configfile1.ctl")
engine.load("configfile2.ctl")
engine.load("configfile3.ctl")
engine.process

And it would parse all the sources/processors/destinations in
configfile1.ctl, configfile2.ctl, configfile3.ctl and process all of them

This would let me use various configuration files as bricks for a bigger
job.

Is it more understandable?

Thanks

- --
Fabien Carrion

() Campagne du ruban ASCII -- Contre les mails en html
/\ contre les pieces-jointes Microsoft
Web: http://fabien.carrion.free.fr/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkxXBVsACgkQTe/bvZWuSM/m1ACfZRKs+YpaBnuIe3KeAIgy4JYN
NdcAn0M0YNInebceUoLjbDaPrxhJcpId
=aF+W
-----END PGP SIGNATURE-----

[Anthony Eden]

That's what .ebf files are for, although I can't recall if they support delayed execution. If not, then I'd suggest that's where the feature needs to be implemented.

-Anthony

On Mon, Aug 2, 2010 at 1:50 PM, Fabien Carrion <fabien....@gmail.com> wrote:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Sure,

Right now we have:
ETL::Engine.process("configfile.ctl")

And it parse the file configfile.ctl and it execute it.


I would need something like:
engine = ETL::Engine.new
engine.load("configfile1.ctl")
engine.load("configfile2.ctl")
engine.load("configfile3.ctl")
engine.process

And it would parse all the sources/processors/destinations in
configfile1.ctl, configfile2.ctl, configfile3.ctl and process all of them

This would let me use various configuration files as bricks for a bigger
job.

Is it more understandable?

Thanks

On 08/02/2010 12:22 PM, Thibaut Barrčre wrote:
> Hello Fabien,
>
>> For my purposes, I would need to describe one job, or one batch in
>> various configuration files. And when all the files are loaded process the job/batch.
>> It seems to me the functionality doesn't exist.
>
> I'm not sure to understand your needs here - can you describe your use
> case a little bit more so we can try to help ?
>
> cheers,
>
> -- Thibaut
>

- --
Fabien Carrion

()  Campagne du ruban ASCII -- Contre les mails en html
/\  contre les pieces-jointes Microsoft
Web: http://fabien.carrion.free.fr/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkxXBVsACgkQTe/bvZWuSM/m1ACfZRKs+YpaBnuIe3KeAIgy4JYN
NdcAn0M0YNInebceUoLjbDaPrxhJcpId
=aF+W
-----END PGP SIGNATURE-----

--
You received this message because you are subscribed to the Google Groups "ActiveWarehouse Discuss" group.
To post to this group, send email to activewareh...@googlegroups.com.
To unsubscribe from this group, send email to activewarehouse-d...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/activewarehouse-discuss?hl=en.

--
GMU/IT d- s: a33 C++(++++)$ UL@ P--- L+(++) !E W+++$ !N o? K? w--- !O M++ V PS+ PE Y PGP t+ !5 X- R tv b++ DI+ D++ G- e++ h---- r+++ y++++**

http://anthony.mp

[Thibaut Barrère]

I second what Anthony said: you can use an .ebf file to gather the sequence of your .ctl files, eg:

require 'common'

run 'first.ctl'
run 'second.ctl' unless some_condition
run 'third.ctl'

Here are a few additional tips:
- you can share common code (any bits of Ruby, really) using a regular require from inside your .ctl

- you can load whatever is needed, including YAML configuration, ENV variables etc

- if you have one .ctl with a large setup you'd like to factor our, you can do this:

before (unique.ctl):

source ..., :table => xxx

transform ...

destination ..., :table => yyy

extract to my_processing.rb:

def do_my_processing(source_table, destination_table)

  source ..., :table => source_table

  transform ...

  destination ..., :table => destination_table

end

then reuse in first.ctl and second.ctl:

# first.ctl

require File.dirname(__FILE__) + '/my_processing'

do_my_processing('A','B')

# second.ctl

require File.dirname(__FILE__) + '/my_processing'

do_my_processing('C','D')

Just the same way, you can extract a group of transform like:

transform :xxx, some_params...

transform :yyy

to 

def macro_transform(some_params)

  transform :xxx, some_params

  transform :yy

end

which makes it easy to reuse a large block (eg: cleaning + lookup operations for instance) accross multiple .ctl files.

Keep in mind that the source/transform/destination etc calls really are "registrations" rather than "executions". Once the .ctl file is fully loaded (and all the .ctl file has been parsed), the execution will start row by row, more or less.

I hope these few tips will help you cover your need - given that you can fallback to Ruby by design (on purpose), it's fairly easy to accomplish all the glue code needed here.

-- Thibaut

[Fabien Carrion]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Thanks for all those tips.

Activewarehouse is working really well for me. I have achieved almost
all my goals.

Let me know if you want to push some commits from my github to the
official repository.

Thanks.

> --
> You received this message because you are subscribed to the Google
> Groups "ActiveWarehouse Discuss" group.
> To post to this group, send email to
> activewareh...@googlegroups.com.
> To unsubscribe from this group, send email to
> activewarehouse-d...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/activewarehouse-discuss?hl=en.

- --
Fabien Carrion

() Campagne du ruban ASCII -- Contre les mails en html
/\ contre les pieces-jointes Microsoft
Web: http://fabien.carrion.free.fr/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkyIHq0ACgkQTe/bvZWuSM9w3gCgw56JYVvH2ZMV49APpG3Cg318
ba4AmwQkIcjoCfk5qU45oFEDnnIyAIFp
=PyXU
-----END PGP SIGNATURE-----