testing ETL tasks

[Oren Mazor]

hey guys,

so I have a fairly straightforward set of ETL tasks and I'm wondering if there is an easy way to test ETL tasks? I'm thinking of wrapping the tasks in rake tasks that do testing.

I played around with the idea of using screens, but there are a lot of scenarios that they'd miss and I'm worried about the performance hit per task.

thoughts?

[Thibaut Barrère]

Hello Oren!

so I have a fairly straightforward set of ETL tasks and I'm wondering if there is an easy way to test ETL tasks? I'm thinking of wrapping the tasks in rake tasks that do testing.

I played around with the idea of using screens, but there are a lot of scenarios that they'd miss and I'm worried about the performance hit per task.

I've never done this kind of acceptance testing.

So far instead, I mix:

- sample screens (pick one well-known record and assert on it)

- global screens (do stats on all the records)

- unit tests on specific components (see https://github.com/activewarehouse/etl-goodies for an example)

Acceptance testing could certainly work, although it would be a bit slow probably, and would require more setup work.

If you dive into this, you may find Aruba::API useful (I used it to write specs for command lines on iban-check here https://github.com/LTe/iban-check/blob/master/spec/command_line_spec.rb).

Hope this helps!

Thibaut

--

http://www.logeek.fr

 

thoughts?

--
You received this message because you are subscribed to the Google Groups "ActiveWarehouse Discuss" group.
To view this discussion on the web visit https://groups.google.com/d/msg/activewarehouse-discuss/-/0ZJK1zXnawYJ.
To post to this group, send email to activewareh...@googlegroups.com.
To unsubscribe from this group, send email to activewarehouse-d...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/activewarehouse-discuss?hl=en.

[Oren Mazor]

Hey Thibaut,

I've set up some simple rspec tests that go something like:

 - load known fixtures into a local db

 - run etl on that local db into a local warehouse

 - run some tests on the resulting warehouse using sql queries

I had some screens, but I'm working on speeding up my ETL so for now I'm avoiding having those per run.

I'm basically just looking for some simple litmus tests I can run regularly during dev to be sure nothing is broken.

thanks!

Oren

Hello Oren!

thoughts?

To unsubscribe from this group, send email to activewarehouse-discuss+unsub...@googlegroups.com.