can we write model validations for the data in etl process

[Jayaram Sudheer]

hai, i want to write the validations for the data in the etl process....like the length of the first_name is 10 characters...can we write these validations in etl control file or in model.rb file.

[Thibaut Barrère]

Hello,

hai, i want to write the validations for the data in the etl process....like the length of the first_name is 10 characters...can we write these validations in etl control file or in model.rb file.

to reply properly, I need to understand more of your needs.

For instance: what is the expected behaviour if on one row, the length is exceeded? Should an error be raised and the processing stopped completely? Should other rows be still processed? If so, do you need to keep a list of "failed" rows?

Let me know and I'll provide more help.

Thibaut

--

http://www.logeek.fr

[Jayaram Sudheer]

yeah if that validation fails then it should raise an error and the process should stopped completely........ 

[Thibaut Barrère]

yeah if that validation fails then it should raise an error and the process should stopped completely........ 

Two ways to do that (but there are many more you could find).

If your target table matches an ActiveRecord model, you could use a before_write instead of a destination, put the validation in your model, and do:

before_write do |row|

  MyModel.new(row).save!

  row

end

so that an error is raised.

Alternatively, you can use whatever verification directly on the rows yourself and raise error accordingly. This is especially useful if you don't use AR as the target, or if you want more complicated validations than AR perform, or if you want to skip AR validations because they become a performance bottleneck.

before_write do |row|

  raise "Value #{row[:field]} is not a number!" unless ....

  row

end

hope this helps,

Thibaut

--

http://www.logeek.fr