Use-case involving stored procedures

[naj...@gmail.com]

Sorry if this seems like a dense question, but I'm a neophyte python programmer, trying to determine whether luigi is a good fit for my use-case.

The majority of the tasks I'm trying to manage are stored procedures sitting on a database (MS SQL Server at the moment, but potentially something else in the future). The conditions for success (i.e., whether a stored procedure has run correctly) are typically varied -- usually we need to check the logs to see if there are any errors, check whether certain tables exists, check whether data is current, check whether row-counts are "normal", check whether certain master data records are present, etc.

Best I can tell, none of the standard luigi modules quite meet my needs. But if I'm understanding it correctly, I should be able to use the standard methods, as long as I can code the correct logic to determine success. e.g., something like this:

class MyFirstTask(luigi.Task):

def requires(self):

## Any preceding Tasks (typically stored procs) that might need to be run

return OtherTasks()

def run(self):

## python code to call the stored procedure

def output(self):

## python code to check various conditions and determine whether the stored
## procedure was a success. Return True if it succeeded return False if it failed.

Am I understanding it correctly?

[Dave Buchfuhrer]

The code to check whether the job succeeded can go into a complete function. Alternatively, you can do that check at the end of the run function, then store a success indicator in a database if it succeeded. That may be necessary if the stuff you're checking for completeness is ephemeral or expensive to check. The output function should instead return a Target object describing what the output of the task looks like. This can be used to determine completeness if you don't write a custom complete function and as input to tasks that depend on this one.

--
You received this message because you are subscribed to the Google Groups "Luigi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to luigi-user+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.