Drake Information

[nmba...@usc.edu]

Hello!

I’m a grad student at USC studying Public Administration. I’m currently exploring various data cleaning tools that can be adopted by government agencies and/or MPA/MPP programs.

We’ve looked at quite a few reviews and Drake seems like a great option. However, given that neither myself nor my team have backgrounds in coding or data science we aren’t too sure on how to evaluate it.

We developed a series of questions for our analysis. But I’m having a lot of trouble connecting with the open source community. I was wondering if anyone here would be willing to answer our questions?

Please feel free to reach out to me via email nmba...@usc.edu for more information or if you might be able to help!

Thank you so much!!

[Aaron Crow]

Hi, 

Drake is not a data cleaning per se... although it can certainly help you string together and manage a workflow that cleans your data. To make that work, you'd almost certainly need someone willing to get their hands dirty with some coding (the coding required to clean your data the way you need).

Feel free to post questions and I'll do what I can to help with some answers...

--
You received this message because you are subscribed to the Google Groups "drake-workflow" group.
To unsubscribe from this group and stop receiving emails from it, send an email to drake-workflo...@googlegroups.com.
Visit this group at https://groups.google.com/group/drake-workflow.
To view this discussion on the web visit https://groups.google.com/d/msgid/drake-workflow/2c100cc6-e0b0-48ce-a04b-c3dc206a9104%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[nmba...@usc.edu]

Hi Aaron,

Thank you so much for your reply! Please see the questions below.

Is the application cloud-based?

What file formats does the technology work with?

What is the average skill-level of users?

What feedback have you heard from users regarding ease of use and overall effectiveness? What are the difficulties clients have when they use the platform?

What are the specific resources that are most popular and useful to governments?

Thank you again!

[Aaron Crow]

Is the application cloud-based?
No. It's designed as a simple command line tool that lives and runs on a single machine, such as your laptop or workstation. That said, it can certainly "live in the cloud" in the same way as any other single-node tool could be hosted in a cloud service.

What file formats does the technology work with?

Drake itself is file format agnostic, meaning, it doesn't particularly care what file formats you are working with. This puts the decision on you, the developer of the data workflow, to choose the file formats you want and then create scripts that work with those formats.

What is the average skill-level of users?

To develop the data workflow (such as a data cleaning pipeline), you will want someone with basic scripting experience. This could be, e.g., experience with Ruby, Python, or similar. To actually run the workflow, you just need some basic knowledge of running unix tools on the command line.

What feedback have you heard from users regarding ease of use and overall effectiveness? What are the difficulties clients have when they use the platform?

Positive user feedback seems to confirm the value of the main design choices and core feature set of Drake (e.g., see the README). Drake is not great for handling long running tasks that require non-trivial management (e.g., complicated Hadoop jobs that might fail often). It can also introduce friction if you have highly custom requirements beyond a deterministic workflow that can be represented with a simple directed graph (e.g., if you have complicated decision branching logic at the top levels of your workflow).

What are the specific resources that are most popular and useful to governments?

Not sure how to answer this one -- I'm not aware of Drake being used by any specific government agency. That said, I'll theorize that if you need to manage a data workflow, such as for cleaning data, your workflow management requirements are probably similar regardless of whether you're in government, the private sector, or just working on a side project.

I hope this helps! Feel free to post follow-up questions, and I hope you'll let us know how it goes!

-Aaron

--
You received this message because you are subscribed to the Google Groups "drake-workflow" group.
To unsubscribe from this group and stop receiving emails from it, send an email to drake-workflo...@googlegroups.com.
Visit this group at https://groups.google.com/group/drake-workflow.

To view this discussion on the web visit https://groups.google.com/d/msgid/drake-workflow/ffbc0292-83dd-43ed-9616-874a2cbe2452%40googlegroups.com.

[Nicole Barbarino]

Will do, thank you so much!

I cannot tell you how helpful this is :)