Airflow to Argo migration at Github

[Jeremy Lewi]

Hi folks,

I was chatting with Michal and Hamel who was talking about the migration under way at GitHub from Airflow to Argo.  We wanted to move this discussion into the open since we thought others would be interested. See below for Michal's initial response.

Michal,

How are folks writing their Argo pipelines? e.g. are they writing YAML directly, using a templating language, using python? 

If you aren't using python, what did folks think of moving away from using python (I'm assuming that's how people used Airflow) to define workflows?

J

Initial response from Michal: 

> We don't do pipelines (yet) because we're building a lot on top of
Workflow CRD so this python overlay that Pipelines uses actually is
hard for us to interact with. For example, we've built another
operator, CronWorkflow which is super simple and allows to run Argo
workflows in cron - important for any data pipeline.
As for airflow vs argo...well k8s itself is great benefit and we have
ton of examples when Argo is actually better to work with.

I can join next Asia-friendly kubeflow meeting and talk about it
(unfortunatly I have permanent conflict on EU friendly one).

As for talk, well, we're still in our infancy with new infra we're
building. Problem is currently we're not even using a lot of deep
learning, so really only big component we use actively is argo + tools
we've built around it. I'll be experimenting with TF2.0 faeatures +
distributed training soon, but haven't done it yet. Let me know how
can I help

[Jeremy Lewi]

[Fix cc list]

[Michał Jastrzębski]

Hey,

So basically our ML team is using yaml to write workflows. Argo has
great library of example workflows, we wrote couple of our own
(tailored towards our internal system and use cases) and some
documentation and howtos, they're pretty comfortable with writing
workflows directly. Argo CLI is great utility to validate syntax and
so far we haven't seen anything hard to debug. Being able to directly
use Argo is great, because fundamentally it allows most control. We've
been writing tools around Argo and we leverage CRDs heavily.

As for Python vs YAML, there is just no added benefit of using python
over yaml. With yaml you have validation in argo CLI, ton of useful
examples in argo repo itself and also getting familiar with general
Kubernetes API (and it's resources syntax in yaml) is great for long
term interactions with infra. Most of code in containers still use
Python, but not all of it, and building this mental model of
disconnecting workflow definition from tools running in each pod is
great.

We had some issues with Argo UI and how to schedule in cron (Argo
Events are very heavy weight and hard to understand for such simple
problem), so we wrote our own versions to handle some of these
problems. Again, being able to natively integrate with k8s api made it
super easy.

Our initial worry that data scientists would have problems writing k8s
manifests (like argo workflows in yaml) was completely unfounded. Some
docs and ton of examples was enough even for people who aren't experts
in linux, not to mention containers, to comfortably write quite
complex workflows.

[Michał Jastrzębski]

We did use Ariflow Operators, but we don't really need this kind of
feature at this moment - we're pretty comfortable calling our
Hive/Presto and HDFS from Python directly.
As for Argo and Events, there were few issues we had with it:

1. It's hard, what we needed really is to trigger workflow every
crontab. With Argo Events that would require creating Gatewat, Sensor
and then in Sensor have nested workflow, which makes it super hard to
read. What we did is simple operator that takes new CRD CronWorkflow,
which is really just workflow definition + 1 spec field - crontab. It
then creates CronJob that will execute workflow in spec. Pretty
simple, all and all ~200 lines of python and makes our lives much
easier.
2. Argo UI really needs some way to filter workflows. One flat list is
unmanagable for people who run them in cron + ad hoc + CI:) We've
created our own UI that pretty much allows listing and searching
workflows. It also integrates with CronWorkflow and allows to show all
executions of particular cronjob with single click. React in official
argo ui proven to be too much for infra engineers and data scienists
:(

On Sun, 24 Mar 2019 at 21:20, Kapas, Mukulika <Mukulik...@intuit.com> wrote:
>
> Hi Michal,
>
> This is great feedback that data scientists are fine with YAML and finding it easy to use Argo.
>
> We got feedback from others also for Argo events and would look into how better to integrate eventing with Argo workflows. Will also give feedback to Blackrock team who are maintaining Argo events.
>
> Were you guys using airflow operators? Are you planning to build similar connectors for data sources that can be invoked from Argo?
>
> Please let us know if you have any questions.
>
> Any talk /blog on this topic would be great whenever you are comfortable.
>
> Jeremy, thanks for getting us connected.
>
> Regards,
> Mukulika
>
> On 3/24/19, 4:49 PM, "Michał Jastrzębski" <inc...@gmail.com> wrote:
>
> This email is from an external sender.

[Adam Leary]

Hi there,

I've got a really dumb and basic question so Argo workflows are used in Kubeflow - I believe that's correct. So looking at Argo as a k8s native airflow and CI/CD which also uses kubeflow? I'm trying to understand the context within kubeflow.

Adam

--
You received this message because you are subscribed to the Google Groups "kubeflow-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubeflow-discu...@googlegroups.com.
To post to this group, send email to kubeflow...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubeflow-discuss/CALc0zUASHfS1fD_0TYrWntyoYBCz5sK2EHwnRzJy1Yqv6oDAdA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

ADAM LEARY
VP, Applied Machine Learning Group

T 415.344.2645     Slack @apl

M 415.858.2566     #i-amlg
235 Second Street, San Francisco, CA 94105

[Michał Jastrzębski]

So Kubeflow is using Argo in 2 ways:

1. CI/CD is entirely built in Argo

2. Kubeflow Pipelines are built on top of Argo. Part of discussion in this thread is why are we using Argo natively rather than Kubeflow Pipelines overlay.

[Jeremy Lewi]

There's a CR in pipelines to regularly run workflows (e.g. a cron job).

https://github.com/kubeflow/pipelines/tree/master/backend/src/crd/controller/scheduledworkflow

I don't know off hand how closely coupled that is to KFP.

[Jeremy Lewi]

Adam,

There is some more information about how Kubeflow does CI/CD here.

We use Prow (a CI system similar to Jenkins) to trigger and track tests. We use Argo to define multi-step E2E test workflows.

J

[Adam Leary]

Thank you Jeremy and Michal! This is very helpful and now I get the discussion context. I really appreciate it. -Adam

To view this discussion on the web visit https://groups.google.com/d/msgid/kubeflow-discuss/CA%2BFTfeD%3DuUBh6yG0oM2PAe7G67xzo5mrtv5JApq1dDhBhtxZcg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.