[OpenLineage] Odpi Egeria overlap

[Shirshanka Das]

Hi folks,

It seems like the goals of the Egeria project also include open metadata models and API-s. 

https://github.com/odpi/egeria/blob/master/open-metadata-publication/website/README.md

How should we think about the OpenLineage initiative versus the Egeria project? It seems like there is some overlap that we should clear up. 

Shirshanka

[Julien Le Dem]

Hi Shirshanka,

This came up as well during discussions with LFAI&Data foundation, which Egeria is also part of now that ODPi merged with the LFAI.

There is a clear distinction in my mind that makes them complementary.

 - OpenLineage is focused specifically on collecting metadata of running jobs. It is closest to the execution and models jobs accordingly. It cares about observability.

 - Egeria focuses on the governance use case. You can see on this picture the much broader area covered: https://github.com/odpi/egeria/blob/master/open-metadata-publication/website/open-metadata-types/README.md

They serve very different users in my opinions.

There is an opportunity to enable the sharing of data collected in OpenLineage to Egeria, but Egeria itself doesn't capture information at the same granularity.

Julien

--
You received this message because you are subscribed to the Google Groups "OpenLineage" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openlineage...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/openlineage/PH0PR21MB191053CB1B885B415D673A28B6F31%40PH0PR21MB1910.namprd21.prod.outlook.com.

[Julien Le Dem]

As a follow up, I had a chat with Mandy, the project lead for Egeria. She is interested in collaborating so that we map the OpenLineage model to Egeria to allow leveraging the OpenLineage metadata there.

She liked the approach of OpenLineage as a project focused on the operational level collection of metadata. She compared this to the Exif of Digital cameras which I thought was nice.

There's a great opportunity to collaborate on this. I'll make sure she is involved in reviewing the spec.

To view this discussion on the web visit https://groups.google.com/d/msgid/openlineage/CAJ_A%2Bf0Lti6eq78dX-WhVNoUBO5t3vHZ2oHfY-b_QtPXVU_AWg%40mail.gmail.com.