API schema registries
611 views
Skip to first unread message
Roy Jacobs
Jun 16, 2017, 8:35:24 AM6/16/17
to API Craft
Hi,
I was wondering if there are any good tools that allow schemas to be registered, documented and browsed.
Some background: I'm working in a company that has been spending a lot of time moving from a few battle-tested monoliths into lots of small microservices. It's mostly going well, but one of the challenges I see that our teams are facing is that it becomes harder to figure out which API's are available.
For REST APIs we expose Swagger UI pages for every microservice, but this doesn't cover things like messaging APIs. Also, we see that there is a lot of overlap in certain domains (e.g. addresses) and different services don't always share the same schemas and end up diverging.
So far the only tool that really seems to be in this space is SwaggerHub but that's very REST focused (not to mention commercial, hosted, etc). I've also found Kafka's schema registry tool to be interesting, but that's (obviously) Kafka/Avro-focused. I was wondering if there are any more general alternatives you could recommend?
Thanks!
Roy
Jørn Wildt
Jun 16, 2017, 9:12:06 AM6/16/17
to api-...@googlegroups.com
Sorry, I cannot help you on the overall question, but this comment got my attention:
> Also, we see that there is a lot of overlap in certain domains (e.g. addresses) and different services don't always share the same schemas and end up diverging.
That is kind of the purpse of microservices or what? Shared schemas accros services means they cannot evolve indenpendently. There *may* be some low level stuff that can be standardized - but then you have to call together a commitee every time you need to do a change.
Microservices allow individual teams to work independently without breaking the monolith. The more you share accross service boundaries the more tighly coupled they get and the closer your solution gets to a distributed monolith.
See also https://particular.net/blog/putting-your-events-on-a-diet which is somewhat related.
/Jørn
--
You received this message because you are subscribed to the Google Groups "API Craft" group.
To unsubscribe from this group and stop receiving emails from it, send an email to api-craft+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/api-craft.
For more options, visit https://groups.google.com/d/optout.
Erik Wilde
Jun 16, 2017, 9:26:31 AM6/16/17
to api-...@googlegroups.com, Roy Jacobs
hello roy.
On 2017-06-16 14:35, Roy Jacobs wrote:
> For REST APIs we expose Swagger UI pages for every microservice, but
> this doesn't cover things like messaging APIs. Also, we see that there
> is a lot of overlap in certain domains (e.g. addresses) and different
> services don't always share the same schemas and end up diverging.
i am not completely sure i fully understand what you mean by schema. but
maybe what you mean is some sort of vocabulary?
http://webconcepts.info/ is meant to document shared concepts. much more
than just schemas, but at least to some extent media types are schemas,
too. right now, this is tracking the constantly evolving web standards
landscape. i have summer plans for it, and maybe these are interesting
for you:
1. one idea is to extend the repo/site so that people can fork it and
add information about which of their APIs uses which concept value. for
example, then it would be simple to find out which APIs in some API
ecosystem are using a particular media type.
(this might get a bit noisy when it comes to documenting which API uses
the HTTP GET method, so finding a good balance here is important.)
2. people should be able to use their own concepts. for example, if you
use many DTDs or XSDs, then it should be possible to add new concepts
("DTD" or "XML Namespace") and register new values (DTD identifiers or
XML namespaces).
currently, web concepts is a bit extensible by adding a new concept to
https://github.com/dret/webconcepts/blob/gh-pages/concepts.xml and
starting to add new specs to
https://github.com/dret/webconcepts/tree/gh-pages/src/specs that define
concept values. this needs to be documented at the very least and
probably simplified. and it needs to move from defining values in a spec
to using them in an API (see #1).
i am not quite sure if this is of any interest to you, and it's not
really something to can use right now. but it's something where i think
more and more companies will want a solution that is agnostic to
specific API technologies, and can be extended as the API landscape evolves.
cheers,
dret.
--
erik wilde | mailto:erik....@dret.net |
| http://dret.net/netdret |
| http://twitter.com/dret |
On 2017-06-16 14:35, Roy Jacobs wrote:
> For REST APIs we expose Swagger UI pages for every microservice, but
> this doesn't cover things like messaging APIs. Also, we see that there
> is a lot of overlap in certain domains (e.g. addresses) and different
> services don't always share the same schemas and end up diverging.
maybe what you mean is some sort of vocabulary?
http://webconcepts.info/ is meant to document shared concepts. much more
than just schemas, but at least to some extent media types are schemas,
too. right now, this is tracking the constantly evolving web standards
landscape. i have summer plans for it, and maybe these are interesting
for you:
1. one idea is to extend the repo/site so that people can fork it and
add information about which of their APIs uses which concept value. for
example, then it would be simple to find out which APIs in some API
ecosystem are using a particular media type.
(this might get a bit noisy when it comes to documenting which API uses
the HTTP GET method, so finding a good balance here is important.)
2. people should be able to use their own concepts. for example, if you
use many DTDs or XSDs, then it should be possible to add new concepts
("DTD" or "XML Namespace") and register new values (DTD identifiers or
XML namespaces).
currently, web concepts is a bit extensible by adding a new concept to
https://github.com/dret/webconcepts/blob/gh-pages/concepts.xml and
starting to add new specs to
https://github.com/dret/webconcepts/tree/gh-pages/src/specs that define
concept values. this needs to be documented at the very least and
probably simplified. and it needs to move from defining values in a spec
to using them in an API (see #1).
i am not quite sure if this is of any interest to you, and it's not
really something to can use right now. but it's something where i think
more and more companies will want a solution that is agnostic to
specific API technologies, and can be extended as the API landscape evolves.
cheers,
dret.
--
erik wilde | mailto:erik....@dret.net |
| http://dret.net/netdret |
| http://twitter.com/dret |
Roy Jacobs
Jun 16, 2017, 9:38:12 AM6/16/17
to API Craft
> Also, we see that there is a lot of overlap in certain domains (e.g. addresses) and different services don't always share the same schemas and end up diverging.That is kind of the purpse of microservices or what? Shared schemas accros services means they cannot evolve indenpendently. There *may* be some low level stuff that can be standardized - but then you have to call together a commitee every time you need to do a change.Microservices allow individual teams to work independently without breaking the monolith. The more you share accross service boundaries the more tighly coupled they get and the closer your solution gets to a distributed monolith.
You're quite right in that we don't necessarily want to enforce these changes. What I would like, however, is a simple way for teams to look at each others' API definitions.
Basically what I'm looking for is not a technical tool to manage API schema evolution or anything like that, it's primarily a communication tool I would like to have. Of course this can be as basic as "every team will upload their OpenAPI definitions to a shared Wiki" but I'd like to make it a bit more automated than that :)
mca
Jun 16, 2017, 11:28:54 AM6/16/17
to api-...@googlegroups.com
along w/ Erik's WebConcepts repository, I and a handful of others are (slowly) working on a way to share domain-specific interface descxriptions -- it's call ALPS for Application-Level Profile Semantics. (https://tools.ietf.org/html/draft-amundsen-richardson-foster-alps-01)
the draft has fallen out of date, but there is still some activity there. you can check out the discussion list (https://groups.google.com/forum/?fromgroups=#!forum/alps-io) for a more up-to-date experience.
in fact, the current convo is around creating APR (Application Profile Registries) and your feedback would be valuable.
--
MattM
Jun 16, 2017, 2:29:45 PM6/16/17
to API Craft
+1 to Jørn, dret and mca
And in terms of defining "message APIs" as you called them, I see a lot of people using Google's protobuf for that (the IDL for gRPC) and also there is a new spec that takes an OpenAPI-like approach to defining async APIs:
https://www.asyncapi.com/
Thanks, m@
And in terms of defining "message APIs" as you called them, I see a lot of people using Google's protobuf for that (the IDL for gRPC) and also there is a new spec that takes an OpenAPI-like approach to defining async APIs:
https://www.asyncapi.com/
Thanks, m@
To unsubscribe from this group and stop receiving emails from it, send an email to api-craft+...@googlegroups.com.
mca
Jun 16, 2017, 2:39:51 PM6/16/17
to api-...@googlegroups.com
It's encouraging (to me) that we're talking about how to create shared contract information for APIs.
in the past, most attempts have veered into very narrow definitions of a single instance of a server somewhere (e.g. the exact URL, protocol, response shapes, etc.) which only works for a single server and not for a class of services (e.g. a general shopping contract *many* services can implement).
i'm hoping that, as we start to do more w/ distributed/networked implementations and that we see the value of "discovery" reaching beyond build/deploy into runtime, we'll see more things like ALPS[0], DCAP[1], JSON Home[3], etc. we'll see more examples of runtime discovery, negotiation, and binding in a way that makes the web of apis just as easy/friendly/reliable as the web of documentts has become over the last two decades.
To unsubscribe from this group and stop receiving emails from it, send an email to api-craft+unsubscribe@googlegroups.com.
Erik Wilde
Jun 16, 2017, 2:50:50 PM6/16/17
to api-...@googlegroups.com, mca
hey mike.
On 2017-06-16 20:39, mca wrote:
> in the past, most attempts have veered into very narrow definitions of a
> single instance of a server somewhere (e.g. the exact URL, protocol,
> response shapes, etc.) which only works for a single server and not for
> a class of services (e.g. a general shopping contract *many* services
> can implement).
that's one of the things web concepts will have to get right. which is
why i have been hesitant how to best add it. many APIs are snowflakes
and to some extent, that may be ok (they may have some schema that only
they are using). but then again, many other aspects of those APIs
hopefully aren't snowflakes (think of HTTP problem reports or JSON home
documents they might use to cover some API basics). so to me, each API
is a mix of generic and specific aspects. or some API designs maybe
reused completely across services, which simply means that the specific
part can go down to zero.
this also means than API designers are challenged: how can they reuse
concepts to make it more easy for developers to understand and use the
APIs? which parts do they need to model and create themselves, and which
ones can they reuse? where do they find popular patterns and how can
they learn about trends in what current APIs are looking like? the more
we expose and promote reuse, the more the scales will tip over in the
direction of reuse over snowflakes, so i too am excited that we are
having this discussion.
On 2017-06-16 20:39, mca wrote:
> in the past, most attempts have veered into very narrow definitions of a
> single instance of a server somewhere (e.g. the exact URL, protocol,
> response shapes, etc.) which only works for a single server and not for
> a class of services (e.g. a general shopping contract *many* services
> can implement).
why i have been hesitant how to best add it. many APIs are snowflakes
and to some extent, that may be ok (they may have some schema that only
they are using). but then again, many other aspects of those APIs
hopefully aren't snowflakes (think of HTTP problem reports or JSON home
documents they might use to cover some API basics). so to me, each API
is a mix of generic and specific aspects. or some API designs maybe
reused completely across services, which simply means that the specific
part can go down to zero.
this also means than API designers are challenged: how can they reuse
concepts to make it more easy for developers to understand and use the
APIs? which parts do they need to model and create themselves, and which
ones can they reuse? where do they find popular patterns and how can
they learn about trends in what current APIs are looking like? the more
we expose and promote reuse, the more the scales will tip over in the
direction of reuse over snowflakes, so i too am excited that we are
having this discussion.
mca
Jun 16, 2017, 3:05:32 PM6/16/17
to Erik Wilde, api-...@googlegroups.com
1+ on all that.
the thing in *my* head is about a way to describe independent of define. that's what ALPS tries to do. I think we need *both* a way to generally describe the interface (the collection of possible data points and actions) and then use that shared understanding as inputs for defining the interface with things like RAML, OAI, WSDL, WADL, etc.
I think you and I also have discussed the idea that ALPS/DCAP/etc URIs can be shared in protocol messages (e.g. HTTP headers) in a way similar to CONTENT-TYPE headers. as a way to confirm the vocabulary/description "in force" on a response. I'd like to use your Profile header for this, but that might not be right (IIUYC).
Pushing the envelope, I think it is possible (valuable?) to use ALPS/DCAP/etc as a way to negotiate in the request similar to the way HTTP uses the ACCEPT header. I've not tried this, tho and I know that not many API developers use ACCEPT as a way to support conneg for API representations now. this might be quite a far-field idea<g>.
finally, while webconcepts.info is collecting registered standards docs from a wide range of places, and schema.org is collection shared terms for use in various places, i have in mind a shared collection of these ALPS/DCAP documents that can make it relatively easy to start creating domain specific descriptions (constraining data and action names) that can be safely reused by multiple API implementors.
we already have the power to negotiate for PROTOCOL (via port # and/or the UPGRADE verb in HTTP)
we already have the power to negotiate for FORMAT (via the accept and content-type headers in HTTP)
i am hoping we can also gain the power to negotiate for DOMAIN-LANGUAGE (via (new) accept-profile and profile headers?)
modeling the data names and actions (not the objects and URLs) allows us to share a level of predetermined knowledge w/o over-constraining a single API to use specific URLs, bodies, or even protocols (e.g. WebSockets, HTTP, MQTT, etc.)
that's my own view anyway<g>
Erik Wilde
Jun 16, 2017, 4:11:47 PM6/16/17
to api-...@googlegroups.com, mca
just pitching in here...
On 2017-06-16 17:28, mca wrote:
> in fact, the current convo is around creating APR (Application Profile
> Registries) and your feedback would be valuable.
from the web concepts point of view, such a registry would just be a
another registry along the lines of existing ones (pretty much all web
concept values are at least partly backed by IANA registries, but there
is no reason this has to be limited to IANA). so if some API landscape
sees ALPS profiles being used and wants to keep track of them, that
would just be another web concept, and the values to keep track of would
be the identifiers of these profiles.
On 2017-06-16 17:28, mca wrote:
> in fact, the current convo is around creating APR (Application Profile
> Registries) and your feedback would be valuable.
another registry along the lines of existing ones (pretty much all web
concept values are at least partly backed by IANA registries, but there
is no reason this has to be limited to IANA). so if some API landscape
sees ALPS profiles being used and wants to keep track of them, that
would just be another web concept, and the values to keep track of would
be the identifiers of these profiles.
Steve Rice
Jul 2, 2020, 3:24:50 PM7/2/20
to API Craft
I know this is an old thread — Roy did you end up finding a solution to what you were looking for?
Roy Jacobs
Jul 3, 2020, 3:14:45 AM7/3/20
to api-...@googlegroups.com
Hi Steve,
I didn't at the time, so we spent some time in our company building our own tool, which unfortunately wasn't really finished due to time constraints. You can see a bit of it here in its docs: https://bolcom.github.io/katalog/home/end-user/intro.html
Our new approach will probably be either:
* Use a tool like SwaggerHub or Stoplight for collaboration during authoring and write a thin layer on top of that
* Still have a fully bespoke tool, but just as an API and have the frontend be part of a tool like Spotify's Backstage (see backstage.io for details)
I hope this helps a little, but it's probably not what you were looking for :)
With regards,
Roy
Op do 2 jul. 2020 om 21:25 schreef 'Steve Rice' via API Craft <api-...@googlegroups.com>:
--
You received this message because you are subscribed to a topic in the Google Groups "API Craft" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/api-craft/fYDyjrSOj1A/unsubscribe.
To unsubscribe from this group and all its topics, send an email to api-craft+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/api-craft/74702c75-fa3b-4ef5-9b78-6dcc3ffa515en%40googlegroups.com.
Steve Rice
Jul 8, 2020, 3:54:50 PM7/8/20
to api-...@googlegroups.com
Thanks Roy! Looks like what you were building was along the lines of what I'm looking for.
I'm checking out Stoplight to see if that'll meet our needs.
Cheers,
Steve
To view this discussion on the web visit https://groups.google.com/d/msgid/api-craft/CAB5JrB4cd9HRsY1rk-TfvmW%2B%2B8KtB1OzsUz9bdtd-8_gLwi%3DQw%40mail.gmail.com.
Reply all
Reply to author
Forward
0 new messages