Generic Web Services Client For Data Integration

[API Dev]

Hi,

For integrating with enterprise systems we are thinking of analyzing the systems we want to connect with (~ 25-30) and build a generic web service client that can pull and push data from/to systems. The challenge with such a feature is to map the output of each of the api responses to our application's data model that is relational in nature. Given that each api would generate the response in different format (the data exchange format - xml/json adds to the complication) it is tricky to map the response to a relational data model. I have a couple of questions

1. How do such generic web service systems handle the mapping of api response to relational data model? 

2. What are some other ways of implementing data integration with many systems?

Thanks.

[Andreas Harth]

Hi,

RDF (Resource Description Format) is a graph-structured data format that is useful in data integration scenarios.
In general, the technologies around Linked Data are well suited for data integration, however, there might be a steep learning curve involved.

Cheers,
Andreas.

[Jørn Wildt]

It sounds as if you are inventing a broker system. Try googling that. It is among other things known for being a troublesome beast to manage when it starts coupling way too many models together.

Sorry for the short reply - I'm on vacation 🍷

/Jørn

--
You received this message because you are subscribed to the Google Groups "API Craft" group.
To unsubscribe from this group and stop receiving emails from it, send an email to api-craft+...@googlegroups.com.
Visit this group at https://groups.google.com/group/api-craft.
For more options, visit https://groups.google.com/d/optout.

--

/Jørn

[mca]

my advice for creating enterprise systems that contain several components need to work are as follows:

THREE THINGS

0) select one (or more) default application-level protocols for sending data between components (HTTP, CoAP, are the ones i see most often)

1) select one (or more) highly structured message formats that *all* components MUST use when sending data between components (HAL, Siren, Collection+JSON, HTML are the most common ones i see for this kind of work)

2) establish a shared vocabulary of terms (Domain-Driven Design style) that *all* components MUST use when sending data between components

INTEROPERABILITY

With these three elements in place you are assured that any component that is created in the future will have a high-degree of INTEROPERABILITY (not tight integration) with each other. This works no matter what internal data models or object models are implemented inside each component. As long as they all "speak" (for example) using the "Accounting" vocabulary formatted into "Collection+JSON" messages sent over "HTTP" all the components will be able to interact with each other.

VOCABULARIES ARE HARD

Note the first two items in my list (protocol and format) are easy to do since the heavy work is already done for you. the last one (vocabulary) is difficult. you'll need to grow your vocabulary from existing (usually implied) domain knowledge over time. if you have a strong governance model (e.g. one committee that controls vocabulary) then you have a chance -- but only if they act to mange the vocabulary in a speedy and safe manner.

I've used this method (w/ variations) several times over many years. At one company, we kept it up for over seven and, even though we grew the vocabulary and updated components many times, parts of the system built in the first year still worked just fine in year seven.

Cheers.

mca

Mike Amundsen

+1.859.757.1449
skype:mca.amundsen
http://amundsen.com/blog/
http://twitter.com/mamund
https://github.com/mamund
http://linkedin.com/in/mamund

http://g.mamund.com/meetme

To unsubscribe from this group and stop receiving emails from it, send an email to api-craft+unsubscribe@googlegroups.com.

Visit this group at https://groups.google.com/group/api-craft.
For more options, visit https://groups.google.com/d/optout.

--

/Jørn

--

You received this message because you are subscribed to the Google Groups "API Craft" group.

To unsubscribe from this group and stop receiving emails from it, send an email to api-craft+unsubscribe@googlegroups.com.

[API Dev]

Andreas - I will read more about RDF but briefly it is about defining web resources. My query is about building a connector that allows the system to pull (and push) data from external systems. Most of the systems do provide api's (REST/XML-RPC etc). How these api's are defined isn't in our control so I am unclear on how could RDF help here. Please correct me if I am misunderstood your input.

Mike - About the three points - application level protocols, message formats & domain driven design - again if I understand it correctly these are the points that should be considered when design our application api's. I did not follow on how these would be useful when integrating with external systems which are not in our control. Could you please clarify?

Appreciate your comments. Thanks.

To unsubscribe from this group and stop receiving emails from it, send an email to api-craft+...@googlegroups.com.

Visit this group at https://groups.google.com/group/api-craft.
For more options, visit https://groups.google.com/d/optout.

--

/Jørn

--
You received this message because you are subscribed to the Google Groups "API Craft" group.

To unsubscribe from this group and stop receiving emails from it, send an email to api-craft+...@googlegroups.com.

[mca]

sorry, I was not clear on that.

in cases where i do not have control over the design/implementation of the interface (e.g. SAP, Saleforce, COTS purchased systems, etc.), these are mediated by a component or, in some simple cases, a proxy, that translates the external APIs to match the internal protocol/format/vocabulary). 

In fact, when introducing the three pillar model to an existing enterprise, I start by adding API mediators for existing internal components that are not yet compliant with the protocol/format/vocabulary model.  As each team is able to upgrade to the new model, I can remove the internal API proxy to simplify and streamline the system.

Also, I do not employ ESB or other "business rules" engines in these cases as I am looking to simply mitigate(translate) the non-compliant implementations, not modify (reprogram/mix) them.

I also do not attempt to mitigate the *entire* interface (e.g. all of salesforoce's API), just the parts that affect my IT operations.

Cheers.

mca

Mike Amundsen

+1.859.757.1449
skype:mca.amundsen
http://amundsen.com/blog/
http://twitter.com/mamund
https://github.com/mamund
http://linkedin.com/in/mamund

http://g.mamund.com/meetme

To unsubscribe from this group and stop receiving emails from it, send an email to api-craft+unsubscribe@googlegroups.com.

[javier hector]

Hi Mike maybe is not the same topic, but OpenAPI previously named Swagger have something called swagger codegen, and it seems using it you can have a kind of generic template client (if the specification of the API is using Swagger). But unfortunately, this specification does not follow Hypermedia API constraints but they call it anyway REST APIs.

 

What should be interesting is to have something similar to swagger-codegen using alps.io.

Do you know if there is any work in progress in that front?

Thanks

/Javier

[API Dev]

Thanks for the details. Can you share a sample or link of a component or the proxy just to get an understanding of how does it look like? That would be of great help to have a clear bigger picture.

[API Dev]

Hi Mike,

The business goal of the project is basically to avoid writing custom implementations for integrating with other systems. The software aims to provide the ability to business users to define integrations with systems. One of the ways to support this data integration is through api's. 

The issue with the proxy or a component is that it requires custom coding with every external API that is configured for integration. This defeats the business goal. 

Thanks,

Amit.