Type Provider format

[Ryan Riley]

I found yet another description language that doubles as a media type format: http://www.markus-lanthaler.com/hydra/spec/latest/core/

Thoughts?

[James D'Angelo]

Looks like I have some reading to do this week.  =)

[Ryan Riley]

Here are some more informative links:

http://json-ld.org/learn.html, and in particular http://www.markus-lanthaler.com/research/on-using-json-ld-to-create-evolvable-restful-services.pdf

[Dmitry Sevastianov]

Interesting find, Ryan, I will be looking into this as well.

[James D'Angelo]

Reading through the document, it really feels a lot like the work put into Simple.Web's linking mechanism (using the idea of HATEOS). If I understand this correctly, it would seem "the framework" would spit out the Hydra vocabulary to describe its resources when it spins up and clients can then use that to map out requests. Or, is it more like Simple.Web in that you get a set of resources back with your request? For example, I could create a resource for a person and making a request to /people/1 using "application/json" would yield something like: { FirstName: "John", LastName: "Smith", Links: [ { Rel: "delete", Title: "Delete this person", Href: "/person/1" }, { Rel: "children", Title: "Get this person's children", Href="/person/1/children" } ] }.

Just trying to wrap my head around the direction.

On Wednesday, June 18, 2014 3:26:50 PM UTC-7, Ryan Riley wrote:

[Ryan Riley]

On Friday, June 27, 2014 2:30:26 PM UTC-7, James D'Angelo wrote:

Reading through the document, it really feels a lot like the work put into Simple.Web's linking mechanism (using the idea of HATEOS). If I understand this correctly, it would seem "the framework" would spit out the Hydra vocabulary to describe its resources when it spins up and clients can then use that to map out requests. Or, is it more like Simple.Web in that you get a set of resources back with your request? For example, I could create a resource for a person and making a request to /people/1 using "application/json" would yield something like: { FirstName: "John", LastName: "Smith", Links: [ { Rel: "delete", Title: "Delete this person", Href: "/person/1" }, { Rel: "children", Title: "Get this person's children", Href="/person/1/children" } ] }.

I think it's a combination of both. As a client, you could request the ApiDocumentation and get an overview of everything. In each response, the result is mapped to an @context and includes properties such as operation that lists allowed operations against the requested URI (essentially embedding an OPTIONS response with every response).

[James D'Angelo]

Ok, that's what it seemed like but I just wanted to clarify what the goal was for this group. I like the idea of being able to ask the server for the API definition--in that way I can totally see where having a type provider that is able to consume it and create a context would be awesome, especially when you spit out JavaScript through something like WebSharper. The picture is becoming a bit more clear now.  =)

[Brad Jones]

Wonderful ideas here Ryan and company ... I especially like your consideration of both Hopac and Hydra. I strongly suggest Markus Lanthaler's doctoral dissertation for a deep dive ... very approachable and highly rewarding! http://www.markus-lanthaler.com/research/third-generation-web-apis-bridging-the-gap-between-rest-and-linked-data.pdf 

[Christopher Atkins]

TL;DR I suggest looking at BED, then kind of ramble a bit to justify asking some impertinent questions and making some unwelcome comments. ;)

Very excited about what you guys are brewing. Is there a wiki or some kind of "lab notebook" out there that I could use to catch up? I'll definitely read through what's already been posted here, but at the risk of repeating something you may have already looked at:

Binary Electronic Data

https://github.com/bed-project/bed/blob/master/spec.ezdoc

https://github.com/bed-project/bed/issues/1

I think BED is interesting because it incorporates links into a self-descriptive binary format, has a small set of simple data types, and has an extension mechanism. There are no implementations, but it could serve as inspiration nonetheless.

 

I'm sure you guys have looked at AtomPub, HAL, Collection+JSON, Siren, UBER, and Ember's REST format (aka jsonapi.org). Then, of course, there are encodings like msgpack, protobufs, BERT, bson, Thrift, etc. It's all kind of overwhelming.

I did some work a while back where I extended ServiceStack with the notion of Resources (unfortunately I wasn't allowed to open source it). All the Services would subclass an abstract type parameterized by a Resource type (TResource).  The ServiceStack Service would handle the appropriate serialization (JSON, XML, HTML etc.) of TResource, and each TResource instance would have a Rels property populated by reflection; public properties of TResource that returned a Resource (or a ResourceCollection) were found via reflection and URLs for those Resources were generated and put on the Rels collection. This worked really well to implement the hypermedia constraint for a multi-provider photo management application. We were able to manually test the application via a web browser and relationships between resources were defined via simple OO (C#) semantics that were already a part of the domain model in most cases.

All that to point out that our ad hoc format was sufficient to get clients going. It wasn't all sunshine and unicorns, though, because there obviously weren't any ready-made gems, nuget packages, cocoapods, npm packages, etc. to consume our format, but most folks were contented with and expecting JSON. The hardest part was evangelizing the hypermedia aspects, as everyone wanted to know what the URL templates were from day 1; a number of clients got cranky when the URLs would change, and that turned out to be a useful lever. Another very unpopular thing was that we did versioning via the Accept headers.

It might have been easier to just write client libraries instead of spreading Restafarianism. In fact, there are often very good reasons to constrain designs RESTfully, but still create client-specific libraries:
https://www.braintreepayments.com/braintrust/when-rest-isnt-good-enough

http://orchestrate.io/blog/2014/07/10/how-to-write-robust-rest-api-clients/

Getting compatibility on format is harder the more general it is. I can still remember clearly those halcyon days of the VB6 SOAP Toolkit and using the SOAP::Lite Perl module together. Everything kind of hung together if you kept what you encoded very simple. Back then all integrations were going to be solved via WSDL. This just ended up being a bottleneck as DBAs/Enterprise Architects/committees decided what the one true representation of their domain was and how ensure the sanctity of that representation wasn't violated. Anyway, even in the most rigorous schemas there were always complications at the domain level (ambiguities, elided info, etc.). It ended up just being a circuitous way to access a database.

When people started doing HTTP APIs, they got things done so much faster because they were just returning POX instead of HTML using the same tools/servers as their websites. The consumers just needed to use one of myriad XML/XQuery libs out there to get the info they wanted. So-called "over-engineering" of media types seems like a threat to that efficacy.

 

All of this begs the question, why a Type Provider for a niche (or novel), potentially abstruse format? Don't get me wrong; I love the idea of a "Intellisense for the web", but if the format is something only easily consumed off-the-shelf by F#, it will only be "Intellisense for F# Web Stack services".  If that's okay (or even desirable), then why look at all these complicated formats and extensions?

I would never discourage intellectual exploration and discourse, and keeping up with e.g. Amundsen's thinking has been enlightening for me personally, but as Dijkstra enjoins, we should relentlessly pursue simplicity. I know I'm preaching to the choir on that point, but it always bears repeating.

I'm going to stop here (I know, about time) without going headlong into "here's what I would do", because 1) I've already overstayed my welcome and 2) I need to read more of what you guys have been discussing this far to add any real value. FWIW, I just really wanted to take the opportunity to highlight the enduring legacy of failure that complexity has wrought and to state anecdotally that real world barriers to REST adoption have less to do with the media types themselves and more to do with lack of ubiquitous tooling to consume HTTP properly.

[mark_la...@gmx.net]

Thanks Brad for CCing me. Since I don't really know yet what exactly you are trying to build in this group here, I don't have to add much at the moment. I just wanted to quickly reach out and let you know that I'm here to help if there are questions regarding JSON-LD [1] (which is already a W3C standard) or Hydra [2] (which is currently being developed in a W3C Community Group [3]).

[1] http://www.w3.org/TR/json-ld/

[2] http://www.hydra-cg.com/

[3] http://www.w3.org/community/hydra/

Cheers,

Markus

---

@MarkusLanthaler

[Dmitry Sevastianov]

It would be great to consolidate experiences in API descriptions. It looks like here we are focusing primarily on fully explorational and completely RESTful formats. 

However, they don't seem to be adopted as widely as traditional SOA-style ones, for example, RAML or Swagger + resource schema, e.g. JSON-schema or XSD. I believe the reasons go beyond mindset inertia of SOA vs REST kind.

I think it is important to recognize varying applicability of API paradigms. Workflows requiring bi-directional communication don't seem to be naturally RESTful. For example, Braintree decided to keep their API closed, providing client libraries instead, and their reasoning looks quite compelling [1]. Simply put, there are things to be done on a client side that a rely hard to enforce by descriptive means.

[1]https://www.braintreepayments.com/braintrust/when-rest-isnt-good-enough

--
You received this message because you are subscribed to a topic in the Google Groups "F# Web Stack" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/web-stack-fs/4ws0UGSr3hc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to web-stack-fs...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ryan Riley]

Thank you, Markus. I'll take some time this evening to write out some of our objectives.

Brad: apologies for missing you Friday. I got tied up in some late meetings. I'll follow up soon.

---

From: mark_la...@gmx.net
Sent: ‎7/‎20/‎2014 7:25 AM
To: web-st...@googlegroups.com
Cc: markus.l...@gmx.net
Subject: Re: Type Provider format

Thanks Brad for CCing me. Since I don't really know yet what exactly you are trying to build in this group here, I don't have to add much at the moment. I just wanted to quickly reach out and let you know that I'm here to help if there are questions regarding JSON-LD [1] (which is already a W3C standard) or Hydra [2] (which is currently being developed in a W3C Community Group [3]).

[1] http://www.w3.org/TR/json-ld/

[2] http://www.hydra-cg.com/

[3] http://www.w3.org/community/hydra/

Cheers,

Markus

---

@MarkusLanthaler

On Thursday, July 17, 2014 3:07:44 PM UTC+2, Brad Jones wrote:

Wonderful ideas here Ryan and company ... I especially like your consideration of both Hopac and Hydra. I strongly suggest Markus Lanthaler's doctoral dissertation for a deep dive ... very approachable and highly rewarding! http://www.markus-lanthaler.com/research/third-generation-web-apis-bridging-the-gap-between-rest-and-linked-data.pdf 

On Friday, July 11, 2014 8:19:46 PM UTC-4, James D'Angelo wrote:

Ok, that's what it seemed like but I just wanted to clarify what the goal was for this group. I like the idea of being able to ask the server for the API definition--in that way I can totally see where having a type provider that is able to consume it and create a context would be awesome, especially when you spit out JavaScript through something like WebSharper. The picture is becoming a bit more clear now.  =)

On Thursday, July 10, 2014 11:54:04 AM UTC-7, Ryan Riley wrote:

On Friday, June 27, 2014 2:30:26 PM UTC-7, James D'Angelo wrote:

Reading through the document, it really feels a lot like the work put into Simple.Web's linking mechanism (using the idea of HATEOS). If I understand this correctly, it would seem "the framework" would spit out the Hydra vocabulary to describe its resources when it spins up and clients can then use that to map out requests. Or, is it more like Simple.Web in that you get a set of resources back with your request? For example, I could create a resource for a person and making a request to /people/1 using "application/json" would yield something like: { FirstName: "John", LastName: "Smith", Links: [ { Rel: "delete", Title: "Delete this person", Href: "/person/1" }, { Rel: "children", Title: "Get this person's children", Href="/person/1/children" } ] }.

I think it's a combination of both. As a client, you could request the ApiDocumentation and get an overview of everything. In each response, the result is mapped to an @context and includes properties such as operation that lists allowed operations against the requested URI (essentially embedding an OPTIONS response with every response).

--

You received this message because you are subscribed to the Google Groups "F# Web Stack" group.
To unsubscribe from this group and stop receiving emails from it, send an email to web-stack-fs...@googlegroups.com.

[Ryan Riley]

I'll write a more complete response later, but in short, the article clearly indicates they are mis-using the term REST, or are at best using it in the meaningless, common use. The relevant quote is:

"If you used the REST API directly there any number of things that could break even from small changes on the server side. It is not hard to imagine an implementation that becomes dependent on the order of elements in an API response. In these situations even additive changes could break a poorly implemented response parser."

Further, RPC APIs can be encoded within a documentation format that supports REST, but the opposite is not true. (Here I mean REST in the formal definition.)

That said, I agree that REST isn't always necessary. RPC is still a useful paradigm. I think our goal should be to provide a format or formats that allow both styles. I also think it useful to generate clients for ease of use, regardless of approach. Both styles require some amount of boilerplate code.

Cheers,
Ryan

---

From: Dmitry Sevastianov
Sent: ‎7/‎20/‎2014 7:58 AM

To: web-st...@googlegroups.com
Cc: markus.l...@gmx.net
Subject: Re: Type Provider format

It would be great to consolidate experiences in API descriptions. It looks like here we are focusing primarily on fully explorational and completely RESTful formats. 

However, they don't seem to be adopted as widely as traditional SOA-style ones, for example, RAML or Swagger + resource schema, e.g. JSON-schema or XSD. I believe the reasons go beyond mindset inertia of SOA vs REST kind.

I think it is important to recognize varying applicability of API paradigms. Workflows requiring bi-directional communication don't seem to be naturally RESTful. For example, Braintree decided to keep their API closed, providing client libraries instead, and their reasoning looks quite compelling [1]. Simply put, there are things to be done on a client side that a rely hard to enforce by descriptive means.

[1]https://www.braintreepayments.com/braintrust/when-rest-isnt-good-enough

On Sun, Jul 20, 2014 at 8:25 AM, <mark_la...@gmx.net> wrote:

Thanks Brad for CCing me. Since I don't really know yet what exactly you are trying to build in this group here, I don't have to add much at the moment. I just wanted to quickly reach out and let you know that I'm here to help if there are questions regarding JSON-LD [1] (which is already a W3C standard) or Hydra [2] (which is currently being developed in a W3C Community Group [3]).

[1] http://www.w3.org/TR/json-ld/

[2] http://www.hydra-cg.com/

[3] http://www.w3.org/community/hydra/

Cheers,

Markus

---

@MarkusLanthaler

On Thursday, July 17, 2014 3:07:44 PM UTC+2, Brad Jones wrote:

Wonderful ideas here Ryan and company ... I especially like your consideration of both Hopac and Hydra. I strongly suggest Markus Lanthaler's doctoral dissertation for a deep dive ... very approachable and highly rewarding! http://www.markus-lanthaler.com/research/third-generation-web-apis-bridging-the-gap-between-rest-and-linked-data.pdf 

On Friday, July 11, 2014 8:19:46 PM UTC-4, James D'Angelo wrote:

Ok, that's what it seemed like but I just wanted to clarify what the goal was for this group. I like the idea of being able to ask the server for the API definition--in that way I can totally see where having a type provider that is able to consume it and create a context would be awesome, especially when you spit out JavaScript through something like WebSharper. The picture is becoming a bit more clear now.  =)

On Thursday, July 10, 2014 11:54:04 AM UTC-7, Ryan Riley wrote:

On Friday, June 27, 2014 2:30:26 PM UTC-7, James D'Angelo wrote:

Reading through the document, it really feels a lot like the work put into Simple.Web's linking mechanism (using the idea of HATEOS). If I understand this correctly, it would seem "the framework" would spit out the Hydra vocabulary to describe its resources when it spins up and clients can then use that to map out requests. Or, is it more like Simple.Web in that you get a set of resources back with your request? For example, I could create a resource for a person and making a request to /people/1 using "application/json" would yield something like: { FirstName: "John", LastName: "Smith", Links: [ { Rel: "delete", Title: "Delete this person", Href: "/person/1" }, { Rel: "children", Title: "Get this person's children", Href="/person/1/children" } ] }.

I think it's a combination of both. As a client, you could request the ApiDocumentation and get an overview of everything. In each response, the result is mapped to an @context and includes properties such as operation that lists allowed operations against the requested URI (essentially embedding an OPTIONS response with every response).

--
You received this message because you are subscribed to a topic in the Google Groups "F# Web Stack" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/web-stack-fs/4ws0UGSr3hc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to web-stack-fs...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Ryan Riley]

As promised, I’ve written a synopsis below of some of the goals of this group.

1. Work together to define good patterns for developing web applications with the F# programming language following the functional style. I recently gave a talk at the Houston Functional Programmers meet up in which I described this to a limited degree.

2. Build libraries together (potentially merging some existing libraries) that facilitate developing F# web applications. We are currently focused on the following libraries because of shared design goals:

• Dyfrig - an Open Web Interface for .NET (OWIN) library for F#
• Frank - a library for building web APIs using System.Net.Http types using a functional style
• HyperF - an excellent library, similar to Frank, for building web APIs using System.Net.Http types that includes several other, useful functional abstractions
• Taliesin - an OWIN routing library built on top of Dyfrig and designed to be a target for a type provider (see next point)

3. Create one or more F# Type Providers that generate client and/or server boilerplate code based off of HTTP API specification documents. Formats so far discussed include RAML, Apiary, ALPS, and JSON-LD+Hydra. Dmitry has already started working on a type provider for RAML, as I understand.

A type provider is a construct available in F# and a few other languages where types can be generated by other means than programmers typing type definitions in the implementation language, e.g. F# in our case. As opposed to classic code generation, these types may not actually exist in the compiled output but are intended to provide design time constraints and discoverability through tools such as IntelliSense/code completion.

I hope this provides some insight into the goals of our group. Please feel free to ask more questions if you would like to learn more. We are still in early stages of collaboration, and so have not made a lot of changes to the existing projects.

Cheers,

Ryan

[Dave Thomas]

Are erasing providers the only option considered.  Generative providers could mean the mechanism could be consumed by Non F# clients.   Just curious. 

[Dmitry Sevastianov]

JsonSchemaTypeProvider I work on is a generative one.
I'll open-source it whenever I have enough time,  though the implementation is rather modest. It only covers Json Schema draft 3 and uses Json.net 6.0.3 for schema parsing,  validation and serialization.
Also,  I do have basic RAML parser which can be used for some kind of framework generation. We use it to generate documentation.  I am not convinced that it makes sense for functional style Web framework though.

You received this message because you are subscribed to a topic in the Google Groups "F# Web Stack" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/web-stack-fs/4ws0UGSr3hc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to web-stack-fs...@googlegroups.com.

[Ryan Riley]

Open to either; I don't have a preference at this time.

---

From: Dave Thomas
Sent: ‎7/‎20/‎2014 3:03 PM
To: web-st...@googlegroups.com

Subject: Re: Type Provider format

[Dmitry Sevastianov]

Re:Braintree, I agree that REST is probably misused as a term in this context. In their case, with API kept private, particular architecture of their API doesn't really matter as they take a burden of hiding it from their clients completely.
Important point here, however, is security, which is at the core of Braintree business. Concerns like message hashing are hard, and I feel they should be accounted for in one form or another in any modern Web framework.

[Markus Lanthaler]

Thanks for the summary Ryan. Not being too familiar with F#, I have a
question regarding your item number

> 3. Create one or more F# Type Providers that generate client and/or
> server boilerplate code based off of HTTP API specification documents.
> Formats so far discussed include RAML, Apiary, ALPS, and JSON-
> LD+Hydra. Dmitry has already started working on a type provider for
> RAML, as I understand.

How do you intend to use these type providers? What exactly do you mean by
"client and/or server boilerplate"?
Also, a Hydra API documentation might change at runtime. Can type providers
react to that for example by refereshing it on cache expiration?


Thanks,
Markus


--
Markus Lanthaler
@markuslanthaler

[aher...@gmail.com]

A type provider is a language extension. Because F# is a statically typed language, it can be a problem to access outside programs that don't have their types translated into F#. More so when the types can be complex, since the combinations may be enormous. A Type Provider creates complex type mappings using a generic type interface. Type mapping code gets executed at design time. The generated code can be seen as "boiler-plate" since it does not have any business logic in it. The idea is to give all the benefits of a dynamic language, and the benefit of a static checking. 

You can recompile a type provider at any time. If the code does not compile the documentation could be flagged for review. It is does take more effort than parsing.

Arturo Hernandez