Sample schema file for API Documentation

[Varsha]

Hello front end team!

I am reaching out to you regarding an  aspect of our project: API documentation.

I have attached the sample schema file that I have created based on the OpenAPI schema to this email.

Before proceeding with the integration of DRF spectacular and Swagger UI into the backend, I’d want to ensure that the generated documentation aligns with your expectations and requirements.

I request you to take some time to review the sample schema file and provide your feedback.

Thanks!

[Kriti Godey]

Hi Varsha,

I don't have any feedback, this seems fine to me. I might have more feedback once the documentation page is up, since it's hard to visualize what the docs will look like.

Thanks!

Kriti

[Sean Colsen]

This is neat Varsha! Thanks for reaching out for feedback. I’m one of the front end engineers, but I’ve never worked with an OpenAPI spec before, and I’m not super familiar with your project. But I’ll weigh in with some thoughts.

TypeScript

Perhaps you’re already be aware, but the front end team maintains TypeScript types for the Mathesar API, and that code has significant overlap with your YAML. For example, see our BaseColumn type which is pretty similar to the content of your YAML file.

It’s worth mentioning that the TS types are fairly robust from a typing standpoint but are a bit disorganized in terms of code cleanliness. Our code structure for these types has been somewhat ad-hoc and in flux, and it’s been tricky to figure out how to organize it.

For a long time we’ve been hoping to have a mechanism to auto-generate TS types from backend code somehow. Your API documentation project could be an important stepping stone in that path!

For other front end engineers reading along, I played around with generating TS types from the YAML, trying two approaches:

• openapi-typescript

  npx openapi-typescript ./openapi.yaml -o schema.d.ts
  

  This one just generates types. It would be the easiest to integrate into our codebase, but I can imagine the finer points of meshing the gears to be a bit tedious.

• openapi-generator

  npx @openapitools/openapi-generator-cli generate -i ./openapi.yaml -g typescript-fetch -o gen
  

  This one generates a full client, and it seems somewhat difficult to disentangle the types from the client. It would be more work to integrate into our codebase but has the potential to be nicer in the end. For example, it transforms all the properties from snake_case to camelCase. Cool!

I don’t want to distract you from your main goal of making an API documentation system and I’m not expecting you to make TS type generation a requirement for your project. But I mention all of the above because I think it’s worth it for you to understand this context and keep this longer term goal of generating TypeScript types in mind as you move forward.

YAML vs empirical observation

I notice some discrepancies between your YAML and the behavior of the API that I’m observing. Let’s examine the response that I get when I create a new Money column.

{
  "id": 15943,
  "name": "n",
  "type": "numeric",
  "type_options": null,
  "display_options": null,
  "nullable": true,
  "primary_key": false,
  "valid_target_types": ["omitted for brevity", "etc..."],
  "default": null,
  "has_dependents": false
}

Compared to your YAML there are a number of differences:

• The default column is null in my observation but specified as string in YAML. I’m curious how OpenAPI encodes this sort of “it might be null” behavior. Is there a way to make YAML more accurately reflect the possibility of null? Within the requestBody I see there is a required field. Perhaps there is something similar for the response body?

  It’s also worth noting that (much to my consternation) Javascript makes a distinction between undefined and null. A response body of { "id": 1, "default": null } will mean that default is null whereas a response body of { "id": 1 } will mean that default is undefined. Does OpenAPI allow us to make such a distinction when specifying these properties? If so, it would be good to take this nuance into account for all properties.

• Beyond the nullability questions of the default property, it also does not seem to be a string when it is present. For example, if I POST with {"name":"c","type":"text","nullable":false,"primary_key":false,"default":{"value":"foo"}} then I get a response where the default property has the value {"value":"foo","is_dynamic":false}. Can we fix this?

• For type_options the YAML only says object. Is there a way to make it more detailed? I recognize that type options are dependent on type and are somewhat polymorphic too. That presents some tricky problems. How are you planning to address that?

• The display_options, nullable, primary_key, valid_target_types, and has_dependents properties are present in my observation but missing in the YAML. Why? Can we fix that?

• Likewise, some properties are present in the YMAL but missing in my observation. It seems like perhaps the YAML is using the request body as the response body?

Process

Given all the discrepancies above, I’m curious what your process will be for ensuring the accuracy of the OpenAPI spec data.

How are you producing the YAML? You are generating it from backend code somehow, right? Are you able to somehow “override” certain parts of the generated spec and insert custom stuff too?

​

Nice work so far, and I'm excited to see where this project goes!

[Pavish Kumar Ramani Gopal]

Hi Varsha, this is a great start!

1. I've used OpenAPI before (with Swagger UI and Redoc for the rendered documentation) and I agree it's a great option for us to go with.

• Answering @Sean Colsen's questions on the spec: 

• > Does OpenAPI allow us to make such a distinction when specifying these properties? If so, it would be good to take this nuance into account for all properties

• > I recognize that type options are dependent on type and are somewhat polymorphic too 

• The OpenAPI spec is quite complete and provides us options to specify all of these. So, it's quite reliable.

2. My major concern is with how the schemas are generated.

• I've not worked with DRF Spectacular before. How accurate is its schema generation?
• As @Sean Colsen  mentioned, the sample schema attached with your mail is not accurate. Was the sample schema autogenerated by DRF Spectacular or did you create it manually?
• If it was not auto-generated, could you provide some samples generated by DRF Spectacular for the same endpoint?

[Varsha]

Thanks Sean for your feedback and for providing additional context about the Typescript types. About the discrepancies you observed, Mukesh and I discussed about it on the meet this evening. Mukesh might be able to summarise the solutions we discussed more effectively.

Also, thank you Pavish for you feedback!

I want to address your concern about the sample schema file that I’ve shared. It wasn’t generated by DRF spectacular, rather, it was written manually. 

I will soon share the schema file that has been generated by DRF spectacular. 

Thanks!

[Varsha]

Hi Frontend Team!

I wanted to share an update regarding the API Documentation project.

While I couldn't find a TypeScript-specific client generator, I have generated a Typescript-fetch client for specifically /datafiles/ endpoint and would want to hear your opinions on it.Please review the client code, experiment with it, and let me know your thoughts.

Thanks!

[Sean Colsen]

Thanks for sending this, Varsha!

I’ve spent some time looking it over, but I’ll be honest I’m not quite sure what to make of it. Maybe Pavish would have a better idea since it seems like he’s worked with OpenAPI before.

From what I can tell, it seems like we’d probably end up using it somewhat like this:

// instantiated somewhere high up
const api = new ApiApi({ /* config options ... */ });

// passed down through context or props
api;

// utilized at lower levels like so:
api.apiDbV0DataFilesList(['mathesar'], /* more args... */);

I like this dependency-injection-style approach because (from what I can tell) it would allow us to mock the API and test larger components in isolation more easily than our current setup where the API code is imported. That said, the amount of refactoring to fully implement that approach would be heavy to say the least, so we’d probably want to find a way to move towards that approach incrementally.

I can see that the apiDbV0DataFilesList method has almost dozen arguments 😲. In python that wouldn’t be so bad because you could use named arguments, but the idiomatic way to “fake” named arguments in JS is to use a single argument typed as an object. Varsha do you think it would be possible to get OpenAPI to generate JS code that uses objects instead of arguments?

For example, instead of:

apiDbV0DataFilesCreate(body?: DataFile, id?: number) {}

it would generate:

apiDbV0DataFilesCreate({body, id}: {body?: DataFile, id?: number}) {}

​