[Feature Request] Go config edit API - ability to CRUD parts of config

[srinivas upadhya]

I wanted to start some conversation about adding APIs to CRUD parts of config XML. I have started discussion with Zabil, hopefully will discuss with more people when I get time. The following is the high level overview of what it could look like.

Approach:

The APIs could have end-points like these:

Repository: /go/api/config/package_repositories/package_repository_id

   Package: /go/api/config/package_repositories/package_repository_id/packages/package_id

        SCM: /go/api/config/scms/scm_id

     Group: /go/api/config/groups/pipeline_group_name

   Pipeline: /go/api/config/groups/pipeline_group_name/pipelines/pipeline_name

   Material: /go/api/config/groups/pipeline_group_name/pipelines/pipeline_name/materials/fingerprint

      Stage: /go/api/config/groups/pipeline_group_name/pipelines/pipeline_name/stages/stage_name

         Job: /go/api/config/groups/pipeline_group_name/pipelines/pipeline_name/stages/stage_name/jobs/job_name

       Task: /go/api/config/groups/pipeline_group_name/pipelines/pipeline_name/stages/stage_name/jobs/job_name/tasks/index

The response will be JSON & request payload for create & update will also be JSON.

Ex:

GET - /go/api/config/groups/group/pipelines/pipeline/stages/stage/jobs/job/tasks/index - will get JSON representation of "group/pipeline/stage/job/" (task at index)

POST - /go/api/config/groups/pipeline_group_name/pipelines/pipeline_name/stages/stage_name/jobs/job_name/tasks/index - with request payload containing JSON representation of "new" task, the new task will be added to "group/pipeline/stage/job/" (at given index)

PUT - /go/api/config/groups/pipeline_group_name/pipelines/pipeline_name/stages/stage_name/jobs/job_name/tasks/index - with request payload containing JSON representation of task at index, the task for "group/pipeline/stage/job/" at given index will be updated

DELETE - /go/api/config/groups/pipeline_group_name/pipelines/pipeline_name/stages/stage_name/jobs/job_name/tasks/index - the task for "group/pipeline/stage/job/" at given index will be deleted

What do you guys think? Does anyone has specific input on this approach? OR Does anyone has alternate approach to suggest?

[Aravind SV]

As with any such APIs, you'll need to consider versioning and good documentation. Also, server configuration seems to be missing. Ability to add a template as well. There must be other areas which are missing as well. Intentional?

--
You received this message because you are subscribed to the Google Groups "go-cd-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[srinivas upadhya]

On Friday, February 6, 2015 at 6:08:01 PM UTC+5:30, Aravind SV wrote:

As with any such APIs, you'll need to consider versioning and good documentation. Also, server configuration seems to be missing. Ability to add a template as well. There must be other areas which are missing as well. Intentional?

No. I thought they will be similar to pipelines (extrapolated). Will create exhaustive list as i work on it. Im thinking of starting with tasks. Lets see how it goes.

[Alexandre Conrad]

Is support for HATEOAS considered for resource discovery? Just curious.

--

You received this message because you are subscribed to the Google Groups "go-cd-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Alex

[srinivas upadhya]

On Saturday, February 7, 2015 at 12:04:27 AM UTC+5:30, Alexandre Conrad wrote:

Is support for HATEOAS considered for resource discovery? Just curious.

 Yes Zabil has suggested having a look at that approach. Will look at it when I get to it.

[zab...@thoughtworks.com]

On Friday, February 6, 2015 at 9:47:26 AM UTC+5:30, srinivas upadhya wrote:

I wanted to start some conversation about adding APIs to CRUD parts of config XML. I have started discussion with Zabil, hopefully will discuss with more people when I get time. The following is the high level overview of what it could look like.

Approach:

The APIs could have end-points like these:

Repository: /go/api/config/package_repositories/package_repository_id

   Package: /go/api/config/package_repositories/package_repository_id/packages/package_id

        SCM: /go/api/config/scms/scm_id

     Group: /go/api/config/groups/pipeline_group_name

   Pipeline: /go/api/config/groups/pipeline_group_name/pipelines/pipeline_name

   Material: /go/api/config/groups/pipeline_group_name/pipelines/pipeline_name/materials/fingerprint

      Stage: /go/api/config/groups/pipeline_group_name/pipelines/pipeline_name/stages/stage_name

         Job: /go/api/config/groups/pipeline_group_name/pipelines/pipeline_name/stages/stage_name/jobs/job_name

       Task: /go/api/config/groups/pipeline_group_name/pipelines/pipeline_name/stages/stage_name/jobs/job_name/tasks/index

I think REST API's must be for resources (groups, pipelines, stages, tasks etc.) and not configuration. 

The end points you posted earlier can look like this.

Base url with versioning (/go/api/v1)

Repository: 

List

GET /repositories/:name

By name (the identifier)

GET /repositories/:name

Package

List

GET /repositories/:name/packages

By name

GET /repositories/:name/packages/:name

SCM

List

GET /scms

By name

GET /scms/:name

Pipeline

      GET groups/:name/:pipeline_name

... and so on.

If you allow filter by criteria support a query param.

 scms/?name=:something

 

The response will be JSON & request payload for create & update will also be JSON.

Ex:

GET - /go/api/config/groups/group/pipelines/pipeline/stages/stage/jobs/job/tasks/index - will get JSON representation of "group/pipeline/stage/job/" (task at index)

not sure if it's a good idea to refer the tasks by index. We need another identifier to uniquely identify the task or query on the index instead.

e.g. 

/go/api/v1/groups/:name/:pipeline/stages/:stage/jobs/:job/tasks?index=1

[Marius Ciotlos]

You are missing Template APIs for the config XML. One of the things we've done using the current config XML api (https://github.com/oanastoia/go-config-management). http://www.go.cd/2014/08/19/cloning-templates.html

[Aravind SV]

We didn't forget you, Marius. :) I asked about templates in the second post of this thread, and Srinivas replied to it, saying he needs to make the list comprehensive.

--

[ivan.r...@gmail.com]

Hi all,

You might be interested that in order to work-around the lack of rest APIs for this sort of thing, my colleague and I wrote Gomatic which provides a lot of this functionality as a Python API.

We implemented this by posting the complete config XML - like you were editing it in the browser.

It would be nice to have a properly supported API for posting the complete config XML rather than our rather horrible reverse-engineered hack to do it (or does that already exist? We didn't find such a thing).

This then allows someone to implement any config change they want, independent of any more granular API that you provide.

Ivan

On Friday, February 6, 2015 at 4:17:26 AM UTC, srinivas upadhya wrote:

[Zabil C M]

Hi Ivan,

 gomatic looks nice :-)

 There's an undocumented API you can use for posting the entire xml. 

 This api won't go anytime soon till we implement it's replacement.

 First, get the xml and the md5 with

 http://[server]/go/admin/restful/configuration/file/GET/xml

 (The md5 will be on the header attribute 'X-CRUISE-CONFIG-MD5') 

 And POST changes to 

 http://[server]/go/admin/restful/configuration/file/POST/xml

  

 with x-www-form-urlencoded params

 xmlFile = (contents of the xml)

 md5  = value from 'X-CRUISE-CONFIG-MD5'

Hope that helps.

 

--

[Ivan Moore]

Hi Zabil,

Excellent! Many thanks; I'll change Gomatic to use that.

Ivan

[Zabil C M]

Also Ivan,

 We are working hard to improve our API's and gradually drop support for undocumented ones. 

 It'll be great if you keep track, adopt and give us feedback here 

 https://github.com/gocd/gocd/issues/1100

Thanks!