Performance enhancements for REST controller?

115 views
Skip to first unread message

Tyler Power

unread,
Jul 18, 2014, 2:57:37 PM7/18/14
to vcap...@cloudfoundry.org
Hi All,

I've enhanced the generic rest object serialization/paginator code to perform better on larger object sets, including when inline-relations is quite deep. I did this to speed up our web console in Stackato. I notice there's been a bunch of activity in this area recently, and that inline-relations depth is now limited to 2, and is perhaps being deprecated at some point. I was looking to submit a pull request but it isn't looking like a straight forward merge anymore as my branch and master have diverged a bit, is it worth spending the time on the pull request? Is this an area people are interested in seeing improvements in?

Some background...

1 .New query param ?orphan-relations which de-duplicates inlined resources by breaking them out (orphaning them) in to a top level hash on the response which is keyed by guid, so objects only appear once, rather then being embedded potentially multiple times inside their parent objects. Something that is quite cool about this is that although the format of the response changes, it can be marshalled back to the original format on the client very easily so it is quite trivial for clients to make use of this without needing substantial changes. This significantly speeds up the API. I can get more numbers of needed but I remember in initial testing with 50 apps (one page) it went from 7MB+ of JSON and many seconds in response time to a few hundred kilobytes and millisecond response times.

2. New ?exclude-relations and ?include-relations exclude or include specific inlined resources e.g. just ask for what you need from the API.

Some high level results based on testing with just 6 apps on the "apps" view in Stackato...

/v2/apps?inline-relations-depth=2 (vanilla upstream cfv2) = 171kb - 714ms
/v2/apps?inline-relations-depth=2&orphan-relations=1 (de-duplicated) = 38kb - 272ms
/v2/apps?inline-relations-depth=2&orphan-relations=1&exclude-relations=events,app_events,stack,service_bindings,routes (me naively looking at an app and picking the things to exclude) = 10.5kb - 83ms
/v2/apps?inline-relations-depth=2&orphan-relations=1&include-relations=space,organization (me looking at what we use on the view and explicitly saying what to include) = 6.5kb - 54ms


Doug Davis

unread,
Jul 18, 2014, 10:00:02 PM7/18/14
to vcap...@cloudfoundry.org

Tyler,
  we'd be very interested in changes in this area.  For the adminUI [1] we've been thinking about switching from the CC REST APIs to accessing the CCDB directly due to the poor performance of the REST APIs.  For our environments we're looking at more than a 30x time difference between the two - we have some pretty big envs.  I don't know if what you're suggesting will be enough for us to not make the switch to the direct DB access, but we'd certainly be interested in seeing if it helps.

[1] https://github.com/cloudfoundry-incubator/admin-ui

thanks
-Doug
________________________________________________________
STSM |  Standards Architect  |  IBM Software Group
(919) 254-6905  |  IBM 444-6905  |  d...@us.ibm.com
The more I'm around some people, the more I like my dog.

Inactive hide details for Tyler Power ---07/18/2014 02:57:43 PM---Hi All, I've enhanced the generic rest object serialization/pTyler Power ---07/18/2014 02:57:43 PM---Hi All, I've enhanced the generic rest object serialization/paginator code to

--
You received this message because you are subscribed to the Google Groups "Cloud Foundry Developers" group.
To view this discussion on the web visit https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/ad7b9bf1-bbc6-41b7-8d80-04d78a3a88c0%40cloudfoundry.org.

To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+u...@cloudfoundry.org.

James Bayer

unread,
Jul 19, 2014, 3:33:54 AM7/19/14
to vcap...@cloudfoundry.org
tyler, i recommend an issue on the cloud controller repo [1] to engage with the dev team. it's likely the community pair for next week will get this request and route you that way.

[1] https://github.com/cloudfoundry/cloud_controller_ng/issues?state=open


To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+u...@cloudfoundry.org.



--
Thank you,

James Bayer

Johannes Hiemer

unread,
Jul 19, 2014, 4:31:56 AM7/19/14
to vcap...@cloudfoundry.org, ramonm...@gmail.com, goran....@gmail.com
I would really like to see this improvement getting merged. For me this is on of the biggest issues with the current REST API implementation. The data redundancy on the response is neither useful, nor practical when using in client side applications (even more in mobile apps). I think it would be much better to reduce the amount and make instead additional requests to gather detailed information for a specific entity - if needed on the client side. 

This week we started building a community build of cf-console (https://github.com/cloudfoundry-community/cf-console) and I think Goran and Ramon would agree as well on this topic. 

To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+unsubscribe@cloudfoundry.org.

--
You received this message because you are subscribed to the Google Groups "Cloud Foundry Developers" group.
To view this discussion on the web visit https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/OF1D8E31B0.AAF8C991-ON85257D1A.000A5FE8-85257D1A.000AF939%40us.ibm.com.

To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+u...@cloudfoundry.org.

Doug Davis

unread,
Jul 19, 2014, 8:42:47 AM7/19/14
to vcap...@cloudfoundry.org, goran....@gmail.com, ramonm...@gmail.com

Johannes,
  there isn't much there in terms of docs - does this overlap with the adminUI?



thanks
-Doug
________________________________________________________
STSM |  Standards Architect  |  IBM Software Group
(919) 254-6905  |  IBM 444-6905  |  d...@us.ibm.com
The more I'm around some people, the more I like my dog.



Inactive hide details for Johannes Hiemer ---07/19/2014 04:32:05 AM---I would really like to see this improvement getting mergeJohannes Hiemer ---07/19/2014 04:32:05 AM---I would really like to see this improvement getting merged. For me this is  on of the biggest issues



From: Johannes Hiemer <jvhi...@gmail.com>
To: vcap...@cloudfoundry.org
Cc: ramonm...@gmail.com, goran....@gmail.com
Date: 07/19/2014 04:32 AM
Subject: Re: [vcap-dev] Performance enhancements for REST controller?




I would really like to see this improvement getting merged. For me this is on of the biggest issues with the current REST API implementation. The data redundancy on the response is neither useful, nor practical when using in client side applications (even more in mobile apps). I think it would be much better to reduce the amount and make instead additional requests to gather detailed information for a specific entity - if needed on the client side.

This week we started building a community build of cf-console (https://github.com/cloudfoundry-community/cf-console) and I think Goran and Ramon would agree as well on this topic.

On Saturday, July 19, 2014 9:33:54 AM UTC+2, James Bayer wrote:

      Tyler,
       we'd be very interested in changes in this area.  For the adminUI [1] we've been thinking about switching from the CC REST APIs to accessing the CCDB directly due to the poor performance of the REST APIs.  For our environments we're looking at more than a 30x time difference between the two - we have some pretty big envs.  I don't know if what you're suggesting will be enough for us to not make the switch to the direct DB access, but we'd certainly be interested in seeing if it helps.

      [1]       https://github.com/cloudfoundry-incubator/admin-ui

      thanks
      -Doug
      ________________________________________________________
      STSM |  Standards Architect  |  IBM Software Group
      (919) 254-6905  |  IBM 444-6905  |        d...@us.ibm.com
      The more I'm around some people, the more I like my dog.

      Tyler Power ---07/18/2014 02:57:43 PM---Hi All, I've enhanced the generic rest object serialization/paginator code to

      From:       Tyler Power <tyl...@activestate.com>
      To:       vcap...@cloudfoundry.org
      Date:       07/18/2014 02:57 PM
      Subject:       [vcap-dev] Performance enhancements for REST controller?




      Hi All,

      I've enhanced the generic rest object serialization/paginator code to perform better on larger object sets, including when inline-relations is quite deep. I did this to speed up our web console in Stackato. I notice there's been a bunch of activity in this area recently, and that inline-relations depth is now limited to 2, and is perhaps being deprecated at some point. I was looking to submit a pull request but it isn't looking like a straight forward merge anymore as my branch and master have diverged a bit, is it worth spending the time on the pull request? Is this an area people are interested in seeing improvements in?

      Some background...

      1 .New query param ?      orphan-relations which de-duplicates inlined resources by breaking them out (orphaning them) in to a top level hash on the response which is keyed by guid, so objects only appear once, rather then being embedded potentially multiple times inside their parent objects. Something that is quite cool about this is that although the format of the response changes, it can be marshalled back to the original format on the client very easily so it is quite trivial for clients to make use of this without needing substantial changes. This significantly speeds up the API. I can get more numbers of needed but I remember in initial testing with 50 apps (one page) it went from 7MB+ of JSON and many seconds in response time to a few hundred kilobytes and millisecond response times.

      2. New ?      exclude-relations and ?include-relations exclude or include specific inlined resources e.g. just ask for what you need from the API.

      Some high level results based on testing with just 6 apps on the "apps" view in Stackato...

      /v2/apps?inline-relations-depth=2 (vanilla upstream cfv2) =       171kb - 714ms
      /v2/apps?inline-relations-depth=2      &orphan-relations=1 (de-duplicated) = 38kb - 272ms
      /v2/apps?inline-relations-depth=2&orphan-relations=1      &exclude-relations=events,app_events,stack,service_bindings,routes (me naively looking at an app and picking the things to exclude) = 10.5kb - 83ms
      /v2/apps?inline-relations-depth=2&orphan-relations=1      &include-relations=space,organization (me looking at what we use on the view and explicitly saying what to include) = 6.5kb - 54ms




    --
    Thank you,

    James Bayer

--
You received this message because you are subscribed to the Google Groups "Cloud Foundry Developers" group.

Johannes Hiemer

unread,
Jul 20, 2014, 3:47:14 AM7/20/14
to vcap...@cloudfoundry.org, goran....@gmail.com, ramonm...@gmail.com
Doug,
yep sorry for that, we just started out on Friday. As far as I can tell the seems to be some overlapping though I must admit we are trying to be more just the UI as known from pivotal, without service plans, etc. (btw. admin UI could get some UI "refreshment" :-))

Goran what do you think?



    --
    Thank you,

    James Bayer

--
You received this message because you are subscribed to the Google Groups "Cloud Foundry Developers" group.
To view this discussion on the web visit https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/7b6ff212-3b36-46b1-a34c-5ec44377abaa%40cloudfoundry.org.

To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+unsubscribe@cloudfoundry.org.

Doug Davis

unread,
Jul 20, 2014, 4:38:16 PM7/20/14
to vcap...@cloudfoundry.org, goran....@gmail.com, ramonm...@gmail.com

Hi Johannes,
  can you elaborate on what you mean by UI refreshment?  We'd welcome pull-requests....




thanks
-Doug
________________________________________________________
STSM |  Standards Architect  |  IBM Software Group
(919) 254-6905  |  IBM 444-6905  |  d...@us.ibm.com
The more I'm around some people, the more I like my dog.

Inactive hide details for Johannes Hiemer ---07/20/2014 03:47:22 AM---Doug, yep sorry for that, we just started out on Friday. Johannes Hiemer ---07/20/2014 03:47:22 AM---Doug, yep sorry for that, we just started out on Friday. As far as I can tell the



From: Johannes Hiemer <jvhi...@gmail.com>
To: vcap...@cloudfoundry.org
Cc: goran....@gmail.com, ramonm...@gmail.com
Date: 07/20/2014 03:47 AM
Subject: Re: [vcap-dev] Performance enhancements for REST controller?




Doug,
yep sorry for that, we just started out on Friday. As far as I can tell the seems to be some overlapping though I must admit we are trying to be more just the UI as known from pivotal, without service plans, etc. (btw. admin UI could get some UI "refreshment" :-))

Goran what do you think?

On Saturday, July 19, 2014 2:42:47 PM UTC+2, Doug Davis wrote:

    Johannes,
     there isn't much there in terms of docs - does this overlap with the adminUI?

    thanks
    -Doug
    ________________________________________________________
    STSM |  Standards Architect  |  IBM Software Group
    (919) 254-6905  |  IBM 444-6905  |      d...@us.ibm.com
    The more I'm around some people, the more I like my dog.





      --
      Thank you,

      James Bayer


--
You received this message because you are subscribed to the Google Groups "Cloud Foundry Developers" group.

Tyler Power

unread,
Jul 20, 2014, 5:06:39 PM7/20/14
to vcap...@cloudfoundry.org
Great, thanks Doug. It sounds like at least a few people would like to see some progress here so I'll look to spend a few days next week working on the PR. James I'll open an issue against the CC when I have some code ready.

Doug/Johannes we too were finding the CC Api was essentially unusable with anything more than a few applications, and short of writing our own REST Api directly on top of the CC database this was the best I could come up with in a reasonable amount of time. We have installations that have thousands of apps so our web console needed to be able to scale appropriately. I think the poor performance is a direct tradeoff of development speed vs runtime performance. The abstracted REST Api in the CC is so simple to add to and extend but doesn't cover many use cases very well yet.

As an example I just grabbed some numbers from our sandbox (medium sized installation, low - mid end hardware), with and without the patches:

GET /v2/apps?inline-relations-depth=2 - 1.5MB, 21s
GET /v2/apps?inline-relations-depth=2&include-relations=space,organization&orphan-relations=1 - 145KB, 3.5s

This was for fetching info for 50 applications which is the default page size. So numbers would vary based on if the page size had been tweaked. In our specific use case here we wanted all the app info plus we needed the name of the space and the organization each app belonged to. So we needed an inline-depth of 2. By using the new query params we were able to de-deduplicate the data in the response, as well as only fetching the specific relations we needed.

So we were able to go from being unusable to being... I guess acceptable is the word. Ideally we would be looking for sub 300ms response times on every Api call, and I think it's certainly possible by optimizing the REST framework further. Which seems entirely worthwhile considering how many people will benefit and how many people it will stop from needing to develop a custom solution.

I'll get started on the PR next week, thanks guys.

Tyler

To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+unsubscribe@cloudfoundry.org.

James Bayer

unread,
Jul 20, 2014, 6:54:53 PM7/20/14
to vcap...@cloudfoundry.org
Tyler, I recommend an issue BEFORE code is ready, I don't want you to waste time on an approach the runtime team may not agree with

Sent from Acompli




To view this discussion on the web visit https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/06aa9c34-b547-4d73-9cf5-4b59b2222708%40cloudfoundry.org.

To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+u...@cloudfoundry.org.

Tyler Power

unread,
Jul 20, 2014, 9:30:44 PM7/20/14
to vcap...@cloudfoundry.org
Sure thing!

Johannes Hiemer

unread,
Jul 21, 2014, 2:53:19 AM7/21/14
to vcap...@cloudfoundry.org, goran....@gmail.com, ramonm...@gmail.com
Doug,
of course. Looking at the source code, I see relatively much JQuery based client side code (datatables etc.) with ruby views (is it a Rails App?). As far as I can tell, the UI does not looks as it would natively run on mobile platforms. As you can see here I forked an old Styx repository and rebuild nearly all parts of styx with Angular/Bootstrap 3.1 and Spring MVC (https://github.com/jhiemer/cf-spring-web-management-console). The UI is much easier to maintain (I cam from JQuery UI Dev and went the Angular way, so I know which pain I left, when switching..). So Goran/Ramon and me are currently evaluating different strategies to build a lean and fresh implementation of the console.

--
You received this message because you are subscribed to the Google Groups "Cloud Foundry Developers" group.
To view this discussion on the web visit https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/1d1606c3-9ffa-42a9-a69a-f8dc8b1bfb5e%40cloudfoundry.org.

To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+unsubscribe@cloudfoundry.org.

Doug Davis

unread,
Jul 21, 2014, 4:44:13 AM7/21/14
to vcap...@cloudfoundry.org

Tyler, yep we noticed similar (and worse) performance as we're looking at tens of thousands, not just hundreds, of records to deal with.  Looking forward to your stuff.



thanks
-Doug
________________________________________________________
STSM |  Standards Architect  |  IBM Software Group
(919) 254-6905  |  IBM 444-6905  |  d...@us.ibm.com
The more I'm around some people, the more I like my dog.



Inactive hide details for Tyler Power ---07/20/2014 05:06:46 PM---Great, thanks Doug. It sounds like at least a few people woulTyler Power ---07/20/2014 05:06:46 PM---Great, thanks Doug. It sounds like at least a few people would like to see  some progress here so I'



From: Tyler Power <tyl...@activestate.com>
To: vcap...@cloudfoundry.org
Date: 07/20/2014 05:06 PM
Subject: Re: [vcap-dev] Performance enhancements for REST controller?




Great, thanks Doug. It sounds like at least a few people would like to see some progress here so I'll look to spend a few days next week working on the PR. James I'll open an issue against the CC when I have some code ready.

Doug/Johannes we too were finding the CC Api was essentially unusable with anything more than a few applications, and short of writing our own REST Api directly on top of the CC database this was the best I could come up with in a reasonable amount of time. We have installations that have thousands of apps so our web console needed to be able to scale appropriately. I think the poor performance is a direct tradeoff of development speed vs runtime performance. The abstracted REST Api in the CC is so simple to add to and extend but doesn't cover many use cases very well yet.

As an example I just grabbed some numbers from our sandbox (medium sized installation, low - mid end hardware), with and without the patches:

GET /v2/apps?inline-relations-depth=2 - 1.5MB, 21s
GET /v2/apps?inline-relations-depth=2&include-relations=space,organization&orphan-relations=1 - 145KB, 3.5s

This was for fetching info for 50 applications which is the default page size. So numbers would vary based on if the page size had been tweaked. In our specific use case here we wanted all the app info plus we needed the name of the space and the organization each app belonged to. So we needed an inline-depth of 2. By using the new query params we were able to de-deduplicate the data in the response, as well as only fetching the specific relations we needed.

So we were able to go from being unusable to being... I guess acceptable is the word. Ideally we would be looking for sub 300ms response times on every Api call, and I think it's certainly possible by optimizing the REST framework further. Which seems entirely worthwhile considering how many people will benefit and how many people it will stop from needing to develop a custom solution.

I'll get started on the PR next week, thanks guys.

Tyler


On Friday, July 18, 2014 7:00:02 PM UTC-7, Doug Davis wrote:

    Tyler,
     we'd be very interested in changes in this area.  For the adminUI [1] we've been thinking about switching from the CC REST APIs to accessing the CCDB directly due to the poor performance of the REST APIs.  For our environments we're looking at more than a 30x time difference between the two - we have some pretty big envs.  I don't know if what you're suggesting will be enough for us to not make the switch to the direct DB access, but we'd certainly be interested in seeing if it helps.

    [1]     https://github.com/cloudfoundry-incubator/admin-ui

    thanks
    -Doug
    ________________________________________________________
    STSM |  Standards Architect  |  IBM Software Group
    (919) 254-6905  |  IBM 444-6905  |      d...@us.ibm.com
    The more I'm around some people, the more I like my dog.

    Tyler Power ---07/18/2014 02:57:43 PM---Hi All, I've enhanced the generic rest object serialization/paginator code to

    From:     Tyler Power <tyl...@activestate.com>
    To:     vcap...@cloudfoundry.org
    Date:     07/18/2014 02:57 PM
    Subject:     [vcap-dev] Performance enhancements for REST controller?



    Hi All,

    I've enhanced the generic rest object serialization/paginator code to perform better on larger object sets, including when inline-relations is quite deep. I did this to speed up our web console in Stackato. I notice there's been a bunch of activity in this area recently, and that inline-relations depth is now limited to 2, and is perhaps being deprecated at some point. I was looking to submit a pull request but it isn't looking like a straight forward merge anymore as my branch and master have diverged a bit, is it worth spending the time on the pull request? Is this an area people are interested in seeing improvements in?

    Some background...

    1 .New query param ?    orphan-relations which de-duplicates inlined resources by breaking them out (orphaning them) in to a top level hash on the response which is keyed by guid, so objects only appear once, rather then being embedded potentially multiple times inside their parent objects. Something that is quite cool about this is that although the format of the response changes, it can be marshalled back to the original format on the client very easily so it is quite trivial for clients to make use of this without needing substantial changes. This significantly speeds up the API. I can get more numbers of needed but I remember in initial testing with 50 apps (one page) it went from 7MB+ of JSON and many seconds in response time to a few hundred kilobytes and millisecond response times.

    2. New ?    exclude-relations and ?include-relations exclude or include specific inlined resources e.g. just ask for what you need from the API.

    Some high level results based on testing with just 6 apps on the "apps" view in Stackato...

    /v2/apps?inline-relations-depth=2 (vanilla upstream cfv2) =     171kb - 714ms
    /v2/apps?inline-relations-depth=2    &orphan-relations=1 (de-duplicated) = 38kb - 272ms
    /v2/apps?inline-relations-depth=2&orphan-relations=1    &exclude-relations=events,app_events,stack,service_bindings,routes (me naively looking at an app and picking the things to exclude) = 10.5kb - 83ms
    /v2/apps?inline-relations-depth=2&orphan-relations=1    &include-relations=space,organization (me looking at what we use on the view and explicitly saying what to include) = 6.5kb - 54ms


--
You received this message because you are subscribed to the Google Groups "Cloud Foundry Developers" group.

To view this discussion on the web visit https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/06aa9c34-b547-4d73-9cf5-4b59b2222708%40cloudfoundry.org.

To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+u...@cloudfoundry.org.

Message has been deleted

Tyler Power

unread,
Sep 29, 2014, 8:33:39 PM9/29/14
to vcap...@cloudfoundry.org
Sorry for the slow reply on this everyone. 

Dieu (runtime PM) just pinged me about it so I've forked the CC and made the tweaks public, you can see them here (note I haven't actually run this code on vanilla CC, only on our Stackato fork, I expect it to work though):


I'm just checking with Dieu to see if this lines up with the direction they want the API to go in upstream and if it's aligned then I will put together some tests and submit a PR.

Cheers,
Tyler

Luan Santos

unread,
Oct 1, 2014, 1:40:30 PM10/1/14
to vcap...@cloudfoundry.org
Hi Tyler,

We took a look at the commit and overall it seems reasonable. Please submit the PR and we'll have a pair review it.

Thanks,

Luan and Jim, CF Runtime Team

tyler...@gmail.com

unread,
Oct 1, 2014, 8:13:36 PM10/1/14
to vcap...@cloudfoundry.org
Reply all
Reply to author
Forward
0 new messages