Further requisition endpoints performance improvements
Sebastian Brudziński
Hello everyone,
we are still investigating performance issues with requisition endpoints and are currently especially interested in requisition search endpoint. We have already looked into reducing the response size, but other than that, the requisition search endpoint makes a lot of calls to the reference data service, mainly to check user rights. The current workflow is: for each found requisition (before pagination) verify user rights, by sending 3 calls to the referencedata service - retrieve user, retrieve right, call 'hasRight' endpoint.
In order to reduce the number of calls made to the reference data service, we are proposing the following:
1. Be smarter about right checking. If we have multiple requisitions coming from the search, let's send an actual request to reference data only for program-facility combo that we didn't ask for already. This could be achieved by having a simple list that holds already checked combinations and checking with the list first, before we ask reference data service.
2. Improve 'hasRight' endpoint contract. For some reason hasRight requires providing user UUID and right UUID in order to verify the user has necessary rights. Other services do not have access to user or right UUIDs, so they need to retrieve them from the reference data service first. Instead of doing all of that, the hasRight could just accept username and rightname what would make the usage of that endpoint simpler for clients.
Do you have any comments or concerns about making those changes? The Malawi team would be up for implementing at least the first one. The second one may take some more time as there are many clients of that endpoint already, but if time allows we could get to that as well.
Best regards,
Sebastian.
Sebastian
Brudziński
Software Developer / Team Leader
sbrud...@soldevelo.com
SolDevelo Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41
Paweł Gesek
--
You received this message because you are subscribed to the Google Groups "OpenLMIS Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openlmis-dev+unsubscribe@googlegroups.com.
To post to this group, send email to openlm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/openlmis-dev/751ceb7a-3e46-8221-0494-eec381f41470%40soldevelo.com.
For more options, visit https://groups.google.com/d/optout.
--
Paweł Gesek
Technical Project Manager
pge...@soldevelo.com / +48 690 020 875
Jake Watson
To unsubscribe from this group and stop receiving emails from it, send an email to openlmis-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/openlmis-dev/CADt-Nu2axUrE%3DqwQM93F0d6ewnehQvTwDktOOS5iW-D3oGuHhg%40mail.gmail.com.
Brandon Bowersox-Johnson
Regarding caching: Yes, caching reference data is part of the discussions and analysis happening now. In the very short term (ie, last 2 sprints), we are finding that we can make easy changes that take less programming time and have larger performance boosts. The most recent example is that we had 1000+ inter-service API calls happening that were reduced to one single call simply because we were making the calls individually in a loop rather than requesting a batch of records in a single API call. (Good work team on finding that using profiling tools and proposing an easy fix!)
Caching is discussed at many of the meetings/conversations we have about performance, so the time is coming when caching moves from discussion to action.
-Brandon
--
To unsubscribe from this group and stop receiving emails from it, send an email to openlmis-dev...@googlegroups.com.
To post to this group, send email to openlm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/openlmis-dev/751ceb7a-3e46-8221-0494-eec381f41470%40soldevelo.com.
For more options, visit https://groups.google.com/d/optout.
--Paweł Gesek
Technical Project Manager
pge...@soldevelo.com / +48 690 020 875
SolDevelo Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41--
You received this message because you are subscribed to the Google Groups "OpenLMIS Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openlmis-dev...@googlegroups.com.
To post to this group, send email to openlm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/openlmis-dev/CADt-Nu2axUrE%3DqwQM93F0d6ewnehQvTwDktOOS5iW-D3oGuHhg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "OpenLMIS Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
openlmis-dev...@googlegroups.com.
To post to this group, send email to
openlm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/openlmis-dev/D566EF9E.F08F%25jake.watson%40villagereach.org.
Josh Zamor
One way to flip this on its head is to attempt a concept that came out of a recent collaboration with ODK. In that collaboration it became apparent that we needed a way to translate our simple RBAC, which is mapped onto a complex Supervisory Node hierarchy, into a set of simple Strings which altogether described the User's rights (aka permissions) in that hierarchy. Such a set of Permission Strings could, I believe, detail all of the different combinations of Rights that a User has on a Program and Facility.
Applied to the question posed here, database paging, there appears to be another use for this set of a User's Permission Strings:
- In order to achieve database level paging, filtering should be done in the database. More specifically, this means that we need to be able to express our "may User X do Y to Z" criteria directly in the databases' criteria. This properly places the filtering work where it belongs and where it's most effecient. RDBMS's are made to index and filter, and we avoid the high expense of turning database rows into Java objects which we then filter and page on.
- A row in the database, for example a Requisition, would need to express a set of Permission Strings that could be used.
The format of these permission strings should be consistent, and should be useful outside of Requisitions. A format which could work looks like the following:
program-facility-right
Where program, facility and right would be UUIDs.
Example
Lets create an example using this supervision structure:
Let's assume the Program we're interested in working with is for ARV, and the Facility W. Clinic is the one we want to see Requisitions at.
Requisition's which were initiated at this facility would have a permission string such as:
ARV-W.CLINIC-viewRequisition
The Storeroom Assistant user has permission strings:
- ARV-W.CLINIC-viewRequisition
- ARV-W.CLINIC-createRequisition
- MALARIA-W.CLINIC-viewRequisition
- etc
The D. Supervisor user has permission strings:
- ARV-W.CLINIC-viewRequisition
- ARV-W.CLINIC-approveRequisition
- ARV-E.CLINIC-viewRequisition
- ARV-E.CLINIC-approveRequisition
- MALARIA-D.HOSPITAL-createRequisition
- etc
In our database paging of Requisition's search, we now pass this set of User permission strings, and use something like a SQL IN to match. A simplified query would look something like:
SELECT * FROM requisitions WHERE permissionString IN ("ARV-W.CLINIC-viewRequisition", "ARV-W.CLINIC-approveRequisition", etc...)
We could even change the Permission Strings which the Requisition list as valid, as needed for filtering. e.g. when the requisition is authorized, we'd remove the string ARV-W.CLINIC-authorizeRequisition from the Requisition, and replace it with ARV-W.CLINIC-approveRequisition. This specific SQL may not work as well within Requisition Service, as the Requisition might want to express multiple permission strings: e.g. view and approve. An approach to this should be quite easy to tackle, even if it meant using another table to store the list of permissions for the requisition.
Other Role Assignments
Moving away from this example and looking at some of our design docs around RBAC, this format would work for the Supervisory Role Assignments (program-facility-right) used above as well as other types of assignment's by dropping pieces off of the front of the string:
- For a Fulfillment Role Assignment (aka Facility Role Assignment) we'd drop the program from the string to create: facility-right
- For an Administrative/Report Role Assignment (aka Direct Role
Assignment) we'd further drop the facility in addition to the program,
and simply create: right
Of course there might be further optimization here that could be made.
- These permission strings are actually UUIDs concatenated by hyphens - not truly strings we'd apply Regex to.
- It also might make more sense to reverse the format, so that it's: right-facility-program, and we'd drop pieces off the back of the string rather than the front for different types of Role Assignments.
- Additionally we'd like to cache a User's permission Strings for a period of time in consuming Services - any change in supervision structure, role definition, facility go-down date, etc would change this list, and we'd need to invalidate it in any caching strategy.
In summary:
- The rights a user has, mapped by any current type of Role Assignment, could be expressed by the Reference Data Service as a set of Permission Strings.
- An entity, such as a Requisition, could be searched for using the databases filtering capability more efficiently if we could compare the Permission Strings required for the entity to the Permission Strings a User has.
- Permission Strings could be further optimized, however we'd like to first validate the concept.
Josh Zamor
--
You received this message because you are subscribed to the Google Groups "OpenLMIS Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openlmis-dev...@googlegroups.com.
To post to this group, send email to openlm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/openlmis-dev/37eacf5b-c447-4785-9280-93c9c97e8100%40googlegroups.com.
Chongsun Ahn
- The permission strings that are associated with a requisition represent the rights that can be performed on that requisition, correct?
- We would store these permission strings for a requisition in the Requisition Service, but based on what you are saying, they wouldn’t be stored in Reference Data, but generated each time the permissionStrings endpoint is called?
- I assume program, facility and right would be UUIDs, which makes sense since I believe it is possible to change code strings for facilities and programs. You mention them as UUIDs, but the example has them as codes.
Shalom,
Chongsun
--
There are 10 kinds of people in this world; those who understand binary, and those who don’t.
On Jun 14, 2017, at 3:58 PM, Josh Zamor <josh....@villagereach.org> wrote:
--
You received this message because you are subscribed to the Google Groups "OpenLMIS Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openlmis-dev...@googlegroups.com.
To post to this group, send email to openlm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/openlmis-dev/37eacf5b-c447-4785-9280-93c9c97e8100%40googlegroups.com.
Josh Zamor
- It’s up to the Requisition service to attach it's meaning to any permission strings during authorization. The first use-case for this is to store the appropriate string(s) which powers a more efficient, database-paginated, /api/requisitions/search. It may be possible for the Requisition service to rely on permission strings for all / near all of it’s authorization checking, however that discovery is left for a later date.
- The ticket OLMIS-2811 calls for Reference Data service to be the place that defines the format and generates the Set for any User. The performance criteria specified (including HTTP level caching support) do I think mean that the Reference Data service implementation will need a clever mechanism to store / cache the generated set, and properly respond to if-none-match in the HTTP header.
- So far the consensus is to use UUIDs for everything, except the right name. The right name being more semantic and unchanging to a Service, this seems reasonable.