More than 2000 NFS products - a blocker in Malawi

[jkon...@soldevelo.com]

Hi all,

Recently, we were asked to add more Non Full Supply products in Malawi, but we discovered that there is a bug - despite having 4500 NFS products in a Requisition, the endpoint only returns 2000 of them. 

I investigated it and it seems that the rest is lost when converting a requisition to DTO:

1. RequisitionDtoBuilder gets ids of available NFS products from a requisition

2. Asks /api/orderables/search for dtos, but only gets 2000

3. In result, a requisition dto is limited to 2000 available NFS products.

It seems that this bug was fixed recently in reference data: https://review.openlmis.org/cru/FEOLMIS-2499#CFR-34994

and there's a ticket for other services: https://openlmis.atlassian.net/browse/OLMIS-4128

However, having more than 2000 products is critical for our stakeholders and we cannot wait for the 3.3 release. Since the patch release is not possible, we are looking for a workaround. 

Do you think that a new microservice acting as a expansion to requisition service could do the job? It could fetch requisition's available product ids with SQL and make a couple of requests to referencedata to fetch them in chunks.

The downside of that solution is that it would take about a week of work, we would also have to update Malawi's UI to use this new endpoint.

Do you have any ideas on how to approach this?

Thanks,

Jakub

[Josh Zamor]

Thanks Jakub,

This is certainly a good point to reflect.  My previous understanding of this Spring framework default was that it was a default page size when asking for a paginated resource and not giving the page size parameter.  Now I understand that this is not just a default page size but a default maximum page size of a paginated resource - a client can’t ever request a page larger than 2000 items.  This is certainly not the behavior of what we wanted in our style guide:

When an endpoint is paginated, and the pagination options are not given, then we return the full collection. i.e. a single page with every possible instance of that resource. It's therefore up to the client to use collection endpoints responsibly and not over-load the backend.

This still feels like the right pragmatic approach for OpenLMIS, though I do wonder if relying on this behavior has made us forget why resources are paginated.

Next steps:

• Agreed that the only two solutions that can solve the issue are either a patch release (of either Reference Data or Requisition), or some middle-ware Service that proxies Requisition’s need of > 2000 Orderables at a time from Reference Data.
• On the second option, a middle-ware service, I’m not clear why this service would lay between Requisition and Reference Data as well as between the UI and Reference Data.  Or why it’d need SQL access to Requisition’s DB.  Couldn’t it intercept the Request to Reference Data’s Orderable’s and put together pages of 2000 items, returning one large page to any client?  Doing so would require some form of re-written rule about the location of the GET Orderables resource held in Consul, but I believe this would be possible without changing Reference Data.  Thoughts?  What am I missing?

I’ll followup later today with more thoughts on timing.

Best,

Josh

SolDevelo Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

--
You received this message because you are subscribed to the Google Groups "OpenLMIS Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openlmis-dev...@googlegroups.com.
To post to this group, send email to openlm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/openlmis-dev/1781d152-7cc0-4b85-aa5f-85bf9115d7ed%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jakub Kondrat]

Hi Josh,

I agree that we could try intercepting the request to Reference Data. The advantage of this approach is that we wouldn't need to execute any sql queries or to modify Malawi's UI.

We would still have set up this new service and re-write nginx rules. If you think that a patch release is possible, it would certainly save time.

Thanks,
Jakub

To view this discussion on the web visit https://groups.google.com/d/msgid/openlmis-dev/0DA85A78-4DC3-4FAB-AB01-D627903C22A4%40villagereach.org.

For more options, visit https://groups.google.com/d/optout.

--

Jakub Kondrat
Software Developer
jkon...@soldevelo.com

[josh....@openlmis.org]

Hi Jakub,

Thanks.  Agreed about the patch release.  While our CI/CD server doesn't yet support patch releases, I think we should do it in this instance - though we'll need to be careful.

To make this patch release I'd expect:

1. Branch a hotfix branch from Reference Data's master branch of the 9.0.0 release, named hotfix-9.0.1.
2. Set the version to 9.0.1.
3. Make the code-change (my understanding is it's mostly a one line change in CustomWebMvcConfigurerAdapter) and commit (please have an OLMIS-* Jira issue to track the commit against).
4. Ensure the build and tests pass.
5. Run contract tests locally and post the logs into a file attached to the OLMIS jira issue.
6. Run performance tests, feel free to take over the perftest server for this for a day.
   
7. If all passes, tag as v9.0.1 on github and push the hotfix release to docker hub as 9.0.1 (you'll need to do this manually most likely).
8. If this change isn't coming back into master (which deviates from gitflow), as we've already made the fix for v10, then let me know.
9. We will lock the hotfix branch, I don't ever want a 9.0.2.  9.0.1's upgrade path will be v10.

Am I missing anything?

Our overall goal is to keep the change minimal, nothing unneeded, and minimize risk by having a clean developer machine.  Lets also post the build and test output's to the jira issue for transparency.

Thanks Jakub for helping get the best solution for Malawi out.  If you have any questions about this then please let me know.

Best,
Josh

To post to this group, send email to ...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/openlmis-dev/1781d152-7cc0-4b85-aa5f-85bf9115d7ed%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "OpenLMIS Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openlmis-dev...@googlegroups.com.
To post to this group, send email to openlm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/openlmis-dev/0DA85A78-4DC3-4FAB-AB01-D627903C22A4%40villagereach.org.
For more options, visit https://groups.google.com/d/optout.

--

Jakub Kondrat
Software Developer
jkon...@soldevelo.com

[jkon...@soldevelo.com]

Hi Josh,

I have followed your steps and released the hotfix. The results of performance and contract tests are in a zip file attached to the Jira issue (OLMIS-4192). 

Kind regards,

Jakub