Starting a discussion about Reporting Stack Services

[Craig Appl]

Hi,

The reporting stack for OpenLMIS v3 includes a number of components, Nifi, Kafka, Druid and Superset. We are planning on creating a docker container for each of these services. I need help figuring out to structure the GitHub repositories associated with the reporting stack.

Here are some questions:

- Should we run the entire reporting stack as a single microservice, or run each component independently?

- There are dependencies between each of the systems that require them all to be up to function properly. Should we register one "reporting stack" with Consul or register each component independently? (Do we even need to register this with Consul?)

- Should we have one GitHub repository for the entire reporting stack, or one repository for each docker container/component?

- Should I create these docker containers on OpenLMIS Docker Hub or is it fine to use third party docker containers? (Ona might even consider creating our own on docker hub)

Thank you,

Craig

[Josh Zamor]

Thanks for starting this Craig, it’s great to see.

Jumping right into the questions:

- Should we run the entire reporting stack as a single microservice, or run each component independently?

I strongly encourage that each runs separately.  This is reflected in our docker practices which is itself a reflection both of docker’s recommended practice and our own hard-learned lessons.  There are a whole number of reasons for this, so I’ll just say every-time (okay not every-time, our Nginx image ignores this and does fine) I haven’t heeded this advice, I’ve regretted it shortly after.

- There are dependencies between each of the systems that require them all to be up to function properly. Should we register one "reporting stack" with Consul or register each component independently? (Do we even need to register this with Consul?)

This is the big question.  I think for the moment we should skip the Consul registration, and especially skip the reverse proxy step.  Both OpenLMIS and the reporting stack will need enough resources in production that fitting it all onto a single instance seems less practical.  For testing and local dev work we’ll likely write special compose files that brings up the basics all in one host, but we’ll expect that they’ll communicate through public addresses.

- Should we have one GitHub repository for the entire reporting stack, or one repository for each docker container/component?

I think we’ll want to version each image separately and so I’ll say separate repos - it makes it easier to have docker hub build that way based on the repos tag.

- Should I create these docker containers on OpenLMIS Docker Hub or is it fine to use third party docker containers? (Ona might even consider creating our own on docker hub)

I think we should either use the external images directly if they’re official images (e.g. the official Nifi), or else we should place them in the OpenLMIS Docker Hub.  This makes sense if the repos will be in the OpenLMIS GitHub org. One question I have though is do we need our own images?  Or do we need to configure the official images?  I’d encourage us to do as little image creation as possible if the official images exist and largely suit our needs.  An example is our Postgres image which starts from an official image and does the bare minimum configuration to make it useful to developers.

Best,

Josh

--
You received this message because you are subscribed to the Google Groups "OpenLMIS Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openlmis-dev...@googlegroups.com.
To post to this group, send email to openlm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/openlmis-dev/9d44e965-09bb-49c1-baf5-7d35d5e79304%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Craig Appl]

Thanks Josh,

I have taken these points and developed a proposed structure document on the wiki: https://openlmis.atlassian.net/wiki/spaces/OP/pages/374702119/Proposed+Reporting+Platform+Repository+Structure Please feel free to review it and I will aim to have a discussion the week of the 14th when Peter returns from Holiday.

Sincerely,

Craig

To post to this group, send email to openl...@googlegroups.com.