End to end notification testing?

Paweł Albecki

unread,

Apr 24, 2018, 11:26:37 AM4/24/18

to OpenLMIS Dev

Hi all,

On one of our Bug triage meetings, Mary Jo asked about automated end-to-end testing around notifications as we encountered few bugs related to notifications during Regression testing. I promised to raise this topic on Dev Forum.

I've also done some research. The simplest way for e2e notification testing would be use our contract tests for this purpose. We could make use of some fake SMPT server (e.g. https://github.com/Nilhcem/FakeSMTP and its docker wrapper https://github.com/digiPlant/docker-fake-smtp) that will intercept sent emails and save them to workspace, then test suite would read and check the email receiver, title, content.

I wonder if it is worth an effort? Do you have any experience with such tests? Please share your opinion.

Best regards,

Paweł.

josh....@openlmis.org

unread,

Apr 24, 2018, 11:40:31 AM4/24/18

to OpenLMIS Dev

Hi Pawel,

Sounds interesting. It'd help to know caused the bugs. As in do we really need to test that we're sending appropriately to an SMTP server, or do we just need to test the contract of the Notification service?

Best,
Josh

Paweł Albecki

unread,

Apr 24, 2018, 12:09:08 PM4/24/18

to josh....@openlmis.org, OpenLMIS Dev

Overall cause of these bugs is that Referencedata does not return correct list of users that should get notification (in some cases it caused error, in other it was silently ignored). The problem is that it is hard to catch such errors in unit tests because Referencedata endpoints work properly, logic in service that sends notification (e.g. CCE using Notification) is fine but the calls are wrong. So what we need to test is that we're sending appropriately to an SMTP server or just correct request to Notification service because there were no bugs related to Notification service itself.

Paweł

--
You received this message because you are subscribed to the Google Groups "OpenLMIS Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openlmis-dev+unsubscribe@googlegroups.com.
To post to this group, send email to openlm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/openlmis-dev/1c42cc72-93e0-4a35-a91e-3917de993e8b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Paweł Albecki
Software Developer
palb...@soldevelo.com

SolDevelo Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

Josh Zamor

unread,

Apr 24, 2018, 12:59:56 PM4/24/18

to Paweł Albecki, Josh Zamor, OpenLMIS Dev

I’d still advise against adding SMTP into the mix as it still sounds like we do not need to test the SMTP contract, but rather that we needed to test another contract(s). Which one I couldn’t say without knowing the workflows involved. You identified the Reference Data service as the root cause. It seems to me like it a test or set of tests could be written to cover the interaction between:

1) The service that contacts reference data service

2) The service that then contacts Notification service

Again I’d only pull in a dummy SMTP tool for tests if we needed to ensure that Notification service is actually properly contacting a SMTP server. That service doesn’t (yet) include any decision logic so it seems a simple set of tests could cover the issue you described. If/when we do get to expanding the Notification service to be multi-channel (email, SMS, in-app, etc), tests that don’t presume that all notifications go out via SMTP would continue to serve our needs in the future.

Obviously I’m commenting a bit in the dark though so if you think I’m missing an important detail please let me know.

Best,

Josh

Paweł Albecki

unread,

Apr 24, 2018, 2:24:53 PM4/24/18

to Josh Zamor, Josh Zamor, OpenLMIS Dev

I don't think you are missing anything important. Regarding testing contract between the service and Reference data/Notification, I don't think we have any good pattern established for this. Impacted code has almost 100% unit test coverage and even if it was 100%, the issue wouldn't be catched because it's more like design issue (like in OLMIS-4580, we didn't expected that supervisory node will be null for some cases) and developer that write unit tests can't always catch such errors. I think we need some approach for automated testing contract between services in which users/domain experts would be more involved.

Regards,

Paweł

To post to this group, send email to openlmis-dev@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/openlmis-dev/1c42cc72-93e0-4a35-a91e-3917de993e8b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Paweł Albecki
Software Developer
palb...@soldevelo.com

SolDevelo Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

--

Paweł Albecki
Software Developer
palb...@soldevelo.com

josh....@openlmis.org

unread,

May 2, 2018, 11:11:35 AM5/2/18

to OpenLMIS Dev

Hi Pawel,

I got delayed. Do we still need a pattern for this? If so could you add the regressions that were found (Jira issues)? I'd like to help if it's still needed however I think some more context would be useful.

Best,
Josh

To unsubscribe from this group and stop receiving emails from it, send an email to openlmis-dev...@googlegroups.com.
To post to this group, send email to openlm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/openlmis-dev/1c42cc72-93e0-4a35-a91e-3917de993e8b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.