External evaluation results of MyChild Solution based on Smart Paper Technology in Afghanistan
Rustam Nabiev
External evaluation results of MyChild Solution based on Smart Paper Technology in Afghanistan. Assessing data quality, operational costs, efficiency gains and transfer of work processes to the existing health system.
In 2015, the Shifo Foundation, the Swedish Committee for Afghanistan (SCA), IKEA Foundation, and the Ministry of Public Health (MoPH) in Afghanistan started a joint collaboration to strengthen child health services in Afghanistan. Data and information are fundamental to inform decisions and assist key stakeholders to allocate appropriate resources to continuously improve the quality of health services. Therefore, one of the main objectives of the collaboration was to strengthen the quality of data and its utilisation in the Expanded Programme on Immunisation using an innovation based on Smart Paper Technology called MyChild Solution.
MyChild Solution is an innovation developed by Shifo Foundation based on Smart Paper Technology. The solution was implemented and evaluated to inform evidence-based decisions on the scale up of the programme. MyChild Solution was implemented in 141 health service delivery points including fixed, outreached, and mobile clinics in the Mehterlam District of Laghman Province in Afghanistan. Currently, using MyChild Solution, more than 45,000 children have ben registered, more than 9,000 children are fully vaccinated and 10,000 children are being followed up with SMS messages which inform parents about vaccination schedules.
From the beginning of the programme, project stakeholders set several programme key success indicators which informed project development and external evaluations. These success indicators measured data quality such as completeness of data, timeliness, internal consistency, and external consistency and analysed if MyChild Solution could be integrated into the existing health system, thus sustained by the government.
In June 2018, two external evaluations were conducted to assess data quality, operational costs, and efficiency gains as well as transfer of work processes to the existing health system. This article summarises the results of these external evaluation reports.
The data quality and review toolkit developed by World Health Organisation was used to evaluate the quality of data generated by MyChild Solution. The assessment showed high quality data generated from MyChild Solution in every indicator, including: completeness (100%), timeliness (91,66%), internal consistency (100%), and external consistency (99,4%). Moreover, the ratio of data recording error was low in the study and ranged from 0.05% to 1.7% for two selected data recording errors.
The second evaluation investigated time efficiency. This evaluation assessed the time health workers spent on administrative tasks during and after delivery of care with MyChild Solution and compared the results with existing Health Management Information (HMIS) tools. Results showed that 64% to 96% of time spent on administration could be reduced with MyChild Solution when compared to the current HMIS.
Incremental cost analysis was done considering two scenarios. The first scenario took into account the monetary value of the reduced time for administration whereas the second scenario was conducted without the time reduction values. The evaluation also took into account two versions of the MyChild Solution. The first evaluation assessed MyChild Forms which is an innovation on facility-based data management tools. The second evaluation assessed MyChild Card which is an innovation modelled after the child health card.
When adding the value of the reduced administration time, the total national cost of MyChild Card was 611,974 USD and the total national cost of MyChild Forms was 316,436 USD. Comparatively, the existing HMIS total national cost was 873,253 USD. Over a five-year period, MyChild Forms would save around 2,938,543 USD and MyChild Card would save around 1,378,875 USD compared to the existing HMIS system. When administration time is removed from the analysis, MyChild Card (501,622 USD) and MyChild Forms (206,126 USD) amounted to be more costly than HMIS forms (195,581 USD). It is noteworthy that when administration time is excluded from the analysis, MyChild Forms were 5% more expensive than HMIS.
The second report displayed the results of the transfer of work processes to the existing health system. This is one of the key elements to evaluate sustainability of the programme as well as how successful management was done by the local stakeholders in Afghanistan.
The results indicate that 95% of the processes essential to the management of MyChild Solution had been transferred to the local stakeholders in Mehterlam District. In most cases, these processes had been transferred in a way that is both accurate and sustainable. The remaining 5% of processes are planned to be fully transferred to Mehterlam by the end of 2018.
The external evaluation reports provide information on the effects of MyChild Solution from four different perspectives which give valuable insights to key stakeholders. These perspectives are data quality, cost, efficiency gains and transfer of work processes to the local level.
Based on the findings derived from these external evaluations Shifo, SCA, and MoPH will collaborate to further expand and investigate the intervention on a broader level to inform decision making for the sub-national implementation of the programme. The programme positively addressed all the key success metrics set in 2015 and brings opportunities to empower health and social workers at all levels of healthcare delivery who continuously work to improve quality of child health services across the country based on reliable and relevant information.
The full reports and additional information about MyChild Solution can be accessed using these links below:
1) Questing The MyChild Solution in Afghanistan- An external evaluation of Data Quality, Operational Cost and Efficiency:
https://shifo.org/doc/rmnch/MyChildExternalEvaluationAfghanistan2018.pdf
2) MyChild Solution in Afghanistan: An External Evaluation - Transfer of Work Processes to Existing Health System: https://shifo.org/doc/rmnch/ShifoExternalEvaluationTransferProcessAfghanistan.pdf
3) About MyChild Solution based on Smart Paper Technology: https://shifo.org/en/solution/
4) Project progress in Afghanistan: https://shifo.org/en/work/afghanistan/
Puta, Chilunga
This is fantastic work and certainly very encouraging results. Were there any challenges related to change management as we have observed that even a good intervention can be limited by health worker perceptions and attitudes. Did you have to deal with any resistance? If so how did you address it? I am really interested in the practical issues of implementing such a program.
Thanks
Chilunga
--
You received this message because you are subscribed to the Google Groups "BID Initiative Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
bidinitiativ...@googlegroups.com.
To post to this group, send email to
bidini...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/bidinitiative/3de5a4ef-385b-4e74-9378-01c7325a53bb%40googlegroups.com.
For more options, visit
https://groups.google.com/d/optout.
Puta, Chilunga
Additionally could you kindly summarize for us in lay man’s terms what Smart Paper Technology is?
Chilunga
From: bidini...@googlegroups.com [mailto:bidini...@googlegroups.com] On Behalf Of Rustam Nabiev
Sent: 12 September 2018 13:10
To: BID Initiative Discussions <bidini...@googlegroups.com>
--
Rustam Nabiev
Puta, Chilunga
Thank you!
To view this discussion on the web visit https://groups.google.com/d/msgid/bidinitiative/5e97433e-b9b0-4cfb-90fa-8ef52c431aba%40googlegroups.com.
caitlin...@swisstph.ch
4) Project progress in Afghanistan: https://shifo.org/en/work/afghanistan
--
You received this message because you are subscribed to the Google Groups "BID Initiative Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bidinitiativ...@googlegroups.com.
To post to this group, send email to bidini...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bidinitiative/3de5a4ef-385b-4e74-9378-01c7325a53bb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "BID Initiative Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
bidinitiativ...@googlegroups.com.
To post to this group, send email to
bidini...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/bidinitiative/5e97433e-b9b0-4cfb-90fa-8ef52c431aba%40googlegroups.com.
For more options, visit
https://groups.google.com/d/optout.
You received this message because you are subscribed to the Google Groups "BID Initiative Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bidinitiativ...@googlegroups.com.
To post to this group, send email to bidini...@googlegroups.com.
Bwakya, Masaina
Dear Rustam,
Many thanks for sharing your experiences, which are very interesting and an eye opener as well.
I have two questions:
1. I would like to know how (the practical strategies used) you have continued to motivate the health workers in encouraging them to use the new solutions and not go back to the old system?
2. How do you hope to sustain this solution so that there is more ownership from the users themselves?
Thank you.
Masaina.
To view this discussion on the web visit https://groups.google.com/d/msgid/bidinitiative/OF84E4E891.833D8BDA-ONC1258307.002566EE-1536821312304%40swisstph.ch.
Mollel, Loishiye
Marius Vouking
--
You received this message because you are subscribed to the Google Groups "BID Initiative Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bidinitiativ...@googlegroups.com.
To post to this group, send email to bidini...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bidinitiative/3de5a4ef-385b-4e74-9378-01c7325a53bb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Marius VOUKING, MPH,
MIVA
Chef de Section
Logistique et Maintenance au GTC PEV, Cameroun
Head of Logistic and Maintenance at CTG EPI, Cameroon
Coordonateur du Hub Cochrane Afrique Francophone au Centre pour le Développement des Bonnes Pratiques en Santé
Coordinator Cochrane Africa Francophone Hub at the Center for Development of Best Practices in Health
Adresses mail: mvou...@gmail.com;
mariu...@yahoo.fr
Tel: +237 677 612 190 (Mobile &
WhatsApp)
http://www.linkedin.com/pub/marius-vouking-zambou/39/899/206
Lukwesa, Peter
Hi Rustam,
Well done for the robust work.
I have questions on system modules and reports.
So apart from registering children, recording fully vaccinated children and defaulter tracing, does the solution also have a vaccine module for vaccine
stock control, and if so how did you work to implement reducing stock balances in real time each time children administered vaccines are recorded?
Are there any reports generated in terms of coverages, dropouts or vaccine usage and how are they generated (triggered or automatically)?
Best wishes
Peter Lukwesa
Systems Implementation Specialist
Mail: PO Box 60045 | Plot No. 2898/207 Highlands, Livingstone Zambia
Street: Off Lusaka Road | Livingstone Zambia
Tel: +260 123 327 233 | +260 977 612 972
www.path.org | www.bidinitiative.org
From: bidini...@googlegroups.com [mailto:bidini...@googlegroups.com] On Behalf Of Rustam Nabiev
Sent: Wednesday, September 12, 2018 1:10 PM
To: BID Initiative Discussions <bidini...@googlegroups.com>
--
Paulo.Nindi
I have two questions i would like to find out.
1. i got interested in the error rate of the system. Following the workflow of the system, it seems most of the conversion to digital data in happening at the district/province and a number of quality checks are happening at this stage. The error rate is pretty impressive i.e standing at almost just 1% (percent) or less. Are there any error trapping mechanisms within the smart forms that ensures data accuracy. Could the quality of training health workers be a big factor here?
2. What could be the pros and cons of using this system in a developing country.
Thanks
Paulo
Rustam Nabiev
Dear Caitlin,
Our lessons of replacing existing paper forms are drawn from Afghanistan, Uganda, Kenya and Gambia, where together with the Ministries of Health (MoH) or Local Governments, we agreed to replace the existing paper forms in the service delivery points where Smart Paper Technology is implemented.
We learned that replacing existing paper forms should be agreed upon as early as possible in the implementation period, in our case even before implementing in the first clinic. We used the following steps to achieve that:
1) Present already available evidence on the benefits/value of SPT compared to existing paper forms.
2) Perform cost analysis prior to starting the project, or during the first months of the project, which informs about the total cost of ownership of the new solution when it is scaled nationally. We only consider costs from MoH perspective, meaning how much MoH should budget annually to operate the solution at the national level. Ideally cost analysis should show how much MoH is spending annually today, and the difference with the new solution. We know that getting the actual cost of the existing solution takes a lot of time, therefore we include printing costs of existing paper forms, which one can get very quickly. This is the most important evidence needed for MoH to make informed decision about replacing existing paper forms. Here is an example we made in Afghanistan, which is done similarly for all other countries as well: https://shifo.org/doc/MyChildCostAnalysisAfghanistan.pdf/(The point is not to be 100% precise in costs, which is never the case, but having an estimation, which can guide the decision-making).
3) Mutually agree on Plan B, C, D in case Plan A doesn’t work. Critical aspect to convince MoH for replacing existing paper forms. For example, what to do if scanning station or internet is not working in the district health office (Plan B), what if backup scanning station or backup internet is not working (Plan C). The last plan should always include a way that health workers can still do their monthly reports correctly. That’s why Smart Paper Forms are designed in such a way that health workers can quickly generate monthly reports manually, in case Smart Paper Forms cannot be scanned and digitized.
4) In case MoH wants to withdraw from the project and go back to existing paper forms, there should be a mechanism to restore all the information, such as register books, tally sheets and monthly report forms, print and deliver them to SDPs. At any moment during the implementation, MoH should be able to exit from the project and all data should be printed and brought to SDPs, so they can continue working with existing paper forms.
5) MoH and Shifo agree from the beginning on the success criteria (indicators and target), how to measure the indicators, and when to do it. Here is an example of Results Framework that defines important indicators and the targets to be achieved: https://doc.shifo.org/display/shareit/32408365/LUTbefbac5a67554221806aaf1a8c1313eeNTP
6) MoH and Shifo agree that during the pilot evaluation, selected SDPs will use only SPT. A separate clause is included in MoU that defines this in details.
7) Start pilot evaluation with one or more selected health service delivery points (SDP), where SPT is introduced and evaluated based on defined success criteria. This step is important. MoH may only be comfortable to pilot in one clinic to begin with and replace existing paper forms there. It’s totally up to MoH on the number of clinics to pilot the new solution.
8) Based on pilot evaluation, MoH decides to withdraw or proceed with expanding SPT to a regional level. In case MoH decides to withdraw and finish the project, Shifo and partners print all the register books, tally sheets and monthly reports and delivers to SDPs, based on which they can continue their work.
9) In case MoH proceeds with expanding SPT, scale-up to selected regions and perform evaluations on a regional level based on success criteria
10) Based on regional scale-up evaluation, MoH decides to withdraw or proceed with expanding SPT to national level
Shifo is convinced that replacing existing paper forms or tools used by health workers from the beginning, making sure health workers only use new solution, is the only way to ensure new solutions are properly introduced, evaluated, scaled-up and 100% integrated into health system structure.
Researcher’s input is important in steps 1, 2, 5, 7, 8, 9, 10.
However, this is what we did at Shifo, and there can be other methods/mechanisms people used and managed to replace existing paper forms. Particularly relevant are experiences of replacing existing paper forms in public health service delivery points. It would be great to learn from other people’s lessons.
Hope I answered your questions.
Kind regards
Rustam
Rustam Nabiev
Dear Masaina,
The first eye opener and most important lessons learned came to us when we understood that Home Based Record is the most important document that health workers use, if they want to know the immunisation history. And in all countries we investigated and observed, health workers screen the child based on Home Based Record. The second lessons learned was that more than 99% of parents bring Home Based Record to immunisation visits. For instance, in Gambia, out of all immunisation visits during February-March 2018, in 0,1% of visits (12/11,519) the Home Based Record was not presented to the health worker. This data was measured in 18 health service delivery points (SDP) where Smart Paper Technology (SPT) was implemented at that time. The third lessons learned was that health workers love their tally sheets and hate register books. We know why J
Equipped with these three insights, we designed Smart Paper Forms in such a way, so that health workers would need to make just one step and be comfortable using the new solution. Simplification of their work processes, eliminating steps they don’t like to do, keeping the steps they like to do was the key to designing a solution that would bring intrinsic motivatation of health workers to use the new tool. We see from observations and quantitative data that there is no motivation for health workers to go back to using the old tools. They don’t miss anything from the old tools.
If you really want to know health workers’intrinsic motivation to use a new tool, an important indicator to measure is data completeness. Measuring it on a monthly basis will inform how motivated health workers are to use the new solution.
If the data completeness is below 95%, it’s already an alert that a health worker is not motivated to use the new tool and wants to go back to the old system.
It’s such an important indicator to measure, so Shifo has divided it into two parts:
1) OI 1.2 – Proportion of fixed sessions performed that are captured in the electronic reports (fixed session data completeness) - Data completeness is defined as data captured from all performed fixed sessions. The schedule provided by clinics for fixed sessions is compared to the dates of sessions for which immunisation data are available in the electronic system. All scheduled, held and missed sessions are detailed on the EPI performance dashboard generated by the solution. A record is kept of cancelled and rescheduled sessions.
2) OI 1.3 – Proportion of outreach sessions performed that are captured in the electronic reports (outreach session data completeness)
Outreach session data completeness is defined as data captured from all performed outreach sessions. The schedule provided by clinics for outreach sessions is compared to the dates of sessions for which immunisation data is available in the electronic system. All scheduled, held and missed sessions are detailed on the EPI performance dashboard generated by the solution. A record is kept of cancelled and rescheduled sessions.
Data completeness is the best indicator to let us know how motivated end-users are to use our solution.
Ownership by health workers of the new solution is equally important. We understand ownership of the new solution by the health workers, through the willingness of health workers to build further capacity of new health workers or any other staff that are using Smart Paper Forms.
In some countries Smart Paper Forms are used for almost three years, such as in Afghanistan. And as turnover is high in health centres, almost all health workers are new members, who did not receive formal training during implementation phase. And despite that, health workers were able to pass on the required knowledge on how to properly use Smart Paper Forms.
Data completeness and incidences of recording error rates made by health workers per health service delivery point gives a good understanding of level of ownership of the new solution by health workers. Recording error rate measures the proportion of data entered incorrectly by health workers. This rate reflects inconsistencies in the data that are most likely due to a mistake made during data recording rather than a mistake made during service delivery. Measured in the health facilities where the solution is implemented. Shifo’s target is that recording error rates should be less than 1% per health centre per month.
We know that local governments don’t have enough resources to train new health workers on how to properly use new tools. Therefore, it was important for us to know that even without a formal training, new health workers can get knowledge from their peers in the same health centre.
Let’s do an experiment. Let me share with you the Smart Paper Forms and Training Manual on how to use Smart Paper Forms. If you figured out just by reading Training Manual what to do, then it means we are in the right direction. If not, then we have a problem.
The training manuals are printed in laminated paper and are kept at each health centre. When new health workers starts the work, he/she goes through the Training Manual. During day to day work, health worker in-charge observes and gives feedback to new health worker.
Link to Smart Paper Forms:
1) MyChild Birth Records Form - https://shifo.bitbucket.io/forms/beta/GMB/1-GMB-06-BIRTH_RECORDS.pdf
2) MyChild Birth Records Update Form - https://shifo.bitbucket.io/forms/beta/GMB/2-GMB-06-BIRTH_RECORDS_UPDATE.pdf
3) MyChild Health Records Form - https://shifo.bitbucket.io/forms/beta/GMB/3-GMB-06-HEALTH_RECORDS.pdf
4) Monthly Return - https://shifo.bitbucket.io/forms/beta/GMB/4-GMB-06-MONTHLY_RETURN.pdf
Link to the Training Manual - https://shifo.bitbucket.io/forms/beta/GMB/GMB-Manual-02.pdf
We assume that health workers are adequately trained on provision of immunisation, immunisation schedule, vaccine stock management
I would appreciate your response.
Kind regards
Rustam
Fred Njobvu
This message is eligible for Automatic Cleanup! (
Dear Rustam,
Many thanks for sharing your experiences, which are very interesting and an eye opener as well.
I have two questions:
1. I would like to know how (the practical strategies used) you have continued to motivate the health workers in encouraging them to use the new solutions and not go back to the old system?
2. How do you hope to sustain this solution so that there is more ownership from the users themselves?
Thank you.
Masaina.
To view this discussion on the web visit https://groups.google.com/d/ msgid/bidinitiative/5e97433e- b9b0-4cfb-90fa-8ef52c431aba% 40googlegroups.com.
For more options, visit https://groups.google.com/d/ optout.
--
You received this message because you are subscribed to the Google Groups "BID Initiative Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bidinitiativ...@ googlegroups.com.
To post to this group, send email to bidini...@googlegroups.com .
To view this discussion on the web visit https://groups.google.com/d/ msgid/bidinitiative/ BN7PR07MB51386CCF1C7732A32575D 33BD01A0%40BN7PR07MB5138. namprd07.prod.outlook.com.
For more options, visit https://groups.google.com/d/ optout.
--
You received this message because you are subscribed to the Google Groups "BID Initiative Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bidinitiativ...@ googlegroups.com.
To post to this group, send email to bidini...@googlegroups.com .
To view this discussion on the web visit https://groups.google.com/d/ msgid/bidinitiative/ OF84E4E891.833D8BDA- ONC1258307.002566EE- 1536821312304%40swisstph.ch.
For more options, visit https://groups.google.com/d/ optout.
--
To unsubscribe from this group and stop receiving emails from it, send an email to bidinitiativ...@googlegroups.com.
To post to this group, send email to bidini...@googlegroups.com.
caitlin...@swisstph.ch
Dear Caitlin,
Our lessons of replacing existing paper forms are drawn from Afghanistan, Uganda, Kenya and Gambia, where together with the Ministries of Health (MoH) or Local Governments, we agreed to replace the existing paper forms in the service delivery points where Smart Paper Technology is implemented.
We learned that replacing existing paper forms should be agreed upon as early as possible in the implementation period, in our case even before implementing in the first clinic. We used the following steps to achieve that:
1) Present already available evidence on the benefits/value of SPT compared to existing paper forms.
2) Perform cost analysis prior to starting the project, or during the first months of the project, which informs about the total cost of ownership of the new solution when it is scaled nationally. We only consider costs from MoH perspective, meaning how much MoH should budget annually to operate the solution at the national level. Ideally cost analysis should show how much MoH is spending annually today, and the difference with the new solution. We know that getting the actual cost of the existing solution takes a lot of time, therefore we include printing costs of existing paper forms, which one can get very quickly. This is the most important evidence needed for MoH to make informed decision about replacing existing paper forms. Here is an example we made in Afghanistan, which is done similarly for all other countries as well: https://shifo.org/doc/MyChildCostAnalysisAfghanistan.pdf(The point is not to be 100% precise in costs, which is never the case, but having an estimation, which can guide the decision-making).
3) Mutually agree on Plan B, C, D in case Plan A doesn’t work. Critical aspect to convince MoH for replacing existing paper forms. For example, what to do if scanning station or internet is not working in the district health office (Plan B), what if backup scanning station or backup internet is not working (Plan C). The last plan should always include a way that health workers can still do their monthly reports correctly. That’s why Smart Paper Forms are designed in such a way that health workers can quickly generate monthly reports manually, in case Smart Paper Forms cannot be scanned and digitized.
4) In case MoH wants to withdraw from the project and go back to existing paper forms, there should be a mechanism to restore all the information, such as register books, tally sheets and monthly report forms, print and deliver them to SDPs. At any moment during the implementation, MoH should be able to exit from the project and all data should be printed and brought to SDPs, so they can continue working with existing paper forms.
5) MoH and Shifo agree from the beginning on the success criteria (indicators and target), how to measure the indicators, and when to do it. Here is an example of Results Framework that defines important indicators and the targets to be achieved: https://doc.shifo.org/display/shareit/32408365/LUTbefbac5a67554221806aaf1a8c1313eeNTP
6) MoH and Shifo agree that during the pilot evaluation, selected SDPs will use only SPT. A separate clause is included in MoU that defines this in details.
7) Start pilot evaluation with one or more selected health service delivery points (SDP), where SPT is introduced and evaluated based on defined success criteria. This step is important. MoH may only be comfortable to pilot in one clinic to begin with and replace existing paper forms there. It’s totally up to MoH on the number of clinics to pilot the new solution.
8) Based on pilot evaluation, MoH decides to withdraw or proceed with expanding SPT to a regional level. In case MoH decides to withdraw and finish the project, Shifo and partners print all the register books, tally sheets and monthly reports and delivers to SDPs, based on which they can continue their work.
9) In case MoH proceeds with expanding SPT, scale-up to selected regions and perform evaluations on a regional level based on success criteria
10) Based on regional scale-up evaluation, MoH decides to withdraw or proceed with expanding SPT to national level
Shifo is convinced that replacing existing paper forms or tools used by health workers from the beginning, making sure health workers only use new solution, is the only way to ensure new solutions are properly introduced, evaluated, scaled-up and 100% integrated into health system structure.
Researcher’s input is important in steps 1, 2, 5, 7, 8, 9, 10.
However, this is what we did at Shifo, and there can be other methods/mechanisms people used and managed to replace existing paper forms. Particularly relevant are experiences of replacing existing paper forms in public health service delivery points. It would be great to learn from other people’s lessons.
Hope I answered your questions.
Kind regards
Rustam
Dear Rustam,Thank you for all the details so far on your work - well done.I am interested to know more about the process of decision making at the Ministry level that enabled action to take place re: replacement of old system with new Smart Paper Forms.I guess my high level starter questions would be - how involved were the researchers in influencing this decision making? What kind of activities took place to support visibility of evidence at the right level?Many thanks,Caitlin
Thank you!
From: bidini...@googlegroups.com [mailto:bidini...@googlegroups.com] On Behalf Of Rustam Nabiev
Sent: 12 September 2018 19:36
To: BID Initiative Discussions <bidini...@googlegroups.com>
Subject: Re: External evaluation results of MyChild Solution based on Smart Paper Technology in Afghanistan
Dear Chilunga,
The biggest challenge was to take away existing paper forms from health workers' work processes. Once that was done, health workers were keen on using Smart Paper Forms and we didn't encounter resistance from them. On the contrary, health workers were keen to integrate the new solution as part of their daily routines, they seem to love it and we understand why.
I would equally love it if someone created a solution which would potentially save 60-70% of my routine admin time.
When MoPH was informed about the total cost of ownership, data quality, admin time saved for health workers and other benefits of Smart Paper Technology, MoPH decided to take away existing paper forms (registers, tally sheets, monthly reporting forms) from health worker's work processes.
Regarding transfer/integration of new work processes into existing health system structure, it took some time to figure out locally adapted processes to update, print, distribute and archive Smart Paper Forms. It was also important to monitor whether health workers were able to build capacity to new team members and community health workers how to properly use Smart Paper Forms, which we measure by amount of errors made per month. It was also possible to transfer maintenance of scanning stations, as scanning stations are placed in district/provincial health office. We know that continuous technical support and continuous need for capacity building are two big bottlenecks for sustaining eHealth interventions.
What we are learning at this stage is how to build a capability of the health system to do continuous quality improvement to improve quality of care (this is Shifo's understanding of data use). It's incredibly important to get this right, as it can have a big potential in improving quality of care and health outcomes.
Shifo's target is that 95% of health facilities would be doing continuous quality improvement to improve indicators that are below the target. Let's see if Shifo's Data4Action interventions can help in reaching that target.
Smart Paper Technology is a solution to digitise hand-recorded smart paper forms, which would create Electronic Birth and Immunisation Register for each health facility. Smart Paper Technology has a potential to digitise health records within Primary Healthcare. Smart paper forms are used at the point of health service delivery to register individual children, assign unique ID and record delivery of child health services. Smart Paper Forms are then scanned and digitised at the scanning centre, and processed, after which digital data is generated and integrated with DHIS2, CRVS, LMIS and other systems. Here is a short description of how this machine works: https://shifo.org/docrmnch/RNMCHSolutionProcess.pdf
https://shifo.org/doc/rmnchMyChildExternalEvaluationAfghanistan2018.pdf
2) MyChild Solution in Afghanistan: An External Evaluation - Transfer of Work Processes to Existing Health System: https://shifo.org/docrmnch/ShifoExternalEvaluationTransferProcessAfghanistan.pdf
3) About MyChild Solution based on Smart Paper Technology: https://shifo.orgen/solution
4) Project progress in Afghanistan: https://shifo.org/en/work/afghanistan
--
You received this message because you are subscribed to the Google Groups "BID Initiative Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bidinitiativ...@googlegroups.com.
To post to this group, send email to bidini...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/dmsgid/bidinitiative/3de5a4ef-385b-4e74-9378-01c7325a53bb%40googlegroups.com.
For more options, visit https://groups.google.com/doptout.
--
You received this message because you are subscribed to the Google Groups "BID Initiative Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bidinitiativ...@googlegroups.com.
To post to this group, send email to bidini...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/dmsgid/bidinitiative/5e97433e-b9b0-4cfb-90fa-8ef52c431aba%40googlegroups.com.
For more options, visit https://groups.google.com/doptout.
--
You received this message because you are subscribed to the Google Groups "BID Initiative Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bidinitiativ...@googlegroups.com.
To post to this group, send email to bidini...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/dmsgid/bidinitiative/BN7PR07MB51386CCF1C7732A32575D33BD01A0%40BN7PR07MB5138.namprd07.prod.outlook.com.
For more options, visit https://groups.google.com/doptout.
You received this message because you are subscribed to the Google Groups "BID Initiative Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bidinitiativ...@googlegroups.com.
To post to this group, send email to bidini...@googlegroups.com.
Bwakya, Masaina
Thanks Rustam. This is very informative and there are many lessons to learn here.
Regards,
Masaina.
To view this discussion on the web visit https://groups.google.com/d/msgid/bidinitiative/1860481695.7626311.1536904321950%40mail.yahoo.com.
Rustam Nabiev
Hi Mollel,
Frequency of scanning smart paper forms varies depending on the country and setup. The minimum requirement is to scan once a month, which satisfies all requirements of routine health service delivery and deadlines for generating monthly data. For instance in Gambia, smart paper forms are scanned once a week, because delivery of smart paper forms is connected with disease surveillance reporting, which happens once a week. In Uganda, large clinics also install scanning stations, which means that smart paper forms are scanned every day, after immunisation sessions are finished.
Shifo usually agrees with MoH on the frequency of delivering smart paper forms, and this is usually connected to already existing routines in the country.
In most countries, all electronic data should be available by 5th of every month. Hence, smart paper forms are brought in the beginning of the month (deadline is 1st of every month), scanned latest by 3rd of every month, and data is quality assured and generated by 5th of every month.
If you noticed in Afghanistan evaluation, one of the months, the data was generated 6th day of the month, therefore, the data timeliness dropped to 91.66%, which is below Shifo’s target of 99% for data timeliness.
Countries also have an option to scan smart paper forms using smartphones. We have started exploring this option in Gambia, where there is at least one health worker who owns a personal smartphone. Therefore, in principle, daily scanning and digitisation can be done in a near future. But Shifo only recommends it in countries where Governments can purchase internet connection to health centres on their own.
Scanning process is relatively fast, one second per two pages. On average, medium size health centres may have 40-50 smart paper forms per month. It takes 20-25 seconds of scanning per health centre and about 5-10 minutes for all health centres in the district. Digitisation of smart paper forms happens in the background and also takes about 20-25 seconds per health centre. Once data is digitised, it goes through automatic and manual quality assurance, so that data quality can get to 99%. With automatic quality assurance, data quality can get up to 95-96%. And in order to reach to 99%, manual data quality assurance is done. Manual data quality assurance takes roughly 5-7 seconds per child visit, which means that for a country with approximately 100 000 newborns per year (e.g. Gambia), equivalent of one person full time should be there to do manual data quality assurance for the whole country. In countries where we work, responsibility for manual data quality assurance is given either to EPI focal person or HMIS focal person in the districts.
Each scanning station can work up to 7-10 years, given that maintenance every 6 month is done. Additionally, every scanning centres are provided with back-up scanning station, which theoretically enables equipment functionality up to 20 years. And we usually put 2-3 scanning stations per district in a cluster, to be close to health centres, so that there is no bottleneck with scanning smart paper forms. In case something happens in the whole district, there is always option to scan in the neighbouring district. In the worst case scenario when no scanning centre in the country works, health centres and districts can still calculate easily all required reports from smart paper forms. When designing smart paper forms, we tried to consider worst case scenario as much as possible.
Hope I answered your questions.
Kind regards
Rustam
Rustam Nabiev
Hi Peter,
Initially when we started our work, we focused on generating electronic immunisation register and all other HMIS monthly reports, which were mandatory. But then we understood that health workers have some challenges calculating monthly stock reports and request notes for next month vaccines and supplies for their health facilities. In order to address this challenge, we developed a component, which automatically generates monthly stock management report, which health workers were supposed to do manually before. Here is an example of the report that is produced automatically by the system: https://cdn-images-1.medium.com/max/1200/1*zd-5uxnamgA8f9dpFJdoTg.jpeg
Given that the system knows the real consumption rate of every health centre, open and closed vial wastage rates, and other important metrics, next step was for the system to forecast necessary amount of vaccines and supplies for the next month, in order to reduce overstocking and understocking of vaccines and supplies. About one year ago, this has been rolled out in Afghanistan and recently we introduced it in Gambia as well. System generates dispatch voucher (in push based model) or request form (in pull based model) and sends to district vaccine store. Based on this information, logistician delivers necessary vaccines and supplies to each health centre. Here is an example of dispatch voucher https://shifo.org/doc/DispatchVoucher.pdf/ and request form template https://shifo.org/doc/VaccinesAndSupplyRequestForm.pdf/
We considered previously used methods (e.g. based on previous consumption or based on target population), to calculate next month needs for vaccines and supplies. We have recently started working on the new prediction method, to improve forecast accuracy. We plan that evaluation of the new prediction method will be completed during Q3 2019.
In regards to other indicators, such as coverage, drop-outs, vaccine usage, the system generates these indicators automatically and sends by SMS to health workers’mobile phones. Based on this, health workers plot those indicators on the Data4Action Notice Boards in the health centres. Shifo agrees with country key stakeholders (MoH, District/Regional Health Office, Health Centres, GAVI, UNICEF, WHO, NGOs) on the indicators to monitor and improve. We believe mutual agreement of key stakeholders on improving set of quality of care indicators is crucial for continuous quality improvement of selected indicators. Here is an example from Gambia of indicators that are generated by the system on a monthly basis and sent by SMS to health workers: https://shifo.bitbucket.io/forms/beta/GMB/D4A/.
If you are curious about how a health worker in Uganda is plotting the run charts based on received SMS, here you can see it: https://drive.google.com/file/d/0B_hcV_XI-yOPUDkxTl90cjlEQ0xZelFVRV9DWGxEU3NzMEE4/view?usp=sharing
Here is an example from Afghanistan: https://drive.google.com/file/d/0B_hcV_XI-yOPaTVyM3hqZUdoZ3VOcFpjTjJCS2k5cW9Jc3RR/view?usp=sharing
Hope I answered your questions.
Kind regards
Rustam
Rustam Nabiev
Hi Paulo,
We know that health wokers will make data entry errors, due to many circumstances. However, as you pointed out, it’s important to set the target and monitor data enrtry errors every month. Shifo’s target is 1% error rate at any given month per health centre. In order to achieve that, we took into account the following points:
1) design of smart paper forms should be as simple as possible, so that health workers know intuitively what is the right thing to do
2) system should have validation algorithms that can automatically capture data entry mistakes and report this information to data quality assurance officer and to the health centres for data quality improvement
3) system should have a mechanism to present data entry errors to data qualtiy assurance officer, who validates and if necessary corrects those mistakes. Some mistakes are automatically corrected based on rules agreed upon with MoH and other stakeholders.
4) On a monthly basis, percentage of data entry errors are generated and sent to health workers and other stakeholders, for continuous quality improvement actions.
Proper training of health workers is important, however, as turnover is quite high and because districts don’t have enough resources to train new health workers, it is important to find more affordable methods of minimising data entry errors, and consistently keeping them low. Shifo is relying on two methods:
1) Fostering health workers to pass on the knowledge and experience to new health workers or students, who help during immunisation sessions
2) Designing intuitive training manual, which can help a new health worker understand how to fill-in correctly smart paper forms
If you have some suggestions based on your own experience, would appreciate to receive your thoughts on what we can improve here.
What could be the pros and cons of using this system in a developing country?
Compared to existing paper forms used, Smart Paper Technology gives countries 1) a capability to generate individual patient level data that is above 99% complete, consistent, timely, 2) generate all required HMIS and LMIS reports automatically, 3) reduce health worker administration time by more than 60%, 4) the solution that can be implemented in any health centre, outreach site and mobile clinics, bypassing infrastructure limitations, 5) at a cost that is affordable to any country to sustain on their own. For instance, recent external evaluation shows that to sustain Smart Paper Technology solution at a national level, Government of Gambia needs to allocate 11 756 USD per year (0,13 USD per child/year). Today, printing existing paper forms cost 8 792 USD per year (0,10 USD per child/year) for Gambian Government.
Considering the existing setup, where scanning centres are established at a district level, this solution satisfies routine primary healthcare service delivery requirements. However, if there is a need to obtain real-time data or digital data should be available every day, for instance for disease surveillance, or for other needs, then with current setup it won’t be able to satisfy those use-cases. And with current setup, health centres need to bring smart paper forms physically to the scanning centres.
We are aware of these limitations, but they were done by design. And depending on the country’s readiness, we recommend an option where smart paper forms can be scanned using smartphones. With this option, data can be generated on a daily basis and there won’t be a need to bring smart paper forms physically to the scanning centres. However, we only recommend using smartphones in countries where at least 95% of health centres are connected to reliable electricity, at least 95% of health centres are using Internet paid by the Government and technical support resources are available at the district level. More and more organisations are realising these essential requirements. PAHO recently published practical considerations, for planning, development, implementation and evaluation of electronic immunization registries. It’s a lengthy but valuable report, and in page 34, you will find a table that includes factors that determine the feasibility of developing an EIR, where they bring up essential factors needed for adopting EIR solutions. Worthwhile to read for people that are implementing Electronic Immunisation Registries: http://iris.paho.org/xmlui/bitstream/handle/123456789/34865/9789275119532_eng.pdf
Hope I answered your questions.
Kind regards
RustamRustam Nabiev
Dear Chilunga,
Another point about resistance is that mostly it happens when one implements a new solution in parallel to the existing solution. This is what happened to Shifo when we implemented MyChild App in parallel to the existing paper forms in Uganda. Until 2-3 months, health workers were excited about the new technology (in our case a netbook/tablet based solution). But after 6 months, we noticed that the usage has dropped dramatically, and health workers were just using existing paper forms. We didn’t give any extra incentives at the time, because project investments came from ourselves (sometimes projects may give unlimited internet bundles, or extra cash) to health workers, which could potentially reduce the resistance for quite some time. When we came to conclusion that even if we would replace existing paper forms, it wouldn’t be feasible for the district/country to sustain MyChild App on their own, we agreed with the district and health centres to transition to Smart Paper Technology.
We recommend to organisations implementing EIR solutions to make sure existing paper forms are taken away from health workers’daily routines as early as possible, so they can have a better experience of using a new solution, hence avoiding the resistance.
Kind regards
Rustam
Puta, Chilunga
Thank you Rustam. This is very helpful information.
Best
To view this discussion on the web visit https://groups.google.com/d/msgid/bidinitiative/39d6e285-5238-4900-9d00-c145928a4d0e%40googlegroups.com.
Mollel, Loishiye
Thank you so much Rustam for you detailed response. You have answered my questions very well, yet am interested to know the scanning technology you are using. Is it possible to share the VIEW of the paper forms how they look like?
Regards
Mollel
To view this discussion on the web visit https://groups.google.com/d/msgid/bidinitiative/ffe7f265-851c-4eb7-bc40-c580b962dc22%40googlegroups.com.
Lukwesa, Peter
Hi Rustam,
Thank you for sharing, this is very informative.
Kind regards
To view this discussion on the web visit https://groups.google.com/d/msgid/bidinitiative/e039b28b-8260-4466-8cc8-77ab6264ac46%40googlegroups.com.