Operability, Acceptability, and Usefulness of a Mobile App to Track Routine Immunization Performance in Rural Pakistan: Interview Study Among Vaccinators and Key Informants

"...provides lessons for LMICS on technological and non-technological factors that require attention to contextualize health provider apps within the local health system."

Despite the proliferation of mobile health (mHealth) apps for immunisations and other public health concerns in low- and middle-income countries (LMICs), not all digital health programmes perform as intended, and assessments of the technology, health systems, and behavioural factors are needed. This study explored experiences with an Android-based immunisation app to improve routine childhood vaccination coverage in a rural, disadvantaged district of Pakistan. The specific objectives were to determine the acceptance and operability of the app as a tool for tracking vaccine encounters within the district health system, assess the data validity and data-related concerns of stakeholders, and understand how the technology is being used within the vaccine delivery system.

Developed by Aga Khan University (AKU) Pediatrics Department, Teeko is an Android-based mobile app that tracks vaccinators' performance. It was co-designed with the sub-national government as part of a larger health system strengthening research programme for routine childhood immunisation and piloted in a rural district of Sindh, Pakistan in 11 union councils (UCs) of the Tando Muhammad Khan (TMK) district. (This district had poor vaccination performance, with a Penta (DTP-HepB/Hib) coverage rate of only 23% and pneumococcal conjugate (PCV) coverage of only 11% at the start of the study; these figures are far lower than the country target of 90% of children completely immunised, set by the Global Vaccine Action Plan - GVAP.) The app generates quality real-time data, and its key features include Global Positioning System (GPS) tracking of vaccinator outreach visits, digital records of the immunisation volume at the static health centre and during outreach visits, creation of the next scheduled immunisation encounter, identification of children who missed vaccination, and communication features for parents. The app has 2 components: data entry and a Web portal for data visualisation.

Teeko, the workings of which are described in detail in the article, was developed over a 6-month period in 2014-2015. Meetings were held during the design phase and continued into the app roll-out. Using a participatory process, the following app features were among those identified: compulsory photo identification; defaulter (children who missed a vaccination) status using traffic light colors; alignment with EPI's management information system; and client data to include the parents' national ID card numbers, household number, and village details. Iterations included the addition of new vaccines, identification of areas of misreporting, adjustment for vaccinations received outside the study area, and adjustment for household migration. To improve the technological features, the study team and technology partner field-tested different versions of the app with vaccinators and the district EPI focal person. Due to poor internet connectivity, the ability to upload data offline was added after the field testing.

The Teeko intervention was implemented over 24 months (2015-2017), after which time the research was conducted. As part of the intervention, vaccinators and their sub-district and district EPI supervisors were provided with smartphones with the Teeko app. (Complementary interventions included facilitation of a district Expanded Program for Immunization (EPI) review platform, microplanning training, fuel support for outreach activities, and co-financing of EPI motorbikes for UCs lacking functional bikes for outreach sessions.) The immunisation programme resulted in a significantly higher number of completed vaccinations at the intervention sites than at the control sites, as measured in terms of Penta 3 and PCV 3 coverage (see Table 1 in the paper).

The research consisted of in-depth interviews with 26 vaccinators posted in the 11 UCs and 7 purposively selected key informants (government district managers) involved with the EPI.

Perspectives of the vaccinators - selected results (see Table 3 for a list of specific responses provided by vaccinators during their interviews):

• Acceptability and operability of the app: All 26 vaccinators used the app to record their vaccination encounters. Nearly all (n=24) took post-vaccination photos of the children; some parents refused the photos. Eighteen vaccinators used all the app features for routine immunisation (RI) awareness, including the awareness video, SMS (text messaging) alerts, and robotic calls. Of the 26 vaccinators, 25 reported that the app was easy to use and they could operate it without difficulty.
• Validity of the app data: Most of the vaccinators (n=23) reported a high likelihood of error with manual documentation of outreach vaccination encounters. The post-vaccination photograph was considered key to ensuring reliable data. Vaccinators reported that the GPS tracking of outreach visits resulted in greater vaccinator vigilance in conducting outreach sessions and improved the reliability of vaccination encounter reports.
• Use of the app data: Vaccinators expressed ownership of the data and a feeling of empowerment when using the data to plan their monthly vaccination rounds. They reported feeling gratified when the best-performing vaccinators were publicly recognised during the review of individual performance levels in the monthly EPI meetings. (In addition to recognition, high-performing vaccinators received EPI motorbikes from the district health office.) Most of the vaccinators reported that this motivated them to do their best. Others reported this was the first time they took their work seriously.

Perspectives of the district stakeholders - selected results (see Table 4 for a list of specific responses provided by these key informants during their interviews):

• Acceptability of the app: District officials found it easy to retrieve RI volume from their cell phones and computers, and the officials used the app to keep track of children who missed immunisations. An unintended knock-on effect was improved coordination and strengthening of EPI review platforms across district stakeholders through digitalised data; e.g., the stakeholders reported that the app led to increased coordination between the EPI and Lady Health Worker (LHW) programme. The health officials expressed an eagerness to continue the tracking initiative and an anxiety that RI coverage would decline again after closure of the project. They also requested that the project be expanded to the other sub-divisions within the district and "were gratified that other districts requested demonstrations and sharing of lessons learned".
• Perception of data validity: Key informants reported that tracking a vaccinator in the field was challenging in routine practice. However, the app data were considered superior in quality to manually collected data.
• Use of the app data: Regular EPI review meetings were convened at the district health office, and the meetings used the real-time app data. In the past, it was challenging to hold the vaccinator cadre accountable for RI performance, making it difficult to improve vaccination volume. To that end, the data from the app were used to improve the number of RI encounters and to locate defaulting children. District stakeholders also expressed a sense of empowerment from using the app to make operational decisions to improve RI in the district.

All in all, to embed digital tracking in vaccine delivery systems, ease of operability, personal recognition, results-oriented mobility support, and empowerment to improvise microplanning are individual-level enables, whereas data transparency and empowerment for district planning are key organisational-level enablers. In terms of constraints, double entry of both manual and digital records was considered time consuming, and aging phone technology created systems issues. (However, the ability for real-time tracking of both individual vaccinators and the entire team through the Web portal considerably reduced the time spent tracking vaccinators during outreach visits.)

In conclusion, this study shows that embedding digital technology into mainstream health systems relies on acceptance by both end users and district stakeholders. "While several of the findings are specific to the study context, the study provides the key parameters for consideration during the current rollout of digital immunization interventions in Pakistan as well as lessons for the introduction of a health provider app in LMICS."