The Child Status Index Usage Assessment

This report evaluates a monitoring and evaluation (M&E) tool that was created in order to assess programmes designed to meet the needs of children who are orphaned and made vulnerable by HIV/AIDS. The Child Status Index (CSI), created in 2008, was designed to meet demand for a tool that could be implemented by low-literate (typically volunteer) community caregivers to periodically capture children's status with regard to the 6 domains of the United States (US) President's Emergency Plan for AIDS Relief (PEPFAR) programmes for orphans and vulnerable children (OVC): food and nutrition; shelter and care; child protection; health; psychosocial; and education and skills training. (Editor's note: To access any of the many components of the CSI tool, click here).

As detailed here, the CSI authors used a community participatory process to create the CSI, and the tool was field-tested for construct validity and inter-rater reliability. Once the tool was complete, a manual, field guide, and booklet for the CSI were published and posted to the MEASURE Evaluation website. A regional conference was held in Kigali, Rwanda, in 2009 to provide training. Participants felt that they would benefit from an online community of practice, so Child Status Net (CSNet) was launched as the primary vehicle for sharing knowledge and experiences regarding the CSI. In addition, a training-of-trainers session was held in Mombasa, Kenya, in the summer of 2010. Otherwise, training for the CSI was conducted informally.

In 2011, in order to systematically assess how programmes implement and use the CSI and to understand programme field needs for additional tools to meet care, support, and M&E demands, the U.S. Agency for International Development (USAID)-funded MEASURE Evaluation conducted in-depth telephone interviews with programme technical leads among large programmes implementing the CSI. A total of 25 organisations in 13 countries were represented in these interviews; 21 interviews were conducted with organisations in sub-Saharan Africa and 4 with those in the Latin America/Caribbean and Asia regions. Participants were asked about materials created to complement the CSI, such as presentations, manuals, training materials, new or additional tools linked to child well-being measurement, and other language translations of the CSI.

The most common uses of the CSI indicated by respondents were: to conduct needs assessments; to monitor the status of individual children; and as a decision support tool for individual children. One interviewee described how that organisation analyses the CSI data to make programme decisions: "An electronic database has been developed by us to generate information on each OVC in a local government area/district. The program is able, at a click, to view the proportion of OVC enrolled with very bad, bad, and fair scores and plan areas of resource needs." Several programmes also reported using the CSI for outcome evaluation, programme evaluation, and performance monitoring.

The CSI has been translated into at least 15 different languages. Most interviewees described making minor changes to the CSI, such as changing pictures or scales. However, several organisations made more significant changes, such as removing/modifying entire domains. In two instances, the CSI had been changed extensively and renamed.

Most organisations conduct step-down or cascade training for the CSI, though a few organisations reported conducting ad hoc or informal training. The amount of training time devoted to the CSI ranged from 45 minutes to three-and-a-half days. Several concerns were raised regarding the cascade training approach, in that the quality of training declines as the training steps down each level. The greatest challenge described by interviewees was that the CSI scores are based on perceptions of the CSI users and caregivers, which led to concern that inter-rater reliability may be low. The achievement of inter-rater reliability requires the CSI users to be sufficiently trained and supported in the field to manage their caseload and assess the needs of children in their care using the CSI.

Despite challenges raised with implementation of the CSI, all interviewees reported that the CSI is useful. Respondents reported that the CSI is comprehensive, allows for monitoring the well-being of children, aids in programme planning, uses a numeric scoring system, and provides a framework for service delivery of programs for children who are orphaned and made vulnerable by HIV/AIDS. Further, four countries have included the CSI as a national tool, and two countries are linking the CSI to their service delivery standards and quality improvement guidelines.

In light of findings, the following recommendations are provided:

• Clarification should be provided on the more challenging aspects of the CSI implementation, including appropriate uses of the CSI, referral protocols, and data management and analysis.
• A training manual for the CSI should be made available.
• Existing documents (manual, field guide, and a "made easy" booklet) for the CSI should be revised to reflect new guidance. The manual should, to the extent possible, provide further explanation on how to provide ratings of 1, 2, 3, and 4.
• Pictorials should be adapted for the Latin America/Caribbean and Asia regions. Additionally, thought should be given to adapting the tool more comprehensively for these regions as the issues, norms, and guidance are different.
• An M&E toolkit should be created to highlight how the CSI complements other M&E tools used by programmes that serve children who are orphaned and made vulnerable by HIV/AIDS. Such programmes would be guided to consider their M&E needs and select from available M&E tools to meet those needs.
• The US Government Orphans and Vulnerable Children Technical Working Group and MEASURE Evaluation should develop a communications strategy that reaches all agencies currently using the CSI and those who may use the CSI in the future.
• In addition to expanding the scope of Child Status Net, additional opportunities and support for those using the CSI should be considered.
• Programmes for children who are orphaned and made vulnerable by HIV/AIDS should collect information at individual child, household, programme, or population levels to meet the information needs of a diverse group of stakeholders, from social welfare workers and home visitors to programme staff and national policymakers.