This blog was written by Kim Viljoen and Willie Ngumi.
It is always exciting to see the reach of mobile services – especially in health. All over the world, life changing information is being shared through SMS, voice and other mobile-based channels. Regardless of how simple or complex these services are, one of the main challenges is how to effectively monitor the service and evaluate the impact of the service on its users. A good Monitoring and Evaluation (M&E) study requires careful user segmentation, which in turn demands a clear understanding of what outcomes you’re trying to track. In this blog post, we will share a few considerations for designing mHealth services with M&E in mind.
Why is user segmentation important for M&E?
As you develop and deploy mHealth services, or any service for that matter, the need to demonstrate impact on your users is essential. For mHealth specifically, changes in user behaviour as a response to exposure to a service is key. Demonstrating this is only made possible by conducting granular M&E with a clearly defined population or samples. Below is a scenario where these segmentation considerations would play out:
| You manage a pregnancy service with 150,000 users registered to date. The service has been running for 12 months, and is providing stage-based messaging along the pregnancy life cycle. You wish to establish the impact of the content that the service is disseminating on knowledge levels and behaviours around the use of vitamin A amongst your users. To do this, you may wish to compare a group of users confirmed to have received message/s on the appropriate use of vitamin A and a counterfactual group of similar users who have not received any messages on vitamin A from your service. You may also wish to establish that both groups have been subscribed to your service for a minimum duration. Appropriate segmentation of these two groups requires pregnancy stage data, data on exposure to relevant content, perhaps information on whether the subscriber is a pregnant woman or a supporter of a pregnant woman etc. |
Three considerations for effective user segmentation:
1. User segmentation by personal information: This plays a key role in defining populations and samples for any user feedback research. Can you segment your users by gender? Age? Stage of pregnancy? Supporter of a pregnant woman or actual pregnant woman? What if you wanted to identify a very specific group of users of your service, for example, pregnant women or male caregivers, by age, or specific lifecycle stage (e.g. pregnancy or postnatal)? Depending on the scale of your service, you may even want to localise the study to regions or districts. This personal data should be collected as soon as possible, typically during registration. The data you collect at registration should only include essential details to deliver a timely and relevant service to end-users (e.g. stage of pregnancy, age of child). Further details, which may be useful for M&E purposes, could be collected at a later stage as the user becomes more familiar with the service. For one of our mNutrition partners, a reduction from a five- to a two-step registration process resulted in an increase of 48 per cent in successful subscriptions. This succinct registration process prevents user fatigue, reduces drop-offs, and allows subscribers to access the content of the service more quickly thereby deriving value early on along their service experience.
2. Segmentation by content exposure: If your service has a content suite of 50 SMS or IVR messages, can your platform easily identify, and segment users who have been exposed to specific messages or topics? Are you able to quantify reach or distribution on a per messages basis, or on a per topic basis? For example, if you have 15 messages on exclusive breastfeeding, can you tell how many unique users any one of these messages has reached? What about all the breastfeeding messages cumulatively?
| One of the first requirements for achieving this is implementing a tagging system that enables the identification of individual messages and/or groups of related messages. There are many ways in which this can be achieved, but a simple suggestion would be to tag a message with a topic reference code and a number indicating its position in a series. For example, the fifth message in a breastfeeding topic series would have an ID starting with BF (for breastfeeding) and its sequence number (e.g. BF05). You could then easily report on all messages tagged BFXX. This is essential for assessing impact of specific content on users who have been exposed to that content. In addition, this could also be useful when demonstrating the reach and impact of your service, and specific information, on a very specific target audience. This is useful when Ministries of Health and even other businesses are interested in sponsoring specific messages in order to reach a very specific audience. |
3. Segmentation by service usage: Knowing the usage rate of a particular subscriber is important to ensure you are comparing ‘like with like’. Some results from our M&E studies conducted by GSMA and Altai Consulting indicate that nutrition knowledge and behaviours improve over time as users engage more with an mHealth service. Figure 1 demonstrates exclusive breastfeeding behaviours for different samples of users of the MTN Care 247 mHealth service in Ghana, showing how behaviours are influenced over time.
Figure 1 – Percentage of MTN Care 247 users who exclusively breastfeed their babies for the first six months*.
| Consider two users who have both received message BF05 that teaches them to exclusively breastfeed for the first six months. User A has been receiving content for over six months, whilst User B subscribed one week ago. User A and B both receive the BF05 message on the same day and you call them a few days later to ask them what a child under the age of six months should be fed. User A asnwers correctly (‘only breastmilk’), but User B does not – surprised? The assumption that these users are similar in that they’ve both received message BF05, and should therefore both give the correct answer to the question, is incorrect. User A is well familiar with the mobile service and the type of advice it provides. She has probably already derived value from previous messages, so when she receives message BF05, she takes it very seriously and commits the information to memory and does everything in her power to exclusively breastfeed her baby. Whilst User B is still trying to figure this mobile messaging service out. Similarly, for on-demand/pull content services, users who have high engagement levels with the service are likely to demonstrate different outcomes for knowledge and behaviours in comparison to less active users. |
The key takeaway – as you build mHealth applications and services, consider the various ways in which you would need to segment users, and ensure that the relevant data is collected, or tracked accurately, in order to best gauge the contribution of your service or content to behaviour change. Over the course of the next few months, the GSMA will be publishing several country case studies and a global report, which will highlight some learnings from the mNutrition programme as well as impact on nutrition outcomes identified through rigorous M&E surveys conducted by the GSMA and Altai Consulting across all eight mHealth markets. Keep an eye on our mHealth resources page or subscribe to our M4D quarterly newsletter to be sure you do not miss these.
*At the time of the survey (February 2017): The Novice User sample included users who had been subscribed to the service for less than a month, Intermediate Users had been receiving content for one to three months and Advanced Users had been receiving content for over 3 months.
This project was funded with UK aid from the British people.