World Bank Q & A Series: The Challenge of Measuring Hunger
The SecureNutrition Q&A series is a new feature that aims to provide more accessible insight into key technical points of World Bank research related to nutrition-sensitive interventions. Authors participate in a structured interview aimed at surfacing the operational and policy implications of their work, and then work with SecureNutrition on a summary targeted towards practitioners.
In this Policy Research Working Paper, World Bank Program Leader Kathleen Beegle, Senior DEC Economist Jed Friedman, and World Bank consultants and professors of economics Joachim De Weerdt and John Gibson compare seven different Household Consumption Expenditure Survey (HCES) methods. The work was carried out among 3,525 households in Tanzania, and the authors looked specifically at differences in reported hunger. Depending on survey method used, the authors calculate hunger figures ranging from 19 to 68 percent in the same villages at the same time. While survey design is a factor itself, the authors find that variation is also correlated with household size, wealth, location (urban v. rural), and education of head-of-household. The paper advocates for more effort to be spent on harmonizing survey design, in order to obtain comparable hunger numbers within and between countries over time.
The following is a Q&A with Kathleen Beegle, Program Leader at the World Bank and one of the authors of The Challenge of Measuring Hunger.
Why is your report relevant to the SecureNutrition community?
We are drawing attention to the potential for household surveys to inform nutrition intake. They could be a real boon on the analytical side of the nutrition agenda.
Even though we all complain about poverty data, the truth is that in most LMICs there are national household surveys every 5 years that quantify food intake from a list of 100 food items. There’s lots of noise; meat categories get collapsed, veggie items get collapsed. But I suspect if you look at other sources of estimating calorie intake, these data may not be worse and they may be better. As someone invested in data production – if I can build demand for these surveys, demand generates supply of funding. I’m interested in making these surveys useful to a new set of data users and policy audiences who’ve not been thinking about them.
The second way this is of interest there is a lot of attention to global estimates of hunger, but maybe not enough as to where those estimates comes from. Until you ask questions about estimates and unpack them, they won’t get better in terms of rigor. The development community at large, including intersections with nutrition, agriculture, poverty, and others, want indicators that are reliable. In Tanzania, there’s something like a difference of 24 million people reported as hungry depending on survey design. That degree of error is exceptionally high.
What was the research objective?
This work was done under a broader initiative in the research group of the Bank, as part of the Living Standards Measurement Study (LSMS) program.
Around 2005 we put forward a proposition to do a suite of studies on survey methods. One problem we wanted to tackle was how we capture consumption in a household (HH) survey. Focusing on food, and under the LSMS survey methods program, we undertook a specific project in Tanzania, thinking let’s review literature about HH food consumption in HH surveys. Most surveys are not 24 hour surveys where we actually weigh food before it is prepared and eaten. Rather, an enumerator goes to a HH and elicits food consumption data from families. We read the literature, and it was clear that what was lacking in lower-income countries were careful evaluations of the tradeoffs between different methods – length, accuracy, and reliability of data.
I look at our paper as one more piece of evidence that what we have now is working poorly. Statistics agencies in the developing world work very hard – but they are often underfunded and get inconsistent advice and competing demands from the international community. There is evidence that they can successfully field surveys – consider the success of the Demographic Health Surveys – and yet we continue to see non-comparable, poorly archived and low quality data when it comes to measuring consumption and monetary poverty. We have not seen an organized approach to tackling the data problem from the international community, although there is lots of rhetoric about it. Even within the World Bank we have lacked resources and vision on this topic, as I think is true in other institutions like the UN, FAO and ILO.
What are some of the challenges in household surveys around consumption?
Most people know the $1.25/day poverty rate; in low-income countries we typically measure whether a HH is above or below that line by asking about consumption patterns, not income. Income is considered noisier to measure. At the same time, there are challenges to measuring consumption. For example, in most developing countries a large share of food is not purchased but is grown by the household, and it is tricky to value that consumed food into a monetary amount. Second is that more people are eating food outside the HH (buying prepared meals) and that is difficult to measure especially if the person being interviewed is not aware of what other members are eating.
In terms of using consumption data to track nutrition, ideally we would have very detailed data, as offered by surveys that actually weigh food. In practice, that approach is not feasible to implement. The national HH surveys can be a 2nd best to get calorie counts.
Your paper references using seven different survey modules – isn’t that a lot to compare in one study?
Yes! And in fact we wanted to have more modules to study but it become too unwieldy. And so we choose a set of different designs that seemed to offer the most insights. With more resources and more time, we might have included more designs. One challenge we had is that we wanted a benchmark – something close to the ‘truth’ – we all want the truth! So, one of the modules was our “gold standard” – a personal HH diary that each adult member maintained over 2 weeks. What makes it ‘gold’ is intensive revisits by enumerators to ensure updates. In effect, we’re asking questions like, “how far off are these different designs from the gold standard?” And, “even if we wanted mean consumption, how much variability is there in recall modules vs. different types of surveys?”
This is about the trade-off in all of statistics – the reality of what you can do. Some things you actually can’t do at any cost, and others you can do for much cheaper and get nearly as good data. We’re scientists, but we’re also dealing with tight budgets. So we wanted to explore options that were less expensive and see how close they get us to our best (and most expensive) measure.
How did you approach the study from a design standpoint?
The underlying research program in Tanzania was through about 4k HH, who were randomly assigned a questionnaire. Some had 14-day recall – so for example, in the last 2 weeks how much rice, maize, etc.? Some had a personal diary plus enumerator visits. Some had detailed long lists of foods, and some short and collapsed lists. Altogether there were eight different types (one was the ‘gold’ standard). We compared these types.
What are the key technical findings of your research?
The key finding is that survey method matters: how many hungry people there are in Tanzania depends greatly on how the questionnaires are designed. Why does this matter? Because across countries there are these differences in design – and sometimes we find them within countries. Also, how design matters is not the same for all households.
Take, for example, HH size. We find that as it gets bigger, the probability that you diagnose the HH as hungry increases if you use recall method vs. the gold standard diary. This is pretty intuitive. If you ask people – usually the spouse or wife – to estimate how much maize they eat, the more people there are in the house the harder it is to estimate. This could go either way – either under- or over-estimating. We find that she underestimates. So it is not just the number of hungry that might change with different surveys, but also the type of households that tend suffer hunger.
More generally, our original intention was to come out and see which surveys work better. Truth be told is we don’t have strong evidence that 1-2 of these designs work well for everything someone wants to do.
In other work, we did conclude that for measuring poverty, a seven-day food recall can work well. But for measuring the number of hungry (those whose food intake is below a calorie threshold), it does not do so well.
What are the policy implications of your findings?
This is a tricky question since this work was not specifically intending to have policy implications, but rather to inform monitoring of hunger by policy makers and other stakeholders. It tries to shine a light on thinking about malnutrition measures and the data problems we face. My hope is that this evidence will help invigorate the agenda towards better survey data, both in international efforts but also at a country level. To that end, the LSMS group has been pursuing a number of other studies related to consumption measurement in other regions, such as Indonesia and Peru, in an effort to continue to identify better ways to collect these data.Learn More
Photo credit: ICRISAT