Ukraine Urban Recovery: Of Mines and Metrics, 2 of 3
Brutal Catalyst: What Ukraine’s Cities Tell Us About Recovery From War, available at: https://www.amazon.com/dp/B0DJ8Q7BGJ
A continuation of discussions drawing on Brutal Catalyst: What Ukraine’s Cities Tell Us About Recovery From War, released October 8, 2024.
Photo adapted from image in Dario Pronesti and Jeoen van den Boogaard, “Ukraine Symposium – Landmines and the War in Ukraine,” March 20, 2023, https://lieber.westpoint.edu/landmines-war-ukraine/. Photo credit: Kyivcity.gov.ua
Relevant metrics need to do more than clear the hurdle of reflecting effectiveness rather than effort expended. We noted in the previous post that good measures must also accurately reflect that effectiveness for decision-makers and funders at all relevant echelons and across all relevant organizations. Metrics should additionally consider metrics’ possible second and higher-order effects. Ideally, metrics motivate positive behavior. But there is another edge to that blade. Two examples touched on in the last blog post demonstrate the point. The first is body count as employed during the Vietnam War. Senior leaders considered the number of enemy dead a measure of progress toward enemy defeat. It instead resulted in some ethically-challenged leaders reporting any deceased Vietnamese as a foe, an obvious inaccuracy as noncombatant casualties are, unfortunately, an inevitable part of war. Second, as the metric could influence performance reports, less honest leaders simply inflated the number of enemy soldiers killed. Last, the metric’s designers were certainly remiss is failing to consider its worst potential consequence: the most unsavory could deliberately kill innocents and report them as part of their unit’s body count. Our second previous example—that of dollars spent in Iraq—posed less of a threat to lives but was no more effective in measuring success. Peter Lippman tells us the problem was not limited to Iraq, nor to government-funded undertakings. The same challenges plagued work in Bosnia-Herzegovina:
Often international NGOs spent money in a hurry toward the end of their budgeting period in the fear that otherwise they could not justify new grants…. This led to wasteful spending and the creation of useless projects…. The international community’s…reckless and wasteful expenditure in the years immediately after the war encouraged local NGOs to sprout like mushrooms. It was crucial for international relief agencies and human rights workers to learn how to assess the sincerity of local NGO workers…. Phantom NGOs with a blank account, and perhaps a brochure, sometimes came into existence solely for the purpose of raking off the dollars and Deutschmarks that were flowing. In addition to these obviously criminal NGOs, there was a spectrum of NGO operations of varied usefulness from feel-good “trauma counseling” with short-lived or nonexistent results to the truly sincere, even heroic, multiethnic organizations. [Peter Lippman, Surviving the Peace: The Struggle for Postwar Recovery in Bosnia-Herzegovina, Nashville, TN: Vanderbilt University Press, 2019, 43]
Returning to the example of demining in Ukraine; Pehr Lodhammar, formerly United Nations Chief of the Mine Action Programme in Iraq, put it succinctly. He and Colonel Gregory Fontenot share an understanding when it comes to measuring effectiveness. “Success in mine action, unlike in the financial world and elsewhere, is not measured in the money we can raise,” Lodhammar related. “It’s measured by what we can do with the money we are entrusted with by the member states.” [from Pehr Lodhammar telephonic interview with Dr. Russell W. Glenn, February 16, 2024] Lodhammar expanded on his observations in an article that provides a reinforcing example demonstrating why donors or others measuring effectiveness in terms of effort (e.g., number of mines removed per dollar donated) miss the point:
Consider the lesson learned from the Fallujah [Iraq] Iron Bridge experience [in which $100,000 was spent removing two items of unexploded ordnance]. If the operation was measured in terms of [only the number of mines removed], the two IEDs destroyed during the week-long effort would seem costly in terms of effort expended yet more than commensurate when valued by the socio-economic impact. Following clearance and repairs to the [bridge], travel time for some residents to the only maternity hospital in a 50 kilometer (31.1 mile) radius [was] reduced from two hours to five minutes. [Pehr Lodhammar, “How Iraq is Changing What We Do: Measuring Clearance in Urban Environments,” The Journal of Conventional Weapons Destruction 22, no. 2 (August 2018): 34 and 35.]
Another expert in the demining discipline reinforced Lodhammar’s observations, independently citing the same example:
We cleared one device [from a bridge in Fallujah, Iraq] and it was $100,000. That doesn’t sound like good value for money, but that allowed the bridge to be repaired which now allows ten thousand vehicles a day to use the bridge and cuts the time for people trying to get to the hospital from four hours to fifteen minutes. So when you turn around and say what was the impact of that $100,000, it was probably ten thousand times what you’d have spent clearing a field of a thousand mines for $10,000. One is $10/per device and the other’s $100,000 per device, but actually the impact was a thousand times more for the $100,000 spent.
[The second interviewee requested anonymity. The two interviewees’ descriptions of the event differ slightly in terms of time saved to reach the hospital and number of devices removed from the bridge. These trivial differences do not diminish the example’s value in getting the point across.]
Creating measures that meet the need to inform decision-makers at every relevant authority level is often less straightforward than might be thought. Some organizations use a “stoplight” chart to reflect status in an effort to standardize presentation of data from what can be very different sources. Green, for example, might mean 90 percent of an organization’s vehicles are operational. Amber (in military-speak; yellow to most of the world) would be 70 to 89 percent are good to go while red reflects readiness of less than 70 percent of vehicles. Can the approach be effective in terms of reflecting a unit’s vehicle readiness (or progress toward goals in other activities)? Yes…but it might well mislead if a single measure encompasses all vehicle types rather than each type. If a fuel delivery outlet for a large corporation has all its sales cars working but only half the trucks that make deliveries, higher headquarters might see an amber or even green rating when in fact the most important of the organization’s services—getting product to customers—is significantly inhibited. Once again, metric design counts.
A US Department of State representative recalled his time in Iraq and how a means of measuring project status could deceive if misunderstood by those reporting or receiving the information. The issue at hand was the success of indigenous police training:
My team said something in a report that [differed from the report by] the brigade. A call came from Camp Victory asking, “Why are you differing? You shouldn’t be differing.”… And I said, “If you ask a local leader, ‘Are the police trained?’ he could truthfully answer, ‘Yes.’” So they have all green lights up on the chart, but I might have a red light because yes, they are trained, but they are going around intimidating the people. In our rush to brevity, we sacrifice accuracy, and then the poor general thinks he is informed, but he is not. [These comments also came from an individual wishing to remain anonymous.]
The point reminds us of those donors who mistakenly think measures such as cost per square meter of mines removed reflects the benefits reaped by their funds. The number of hours spent training the police provided no measure of benefit. What those law enforcement personnel learned, internalized, and how they applied their knowledge was the real metric of value. It was also one far harder to design and measure, just as in the case of gaging the effects of demining.
Next time: Some thoughts on better metrics (third of three posts on mines and metrics).
Last blog post’s trivia question: How many women are buried in Normandy American Cemetery?
Answer: There are four women buried in the Normandy American Cemetery. For those interested in more detail:
From https://www.spearhead-tours.com/post/5-interesting-facts-about-normandy-american-cemetery:
“Amongst the [over 9,300] people resting at [the cemetery,] four are women. One of them, Elizabeth Richardson, was a Red Cross volunteer…. The other three women, Mary Bankston, Mary Barlow, and Dolores Browne were WACs (Women’s Army Corps) and served with the 6888th Postal Battalion. While 'Liz' died in a plane crash on July 25, 1945, the WACs were involved in a jeep accident on the outskirts of Rouen, just a few weeks before, on July 8, 1945. In a strange twist of fate, all four died in July 1945 so after VE-Day, their deaths resulting from accidents, and all four were initially buried in the same temporary cemetery at St. Andre.”
Additional note: The three members of the 6888th Postal Battalion were African-American. Greater detail on the unit is available at https://www.nps.gov/articles/000/6888th.htm and the following from https://www.norwichbulletin.com/story/news/local/2014/06/01/historically-speaking-2-conn-women/37222499007/:
“African-American women were eager to serve their country. Author Brenda Moore, in her book, To Serve My Country, To Serve My Race, tells more about the [6888th Postal Battalion]…. The battalion was formed at the urging of first lady Eleanor Roosevelt and the unit was part of the Woman's Army Corps. The WAC was created by legislation in 1942. Col. Charity Adams was the battalion commander, the highest ranking African-American woman in the Army at that time. Later, in 1982, Adams was listed by the Smithsonian Institution as one of the 110 most important Black American Women of all time.”
If you are interested, the above book is available new at https://www.amazon.com/Serve-Country-Race-African-American-Stationed/dp/0814755879
or used at:
Next question: There are 45 sets of brothers buried in the Normandy American Cemetery. Two of those brothers have the last name "Niland." Why might you know of them...but by a different name?
###
Russell W. Glenn, Brutal Catalyst: What Ukraine’s Cities Tell Us About Recovery From War (Boulder, CO: KeyPoint Press, 2024). Available in hardback, paperback, and eBook at https://www.amazon.com/dp/B0DJ8Q7BGJ.
For media inquiries and review copies, email: editor@keypointpress.com
Discover more about the war in Ukraine and recovery at: www.ukrainecities.com
KeyPointPress: www.keypointpress.com
