New method matters in reframing policy and management [updated and newly added]

1. Key method questions in complex policy and management

2. Methodological implications of using triangulation in complex policy and managment.

3. Methodologically, analogies without counter-cases are empty signifiers

4. Peer review: another area where error avoidance is the method for reliability management

5. Not all the “can’s, might’s, may’s, or could’s” in the worlds of policy and management add up to one single “must”

6. To paraphrase Wittgenstein, an infinite regress explains nothing

7. The Achilles heel of conventional risk management is the counterfactual

8. What “calling for increased granularity” means

9. Seven differences in method that matter for reliable policy and management

10. The methodological relevance of like-to-like comparisons for policy and management

11. The methodological importance of “and-yet” counternarratives (newly added)

---

1. Key method questions in complex policy and management

I

A young researcher had just written up a case study of traditional irrigation in one of the districts that fell under the Government of Kenya’s Arid and Semi-Arid Lands (ASAL) Programme. (We’re in the early 1980s.) I remember reading the report and getting excited. Here was detailed information about really-existing irrigation practices and constraints sufficient to pinpoint opportunities for improvement there.

That was, until I turned the page to the conclusions: What was really needed, the author stated, was a country-wide land reform.

Huh? Where did that come from? Not from the details and findings in the report!

This was my introduction to “solutions” in search of problems they should “solve.” Only later did I realize I should have asked him, “What kind of land reform for whom and under what conditions at your research site?”

II

Someone asserts that this policy or approach holds broadly, and that triggers your asking:

• Under what conditions?
• With respect to what?
• As opposed to what?
• What is this a case of?
• What are you–and we–missing?

Under what conditions does what you’re saying actually hold? Risk or uncertainty with respect to what failure scenario? Settler colonialism as opposed to what? Just what is this you are talking about a case of? What are you and I missing that’s right in front us?

2. Methodological implications of using triangulation in complex policy and management

I

Triangulation is the use of multiple methods, databases, theories, disciplines and/or analysts to converge on what to do about the complex issue. The goal is for analysts to increase their confidence–and that of their policy audiences–that no matter what position they take, they are led to the same problem definition, alternative, recommendation, or other desideratum. Familiar examples are the importance in the development literature of women and of the middle classes.

In triangulating, the analyst accommodates unexpected changes in positions later on. If your analysis leads you to the same conclusion regardless of initial positions that are already othogonal, then the fact you must adjust that position later on matters less because you have sought to take into account utterly different views from the get-go.

Everyone triangulates, ranging from cross-checking of sources to formal use of varied methods, strategies and theories for convergence on a shared point of departure or conclusion. Triangulation is thought to be especially helpful in identifying and compensating for biases and limitations of any single approach. Obtaining a second (and third. . .) opinion or soliciting the input of divergent stakeholders or ensuring you interview key informants with divergent backgrounds are common examples.

Detecting bias is fundamental, because reducing, or correcting and adjusting for bias is one thing analysts can actually do. To the extent that bias remains an open question for the case at hand, it must not be assumed that increasing one’s confidence automatically or always increases certainty, reduces complexity, and/or gets one closer to the truth of the matter.

II

Return now to our starting point: The approaches in triangulation are chosen because they are, in a formal sense, orthogonal. This has another methodological implication: The aim is not to select the “best” from each approach and then combine these elements into a composite that you think better fits or explains the case at hand.

Why? Because the arguments, policies and narratives for complex policy and management already come to us as composites. Current issue understandings have been overwritten, obscured, effaced and reassembled over time by myriad interventions. To my mind, a great virtue of triangulation is to make their “composite/palimpsest” nature clearer from the outset.

To triangulate asks what, if anything, has persisted or survived in the multiple interpretations and reinterpretations that the issue has undergone over time up to the point of analysis. Indeed, failure to triangulate can provide very useful information. When findings do not converge across multiple and widely diverse metrics or measures (populations, landscapes, times and scales…), the search by the analysts becomes one of identifying specific, localized or idiographic factors at work. What you are studying may be non-generalizable–that is, it may be a case it its own right–and failing to triangulate is one way to help confirm that.

3. Methodologically, analogies without counter-cases are empty signifiers

The relentless rise of modern inequality is widely appreciated to have taken on crisis dimensions, and in moments of crisis, the public, politicians and academics alike look to historical analogies for guidance.

Trevor Jackson (2023). “The new history of old inequality.” Past & Present, 259(1): 262–289 (https://doi.org/10.1093/pastj/gtac009)

I

I bolded the preceding because its insight is major: The search for analogies from the past for the present is especially acute in turbulent times.

The methdological problem–which is also a matter of historical record–is any misleading analogy. Jackson, by way of illustrating this point, provides ample evidence to question the commonplace that the US is presently in “the Second (New) Gilded Age,” with rising inequality, populism and corruption last seen in the final quarter of our 19th century.

II

Even here, we are still stuck with the fallacy of composition: Just because a tree is shady does not mean each leaf provides shade. Not all of the country was going through the Gilded Age, even when underway. And doubtless parts of the country are now going through a Second Gilded Age, even if not nationally.

The upshot is that we must press the advocates of this or that analogy to go further. The burden of proof is on the advocates to demonstrate their generalizations hold regardless of the more granular exceptions, including those reframed by other analogies.

Why would they concede exceptions? Because we, their interlocutors, know empirically that micro and macro can be loosely-coupled, and most certainly not as tightly coupled as theory and ideology often would have it. Broad analogies that do not admit granular counter-cases float unhelpfully above policy and management.

III

A fairly uncontroversial upshot, I should have thought, but let’s make the matter harder for us.

The same day I read Jackson’s article, I can across the following analogy for current events. Asked if there were any parallels to the Roman Empire, Edward Luttwak, a scholar on international, military and grand strategy, offered this:

Well, here is one parallel: after 378 years of success, Rome, which was surrounded by barbarians, slowly started admitting them until it completely changed society and the whole thing collapsed. I am sure you know that the so-called barbarian invasions were, in fact, illegal migrations. These barbarians were pressing against the border. They wanted to come into the Empire because the Romans had facilities like roads and waterworks. They knew that life in the Roman Empire was great. Some of these barbarians were “asylum seekers,” like the Goths who crossed the Danube while fleeing the Huns. 

https://im1776.com/2023/10/04/edward-luttwak-interview/?ref=thebrowser.com&utm_source=substack&utm_medium=email

Of course, some read this as inflammatory and go no further. Others, of course, dismiss this outright as racist, adding the ad hominem “Just look at who is writing and publishes this stuff!”

But the method to adopt would be to press Luttwak for definitions and, most importantly, counter-examples.

4. Peer review: another area where error avoidance is the method for reliability management

Below is part of an interchange in the Comments section of a recent Financial Times article on scientific fraud:

Comment: I am a scientist. I spend all my time trying not to be wrong in print. Even then, occasionally I am. It is the same for all of us. Furthermore, some scientists are very poor at dealing with statistics and are thus wrong more than others. Our common incompetence is different from actual fraud. The proportion of frauds has probably held steady since the time science became a profession and has grown as the number of scientists has grown. I find it unlikely that the proportion of scientists with this character flaw has increased recently. Possibly much more common than fraud is ripping off your collaborators, or stealing ideas during reviews of manuscripts and grant applications. That is quite hard to prove and so it seems to be popular among certain character types but again, there is no reason to think their proportion has increased.

Reply: It actually doesn’t matter if the proportion is remaining steady – even though it almost certainly is growing, with so much more financial, career and political pressure on academics these days, and a for-profit publishing system that reduces public oversight and is massively biased towards positive outcomes.

The goal should remain zero.

It’s unacceptable for scientists to publish errors due to being ‘poor at statistics’. Huge amounts of money is being wasted, lives are being lost – the least people can do is get training, or work with someone else who IS good at them.

https://www.ft.com/content/c88634cd-ea99-41ec-8422-b47ed2ffc45a

Upshot: If peer review isn’t solely about error avoidance, how can it aspire to be reliable?

5. Not all the “can’s, might’s, may’s, or could’s” in the worlds of policy and management add up to one single “must”

Consider the following example (my bolding):

Our expert-interview exercise with leading thinkers on the topic revealed how climate technologies can potentially propagate very different types of conflict at different scales and among diverse political actors. Conflict and war could be pursued intentionally (direct targeted deployment, especially weather-modification efforts targeting key resources such as fishing, agriculture, or forests) or result accidently (unintended collateral damage during existing conflicts or even owing to miscalculation). Conflict could be over material resources (mines or technology supply chains) or even immaterial resources (patents, soft- ware, control systems prone to hacking). The protagonists of conflict could be unilateral (a state, a populist leader, a billionaire) or multi- lateral in nature (via cartels and clubs, a new “Green OPEC”). Research and deployment could exacerbate ongoing instability and conflict, or cause and contribute to entirely new conflicts. Militarization could be over perceptions of unauthorized or destabilizing deployment (India worrying that China has utilized it to affect the monsoon cycle), or to enforce deployment or deter noncompliance (militaries sent in to protect carbon reservoirs or large-scale afforestation or ecosystem projects). Conflict potential could involve a catastrophic, one-off event such as a great power war or nuclear war, or instead a more chronic and recurring series of events, such as heightening tensions in the global political system to the point of miscalculation, counter-geoengineering, permissive tolerance and brinksmanship. . . .

States and actors will need to proceed even more cautiously in the future if they are to avoid making these predictions into reality, and more effective governance architectures may be warranted to constrain rather than enable deployment, particularly in cases that might lead to spiralling, retaliatory developments toward greater conflict. After all, to address the wicked problem of climate change while creating more pernicious political problems that damage our collective security is a future we must avoid.

https://www.sciencedirect.com/science/article/pii/S2211467X22002255 (my bolds)

Let’s be clear: All such “could’s-as-possibilities” do not add up to one single “must-as-necessity.”

The only way in this particular passage that “could” and “can” link to “must” would mean that the article (and like ones) actually began with “We must avoid this or that” and then proceeded to demonstrate how to undertake really-existing error avoidance with respect to those could-events and might-be’s.

6. To paraphrase Wittgenstein, methodologically an infinite regress explains nothing

Stop fossil fuels; stop cutting down trees; stop using plastics; stop this defense spending; stop being imperialist; stop techno-solutionism; start being small-d democratic and small-p participatory; dismantle capitalism, racism and the rest; transform cities; save biodiversity; never forget class, gender, race, inequality, religions, bad faith, identity. . .and. . .and. . .

7. The Achilles heel of conventional risk management methods is the counterfactual

I

For me, the crux of counterfactual history is a present always not one way only. You want a counterfactual, but for which current interpretation or set of historical interpretations?

What are at work are the two blades of a scissors.

One devotes time and attention to “what is happening,” a state of affairs that can be interpreted in multiple different ways. The other devotes time and attention to “what has happened,” a state of affairs that could have turned out differently and so too the allied interpretations. Where the blades slice, they open up not only the contingent nature of events past and present, but also the recasting of what is and has been.

II

For example, consider a city’s building code. Viewed one way, it is sequential interconnectivity (do this-now followed by then-that). But if cities also view their respective building codes as the means to bring structures up to or better than current seismic standards, then the code becomes a focal mechanism for pooled interconnectivity among developers and builders.

That neither is guaranteed should be obvious. That you in no way need recourse to the language of conventional risk management to conclude so should also be obvious.

8. What “calling for increased granularity” means

I

When I say concepts like regulation, inequality, and poverty are too abstract, I am not criticizing abstraction altogether. I am saying (1) that these concepts are not differentiated enough for an actionable policy or management and (2) that this actionable granularity requires a particular kind abstraction from the get-go.

What then does “actionable granularity” mean?

II

I have in mind the range of policy analysis and management that exists between, on the one side, the adaptation of policy and management designs and principles to local circumstances and, on the other side, the recognition that systemwide patterns emerging across a diverse set of existing cases inevitably contrast with official and context-specific policy and management designs.

Think here of adapting your systemwide definition of poverty to local contingencies and having to accommodate the fact that patterns that emerge across how really-existing people identify poverty differ from not only system definitions but also from localized poverty scenarios based in these definitions.

III

One implication is that cases that are not framed by emerging patterns and, on the other side, by localized design scenarios are rightfully called “unique.” Unique cases of poverty cannot be abstracted, just as some concepts of poverty are, in my view, too abstract. Unique cases stand outside the actionable granularity of interest here for policy and management.

Where so, then there is the methodological problem of cases that are assumed to be unique or stand-alone, when in fact no prior effort has been made to ascertain (1) systemwide patterns and local contingency scenarios in which the case might be embedded along with (2) the practices, if any, of adaptation and modification that emerge as a result.

From a policy and management perspective, such cases have been prematurely rendered unique: They have been, if you will, over-complexified so as to permit no abstraction. Unique cases are not themselves something we can even abstract as sui generis or even “‘a case’ in its own right.”

I stress this point if only because of the exceptionalism assigned to “wicked policy problems”. Where the methodological problem of premature complexification isn’t addressed beforehand, then by definition the so-called wicked policy problem is prematurely “wickedly unique.” Or, more ironically, uniquely wicked problems are abstracted insufficiently for the purposes of systemwide pattern recognition and design scenario modification.

9. Seven differences in method that matter for reliable policy and management

When I and others call for better recognition and accommodation of complexity, we mean the complex as well as the uncertain, unfinished and conflicted must be particularized and contextualized so that we can analyze and manage, if only case-by-granular case.

When I and others say we need more findings that can be replicated across a range of cases, we are calling for identifying not only emerging better practices across cases, but also greater equifinality: finding multiple but different pathways to achieve similar objectives, given case diversity.

What I and others mean by calling for greater collaboration is not just more teamwork or working with more and different stakeholders, but that team members and stakeholders “bring the system into the room” for the purposes of making the services in question reliable and safe.

When I and others call for more system integration, we mean the need to recouple the decoupled activities in ways that better mimic but can never fully reproduce the coupled nature of the wider system environment.

When I and others call for more flexibility, we mean the need for greater maneuverability across different performance modes in the face of changing system volatility and options to respond to those changes. (“Only the middle road does not lead to Rome,” said composer, Arnold Schoenberg.)

Where we need more experimentation, we do not mean more trial-and-error learning, when the systemwide error ends up being the last systemwide trial by destroying the limits of survival.

While others talk about risks in a system’s hazardous components, we point to different systemwide reliability standards and then, to the different risks and uncertainties that follow from different standards.

10. The methodological relevance of like-to-like comparisons for policy and management

Assume you come across the following typology, a 2 X 2 table identifying four types of confidence you have over empirical findings for policy analysis and policymaking:

It’s fairly easy to question the above. Do we really believe, for example, that well-established evidence and high certainty are as tightly coupled and correlated? In fact, each dimension can be problematize in ways relevant to policy analysis and policymaking.

But the methodological issue at stake here is to compare like to like.

That is, interrogate the cells of the above typology using the cells of another typology whose overlapping dimensions also problematize those of the above. Consider, for example, the famous Thompson-Tuden typology, where the key decisionmaking process is a function of agreement (or lack thereof) over policy-relevant means and ends:

This latter world has a few surprises for the former one. Contrary to the notion that inconclusive evidence is “solved” by more and better evidence, the persistence of “inconclusive” (because, say, of increasing urgency and interruptions) implies eventually lapsing, it is hypothesized, into decisionmaking-by-inspiration. So too the persistence of “unresolved” or “established but incomplete” shuttles, again and again, between decisionmaking by majority-rule and compromises. More, what is tightly coupled in the latter isn’t “evidence and certainty” as in the former, but rather the beliefs over evidence with respect to causation and the preferences for agreed-upon ends and goals.

In case it needs saying, methodological like-to-like comparisons of typologies need not stop at a comparison of two only. Social and organizational complexity means the more the better by way of finding something more tractable to policy or management.

11. The methodological importance of “and-yet” counternarratives (newly added)

The paragraph I’ve just read is immediately bookended by two quotes:

Just before: “Therefore, rather than being schools of democracy, ACs [associative councils] may be spaces where associative and political elites interact and, therefore, just reproduce existing political inequities (Navarro, 2000). Furthermore, these institutions may have limited impact in growing and diversifying the body of citizens making contributions to public debate (Fraser, 1990).”

Just after: “The professionalised model results from a complex combination of inequalities in associationism and a specific type of participation labour. Analysing the qualitative interviews, regulations and documents was fundamental to understanding the underlying logic of selecting professionals as the main components.”

Now try to guess the gist of the paragraph in between. More of the same? Well, no. Six paragraphs from the article’s end emerges an “and-yet” that had been there from the beginning:

Nevertheless, an alternative interpretation of professionalisation should be considered. The fact that ACs perform so poorly in inclusiveness does not mean that they are not valuable for other purposes, such as voicing a plurality of interests in policymaking (Cohen, 2009). In this respect, participants can act as representatives of associations that, in many cases, promote the needs of oppressed and exploited groups (De Graaf et al., 2015; Wampler, 2007). Suffice it to say, for example, that labour unions or migrants’ associations frequently send lawyers or social workers to ACs to defend their needs and positions. Problems with inclusion should not take away from other purposes, that is, struggles to introduce critical issues and redistribution demands to the state agenda. Other studies have already shown that groups make strategic decisions to achieve better negotiation outcomes in the context of technical debates (Grillos, 2022). Thus, the choice of selecting professionals can be a strategy to improve the capacity of pressure in institutional spaces dominated by experts. (my bold; accessed online at https://journals.sagepub.com/doi/10.1177/00323217251319065)

Methodological upshot: What the counterfactual is to economic analysis, the and-yet counternarratives are to policy analysis. What