Because I’ve had a career as a practicing policy analyst, I’ve also had a healthy respect for the utility of different methods of problem-solving. What I wasn’t prepared for were all the “vs”: quantitative versus qualitative, positivist versus post-positivist, reductionist versus holistic, and such. The complexity of the policy issues meant that all and more could be required–and anyway these “versus’s” weren’t granular enough to identify which specific methods were being compared or not.
Once the analyst starts thinking in terms of different specific methods, the horizon of candidates broadens considerably. In my practice, methods came to include applications and modifications of middle-range frameworks from one area to another in order to reframe more tractably difficult problems in the latter. New ways to recast the benchmarks and metrics for managing risk, uncertainty and unknowns have been of interest (for other examples, see my Guide).
Below are twelve examples to add to that earlier discussion for a methodologically expanded policy analysis and public management. These map out for the analyst-as-a-professional additional working space not in graduate school training. As always, your comments are welcome.
1. Key method questions in complex policy and management
2. Methodological implications of using triangulation in complex policy and management.
3. Methodologically, analogies without counter-cases are empty signifiers
4. The methodological challenge of empty and floating signifiers to analyzing policy (newly added)
5. Peer review: another area where error avoidance is the method for reliability management
6. Not all the “can’s, might’s, may’s, or could’s” in the worlds of policy and management add up to one single “must”
7. To paraphrase Wittgenstein, an infinite regress explains nothing for the purposes of policy and management
8. The Achilles heel of conventional risk management is the counterfactual
9. What “calling for increased granularity” means
10. Seven differences in method that matter for reliable policy and management
11. The methodological importance of “and-yet” counternarratives
12. Etcetera’s are the crisis kitsch of policy analysis (newly added)
1. Key method questions in complex policy and management
–A young researcher had just written up a case study on traditional irrigation in one of the districts that fell under the Government of Kenya’s Arid and Semi-Arid Lands (ASAL) Programme. (We’re in the early 1980s.) I remember reading the report and getting excited. Here was detailed information about really-existing irrigation practices and constraints sufficient to pinpoint opportunities for improvement there.
That was, until I turned the page to the conclusions: What was really needed, the author stated, was a country-wide land reform. Huh? Where did that come from? Not from the details and findings in the report!
This was my introduction to “solutions” in search of problems they should “solve.” Only later did I realize I should have asked him, “What kind of land reform for whom and under what conditions at your research site?”
–Someone asserts that this policy or approach holds broadly, and that triggers your having to ask:
- Under what conditions?
- With respect to what?
- As opposed to what?
- What is this a case of?
- What are you–and we–missing right here?
Under what conditions does what you’re saying actually hold? Risk or uncertainty with respect to what failure scenario? Settler colonialism as opposed to what? Just what is this you are talking about a case of? What are you and I missing that’s right in front us?
2. Methodological implications of using triangulation in complex policy and management
I
Triangulation is the use of multiple methods, databases, theories, disciplines and/or analysts to converge on what to do about the complex issue. The goal is for analysts to increase their confidence–and that of their policy audiences–that no matter what position they take, they are led to the same problem definition, alternative, recommendation, or other desideratum. Familiar examples are the importance in the development literature of women and of the middle classes.
In triangulating, the analyst accommodates unexpected changes in positions later on. If your analysis leads you to the same conclusion regardless of initial positions that are already orthogonal, then the fact you must adjust that position later on matters less because you have sought to take into account utterly different views from the get-go.
Everyone triangulates, ranging from cross-checking of sources to formal use of different methods, strategies and theories for convergence on a shared point of departure or conclusion. Triangulation is thought to be especially helpful in identifying and compensating for biases and limitations of any single approach. Obtaining a second (and third. . .) opinion or soliciting the input of divergent stakeholders or ensuring you interview key informants with divergent backgrounds are common examples.
Detecting bias is fundamental, because reducing, or correcting and adjusting for bias is one thing analysts can do. To the extent that bias remains an open question for the case at hand, it must not be assumed that increasing one’s confidence automatically or always increases certainty, reduces complexity, and/or gets one closer to the truth of the matter.
II
Return now to our starting point: The approaches in triangulation are chosen because they are, in a formal sense, orthogonal. This has another methodological implication: The aim is not to select the “best” from each approach and then combine these elements into a composite that you think better fits or explains the case at hand.
Why? Because the arguments, policies and narratives for complex policy and management already come to us as composites. Current issue understandings have been overwritten, obscured, effaced and reassembled over time by myriad interventions. To my mind, a great virtue of triangulation is to make their “composite/palimpsest” nature clearer from the outset.
Indeed, failure to triangulate can provide very useful information. When findings do not converge across multiple and widely diverse metrics or measures (populations, landscapes, times and scales…), the search by the analysts becomes one of identifying specific, localized or idiographic factors at work. What you are studying may be non-generalizable–that is, it may be a case in its own right–and failing to triangulate is one way to help confirm that.
3. Methodologically, analogies without counter-cases are empty signifiers
The relentless rise of modern inequality is widely appreciated to have taken on crisis dimensions, and in moments of crisis, the public, politicians and academics alike look to historical analogies for guidance.
Trevor Jackson (2023). “The new history of old inequality.” Past & Present, 259(1): 262–289 (https://doi.org/10.1093/pastj/gtac009)
–I bolded the preceding because its insight is major: The search for analogies from the past for the present is especially acute in turbulent times.
The methodological problem–which is also a matter of historical record–is any misleading analogy. Jackson, by way of illustrating this point, provides ample evidence to question the commonplace that the US is presently in “the Second (New) Gilded Age,” with rising inequality, populism and corruption last seen in the final quarter of our 19th century.
–Even here, we are still stuck with the fallacy of composition: Just because a tree is shady does not mean each leaf provides shade. Not all of the country was going through the Gilded Age, even when underway. And doubtless parts of the country are now going through a Second Gilded Age, even if not nationally.
The upshot is that we must press the advocates of this or that analogy to go further. The burden of proof is on the advocates to demonstrate their generalizations hold regardless of the more granular exceptions, including those reframed by other analogies.
Why would they concede exceptions? Because we, their interlocutors, know empirically that micro and macro can be loosely-coupled, and most certainly not as tightly coupled as theory and ideology often would have it. Broad analogies that do not admit granular counter-cases are unhelpfully empty for policy and management.
–A fairly uncontroversial upshot, I should have thought, but let’s make the matter harder for us.
The same day I read Jackson’s article, I came across the following analogy for current events. Asked if there were any parallels to the Roman Empire, Edward Luttwak, a scholar on international, military and grand strategy, offered this:
Well, here is one parallel: after 378 years of success, Rome, which was surrounded by barbarians, slowly started admitting them until it completely changed society and the whole thing collapsed. I am sure you know that the so-called barbarian invasions were, in fact, illegal migrations. These barbarians were pressing against the border. They wanted to come into the Empire because the Romans had facilities like roads and waterworks. They knew that life in the Roman Empire was great. Some of these barbarians were “asylum seekers,” like the Goths who crossed the Danube while fleeing the Huns.
https://im1776.com/2023/10/04/edward-luttwak-interview/?ref=thebrowser.com&utm_source=substack&utm_medium=email
Of course, some read this as inflammatory and go no further. Others, of course, dismiss this outright as racist, adding the ad hominem “Just look at who is writing and publishes this stuff!”
But the method to adopt would be to press Luttwak for definitions and, most importantly, counter-examples.
4. The methodological challenge of empty and floating signifiers to analyzing policy (newly added)
I
The usual way floating and empty signifiers are distinguished is that the former are terms or concepts with (too) many meanings, while the latter have none. The ready example of a floating signifier in today’s policy discussions is “capitalism,” since it’s
laissez-faire capitalism, monopoly capitalism, oligarchic capitalism, state-guided capitalism, party-state capitalism, corporate capitalism, corporate-consumerist capitalism, bourgeois capitalism, patrimonial capitalism, digital capitalism, financialized capitalism, political capitalism, social (democratic) capitalism, neoliberal capitalism, crony capitalism, cannibal capitalism, wellness capitalism, petty capitalism, platform capitalism, cloud capitalism, infrastructural capitalism, algorithmic capitalism, aspirational capitalism, welfare capitalism, compliance capitalism, authoritarian capitalism, imperialistic capitalism, turbo-capitalism, post-IP capitalism, green (also red and brown) capitalism, climate capitalism, extractive capitalism, libidinal capitalism, regulatory capitalism, clickbait capitalism, emotional (affective) capitalism, tech capitalism, American capitalism, British capitalism, European capitalism, Western capitalism, transnational capitalism, agrarian capitalism, disaster capitalism, rentier capitalism, industrial capitalism, post-industrial capitalism, fossil capitalism, settler-colonial capitalism, supply chain capitalism, asset manager capitalism, information (data) capitalism, cyber-capitalism, cybernetic capitalism, necro-capitalism, bio-capitalism, war capitalism, crisis capitalism, managerial capitalism, stakeholder capitalism, techno(scientific)-capitalism, pandemic capitalism, caring capitalism, zombie capitalism. . .
Nothing would seem more policy relevant than capitalism, but even a cursory study of the list identifies no shared or central focus on, importantly, markets and prices. The cumulative implication for policy analysis, my profession, is obvious: Just what is the–or even “an”–anti-capitalist alternative?
II
If capitalism as a floating signifier makes a hash of what it is to be “anti-capitalist,” the opposite is the case for the empty signifier of that equally policy relevant neoliberalism. Neoliberalism is almost always a term used by those whose position is anti-neoliberalism. “Recent scholarship has suggested that ‘neoliberalism’ functions almost exclusively a term used by its critics,” where
An empty signifier, in Laclau’s vocabulary, is a name progressively emptied of specific content so
that it can stand for an entire chain of heterogeneous demands: the poorer its intension, the wider the
coalition it can represent (Laclau, 1996, 2005). . . .Laclau was entirely lucid about the epistemic price and entirely explicit that it was worth paying, because the point of such signifiers is not to describe but to assemble [and articulate]. . . .Defenders of the concept argue that neoliberalism names a real macro-phenomenon and that the term’s breadth reflects the breadth of the thing (Peck, 2013; Laruffa, 2024). Our argument does not require denying this: real processes of marketisation, privatisation and financialisation occurred. The question is why they came to be gathered under this name, in this form, on this timeline. . .
A concept that functions as an articulating signifier is unlikely to be disciplined by demonstrations of its descriptive failure, because description was never its principal function, and the productive question may therefore be less ‘what is the correct definition of neoliberalism?’ than ‘what work is the concept doing, for whom, and at what analytical cost?’
Paniagua, Pablo and Vergara, Álvaro (2026). “Neoliberalism As Adversary: The Latin American Construction of an Empty Signifier”. My underlining. https://ssrn.com/abstract=7316099)
The upshot: The analytic costs in relying on and using “neoliberalism”–and by implication other empty signifiers–understandably trouble the policy analyst more than the policy advocate intent on articulating who the adversary is and assembling coalitions to oppose it.
III
Floating and empty signifiers are problematic for policy analysts when these terms, capitalism, neoliberalism and the like (e.g., “cooperative global governance”), are exiled from their specifics and particularities as events, situations and contexts. More, the identification of systemwide patterns (if any) over a range of different events, situations, and contexts–including but not limited to those self-identified as neoliberal or anti-capitalist–is also unavoidable for the analyst.
Why? Because practices surface in pattern recognition and become the signifier of interest–the center of analytic gravity–for the analyst. It’s the practice of protesters waving or holding blank sheets of paper that is significant. Practices for managing assets (A) differ from the older practices of commodification (C), which in turn has changed Marx’s classic formulation for making more money (M’) from M–C–M’ to M–A–M’.
IV
So what? Two implications follow from this reshifted focus to practices.
(1) First, centering the analysis of policy on existing practices helps explain why human values and beliefs are more often the floating and empty signifiers of choice in policy advocacy. Policy analysts are, in my experience, keener at describing the formal and informal routines, procedures, mechanisms and improvisations key critical infrastructures undertake in order that real-time market exchange and transactions take place. In contrast, the policy advocates with whom I’m familiar are more comfortable in focusing on, say, “human greed” and “belief in markets” to explain that which takes place.
The latter quotation marks are not meant to be pejorative, but rather to mark that both signifiers need to be recognized as socially constructed before taking further action (rather than calling for greater detail tout court).
(2) Second, the response of policy analysts to policy advocates’ reliance on empty and floating signifiers isn’t to insist advocates provide these details. Policy advocacy is an entirely different genre than policy analysis. Rather, the burden of proof is on the advocates to demonstrate their signifiers hold regardless of the more granular qualifications or exceptions provided by analysts. (See the point made with respect to the Luttwak quote in the preceding #3.)
5. Peer review: another area where error avoidance is the method for reliability management
Below is part of an interchange in the Comments section of a recent Financial Times article on scientific fraud:
Comment: I am a scientist. I spend all my time trying not to be wrong in print. Even then, occasionally I am. It is the same for all of us. Furthermore, some scientists are very poor at dealing with statistics and are thus wrong more than others. Our common incompetence is different from actual fraud. The proportion of frauds has probably held steady since the time science became a profession and has grown as the number of scientists has grown. I find it unlikely that the proportion of scientists with this character flaw has increased recently. Possibly much more common than fraud is ripping off your collaborators, or stealing ideas during reviews of manuscripts and grant applications. That is quite hard to prove and so it seems to be popular among certain character types but again, there is no reason to think their proportion has increased.
Reply: It actually doesn’t matter if the proportion is remaining steady – even though it almost certainly is growing, with so much more financial, career and political pressure on academics these days, and a for-profit publishing system that reduces public oversight and is massively biased towards positive outcomes.
The goal should remain zero.
It’s unacceptable for scientists to publish errors due to being ‘poor at statistics’. Huge amounts of money is being wasted, lives are being lost – the least people can do is get training, or work with someone else who IS good at them.
https://www.ft.com/content/c88634cd-ea99-41ec-8422-b47ed2ffc45a
Upshot: If peer review isn’t solely or primarily about error avoidance, how can it aspire to be reliable?
6. Not all the “can’s, might’s, may’s, or could’s” in the worlds of policy and management add up to one single “must”
Consider the following example (my bolding):
Our expert-interview exercise with leading thinkers on the topic revealed how climate technologies can potentially propagate very different types of conflict at different scales and among diverse political actors. Conflict and war could be pursued intentionally (direct targeted deployment, especially weather-modification efforts targeting key resources such as fishing, agriculture, or forests) or result accidently (unintended collateral damage during existing conflicts or even owing to miscalculation). Conflict could be over material resources (mines or technology supply chains) or even immaterial resources (patents, soft- ware, control systems prone to hacking). The protagonists of conflict could be unilateral (a state, a populist leader, a billionaire) or multi- lateral in nature (via cartels and clubs, a new “Green OPEC”). Research and deployment could exacerbate ongoing instability and conflict, or cause and contribute to entirely new conflicts. Militarization could be over perceptions of unauthorized or destabilizing deployment (India worrying that China has utilized it to affect the monsoon cycle), or to enforce deployment or deter noncompliance (militaries sent in to protect carbon reservoirs or large-scale afforestation or ecosystem projects). Conflict potential could involve a catastrophic, one-off event such as a great power war or nuclear war, or instead a more chronic and recurring series of events, such as heightening tensions in the global political system to the point of miscalculation, counter-geoengineering, permissive tolerance and brinksmanship. . . .
States and actors will need to proceed even more cautiously in the future if they are to avoid making these predictions into reality, and more effective governance architectures may be warranted to constrain rather than enable deployment, particularly in cases that might lead to spiralling, retaliatory developments toward greater conflict. After all, to address the wicked problem of climate change while creating more pernicious political problems that damage our collective security is a future we must avoid.
https://www.sciencedirect.com/science/article/pii/S2211467X22002255 (my bolds)
Let’s be clear: All such “could’s-as-possibilities” do not add up to one single “must-as-necessity.”
The only way in this particular passage that “could” and “can” link to one or more “must” would mean that the article (and like ones) actually began with “We must avoid this or that” and then proceeded to demonstrate how to undertake really-existing error avoidance with respect to those (could- and might-be) events.
7. To paraphrase Wittgenstein, methodologically an infinite regress explains nothing for the purposes of policy and management
Stop fossil fuels; stop cutting down trees; stop using plastics; stop this defense spending; stop being imperialist; stop techno-solutionism; start being small-d democratic and small-p participatory; dismantle capitalism, racism and the rest; transform cities; save biodiversity; never forget class, gender, race, inequality, religions, bad faith, identity. . .and. . .and. . .
8. The Achilles heel of conventional risk management methods is the counterfactual
–For me, the crux of counterfactual history is a present always not one way only. You want a counterfactual, but for which current interpretation or set of historical interpretations?
What is at work are the two blades of a scissors. One devotes time and attention to “what is happening,” a state of affairs that can be interpreted in multiple different ways. The other devotes time and attention to “what has happened,” a state of affairs that could have turned out differently and so too the allied interpretations. Where the blades slice, they open up not only the contingent nature of events past and present, but also the recasting of what is and has been.
–For example, consider a city’s building code. Viewed one way, it is sequential interconnectivity (do this-now followed by that-then). But if cities also view their respective building codes as the means to bring structures up to or better than current seismic standards, then the code becomes a focal mechanism for pooled interconnectivity among developers and builders.
That neither is guaranteed should be obvious. That you in no way need recourse to the language of conventional risk management to conclude so should also be obvious.
9. What “calling for increased granularity” means
I
When I say concepts like regulation, inequality, and poverty are too abstract, I am not criticizing abstraction altogether. I am saying (1) that these concepts are not differentiated enough for actionable policy or management and (2) that this actionable granularity requires a particular kind of abstraction from the get-go.
What then does “actionable granularity” mean?
II
I have in mind the range of policy analysis and management that exists between, on the one side, the adaptation of policy and management designs and principles to local circumstances and, on the other side, the recognition that systemwide patterns emerging across a diverse set of existing cases inevitably contrast with official and context-specific policy and management designs.
Think here of adapting your systemwide definition of poverty to local contingencies and having to accommodate the fact that patterns that emerge across how really-existing people identify poverty differ from not only system definitions but also from localized poverty scenarios based in these definitions.
III
One implication is that cases that are not framed by emerging patterns and, on the other side, by localized design scenarios are rightfully called “unique.” Unique cases of poverty cannot be abstracted, just as some concepts of poverty are, in my view, too abstract. Unique cases stand outside the actionable granularity of interest here for policy and management.
Where so, there is the methodological problem of cases that are assumed to be unique or stand-alone, when in fact no prior effort has been made to ascertain (1) systemwide patterns and local contingency scenarios in which the case might be embedded along with (2) the practices, if any, of adaptation and modification that emerge as a result.
From a policy and management perspective, such cases have been prematurely rendered unique: They have been, if you will, over-complexified so as to permit no abstraction. Unique cases are not themselves something we can even abstract as sui generis or “a case in its own right.”
I stress this point because of the exceptionalism assigned to “wicked policy problems”. Where the methodological problem of premature complexification isn’t addressed beforehand, then by definition the so-called wicked policy problem is prematurely “wickedly unique.” Or, more ironically, uniquely wicked problems are abstracted insufficiently for the purposes of systemwide pattern recognition and design scenario modification.
10. Seven differences in method that matter for reliable policy and management
When I and others call for better recognition and accommodation of complexity, we mean the complex as well as the uncertain, unfinished and conflicted must be particularized and contextualized so that we can analyze and manage, if only case-by-granular case.
When I and others say we need more findings that can be replicated across a range of cases, we are calling for identifying not only emerging better practices across cases, but also greater equifinality: finding multiple but different pathways to achieve similar objectives, given case diversity.
What I and others mean by calling for greater collaboration is not just more teamwork or working with more and different stakeholders, but that team members and stakeholders “bring the system into the room” for the purposes of making the services in question reliable and safe.
When I and others call for more system integration, we mean the need to recouple the decoupled activities in ways that better mimic but can never fully reproduce the coupled nature of the wider task environment.
When I and others call for more flexibility, we mean the need for greater maneuverability across different performance modes in the face of changing system volatility and options to respond to those changes. (“Only the middle road does not lead to Rome,” according to composer Arnold Schoenberg.)
Where we need more experimentation, we do not mean more trial-and-error learning, when the systemwide error ends up being the last systemwide trial by destroying the limits of survival.
While others talk about risks in a system’s hazardous components, we point to different systemwide reliability standards and then, to the different risks and uncertainties that follow from different standards.
11. The methodological importance of “and-yet” counternarratives
The statement that I will be discussing is immediately book-ended by two quotes:
Just before the missing statement: “Therefore, rather than being schools of democracy, ACs [associative councils] may be spaces where associative and political elites interact and, therefore, just reproduce existing political inequities (Navarro, 2000). Furthermore, these institutions may have limited impact in growing and diversifying the body of citizens making contributions to public debate (Fraser, 1990).”
Just after the missing statement: “The professionalised model results from a complex combination of inequalities in associationism and a specific type of participation labour. Analysing the qualitative interviews, regulations and documents was fundamental to understanding the underlying logic of selecting professionals as the main components.”
Now try to guess the gist of the paragraph in between. More of the same? Well, no. Six paragraphs from the article’s end emerges an “and-yet” that had been there from the beginning:
Nevertheless, an alternative interpretation of professionalisation should be considered. The fact that ACs perform so poorly in inclusiveness does not mean that they are not valuable for other purposes, such as voicing a plurality of interests in policymaking (Cohen, 2009). In this respect, participants can act as representatives of associations that, in many cases, promote the needs of oppressed and exploited groups (De Graaf et al., 2015; Wampler, 2007). Suffice it to say, for example, that labour unions or migrants’ associations frequently send lawyers or social workers to ACs to defend their needs and positions. Problems with inclusion should not take away from other purposes, that is, struggles to introduce critical issues and redistribution demands to the state agenda. Other studies have already shown that groups make strategic decisions to achieve better negotiation outcomes in the context of technical debates (Grillos, 2022). Thus, the choice of selecting professionals can be a strategy to improve the capacity of pressure in institutional spaces dominated by experts. (my bold; accessed online at https://journals.sagepub.com/doi/10.1177/00323217251319065)
Methodological upshot: What the counterfactual is to economic analysis, the “and-yet” counternarratives are to policy analysis. What would have happened in the former is contrasted to what else is happening during the latter.
12. Etcetera’s are the crisis kitsch of policy analysis (newly added)
Start here: “Policymakers also need to worry about those other factors—societal, political, economic, historical, cultural, geographical, governmental, psychological, technological, ethical, religious etc—that are so undeniably part of policy analysis and management.” Or: The very same people who question the use of GDP as a measure of human health and the environment end up being among the first to urge “Increase government budgets by x% of GDP for health and the environment and social protection and this and that, etc. . .”
That pincushion of etcetera’s indicates things are just critical enough to get a nod, but not that critical to actually have to name. So too Samuel Taylor Coleridge argued “matter” was treated like a pincushion whose surface was hidden by all the sensations, thoughts and properties stabbed helter-skelter into it.
You ask today, “But just what are the important factors?,” and you still get a pincushion of “etc’s” Methodologically, they are the crisis kitsch of policy analysis. But in reality, as Wittgenstein writes: “Again and again, my ‘etc’ has a limit.
