Practical Research Planning And Design 12th Ed

[Ezilda Newnam]

Todo that work well I did of course engage in some depth with relevant academic work, including the research on longtermism both from within academia and on this forum. And lo and behold it seemed to me like the space was divided into two distinct camps: longtermist theorists and long-term practitioners. The theorists wonder why policy makers do not listen to them and the policy practitioners wonder why academics are not producing work relevant to them. As a practitioner, it seemed that on some days I would say something that was obvious to me and a researcher would be excited by how novel and useful it is, yet on other days I just could not understand the things longtermist researchers were doing and why it mattered. This post is an attempt to bridge this divide.

Section A is descriptive. I invite you to look around my world, at the politicians, policy makers and risk planners who think long-term. I draw examples from fields as diverse as defence, forestry, tech policy and global development looking for common threads and patterns that give us some idea of how we should be making our long-term plans and decisions. My hope is to both be informative about current best practice in long-term planning but also to give a sense of where I am coming from as a practitioner thinking about the long-term.

Section C is constructive. I reflect on how my experience as a practitioner of longtermism shapes my view of the academic longtermist community. I then make some recommendations about how longtermists can better produce useful practical research.

Imagine that you are a politician or policy maker. You believe that the future matters a lot and that preventing existential risk is important, but you are uncertain about how best to achieve long-term goals. So for a starting point you look for existing examples of good long-term planning and long-term decision making.

At first good examples of long-term policy thinking can be hard to spot. Political incentives that push policy-makers towards short-term plans [1] or towards making long-term decisions primarily based on ideology [2]. There are however places where there is seemingly good long-term policy making to learn from, especially a step away from the most politicised topics. And if we look across enough institutions we start to get a picture of a best practice approach to long-term thinking.

One key thing to think about for long-term planning is that you do want your long-term vision to be broadly agreed by those who will use it and to not change too much over time. For policy institutions that need to plan across multiple domains and political changes this means as much as possible building support across party boundaries and ensuring broad public buy-in to the vision. This may require broad public consultations to build a vision that maps out a consensus view of the whole population. For smaller institutions this may require using consensus decision making tools among those most involved or affected by the institution.

Long-term plans almost always have the following pattern. They start with a long-term vision, often time independent. There is then a long-term high-level target or set of targets, maybe 10-30 years ahead. There are then sets of medium term targets a few years ahead that go into more detail. Finally there are the implementation plans for achieving those targets. If well managed there will also be oversight and accountability and check-in mechanisms to make sure the whole process is working.

I would also note that agreeing the medium-level goals likely requires some combination of research, taking stock of progress towards the high-level goals and consensus building among key decision makers.

To make adaptive plans work may require ongoing monitoring of situations and awareness of the triggers that might cause your plans to change and designing actions that maintain option value and flexibility. Adaptive plans are not very common but they may look longer ahead.

To make good long term plans and to support future focused decision making there are a host of futures tools that practitioners use. This includes tools such as: horizon scanning, Delphi method, discount rates, scenario planning, red-teaming, prediction markets, reference class forecasting, Superforecasting and many more.

I am not going to introduce every single futures tool here but I will run through at a high level what various tools are doing. There are a variety of different ways to explain Futures work and there are better sources than me, maybe see this NESTA report or the UK government futures toolkit or the Society for Decision Making Under Deep Uncertainty

Mapping out a broad range of possible futures to ensure that we are asking the right questions, not neglecting key possibilities and able to design options that are robust to a range of future scenarios. Tools include: tabletop exercises, exploratory modelling, brainstorming, red-teaming, trend analysis, scenario planning. (Examples of UK good practice include the work of the Emergency Planning College which regularly runs emergency exercises for civil servants and the in-depth Global Strategic Trends report.)

Analysts and forecasters are in many cases not the final decision makers and they will often need to communicate their images of future worlds to others in a way that supports good decisions. This can be achieved by senior decision makers participating in futures exercises, scenarios that describe the most decision relevant future (a nice example here), or constructing a believable inspiring shared vision to allow large teams to work together towards the same goal (see examples above in section A.1).

The standard way that policy decision makers make decisions about the future is to use quantitative assessments (such as expected value calculations) to compare current and future or a variety of future options. In policy analysis this often looks like converting current and future costs and benefits of various options into a Net Present Value (NPV) or a net present social value (NPSV). For example see the Regulatory Impact Assessment Calculator (the calculator is useful but unsurprisingly for a government tool it cuts off any consideration of effects after 10 years).

Longtermists would almost universally reject the pure time preference part of the discount rate. When considering extinction risks the economic growth part also drops out. The risk parts remain and as such discount rates could be a useful tool for longtermist decision making. [4]

In low stakes situations or when there are sufficiently good forecasts and low-uncertainty, a simple quantitative assessment might be sufficient. But to make a good long term decision, especially on greater time scales, a few more tools are needed.

Decisions about the future are often decisions about situations of high uncertainty. The overarching principle I see applied to making good decisions about an uncertain future is to ensure that those decisions are robust to a broad range of types of evidence, to a broad range of assumptions and to various future scenarios.

If making a decision about the future (or about anything with high-uncertainty) decision makers should ensure that they are drawing on a broad range of decision making tools. I think this is best explained by Joey here or by Holden here. The more different decision tools that you can use and the greater the extent to which they converge the more certain one can be in their decision. This could mean using quantitative assessment(s), expert consensus, strategic analysis, common sense and so on. Decision makers should also be aware that more bespoke decision tools outperform general decision tools [6], for example for managing risk you could use a vulnerability assessment approach which is designed for understanding risks.

These tools require identifying a range of possible future scenarios. These might be the scenarios where current assumptions about the future are wrong, the most likely scenarios, the scenarios suggested by considering critical uncertainties, high-risk scenarios, or something else. Decisions makers then build a detailed understanding of each of the important envisaged future scenarios. They then aim to make a plan that should work (or not fail horribly) in all the identified future scenarios.

These tools require trying to identify the critical assumptions that underlie the decision that might be wrong. Decision makers then try to design a plan that would still work if any of those assumptions were incorrect. This reduces the chance that the plan will fail.

As mentioned this may look like taking a satisficing rather than maximizing approach. So rather than taking the option with the highest expected value robust decision makers take the option that is satisficing good on a range of future scenarios or assumptions. This helps ensure that plans is robust to a range of possible points of failure (the theoretical case for this is set out in this GPI paper), that the plan is more likely to work in the case of unknown unknowns and that the plan is stronger to poor decision making (similar to how engineers will design a bridge made to withstand many times the forces it is expected to face, to adjust for uncertainty and overconfidence).

These tools are used but are often not published so it is harder to point to examples. Robust use of evidence is more of a way of thinking than a specific tool, and assumption based planning is not common in the UK policy spaces I am familiar with. Furthermore the disconnect between analyst and policy maker means there are sometimes challenges to get the outputs to feed into final decision making. Some examples are:

Maybe you disagree and think existing practice can teach us very little about how to make decisions. But agree or disagree or something in between, let's move on. I have shown you around my world and at the tools, approaches and ideas that make up the best examples of long-term thinking I can find. The question to explore now is what might this tell us about how to approach longtermism.

3a8082e126