Public engagement and hard policy evidence

by Kent Aitken

---

Last week I pressed send on my dissertation, on which I'll blame my lack of posting. The topic was public engagement in Canada, and particularly, the role of economic analysis. My plan is to reduce the interesting parts of that research into a readable length, but I thought I might share six points that fell out of my conclusion.

1. Public engagement on policy, program, and service development is a thing. There's always a slate of ongoing public consultations in Canada, but the pace has picked up in the last year and the major difference is that there are far more that are intended for a broad public audience, rather than niche stakeholder groups. There are pros and cons to this.

2. As a general rule, government consultations are designed to understand what citizens value, but in a qualitative, rather than quantitative, way. That is, public input is viewed as a source of ideas and general feedback, not as empirically rigourous data. As currently practiced, public engagement is better suited for generating general insights, achieving social licence for policies, and avoiding major pitfalls than it is for systematically adding to the evidence base for policy options.

3. Each public engagement activity is important. Each represents an experience through which citizens' trust in government, and their willingness to participate in future engagement, can rise or fall. Public perception of the value of these engagements is crucial. The major variables here are the extent to which public input can theoretically influence policy, and the extent to which governments can prove that input was meaningfully considered. 

4. While it can be appropriate for governments to seek public input for general ideas and feedback, there's a massive downside. The greater the extent to which public input can be considered hard evidence, the easier it is for governments to incorporate that input in policy decisions, and the easier it is for governments to demonstrate how it influenced policy. There are many goals to engagement, including education, consensus-building, and legitimacy, but insofar as better policy is a central goal, engagement should be designed to produce data.

5. Public engagement is complex. There are hundreds of studied formats, each requiring a set of detailed design decisions, to align governments' needs with citizen satisfaction while producing the required data and insights. However, the way public engagement is governed, most of these design decisions are lost. If I may be blunt, it's essentially like a first-time homeowner overruling an architect on how their plumbing and electrical will work.

6. Governments need to build capacity for public engagement, particularly digital, but as per the above they may already have more capacity than they realize. So they must also develop governance that prioritizes expertise and good practices over ad hoc goals. The strongest version of this would be governance that includes public input and oversight on how engagement is designed and evaluated.

On Somewhat Simpler Taxonomies

by Nick Charney

---

A few weeks ago I shared some of the thinking I was doing with my team about new policy instrument taxonomies (See: On Possible Taxonomies of New Policy Instruments and Approaches); since then I've come up with a more plain language version that I wanted to share.

I've already had a couple of conversations with folks about it and while it's closer to being about outcomes it still doesn't quite crack the nut:


I want to use government's ...

	Nodality	Authority	Treasure	Organization
Substantive	convening power to create public value E.g. social networks, information networks, coordinating bodies, etc.	legal authorities to create public value E.g. law, regulations, standards, taxation, etc.	financial resources to create public value E.g. grants, contributions, transfer payments, etc.	non-financial resources to create public value E.g. people, skills, land, buildings, technologies, etc.
Procedural	convening power to explore how public value is defined and/or created E.g. social networks, information networks, coordinating bodies, etc.	legal authorities to explore how public value is defined and/or created E.g. law, regulations, standards, taxation, etc.	financial resources to explore how public value is defined and/or created E.g. grants, contributions, transfer payments, etc.	non-financial resources to explore how public value is defined and/or created E.g. people, skills, land, buildings, technologies, etc.

Essentially it breaks down the approach vector into wanting to innovate within the current confines of the system (substantive) and innovating on the system itself, which in my mind is an important distinction. That said, upon reflection, this is akin to the policy innovator's dilemma I've articulated previously (See: Policy Innovator's Dilemma).

It needs another iteration, but like I said, I wanted to share and solicit additional feedback. So, your thoughts?

Culture and risk

by Kent Aitken

---

"Culture, as the saying goes, eats strategy for breakfast, apparently it also eats technology, and probably has a taste for deliverology as well."
- Thom Kearney [You can change culture now]

The culture-eats-strategy theme was thickly present at last week's Canadian Open Dialogue Forum, a direction-setting conference about citizen engagement in Canada.

Citizen engagement is messy. It's uncertain. And it's in the open. So naturally, in lockstep with the culture-eats-strategy theme was the question of whether government is prepared for risks (real or perceived) associated with citizen engagement and open government.

Former Clerk of the Privy Council Wayne Wouters spoke about culture and risk, asking "How can you do something as a public service employee if you feel 'I'm breaking a rule'?" He disparaged the stock answer to mistakes in the public sector, which is to create a rule to ensure that X never happens again.

The problem is that any system, no matter how reliable, will generate errors with enough repetition - a fact that's at odds with a previous Clerk, Paul Tellier, who called for "an error-free administration." As Deputy Premier of Ontario Deb Matthews lamented, “We’re not allowed a failure on version 1.0 in government.” Unfortunately, that's the culture that has stuck.

A couple stories

Last year, a handful of public servants wrote a letter of praise, intending to send it to the managers of a colleague who'd been doing an amazing job and who was really helping out the broader community through sharing information and advice. When asking for signatories, a number of people said this: "This could backfire - collaboration may not be universally seen as  positive." That is, people were worried that drawing attention to an employee's collaborative, whole-of-government approach would diminish that person's standing in the organization.

Collaborative, networked, whole-of-government approaches are the strategy. Culture 1, strategy 0.

More recently. an NGO called In With Forward came to Ottawa to conduct a design lab with policymakers, exploring ways to support street-involved adults. From their blog:

We were testing what it would take to add ethnographic data to policy briefs. How could we give people in power direct access to the experiences of street-involved adults, and how could they use this information in the decision-making process? An oft repeated response was, “We can’t use stories. That’s not what we are asked to provide up the line. I wouldn’t even try to get it through the approval process.”

Design thinking, social innovation, and user research are part of the strategy. Culture 2, strategy 0.

What gives strategy something to chew on?

Ryan Androsoff and Xenia Menzies were exploring a possible hierarchy on Twitter throughout the conference: 

structure > incentives > culture > strategy

Strategy, in this model, has somewhat of an uphill battle. The left side, if poorly aligned with strategy, represents "organizational debt" that has to be addressed before you can make investments and start gaining ground (see: Nesta on Innovation in the public sector: Is risk aversion a cause or a symptom?)

Simply telling people that it's okay to take risks only works on the margins. And, like in the letter example, I'd even argue that it can backfire, leaving employees conflicted between what they're hearing and what they're experiencing.

Structural and systematic biases - in this case, a bias against risk - need structural and systematic responses. Governments have done this with Official Languages and Employment Equity, but we're never going to have a Key Performance Indicator for risk tolerance. Governments can't have risk quotas to meet (I'd dread the reporting: "We undertook 100 activities this year and 5% of them were classified as high-risk.")

Which means we need to dissect the structure, incentives, and culture to figure out the DNA of why public sector employees and executives make the decisions they do.

That said, in the meantime I'd propose a natural starting point: risk and hierarchies don't play well together. Short of calling for removing layers, I'd suggest that we revisit the assumption that hierarchies and decision-making chains have to be the same thing. In Australia, for instance, policy directors send advice and briefs directly to Ministers; the senior executives focus on coordination and administration. There are alternatives.

On Organizing Principles: Policy or Delivery?

by Nick Charney

---

A long time ago in a galaxy far, far away Mike Bracken – former Executive Director of the Government Digital Service in the UK Cabinet Office – delivered a speech entitled On policy and delivery at the Institute for Government in London. The video immediately made the rounds on social media and was the subject of much discussion among my peers. While I had an immediate and almost visceral reaction to the video when it hit the web (as Kent can attest to) I ultimately decided to pass on the opportunity to share it because I didn't think ruffling people's feathers at the time was worth it. I was cleaning out my Evernote recently, re-read my notes and re-watched the video; here's the redux.


If you haven't watched the video or read the transcript, I'd strongly encourage you to do so, as everything that follows below flows directly from it. If you need a TL;DR version, its: government ought to be organized around delivery not policy, because Internet. Its worth noting that Bracken positions his speech as a response to an early speech entitled The Positive Neutrality of Civil Servants given by Martin Donnelly, Permanent Secretary for the Department of Business, Innovation and Skills, as a part of the same lecture series. Unsurprisingly, Donnelly's speech wasn't shared as widely on social media. This is likely because of its more traditional view and the fact that Bracken's speech is more conducive of both the medium and the mindset of its user base.

That said, the dialogue between the two is important because it demonstrates that the world doesn't end when two civil servants speak publicly about their professional non-partisan views of the future of the civil service and that they can disagree without losing the confidence of their political masters or their (presumed) respect for each other. Or, in Bracken's words, "It's tribute to the civil service that it talks openly and occasionally critically, and from the inside, about digital transformation in government".

A deeper dive on re-orienting the organizing principle from policy to delivery

In general, I agree with Bracken's assertions that the civil service does a lot of good work that largely goes unheralded, that governments aren't immune to the pressures of digital technologies and that for far too long (and to our detriment) we have made 'digital' the purview of IT professionals. I also agree that there are tremendous opportunities to improve service delivery by making better use of technology and that the civil service needs to retain the ability to have a direct relationship with the public that it serves. He's even bang on when he advocates for closing the gap between policy and delivery. Despite agreeing with his line of reasoning, I find myself arriving at a different conclusion. I don't think that the civil service ought change its organizing principle from policy to delivery, but rather more directly come to terms with the larger governance challenges that are bubbling up wherever new technologies are rubbing up against governing institutions.

In fairness, some of my problems with Bracken's approach are partially rooted in his invocation of the Internet as a force majeure changing everything it touches for the better. Admittedly some of my disagreement stems from my reading of Evgeny Morozov's Click Here to Save Everything (See: Impossible Conversations: Click Here to Save Everything). I don't want to walk too far down that path but will admit that I've learned to reflect more critically on my own Internet-boosterism and the public discourse around the democratization of technology. While I'll spare you a lengthy explanation of Morozov's 'Internet-centrism' and 'technosolutionism' (again, see the book review), I will say that if you are sympathetic to either of those critiques (as Morozov levies them in the book) you will likely find Bracken guilty of both. While there are many instances where one could mount either charge against him in this regard (e.g. "I'm from the Internet", "The Internet has changed everything", "The Internet is changing the organizing principle of everything it touches, mostly for the better", "the Internet has 100% track record of success", etc.) the most obvious to me are "the Internet will reject that premise" and "because Internet".

Now let me clear, the point here is not to levy an attack on Bracken (especially not a personal one) but to engage with the ideas he puts on the table, ideas that I'm sure someone like Morozov would have strong views about. Yes, the Internet is changing things. Yes, it is creating new opportunities, but it's also decimating old business models. Ask someone who's lost their job security to the disruption (e.g. a taxi driver in Paris) whether or not disruptive innovation is inherently and unequivocally a net positive. You might find they have a different view. Disruption creates trade-offs that ought to be discussed and debated in the public sphere rather than simply accepted as inherently good for it. It's a view that silicon-valley contrarians like Morozov and Andrew Keen (Read: Cult of the Amateur: How Today's Internet is Killing Our Culture or See: New Thoughts From an Early Adopter) are likely to agree with.

Raising the profile of delivery innovation

Differences in world view aside, I think Bracken does an amazing job in raising the profile of innovation in delivery in his talk, especially at a time where (at least in Canada) so many seem to be focused on policy innovation (See: Is Innovation in Service Delivery a Blind Spot in Canada?).That said, I'm of the opinion that the primary driver here is "because people" not "because Internet". The difference is important because the former lends itself to higher order questions about the evolving nature of governance whereas the latter is likely to take us down the road of tools, techniques and technologies. What makes digital technologies truly important aren't the opportunities they present to improve delivery, but rather the fundamental questions they raise about governance writ large: Who has power? Who makes decisions? How do other players make their voices heard and how is account rendered? At least, these are the questions I'd rather be focusing on.

Moreover, "because Internet" ignores opportunities to improve delivery that have little if anything to do with digital technologies (e.g. behavioral economics). I fear that if we focus too narrowly on how to better lever digital assets to improve delivery we may crowd out ideas on how to better lever analogue ones (people, physical space, etc.). Surely there are tremendous opportunities to improve service delivery and drive better outcomes that have nothing to do with digital technologies: we can tweak the proximity, frequency, or type of intervention. Finally – and this may be unwarranted – civil service with delivery as its organizing principle risks making it primarily a vehicle for service delivery in the public eye and reinforce the archetype of citizen as consumer rather than a co-creator in a broader system of governance.

Does the Internet break policy-making?

In the speech, Bracken argues that:

"In a digital age, traditional policy-making is largely broken. It is slow, inflexible, unnecessarily complicated, afraid of technology and afraid of change. The cycle of green paper, white paper, draft bill, and secondary legislation is no longer the best way to decide to create or develop new services because user needs are given scant consideration, however necessary the process may be for Parliament."

Policy-making is hard because its rife with tradeoffs. Tradeoffs imply difficult choices and may inform inflexible processes, but does that make it broken? That I'm less sure of. What if the lack of speed, flexibility and complexity are the result of choices we've made in the past? Moments in time where we learned that speed creates propensity for error, flexibility room for exploitation and simplicity opportunities for untoward influence? In other words (Morozov's words) are these features of policy-making or are they bugs? This is an especially interesting and important consideration when we apply the project management triangle to the field of policy-making – the classic tradeoff matrix whereby you are constrained to choosing two of three options: Fast, Good and Cheap – and told that based on the fiscal climate one of your two choices had better be Cheap. Good policy-makers have always needed to understand the issue in context and if the world is as complex as everyone describes, its only natural that understanding that context may take more time. Think about the famous Einstein quotation about spending the bulk of the effort on defining the problem and the smallest possible fraction of it on actually solving for it. Obviously we can do a better job of involving service benefactors in service design but that doesn't necessarily solve the hard problem of choosing who those benefactors are or what that service is in the first place.

The Internet only breaks traditional policy-making if and when policy-makers fail to use it to their advantage as an input, a tool for outreach, and a means for implementation. In other words we ignore the Internet at our peril. That said, perhaps part of the challenges we face with respect to policy-making today isn't that we cling to traditional policy-making but rather we've too readily embraced the fleeting nature of Internet culture, that we don't value subject matter expertise as much as we once did, that policy work has been largely replaced with issues management and that as a result breadth not depth has defined career path of the next generation of policy wonks. In many ways the immediate and often emotional nature of discourse on the Internet is not a natural compliment to slow and deliberate policy discussion (See: Thinking, Fast and Slow About Online Public Engagement).

Furthermore, policy is about more than service generation. It's a proxy for discussions about decision-making, governance writ large, societal trade-offs, and the ongoing negotiation of the social contract through the iterative process we call participatory democracy. Why is a given (wicked) problem worth solving? How can it best be addressed? What does that intervention entail? My underlying concern with Bracken's approach is that I don't see these fundamental questions as issues that public institutions organized around delivery would be able to answer. If we take the current crisis in Syria as a tangible example and hypothetically overlay Bracken's organizing principle over the department responsible (Citizenship and Immigration Canada) we would likely see the department channel all of its resources to easing the flow of immigrants. Now, undoubtedly this would make an important and meaningful difference in the lives of hundreds if not thousands of people (and in all reality, much of this work is already marching full steam ahead). That said, is a delivery organization likely to help the government come to a decision on the number of refugees coming to Canada? In my view the number of refugees the government chooses to accept is fundamentally a matter of policy choices with some concurrent (e.g. processing capacity of the department) and many downstream (e.g. skills development programs, foreign credential recognition, etc.) delivery implications.

The point that I am driving at is that we can easily bring people more firmly into policy discussions early and often and involve them more directly in both the policy and delivery but ultimately to govern is still face the difficulty of having to choose (See: To Govern is to Choose). In summary (TL;DR), Bracken says he wants to fix the shop floor and the shop window, but I think the real question is ought we have the shop in the first place, and if so what ought it do and for how long?

Innovation is Information

by Kent Aitken

---

In August, the Clerk of the Privy Council delivered a speech titled “A National Dialogue On Policy Innovation.” Elsewhere, #policyinnovation is one of the most used hashtags by Canadian public servants. It’s somewhat of a hot topic right now. But what is "policy innovation" in the first place? 

For starters, it could refer to "new and interesting ways of developing policy." Or, "new and interesting policy." (See: On Prioritizing Policy Innovation) We tend to use both versions almost interchangeably, but this post tilts towards the former usage. I’ve heard the term used to refer to crowdsourcing and challenge prizes, deep dives into technological and social trends, improvements to government services, behavioural economics, and much more.

But within that nebulous concept, I think there's a central core to the entire idea that may be a useful way to think about how we gather and understand evidence, and how we make and implement decisions. It's all about information.

More options means more precise application

To back up slightly, let’s consider another arc of innovation that is both an analogy and a predecessor, that of telecommunications. We’ve gone from letter-writing to printing presses, telegraphs, telephones, the internet, and now to low-cost ubiquitous mobile connections. Every combination of one-to-one, one-to-a-select-few, one-to-many, public forums, with every combination of attributed or anonymous, for every combination of formats, all at a vanishingly small cost.

But here's the key: at one point, to communicate long-distance you had one option: handwriting a letter. Later, you had two: handwriting a letter, or paying to have something reproduced many times on a printing press. You didn't have to rely on a letter when it wasn't the best option. As more and more options became available, you could match your communications goal more precisely to different ways to achieve it.

Likewise, now we have a wider range of policy development approaches and policy instruments, which means there’s a greater chance that we can match the right approach to the right situation. We have a wider range of options partially because we get inventive over time, but far more so because policy development and implementation often is communication and so we’re simply piggybacking on telecommunications advances.

The information

Which isn’t much of an insight, I recognize. Yes, the internet opens up options for how government does things. But if we start to think of policy innovation as communication, instead of as enabled by communication, it starts to shed light on what we’re really trying to accomplish, and where “innovative” approaches fit in more “traditional” approaches. Using the terms in quotations lightly.

Basically, the approaches that get pegged as "policy innovation" often boil down to two key actions:

• transferring information between people
• arranging information for people

It’s the crux of crowdsourcing, policy or service jams, innovation labs, open data, design thinking, challenge prizes, and citizen engagement approaches like consultations, townhalls, and social media chats. Someone has information that policymakers can use: ideas, problems, slogans, lived experience, or academic expertise (see: The Policy Innovator's Dilemma). Then it’s a matter of finding the best way to access it, which is a question of format. You just have to learn the formats. Similarly, once you've crossed the threshold and learned a new telecommunications approach (case in point might be parents and grandparents on Facebook), it becomes part of a passive mental algorithm that takes a need or goal and instantly knows how best to accomplish it.

Talk of policy innovation tends to go hand-in-hand with the idea that policy issues increasingly cross jurisdictional or societal boundaries, and are a part of an increasingly complex environment (see: Complexity is a Measurement Problem or On Wicked Problems). Which is where arranging information becomes invaluable.

Let's say  you get ten informed stakeholders of a given policy question in a room, and ask each for their concerns. They each reveal a different way of looking at the issue, revealings its complexity and pointing out legitimate pitfalls for policy options. The problem is that by the time the tenth stakeholder spoke you forgot the concerns of the first five, so it's impossible to understand all ten in context. It's Miller's Law: human beings can only hold seven things, plus or minus two, in our working memory. Which is where techniques like journey mapping, system mapping, and sticky noting everything are crucial for policy. They're the policy landscape equivalent of doing long division on paper so you can remember everything in play - what we might call mental scaffolding. 

Many approaches include both transferring and arranging information. For instance, a public consultation might include a call for ideas with a voting mechanism that creates a ranking, signaling importance. Some deliberation platforms include argument mapping systems that use algorithms to arrange the discussions for participants, almost like Amazon bringing complementary products to the forefront. ("Are you outraged at your government about X? Many people outraged about X are also outraged about Y, perhaps you should consider lambasting them on that topic too.")

In other cases, governments can (and should) map out what they already know about a given policy issue to get it out of working memory and focus on change drivers and relationships between forces. This will become increasingly important if we truly want to get out of siloed policy-making, find hard-to-see connections between once-distinct policy areas, and genuinely understand entire systems. Our governance model was built for a world we falsely believed was simpler than it was, and within that we're running into our own cognitive limits. We literally cannot hold all the elements of a complex policy issue in our heads without some kind of mental scaffolding, be it tools, other people, or paper.

Metadata

Two notes on metadata, or information about information (an example would be how DSLR cameras automatically include date stamps, aperture, shutter speed, iso, and more information in image files).

First, some approaches that get lumped in with policy innovation don't fit perfectly with the transferring and arranging information categories. Behavioural economics, for instance (and its service delivery cousin of user testing), seems more like creating new information through research. But viewed from a policy lens, I'd suggest it's actually more like metadata.

Let's say government wants to maximize the rate of tax returns, so tweaks the language on letters to taxpayers to see what framing resonates with people. Here's the UK example:

"...replacing the sentence “Nine out of 10 people in the UK pay their tax on time” with “The great majority of people in [the taxpayer’s local area] pay their tax on time” increased the proportion of people who paid their income tax before the deadline."

The core policy instrument here is a law, and the letter sent to taxpayers is supporting education about the importance of filing tax returns. In this case, the information is in the letter. The behavioural economics piece is metadata about that information: how many, and which, people acted upon the information they received. It's still really about transferring information between people, which puts tools like behavioural economics and data analytics in this common framework and may help practitioners navigate between possible approaches.

Second, there's a meta-level to the idea of transferring and arranging information that changes the value of different approaches and formats. We might call it "conspicuous innovation" or "conspicuous engagement." Basically, the transfer and arrangement of information is not the only goal achieved by these approaches - someone emailing a policymaker a vital piece of information for a policy question is worth less than that same person posting it publicly during an official consultation. The metadata for that piece of publicly posted information includes the number of views from other people, the signals about government's attitude towards governance and transparency, and the future value to others. 

So what?

The "policy innovation" toolkit centers around two actions: transferring information between people and arranging information for people. Past this common core, it's often a question of forums and formats (increasingly, but not uniquely, about how we transfer information from non-governmental actors) (with exceptions, of course). So what?

One, I think it's worthwhile to examine what binds the idea of policy innovation together, to refine our working concept of the term.

Two, I think thinking in these terms highlights what we're actually trying to accomplish through these approaches, and might make it easier to choose between them.

Three, putting them in a historical context puts the perceived risk in context. I mean two things here: first, that policy innovation is very similar to our personal experience with telecommunications advances: more options allows more niche approaches, and eventually they become routine. Second, that if some of these approaches are at a fundamental level analogous to things government has been doing for ages, they seem less daunting. For instance, there are dozens of consultations ongoing at http://www1.canada.ca/consultingcanadians at any given time. It's just a different way of transferring information between people and policymakers.



Thank you to Blaise Hebert and Nick Charney for super interesting conversations on this topic.

Also, two recent posts from Melissa that are good general fodder here: What Innovation Feels Like, Part 1: Fear; and Part 2: Lack of Trust

Does Government Need a Prediction Market for Policy Options?

by Nick Charney

---

I'd like to spend some time exploring whether or not the idea of a Policy Analysis Market (or if you prefer a prediction market for policy options) has legs in the broader public policy context. In so doing, I will be drawing on a handful of papers I've read on the subject, citing them heavily, reserving most of my own commentary for the end. Each of the quotations are linked to the source and I've provided the references as a laundry list of recommended reading at the end of this piece.

What is a prediction market?

"Prediction markets are forums for trading contracts that yield payments based on the outcome of uncertain events. There is mounting evidence that such markets can help to produce forecasts of event outcomes with a lower prediction error than conventional methods... Several researchers emphasize the potential of prediction markets to improve decisions. The range of applications is virtually limitless – from helping businesses make better investment decisions to helping governments make better fiscal and monetary policy decisions ... These markets could assist private firms and public institutions in managing economic risks, such as declines in consumer demand. and social risks, such as flu outbreaks and environmental disasters, more efficiently." (The Promise of Prediction Markets)

"The theories underlying [Policy Analysis Market] and other prediction markets are the Efficient Capital Markets Hypothesis (ECMH) and the Hayek hypotheses. These hypotheses explain how information is aggregated such that market prices provide accurate estimates on the likelihood of future outcomes." (Using Prediction Markets to Enhance US Intelligence Capabilities)

"The power of prediction markets derives from the fact that they provide incentives for truthful revelation, they provide incentives for research and information discovery, and the market provides an algorithm for aggregating opinions." (Prediction Markets)

How do prediction markets work?

"Trading in prediction markets is similar to any haggling kind of transaction: buyers and sellers exchange offers and counter-offers until they agree on a price. In a double auction, the most common mechanism used to clear prediction markets, buyers submit bids and sellers submit asking prices, which are ranked from highest to lowest to generate supply and demand curves. Trades are executed when two prices match (i.e., bid-ask spread is zero or supply intersects demand) ... [and] payoffs are determined by the occurrence (or lack thereof) of outcomes." (Using Prediction Markets to Enhance US Intelligence Capabilities)

Generally speaking:

"Market prices for contracts can be interpreted as probabilities of an expected outcome ... [for] example, a contract closing at 67 cents would mean there is a 67 percent probability." (Using Prediction Markets to Enhance US Intelligence Capabilities)

What purpose do prediction markets serve?

"Numerous studies have suggested, however, that markets do lead to predictions that are more accurate than traditional forecasting techniques, including those that rely on expert opinions. (Using Prediction Markets to Enhance US Intelligence Capabilities)"

"Exploring the possibilities of prediction markets further, others have proposed that these markets should serve as mechanisms to help decide which of several policies options should be implemented." (Using Prediction Markets to Enhance US Intelligence Capabilities)

"The 9/11 Commission, in its discussion of how to reorganize the US Intelligence Community, cited the lack of unity of effort in information sharing as the “biggest impediment to all-source analysis—to a greater likelihood of connecting the dots.” The lack of information sharing is further compounded by a culture that emphasizes information compartmentalization, suffers from stovepipe mentalities, and bureaucratic distrust. One way to solve these problems is to work on [Intelligence Community]-wide software and databases and develop improved protocols for accessing classified information and for providing better coordination of interagency analyses. Another way is to use prediction markets to aggregate information and analyses ... information and judgments from different corporate divisions into probabilistic estimates of future outcomes, a prediction market could perform the same function for the Intelligence Community." (Using Prediction Markets to Enhance US Intelligence Capabilities)

Have prediction markets worked before in a policy context?

The Defense Advanced Research Project Agency (DARPA) experimented with this approach back in 2001 when it created a Future Markets Applied to Prediction (FutureMAP) program that tested whether prediction markets, could be used to improve upon existing approaches to preparing strategic intelligence. Long story short, the program was cut short when congressional (faux) outrage took over and shut down the project; apparently the optics of intelligence officers placing wagers on terrorist activities didn't sit well with them and they thought such activities might actually incite actors to take measures they otherwise wouldn't have. That said, much of the research I've done have indicated considerable upside to the approach:

"Prediction markets can function as powerful complements to the traditional process by which long-term estimates are performed. Their power is further multiplied when one imagines that the time and resources saved in running such markets means that several long-term estimates can be run concurrently and updated periodically. ... by allowing analysts to hedge their estimates in the form of conditional contracts, policymakers gain valuable probabilistic estimates, as opposed to wishy-washy judgments which policymakers may easily ignore." (Using Prediction Markets to Enhance US Intelligence Capabilities)

"Prediction markets could also be used to make ex-ante evaluation of policies. Take the question of whether the United States should continue to fund the Andean Regional Initiative (ARI). Analysts could bet on two futures contracts: (1) the tons of cocaine that will be exported from the countries affected by the ARI to the United States in 2009, conditional on the United States continuing ARI; and (2) the tons of cocaine that will be exported if ARI is terminated.The difference in the two estimates would tell policymakers how much of a reduction (or increase) in cocaine analysts expect from the implementation of ARI. A more realistic assessment would most likely involve analysts speculating on several futures contracts with different expiration dates." (Using Prediction Markets to Enhance US Intelligence Capabilities)

Who ought to participate in such a prediction market?

"Since the objective here is to effectively aggregate information and analyses of the entire Intelligence Community, implementation of prediction markets on a community-wide basis is preferable to intra-agency markets. Ideally, anyone with the relevant information should trade. If the traded contract relates to aerial suicide bombs, then even airport luggage screeners, in addition to homeland security analysts, are potential market participants. This necessarily means that expert knowledge on a particular subject is not required before making a bet.

A more difficult question is whether there are circumstances under which the general public should be allowed to trade. Certain issues might require the aggregation of information and opinions on subjects intelligence officers may know little or not enough about. On the other hand, making public certain markets might be inadvisable because doing so might signal adversaries about intelligence interests.

A compelling case can be made for making diversity a key criterion. Diversity means that market participants have different pieces of information about their surrounding environment and consequently different judgments on the event for which they are betting. The HP experiment aggregated information across several corporate divisions. Economic theory and empirical evidence suggests that “thick” markets are preferable to “thin” ones."(Using Prediction Markets to Enhance US Intelligence Capabilities)

Moreover,

"Ambiguous public information may be better in motivating trade than private information, especially if the private information is concentrated, since a cadre of highly informed traders can easily drive out the partly informed, repressing trade to the point that the market exists" (Prediction markets for business and public policy)

What would a policy options prediction market look like in the Canadian policy context?

At the systems level, a cross government policy options prediction market holds out the promise of providing a better interdepartmental view on success/failure probabilities of different policy interventions because the aggregating forces of the market helps overcome the inherent information asymmetries built into Westminster accountability structures. It could also provide tremendous insight into how different departments view a given issue because you could run analysis on whom (point of origin) is investing where.

If such a market was opened up to broader public participation it would provide policy makers with a better overall societal view of the options on the table. Further, there's an interesting argument to be made here about whether or not such a market could help redefine participative democracy in a digital era and improve overall public engagement by offering continuous options for engagement across a spectrum of issues rather than limiting it public participation to issue specific and time-limited opportunities for feedback. In so doing this approach could also give policy makers valuable insight into public preference on a particular subset of issues, say whether or not the public prefers prevention or remediation strategies, by how they invest (prioritize) between competing options.

An interesting place to test this type of broader approach may be in the field of social finance where it could be used to see how different players (funders, service providers, stakeholders, clients, and government officials) see the probability of success of a given initiative and/or how they value the other actors in the ecosystem.

Overall, I think the idea has merit.


Three closing but related caveats about design

First, the research clearly shows that contracts in such a market would need to be clear and specific, written in plain language so as to be easily understood by participants.

Second, the research also shows that the market's success is contingent on participants' motivation to trade and that the profit motive is usually enough to spur activity; this tells me that the design of the overall prediction market needs to be gamified in some way.

Third, an interesting place for people interested in this type of design might be Empire Avenue, which has a whole bunch of those types of design decisions built into it under the hood. It's a social media service that a bunch of us got into years ago but subsequently walked away from. That said, it may be worth a second look in the context of this conversation around the applicability of prediction markets to policy options.


Recommended Reading / Resources

• Using Prediction Markets to Enhance US Intelligence Capabilities: A "Standard and; Poors 500 Index" for Intelligence 
• Prediction Markets by Justin Wolfers and Eric Zitzewitz, National Bureau of Economic Research
• Prediction markets for business and public policy, Andrew Leigh and Justin Wolfers, The Melbourne Review
• The Promise of Prediction Markets, Science Magazine 

The Future of Policy Work

by Nick Charney

---

***Updated Nov 28 to reflect changes to remarks between the original draft and the speech I delivered.***

On Tuesday next week I will have the privilege of addressing a large number of policy professionals in the Ontario Public Service as a part of Polivery V: The Future of Policy Work. What follows is a first draft of my remarks, pardon the style, it helps me deliver. As always, your comments are more than welcome... (it's late, this is a draft, and I'm tired, etc).



Good morning everyone.

I'm not sure you know how lucky you all are to have such an esteemed panel before you today.

I'm not really sure how I snuck on to the bill.

My name is Nicholas Charney.

I'm the Director for Engagement and Innovation at the Institute on Governance.

We are small not for profit organization whose mission is to advance better governance in the public interest.

We do this through our ongoing advisory work, learning activities and by conducting primary research with academic partners.

I’m a policy professional and currently on interchange from the Government of Canada where I've spent the last 8 years working at the confluence of people, public policy and technology.

There are many things that I could say the future of policy work.

I could start by saying that in the future having the right skills will be essential.

Or, that a talent-focused culture will be critical.

Or, that organizational agility is the key to effective outcomes.

But I could say all of that and have said nothing.

I'd rather start out by saying that future of policy work is still being written.

That there is no shortage of wicked problems, demand for ideas, or need to bring them bear.

That technology and Zeitgeist are changing the nature of public policy.

That these changes are creating a number of challenges and opportunities.

And that how you deal with them today will ultimately determine what your future holds.

First the challenges.

Challenge #1 - The hollowing out of capacity.

Strategic policy shops have quickly become issues management shops.

Driven by increased transparency and a 24/7 news cycle.

We often sacrifice the long-term health of our democracy to deal with that which is immediately before us.

This is the fast food approach to public policy.

It might taste good at 2 o'clock in the morning, but ultimately it's terrible for our health.

As my mother used to say to me when I came home late at night, we need to make better choices.

Policy makers need to re-claim their relationships with the media, with elected officials, and with each other.

They need to stand by less and stand for more.

The faceless bureaucrat is no longer a tenable position in this environment.

Challenge #2 - Innovation by check box.

Everyone is suffering from innovation fatigue.

When everything is innovative, nothing is.

Labeling something as such is as meaningless as labeling it as secret in today's environment.

Yes - innovation hubs, labs, dragon's dens and hackathons are all in vogue right now.

But the true test isn't what goes into them, but rather what comes out of them.

Too often our best and brightest are put to work on matters of process rather than substance.

Let's put more smart people next to hard problems and stop treating problems as puzzles to be solved.

That metaphor assumes that all the necessary pieces are already on the table.

That they just need to be rearranged or reprogrammed.

But that’s not true.

‘Policy Innovation’ defined as moving the pieces around or adding more processing power won't disrupt the status quo.

That is the status quo

Challenge #3 - Hyper-bureaucratics

Process has always been the bureaucratic panacea.

But by now we must be fast approaching what I like to call peak bureaucracy.

The point where we simply cannot add any additional layers without incurring untenable costs.

Be wary of those who refuse to do the hard work of flattening hierarchies, simplifying processes and minimizing barriers.

Be wary of those who would rather establish processes to diffuse blame than simplify them to consolidate responsibility.

We need more decisions and less diffusion.

Challenge #4 - The loss of monopoly & increased competition

We have new roles.

We've moved from that of a monopoly provider to something more akin to a sensor, sense-maker, connector, a validator.

It can be unnerving but don't panic.

Embrace the fear and explore the new opportunities.

Opportunity #1 - Bask in the complexity

We have never had a better understanding of how things are interconnected.

But focusing solely on technology or innovation actually prevents us from realizing the art of the possible.

We know that connecting people and ideas has never been easier.

Yes the policy shop of the future deploys technologies to connect people around ideas but also employs people to do the same.

It asks people to lean in and slog through the messy stuff: the history, the economics, the philosophy, the art, the ambiguities, the contradictions, the trade-offs.

The stuff technology can't fix.

This takes time and effort.

The policy shop of the future retains the time-honored tradition of subject matter expertise and encourages depth, not just breadth, of experience.

Opportunity #2- Engage in social media

Listen to what people are saying.

Find the experts.

Weigh their analysis.

Read what they read.

What's the Zeitgeist telling you?

Be curious.

Create content don't just consume it.

Lean in and slog through the hard stuff yourself.

Write things down and work through problems.

And don’t forget to take the time to unplug once in a while.

Put down your phone

Put down the remote and read a book.

Like a paper book.

Break its spine, dog ear the pages and write in the margins.

Opportunity #3 - Experiment with data

Find, verify and link or liberate useful data sets inside your organization or within your field of work.

Explore what happens at the margins where different data sets interact.

Create visualizations that cast an old problem in a new light.

If you find something interesting broaden the tent and engage others.

I did this recently by visualizing all 278 instruments in the Federal Government’s Treasury Board Policy Suite.

I did it out of interest, shared it to the Internet and wound up in front of the ADM responsible for the suite within a week.

Don't worry, it was a good meeting.

If you don't have the skills to do this or the time to learn, find people who do, and work with them.

Opportunity #4 - Use design thinking

Empathize with problem.

Be creative when thinking about solutions.

Be rational when mapping the solution to the problem.

Match people's needs with what is feasible.

This is something we are teaching right now in collaboration with the GovLab @ NYU.

Its surprising how effective a two day deep dive on a problem can be if you approach it with the right methodologies.

In case you are interested, both NYU and the d.school at Stanford have a number freely available methodologies online.

Opportunity #5 - Read up on behavioural economics

Commonly referred to as nudge.

There are a lot of books on the matter and some interesting work has been done recently in the UK.

Long story short, slight tweaks in your approach vector can drive vastly superior outcomes.

Behavioural Economics brings sentiment, analytics, and design to ground by emphasizing what people actually do when faced with a given situation rather than what we think they ought to do.

The US government just invested $100 million dollars in a simple nudge.

They doubled the value of the SNAP benefits – or food stamps – for people who use them to buy local fruits and vegetables at farmers markets.

They got the idea itself came from a farmers market that had been doing it on its own without the government subsidy.

In conclusion I want to say

How you go about your work will continue to change but ultimately being able to frame up advice that helps leaders make good decisions will always be a critical skill for policy makers.

Indeed, it always has been.

Now if you recall in my opening remarks, I told you that the future of policy work is still being written.

In closing I want to appeal to your sense of agency and remind you that when it comes to policy advice you literally have the pen.

Invest that pen.

Familiarize yourself with the trends and new techniques, but don't chase breadth at the expense of depth.

Do you best to balance both, stay curious, and remember, the pen is mightier than the sword.

Thank you