Showing posts with label experimentation. Show all posts
Showing posts with label experimentation. Show all posts

Friday, 21 October 2016

The Many Challenges of Convening the Policy Innovation and Experimentation Ecosystem


by Nick CharneyRSS / cpsrenewalFacebook / cpsrenewalLinkedIn / Nick Charneytwitter / nickcharneygovloop / nickcharneyGoogle+ / nickcharney

To be fair, I'm not sure such a thing really exists. Sure there's a lot of chatter about all the interconnect moving pieces but I'm not sure its quite as tangible or coherent a thing as perhaps many of us make it out to be.

First, there's lots people inside the innovation bubble (the "us" I referred to above): dedicated hubs and labbers, reverse mentors to bodies like the Deputy Minister's Committee on Policy Innovation (DMCPI), folks working in central agencies on the experimentation file, folks in line departments working on advancing a particular innovative instrument or approach (e.g. social finance), or anyone else with 'innovation' (however defined) as a core responsibility. Generally speaking those inside the bubble speak their own language, enjoy the privilege of proximity to decision makers, and are well connected across the system to other folks inside the bubble. In many ways its not much different than any other functional area of expertise. What we are witnessing is simply the formation of a network of 'public sector innovators' as the vocation of 'public sector innovation' goes through some sort of piecemeal and un-managed professionalization. This is apparent when one looks at the inherent definitional challenges within the ecosystem. There's not a shared vision of what policy (and program?) innovation and experimentation is. So much of the conversation around these issues is either non-existent, conceptual, or conflictual. Moreover, experimentation -- while appearing in a mandate letter -- remains loosely defined and as a result could run the gamut from trying something new to running a randomized control trial. More importantly, the term itself is problematic, one wouldn't want to conduct a policy experiment on a vulnerable population and/or indigenous community for obvious reasons. The term is sticky because -- like I said -- its in a mandate letter and was likely included to encourage the development of evidence based policy but its problematic in some very important contexts (e.g. reconciliation with Canada's indigenous peoples). Personally, I parse it as 'policy and program innovation' (which includes regulation) and think it manifests in two ways: (1) how we design policies and/or programs and/or (2) how we deliver policy and programs. Innovation then is about improving our policy inputs and/or our policy outcomes. Whereas I see experimentation as gathering an evidence base that can be measured against a counterfactual to determine which of the two options generates greater public value. (Caveat: I don't see much room for policy innovation and/or experimentation to occur at the point of the decision to enact (or not) a particular policy. That nexus is largely the purview of elected officials, and short of electoral reform -- which is obviously highly politicized -- there's little public servants can be doing to innovate around the decision making nexus (See: To govern is to choose).)

Second, while the innovation bubble is incredibly important to those working inside or beside it, it remains largely unknown to those working outside it in the regulatory and/or programmatic spaces of government. Perhaps my evidence is anecdotal but my overwhelming experience has been that whenever I speak about what is going on inside the bubble with those outside it that its the "first they've heard of any of it" and that it is either "fascinating" or "completely irrelevant". The truth is likely that a more concerted effort needs to be made by both those inside and outside the bubble to meet somewhere in the middle and grow the linkages between the two. However both sides face tremendous pressure to deliver within their own spheres and if the incentives for more purposeful communication and collaboration aren't in place then we'd be foolish to expect them to do so of their own volition. We need more connectors. People's whose job it is to build bridges between the players, ensure alignment, new create opportunities, and scale existing ones. I doubt we will see the necessary investment in these types of people/roles because there's a lack of understanding of their value. This lack of understanding is embedded in the general consensus that these types of connections will form spontaneously in the network because of the proliferation of communications channels, the availability of information, the wisdom of the crowd to elevate that which needs elevating, and everything else taken for granted under the larger rubric of the ethos of the internet age. However, if there is something I've learned in my current position it is that there is still a very strong role for connection and curation in this environment. The proliferation of communications technologies has shifted the friction, but make no mistake, it surely still exists.

Third, there's a self-selection problem -- and not in the way you'd think. It's not so much about people falsely self-selecting into the 'innovators' category but rather that innovative folks are self-selecting out of it completely. Few people doing innovative work actually self-describe their work as such. They are simply trying to do better. They weren't given an innovation mandate or draw resouces from a centralized or dedicated innovation fund, they simply chose to use their resources differently within their own particular context. This means that they might not even know the value of what they are doing to the rest of the ecosystem. Or, they may be actively hiding, staying below the radar, because with greater attention comes greater scrutiny, and with greater scrutiny comes the loss of flexibility that allowed them to be innovative in the first place (See: Asymmetric Scrutiny, Superforecasting, and Public Policy). This was in fact one of the findings of an innovation survey conducted by DMCPI.

Fourth, like any ecosystem it is both collaborative and combative -- there are things that work well together, but there is also competition for scarce resources (e.g. money, talent, the attention of senior management, etc.) and there are (implicit and/or cultural) established food chains (e.g. Central Agencies > Line Departments, Strategic Policy > Program Policy, Ottawa > Regions, etc.). Often when we use the word ecosystem we invoke it in such a way that emphasizes the collaborative elements and de-emphasizes the competitive ones, but we all know that ecosystems are highly competitive environments. For example, there are currently 18 instruments/approaches covered off in the policy innovation portal my team stood up, the fact that these approaches are innovative brings them together under the portal's banner of 'policy innovation', however we can't expect them (or more importantly their champions) to always work in complimentary ways. Open data and pay-for-performance may be two innovative approaches but they may also not have all that much to do with one another at an operational level; this leaves little room for collaboration and creates room for competition over scarce resources. This is where the emerging literature in public administration on the rise of instrument constituencies is particularly compelling (See: On the Rise of Instrument Constituencies).

I don't want to belabour the point but did want to say that all of these interconnected factors -- and their obvious exceptions, caveats, counter-examples, and side-steps and twists -- make up but a handful of the many challenges we are facing as an organization whenever we try to convene the players, increase alignment, and identify opportunities for collaboration and scale. Finally, we need to carve out more time for coordination, thoughtful deliberation, and purposeful conversation within and around the ecosystem as a whole if we are to rely on it to deliver our organization into the 21st century.

Friday, 12 August 2016

The transparency antidote to risk

by Kent AitkenRSS / cpsrenewalFacebook / cpsrenewalLinkedIn / Kent Aitkentwitter / kentdaitkengovloop / KentAitken

A couple years ago my team worked on a project that got some negative attention. I'll go light on the details, but we took some criticism and my teammates got worked up, looking for ways to defend the outcomes. I had the opposite response: now we had some ammunition to do things better in the future.

Across the public sector, the meme seems to be that people are scared of transparency (e.g., open data excuse bingo). Which is, at first blush, understandable: it might mean more scrutiny, more criticism, a loss of control over the conversation.

However, there's another meme that bureaucracies are risk-adverse and that they stick to the status quo. Transparency about public programs and services reveals the risk in the way things are now, and gives decision-makers an even playing field to make decisions about change.

Here's an example. Edmonton has had a public dashboard of public policies and services for years: transit ridership, 311 call response times, growth in small and medium businesses, etc. And some are in the yellow and red, not meeting the benchmarks set.


No one signs up for public service to deliver a service that doesn't meet the public's needs. Having the performance freely available on a struggling program means that the program managers will have to explain why things are the way they are, but it also means that the program will receive the support it needs to improve - or change. It's short term stress for the sake of being a part of something worthwhile in the long run.

And the public gets a better shot of getting what they need, sooner - both in terms of information and public outcomes. Where there's little transparency about current performance, potential changes from the status quo get disproportionately scrutinized. And given that even long-running activities - e.g., how governments have been doing IT - are still experiments with uncertain long-term outcomes, we need to put those activities on an equal footing with their alternatives.


Friday, 27 November 2015

Experimenting with Policy Development

by Nick CharneyRSS / cpsrenewalFacebook / cpsrenewalLinkedIn / Nick Charneytwitter / nickcharneygovloop / nickcharneyGoogle+ / nickcharney

A few weeks ago I pitched a experimenting with open policy making at Policy Ignite here in Ottawa with my friend and colleague Todd Julie. The rationale for the presentation was pretty straight forward:
The policy monopoly of elites no longer exists. Opening up of policy process creates competition between options and questionable interpretation of the facts and contexts upon which they are based. In the future, data analytics, social media and citizen consultation should allow the triangulation of different perspectives in a way that respects factual analysis and arrives at common solutions. However in the interim, as policy tools, players and perspectives continue to change, governments must choose between competing policy priorities. Foreseeing the wider implications of these choices is critical to addressing today's "wicked" problems. However, very little is known about the different policy avenues available and the type and quality of policy advice that can be reasonably expected if it is pursued.  

In the presentation we argued that, if you accept the rationale than you are also likely to accept experimentation with policy development to facilitate learning; leading us to propose the following 6 step policy making experiment:

Step 1: Pick a wicked problem

Step 2: Frame the policy question

Step 3: Set a time limit

Step 4: A/B test different approaches:

  • Traditional, internal, institutionally led
  • Open, external, crowd driven
  • Contestable, outsourced, single private firm 
Step 5: Evaluate the results vis-à-vis common characteristics of any good policy:
  • It serves the public interest.
  • It follows appropriate laws and is enforceable.
  • It aligns with the organization’s mandate and direction and accountabilities are clearly defined.
  • It is evidence-based; assumptions, options, risks, and intended outcomes are clearly articulated.
  • Stakeholders were included in the development process and ideas have been tested prior to implementation.
  • It is historically informed and addresses both long-term interests and short-term concerns.
  • It is cost effective and there is capacity to evaluate outcomes.
Step 6: Socialize the findings to spread lessons learned and inculcate a wider culture of policy experimentation

Essentially, if you had to boil the presentation down to a TL;DR it would be:
We don't know what we don't know when it comes to the different policy-making approaches that are available to us; and if we want to know, then we ought to experiment.

Caveats

There's a couple of related caveats worth mentioning re: the need for greater experimentation:

  • We also don't have a good sense of levels of effort related to different policy development approaches and evaluating/implementing their outcomes. This will require some practical study.
  • We might need to rethink our approach to online engagement as a policy input (See: Thinking, Fast and Slow about Online Public Engagement) because the current tool set may fall short.
  • Regardless of what the different outcomes are in a given experiment, the last mile always belongs to elected officials and their senior civil servants, the best the rest of us can hope for is a more robust evidence base to support existing and emerging tools (See: To Govern is to Choose).
Its worth mentioning that while there has been little activity on this front in Canada, the UK has some experience with it. If this is an area that interests you, I'd recommend looking at the details on the UK Contestable Policy Making Fundthe correspondong press release from its first use, and the Institute for Government's assessment of the fund.


Non-Sequitors

Thank you to the Policy Ignite team for putting on a bang up event, screening us in, and moving us up to the first slot (at the last minute, presumably to warm up the crowd, which I think we accomplished).

Thank you to Minister Catherine McKenna for stopping by the event the day after her swearing in to offer words of encouragement.

If anyone is looking to pick up a good thinker and a talented researcher, you should get in contact with my co-presenter Todd Julie. While I've had a blast thinking through many of the issues facing governing institutions in the digital era with him, he's looking to relocate to Toronto for personal reasons and I'd love that to happen for him.

Wednesday, 15 July 2015

A Government that Learns by Design


by Melissa TullioRSS / cpsrenewalFacebook / cpsrenewaltwitter / creativegov

In an earlier blog post, I talked about values and our social contract with people (see: Open Gov, Values, and the Social Contract). I touched briefly on an idea that Nesta posted on their blog about cognitive government. Nesta posed an interesting challenge in that post:
How can governments shorten their learning curve to more effectively adapt to the technological changes that surround them?
It will come as no surprise to people who have worked with me, or even if you've read a couple of my posts here on this blog, that I think we need to begin with examining what values we support and demonstrate inside government in order to tackle Nesta's challenge. An assumption embedded in their challenge is that government has the kind of learning culture to support adaptation, so I think we need to dig deeper.

Based on my seven years exploring the government culture, an assumption I've been able to test (and confirm) is that innovation and creativity are things that governments don't inherently value (they usually require you to fill out a business case template to consider such things <tongue-in-cheek>). So I have another challenge I've been thinking about related to cognitive government.
How might gov support a learning culture that allows for experimentation and creative ways of approaching the way we do our work?
My hypothesis is that a government that learns by its very design, and values creative mindsets, will be a government that adapts better to technological changes (and other shifting patterns and citizen expectations). My big idea would be to create opportunities in government, from the inside-out, to think and act like a lab.

Gov that Thinks and Acts like a Lab

Design thinking principles, from d.school's bootcamp bootleg
Because I've drunk the design thinking Kool-aid, so to speak, naturally, I believe that that's the approach we need to start applying in order to change the way we build and deliver government programs. There's a lot of lab and lab-like work going on across the globe to start transforming how government delivers services and programs to citizens.

In Nesta's 2014 report on i-teams — units that are established inside government to enable innovation in service design — successful teams have a few things in common with design thinking principles. One of the ten principles identified that i-teams possess is to "have a bias towards action and aim for rapid experimentation."

This is where you tell me "it'll never happen." Government isn't a fast-moving animal; it's the sloth1 of the services kingdom. But I'd argue that it's not about speed of overall change, or about being the first to play with the latest shiny object; it's about figuring out quickly how it might add value to existing processes.

The "figuring out quickly" part is where we can use (that is, demonstrate that we value) creativity — it's where, inside government, we should find ways to experiment, learn, and use those lessons to feed into existing programs and processes. And for that, you need people who understand the system, and are able to learn and apply some creative thinking.

People who Think and Act like a Lab

Any organization is as nimble and adaptable as the people it fosters. A learning culture supports experimentation; a culture that values creative mindsets is where innovative people are driven, and where they thrive. If you've followed me this far in the post and agree that gov should think and act more like a lab, we need to start thinking about how to attract and keep people who value innovation and creativity.

What would move us closer to adopting a culture that supports i-teams? What barriers are in the way, and how do we start getting around them or removing them completely? And if we can carve out a tiny space for them somewhere, what experiments might we run, on a small scale and in much needed parts of government (e.g., procurement? IT? Budget?), to test my hypothesis above?

This, of course, is unfair to sloths. I saw a little guy on Meet the Sloths come back from having a leg amputated, and he re-learned how to climb within a day after his bandages were removed. But the sanctuary does support a learning culture (and experimentation), so that might have been part of the reason for his success.

Wednesday, 3 June 2015

You're Experimenting Right Now


by Kent AitkenRSS / cpsrenewalFacebook / cpsrenewalLinkedIn / Kent Aitkentwitter / kentdaitkengovloop / KentAitken



You may be familiar with the trolley problem in ethics:

There is a runaway trolley barreling down the railway tracks. Ahead, on the tracks, there are five people tied up and unable to move. The trolley is headed straight for them. You are standing some distance off in the train yard, next to a lever. If you pull this lever, the trolley will switch to a different set of tracks. However, you notice that there is one person on the side track. You have two options: (1) Do nothing, and the trolley kills the five people on the main track. (2) Pull the lever, diverting the trolley onto the side track where it will kill one person. Which is the correct choice?

The latter option is better from a purely utilitarian perspective, but many people lean towards the first option, which absolves them of participation or responsibility in the outcome.

The current public administration zeitgeist would more likely point to behavioural economics to prove the power of defaults to provide a level of comfort and confidence in people’s decisions (see: How Nudges Work for Government). But the point stands: we are unduly comfortable with the status quo, and we wrongly absolve ourselves of culpability for outcomes generated through the status quo. In the trolley problem, doing nothing is a big decision*.


Duty and Responsibility

Let's say you're debating a pivot in your career. You've been working in a field for a few years, and you're considering trying something new. It could just be a different job within your organization, or it could be throwing everything out the window, including yourself, and taking a leap of faith. Everything along that spectrum comes with uncertainty, discomfort, and perhaps a degree of fear.

Or, let's say you're an politician debating a policy change: it could be a minor adjustment or something major like mandatory voting or a guaranteed basic income, both of which have been proposed in Canada lately. It's not the sort of thing you can pilot in a vacuum; you have to change the way things things work to gauge how people react. Like the career pivot, it's  uncertain, uncomfortable, and scary.

Who knows how such experiments will work out? Will they be worth the risk? It's impossible to say with 100% certainty, which is the nature of experiments.

It’s tempting to think that those changes are experiments, whereas the course we are on is not. But the status quo is not a valueless, neutral starting point. It’s an experiment. It represents a plethora of design decisions, all of which influence how people behave and make decisions. And you are — we all are — complicit. As Richard Thaler, co-author of Nudge has put it, “There’s no avoiding nudging. Like in a cafeteria: You have to arrange the food somehow. You can’t arrange it at random. That would be a chaotic cafeteria.”

You’re Experimenting Right Now

You've never gone through a career on your current track before. No one ever has, in today's particular environment. Are you in digital media, for instance? Exactly zero people ever have put in a 30-year career in that field.

Likewise for policy. Canada has never entered the 21st century before, our policies have never stood up in the economic, demographic, or technological context they're about to face.

You're experimenting right now. We all are. And we have to weigh the costs and benefits of both the changes we’re considering and the track we’re already on.

* If you find yourself finding holes and rationalizations, Michael Sandel will cure that in his amazing lecture on ethics.