Uncategorized

Superforecasters and Dragonfly Eyes: Booknotes

Despite my best intentions to get through an ever-growing stack of books, a brand new one crept into the mix and demanded my immediate attention, so here goes, with a few notes on it:

Illustration: Watercolor and Platinum Carbon black pen and ink sketch by Black Elephant Blog author

Illustration: Watercolor and Platinum Carbon black pen and ink sketch by Black Elephant Blog author

Superforecasting:  The Art and Science of Prediction, by Philip E. Tetlock and Dan Gardner, (Crown Publishers: New York, 2015).

In this book, the authors, Tetlock, a professor of psychology, political science and business and Gardner, a journalist and author, note that “we are all forecasters,” in the sense that we need to make decisions that involve uncertainty (as when we buy a home or make an investment or decide to relocate, etc.).

When it comes to really big events, like market crashes, wars, etc., however, we expect to turn to “experts.” Unfortunately,  according to the authors’  research results, the experts we might most expect to be able to “forecast” events with precision are less able to do so (against certain types of problems) than “ordinary” well-informed people who are not experts in the subject matter.

These “ordinary” people have some extraordinary characteristics, the authors realized when they analyzed their research results.  These include an ability to step outside of themselves and get a different view of reality, something the authors note is really hard to do.  But the ordinary people who did the best in the forecasting tournaments run by the authors, exhibited a remarkable ability to do just this:

“Whether by virtue of temperament or habit or conscious effort, they [the successful forecasters] tend to engage in the hard work of consulting other perspectives.”

In conducting U.S. government-backed research, the authors found that people such as a retired computer programmer with no special expertise in international affairs  could successfully answer very specific questions such as “Will the London Gold Market Fixing price of gold (USD per ounce) exceed $1850 on 30 September 2011?” People they worked with, such as this individual,  were enabled by the rules of the research project to update their forecasts in real time, incorporating new information in their estimates as they came across it.  (The process is explained in detail in the book.)  Over time, “superforecasters,” such as this retired computer programmer stood out among the pack.  Such people, write the authors:

“…have somehow managed to set the performance bar so high that even the professionals have struggled to get over it…”

The results made the authors inquire into the reasons for the “superforecasters'” better performance.  They write that “It’s hard not to suspect that [so-and-so’s] remarkable mind explains his remarkable results.”

Indeed, some of their superforecasters have multiple degrees in various subjects from various top-notch universities, speak several languages, and lived or worked abroad, and are voracious readers.  But, assuming that knowledge and intelligence drive strong forecasting performance would send us down the wrong path, concluded the researchers.  To be a superforecaster “does not require a Harvard PhD and the ability to speak five languages,” they concluded.  Many very well-educated and intelligent participants in their study “fell far short of super forecaster accuracy.”  They continue:  “And history is replete with brilliant people who “made forecasts that proved considerably less than prescient [citing Robert McNamara — defense secretary under Presidents Kennedy and Johnson as one example].”  So, the authors conclude:

“Ultimately, it’s not the [data/brain etc] crunching power that counts. It’s how you use it.”

Well, duh, you might say.  Isn’t this obvious?  Apparently not.

Dragonfly Forecasting So how do these superforecasters do it?  What do they have in common?  The authors survey a number of case studies from their research to provide some insights.  What they discovered is a capability they call “dragonfly forecasting.”  The researchers observed that the super forecasters, while “ordinary” people, have an ability to synthesize a large number of perspectives and to cope with a lot of “dissonant information.”  They have more than two hands, write the authors, because they are not limiting themselves to “on the one hand or the other hand thinking.” (Sidebar:  I just attended a seminar on energy and climate challenges where one of the speakers, an engaging, colorful and normally compelling orator, clearly), made the comment that “on one hand you have total environmental disaster or, on the other hand, total commercial disaster,” concluding that “we need to get on the right side of this.”

Illustration: Seminar sketch by Black Elephant Blog author

Illustration: Seminar sketch using Black Sharpie pen on Stone Journal notepaper by Black Elephant Blog author

This sort of binary thinking can be quite limiting, particularly when there is no “right side” as is the case, more often than not, when facing a world of increasingly complex challenges.  I heard more examples of this “either-or” thinking problem again just yesterday in an all-day conference, with people literally saying that they don’t see an option beyond the frame they’re in.)

“I’ve Looked At Things From Both Sides Now” 

By contrast, the dragonfly eye in operation, according to the authors, is “mentally demanding.”  (Already,in this mere statement, we run up against some cultural and cognitive realities in many large organizations where everyday urgent matters and matters only perceived as urgent (possibly because of this very binary winners vs. losers thinking) take up almost all available bandwidth.)

Superforecasters “often think thrice–and sometimes they are just warming up to do a deeper-dive analysis.”  Forecasting is their hobby, write the authors.  They do it for fun and also because they score high in “need-for-cognition” tests.  These tests rate people who have a tendency to “engage in and enjoy hard mental slogs.”

There also is an element of personality likely involved, they conclude.  The traits involve “openness to experience” which includes “preference for variety and intellectual curiosity.”

The authors conclude, however, that this dragonfly eye capability, which involves synthesizing a growing number of perspectives, has “less to do with the traits someone possesses and more to do with behavior.”  These behaviors include “an appetite for questioning basic, emotionally charged beliefs.”  Interestingly, the researchers have concluded that, without this behavior, individuals (forecasters or not) “will often be at a disadvantage relative to a less intelligent person who has a greater capacity for self-critical thinking.  [emphasis added]”

Those with a dragonfly eye cultivate their ability to encounter different perspectives.  They are “actively open-minded,” write the authors.  There is an actual psychological concept around this cognitive behavior.  For superforecasters, therefore, “beliefs are hypotheses to be tested, not treasured to be guarded,” conclude the authors.

There are too many implications of this work–important implications–to cover in a blogpost.  But it must be said that the book raises implicitly at least as many questions about how to proceed in a complex interconnected world as it attempts to answer.  For instance, fewer enduring problems of real consequence can be addressed with a simple forecast, no matter how accurate, in a bounded time-wise constraint.  Inherently complex “super wicked problems” discussed earlier on this blog do not lend themselves to this sort of forecasting.  Tougher choices involve immersing ourselves in deeper questions of values and longer-term perspectives.

Nonetheless, what the authors have demonstrated with their research offers us the opportunity to pursue these challenges with greater awareness of individuals’ different cognitive and philosophical outlooks, and perhaps–from a corporate human resources point of view–to allocate jobs and tasks to people based on comparative evaluations of their cognitive and behavioral strengths.

As more and more issues require deeper thinking and appreciation of systemic interconnections, it may become ever more important (even if not acknowledged in organizational priorities) to find ways to incorporate “dragonfly eye” sense-making behaviors.   The authors have observed that “belief perseverance” can make people “astonishingly intransigent–and capable of rationalizing like crazy to avoid acknowledging new information that upsets their settled beliefs.”  When people have a greater investment in their beliefs, it is harder for them to change their views.

There is important stuff in this book which requires a great deal more reflection. So, this thread of inquiry will continue in the next post’s look at another new book called Nonsense: The Power of Not Knowing, by Jamie Holmes (Crown Publishers, New York, 2015).    Not at all “nonsense,” thinking about thinking matters.  Even if these books fail to provide us with concrete next steps, the relevance of these works to current challenges facing decisionmakers, and their advisors, in all sectors cannot be overstated.

Standard
Uncategorized

The Sketcher’s Dilemma

Sketcher's Dilemma

Illustration: Micron and Faber Castell Pitt artist pens and watercolors by Black Elephant Blog author

We’ve heard of the “innovator’s dilemma,” of course, and here comes the “sketcher’s dilemma” based on a true story (as reported to this writer) from someone on location in American suburbia.

Standard
Uncategorized

Oyster Heartbeats at the World Bank

Yesterday an international audience of development, media, business, NGO, and technology experts attending a session of the 2015 Meetings of the World Bank Group and International Monetary Fund listened to an oyster’s heart beating.  It was a surprising session in many ways, and relevant to this blog’s focus.

Illustration:  Pen and ink by Black Elephant Blog author

Illustration: Pen and ink by Black Elephant Blog author

The session was “Big Data for a More Resilient World,” (video coverage here) where one of the keynote speakers, Ros Harvey, Chief Strategy Advisor of the Knowledge Economy Institute, described how the “Internet of Things” enables oyster farmers in Tasmania to integrate data about the heartbeats and other biorhythms of oysters to weather and water temperature data. Harvey’s presentation, as well as those of other speakers–from Intel, Google, the World Economic Forum, Caribou Digital as well as the author of Resilience-Why Things Bounce Back, Andrew Zolli–all emphasized the need for those grappling with so-called “big data” to find ways to put that data to use by the people generating it.

One take-away, definitely, was a sense of the embryonic nature of this topic for people, even specialists (if there are any yet) in “big data,” in all industries. A strong dose of humility about mankind’s collective readiness for this world–surprisingly (and certainly refreshingly)–was evident throughout the session, in this attendee’s view. Presentations and comments toggled frequently between the opportunities and dangers involved.  All speakers emphasized the need to move beyond “data” to focusing on “information,” complex systems and business models, thus tying the subject to the topic of resilience, a topic explored earlier on this blog.  Hence, the post about this event, with regard to some of the most thought-provoking insights from the event’s speakers.

zolli 2

Image credit: Photo of Andrew Zolli’s presentation given at the World Bank on Big Data and The Future of Resilience, 15 April 2015

The organizers noted that Big Data is one of the most important topics in the development world today but more than half of development experts think their sector is not prepared to use these possibilities.  Zolli presented a slide showing the “grand synthesis” of the elements of Big Data (a photo of which taken at the event is provided here).

Zolli, the author who has recently joined Planet Labs (where, he noted, one of the first employees of the companies was an “artist-in-residence”), noted that “we live in this world of super wicked problems, entangled, complicated..”  How people can persist, recover, and thrive in this world is “our collective resilience challenge.”  The challenge is that change is happening all around us and we can’t see it, he said.  Zolli demonstrated our inability to see this change with a short visual game he played with the audience.

In a world “dominated by the giant hairball” (of complexity), data has a role to play in helping us to be resilient. But what’s important, he says, is that we want to move from the “realm of big data” to the “realm of big indicators.” The Elements of Resilience–each of which is interconnected with the others–presented by Zolli include:

Building Regenerative Capacity

Sensing Emerging Risks

Responding to Disruption

Learning and Transformation

To move to the “realm of big indicators,” Zolli emphasized that “entirely new social architectures” need to be built and that we “need to reengineer relationships as much as we have to reengineer our institutions.” Some of the examples he cited of this at work, in addition to Planet Labs, were other geospatial providers, such as Digital Global and Zooniverse, which he described as “sophisticated crowdsourcing platforms”.  These are using citizen science for disaster response, such as Planet Labs’ work before and after the Tacloban disaster in the Philippines.

Wicked problems, said Zolli, “resist mere cleverness.”  Big Data is not simply “push-the-button.” It is necessary to drag unconventional players into the effort, into “adhocracies.”  Only in this way, might wicked problems “yield to mass cleverness.”

Illustration:  Pencil and ink by Black Elephant Blog author

Illustration: Pencil and ink by Black Elephant Blog author

Harvey was next up, with a short video of the oyster farmers at work in Tasmania. She said:  that big data is not only about connecting people, processes, and things but also animals.   Being able to measure the oysters’ heart rates and integrate that data with weather and water conditions, for instance, epitomized the potential of the “Internet of Things,” she said. The challenge is “how do we ‘architect’ technology so the benefits accrue to the many?”  How can we create public good with private effort?  According to Harvey, “It is the new business models that will drive the disruption from the technology.”   Working together from a common data source creates new value, but there is a need to design systems based on 3 principles, known in shorthand as “SOS:”

Sustainable

Open Innovation

Scalable

“We need to understand that much of the world’s data is in the private sector” but “open innovation” requires that many people work on the problems.  Also the efforts must be sustainable.  They should not depend on funding that “comes and goes.” The Knowledge Economy Institute where Harvey works  focuses on such ways to solve complex problems through collaboration and innovation.

Nigel Snoad from Google, was next and said his work is focused on how to make critical information more accessible in times of crisis.  Snoad noted that the unexpected happens when you give people an “open tool” and “open APIs” are “where we’re going.” He cited Google Flu Trends as an example, but emphasized the Big Data is not a silver bullet.  This is because Big Data comes from “very complex systems.”  It is therefore necessary to “understand the systems” behind Big Data.

The moderated panel closing the session featured the founder of Caribou Digital, Chris Locke; Associate Director of the World Economic Forum’s Telecommunications Industry William Hoffman; and Senior Principal Engineer of the Strategy Group at Intel, Tony Salvador.  Key highlights among their comments included:

  • The poor don’t need more surveillance;
  • We need to get data on things that the people on the ground actually care about.
  • It’s not a technology problem; it’s a business model problem.
  • These technologies have ‘interpretive flexibility’ that can be used to concentrate power.
  • We are just beginning to understand this.  Now we understand that it’s about complex systems.
  • We need to talk about “information” and understand that it is a social construct.
  • There is a growing recognition of public-private partnerships.
  • The potential is there but it’s about Governance. This is in the next set of ‘grand challenges’.
  • We need to examine some of the fundamentals that underline the systems we have today.
  • What is the context of production [of Big Data] and what is the context of analysis once we have that data?
  • Sometimes we are looking through the wrong end of the telescope.
  • How can we unlock that data and make it valuable to the people it’s coming from?
  • We need to listen. We don’t want to go down the path of reinventing our own assumptions.
  • Big data is potentially big power…These [business] models need to be liberated from, and [sustained?] above, the level of individual institutions.
  • We will see models emerge that will be surprising.
  • We need to follow up with social encouragement and deep engagements.
  • APIs that speak up out of the platforms
  • Data can have value and merit right where it’s being collected
  • Due process is an important topic. Things will go wrong. What will the most vulnerable do when things go wrong?

All in all, a rich hour-and-a-half session at the World Bank Headquarters yesterday, and a contribution to the state of understanding on this topic which affects everyone, even oysters.

Standard