Why Serial Innovators Matter

Image: Photo taken by Black Elephant Blog author.

A trip to the historic landmarks of Philadelphia reminds one why serial innovators matter.  The list of Ben Franklin’s (January 17, 1706 – April 17, 1790) accomplishments alone testifies to this: how one person could pioneer so many new initiatives in one lifetime is amazing!  (According to Wikipedia, Franklin was a polymath, a person whose areas of expertise span many domains.) But it took commitment, courage, curiosity, collegiality, and a large team composed of diverse skills, perspectives, experiences, and relationships to forge a Declaration of Independence and a new Constitution, as well as a new flag, for a new country.  These were just a few of the innovations occurring in Philadelphia in the late 18th century.  Those involved took enormous risks.

Unforgettably well-informed and  eloquent guides (who clearly cared about what they were doing) from the U.S. National Park Service remind visitors to Independence Hall that the leaders who signed those founding documents could not know ahead of time if their efforts would succeed. We tend to forget, said the guides, the risks they were taking because we have the benefit of hindsight.  But, at the time, Ben Franklin famously said, “If we don’t hang together, we will hang separately.”  It was definitely not business-as-usual.

But not all of the otherwise clearly remarkable founders of the United States were “serial” innovators, as exceptional as they each probably were as individuals.  So, what is a “serial innovator,” why do serial innovators matter, and what kinds of conditions enable them to succeed?  Would Ben Franklin have been as productive as an employee making his way up through the ranks of a modern multinational corporation or public sector institution today?  What would it be like to be the manager of Ben Franklin?

Fortunately, many of these questions are addressed directly or indirectly in the careful research presented in the book, Serial Innovators:  How Individuals Create and Deliver Breakthrough Innovations in Mature Firms by Abbie Griffin, Raymond Price, and Bruce A. Vojak (Stanford Business Books, 2012).   Griffin holds a Chair in Marketing at the University of Utah’s David Eccles School of Business; Price holds a Chair of Human Behavior in the College of Engineering at the University of Illinois at Urbana-Champaign and is the Co-Director of the Illinois Foundry for Innovation in Engineering Education; and Vojak is Associate Dean for Administration in the College of Engineering at the University of Illinois at Urbana-Champaign and Adjunct Professor of Electrical and Computer Engineering and of Industrial and Enterprise Systems Engineering.

The authors interviewed over fifty serial innovators (providing examples and case studies from these interviews) and a larger number of their coworkers, managers, and human resource managers to develop their research findings.  The organizations studied for this research were firms with substantial R&D programs, like Proctor & Gamble.

This book provides some answers and guidelines for organizations, managers, and “serial innovators” that would be useful anywhere where traditional approaches might not be sustainable or standard ways of doing things may no longer meet new requirements. According to the authors, such serial innovators exist in many organizations–although in small overall numbers–and have an impact “that greatly exceeds the frequency with which they appear.” Caterpillar, Hewlett-Packard/Agilent, Proctor&Gamble and Alberto-Culver have made hundreds of millions of dollars of profit from the products that four serial innovators named in this book have invented and commercialized.

Managing these innovators requires special skills, conclude the authors. Because “the Serial Innovator’s interpretations usually contradict how the majority view the same data, paradigm changes are more challenging and more likely to produce significant conflict in the organization,” write the authors.

According to the authors, serial innovators are “individuals who have conceived ideas that solve important problems for people and organizations, have developed those ideas into breakthrough new products and services, inventing new technologies to do so as needed, and then have guided those products and services through the corporation’s commercialization processes and into the market.”

Serial innovators are important to corporations because “they can generate millions of dollars of revenue” and their products frequently “change the lives of millions of people for the better.”  While some serial innovators, such as those in the “creative arts” (like Paul McCartney, say the authors), innovate independently or with a friend, the focus of this book is on serial innovators who work in large mature firms.

• Different types of innovation include:  innovating to support the ongoing business; moving firms into new competitive space; and creating breakthrough innovation.
• Different roles of innovators require different and complementary skill sets; inventors, champions, and implementers need to have different degrees of technical savvy, market insight, “political” savvy, and project facilitation knowledge.

One of the defining personality characteristics of serial innovators is that they are “systems thinkers,” according to the authors.  “For them, the whole is much more than just the sum of the parts.”  They tend as well to have a blended perspective that is both “business-oriented and idealistic.”  They have a high motivation to create, and “it is a strong and interacting combination of external and internal forces that motivates” them.

Serial innovators’ processes are:  1) highly-dynamic across domains; 2) nonlinear, with much more overlap, iteration, and feedback “than is found in a firm’s typical linear product development process”; and 3) “are more far-reaching” than typical processes.  These processes are the subject of close study by the authors who share the results of their research in several chapters in the book, placing it within the context of mainstream technological innovation and personnel management issues.

In the “nature” versus “nurture” debate, the authors found that “the characteristics associated with personality are those that align most closely with nature.”  The qualities of serial innovators tend to be “inherent rather than cultivated.”   These include:

Curiosity:   These type of innovators “naturally need to understand ‘why.'”  They have a broad range of interests and an ability to dive deeply into subjects when their interest is piqued, write the authors.

Intuition:   They have “an informed or expert understanding of something based on experience, deep knowledge in a domain, and a keen sense of ‘what might happen'”, write the authors. This intuition enables the innovators to develop “hunches about what ideas to pursue.”

Creativity:  Serial innovators  “generate many ideas and [can] contribute from different viewpoints or domains.”  They tend to “reframe” or redefine the problem.  Almost every Serial Innovator the authors interviewed “did some type of reframing to approach a problem from a new angle, which enabled them to create an unconventional solution.

Systems thinking:  Serial innovators “focus on ‘making sense’ of complex situations,” write the authors.  They also are invested in the “greater good,” typically trying the balance the needs of all interested parties.

These serial innovators truly value the contributions others can make and their “positive perspectives about people enable [them] to get others to join them on challenging, difficult projects,” write the authors.

The nature of breakthrough innovation “requires a long-term perspective,” write the authors.  Those who care about staying ahead in a topsy-turvy world of change will want to read this book for its well-researched tips on what it takes to manage a serial innovator in an established organization. As one might expect, it isn’t easy for anyone, not least the serial innovators themselves.

Future posts on this blog will return to these issues as this book’s findings implicitly raise important questions about other endeavors in a world of nonlinear systems.

Black Swans and Serial Innovators

Image: Wikimedia

During a recent visit to the Philadelphia Museum of Art, I had a chance to pick up the new book, Edgar Degas:  Drawings and Pastels   (Thames and London Ltd., 2014), by Christopher Lloyd, formerly surveyor of the Queen’s Pictures in the British Royal Collection and a widely published author on Western art.  In an unusually crisp and absorbing writing style complemented with high-quality illustrations, this book traces the education and professional evolution of Edgar Degas, the artist, who lived from 1834 to 1917.  It is the story of how a naturally gifted individual moved from the bounds of convention, and the conventional, to create original ways to view everyday realities. Degas (seen below) was a compulsive draughtsman born in Paris into a cultured (and multi-cultural family) with close family relations in both New Orleans and Italy.

Image: Wikipedia

The poet and critic, Paul Valery, who knew Degas in his final years, said of Degas’ approach to art that:

“The sheer labour of Drawing had become a passion and a discipline to him, the object of a mystique and an ethic all-sufficient in themselves, a supreme preoccupation which abolished all other matters, a source of endless problems in precision which released him from any other form of inquiry.  He was and wishes to be a specialist, of a kind that can rise to a sort of universality. [emphasis added].”

Degas made a conscious attempt to select as role models artists from whom he thought he could learn the most. He began his career as a painter of historical scenes then traditional in the art society of the 1860s, but it wasn’t long before his innately independent and original character took over and propelled him into uncharted domains.  Early in his career, he invested years to master the conventional approaches to drawing and painting as represented by the dogma of the Ecole des Beaux-Arts at the time.  Nonetheless, he demonstrated a lifelong loyalty to maintaining a distance and independence from the schools and art movements of his time.  Lloyd notes that Degas’ determined individuality and preference for  working within the privacy of his studio enabled him to “deploy his creative resources.”  Lloyd continues:  “It is as a result of his single-mindedness that he was able to experiment without fear of failure.”

The years of investment in first learning the technical skills as a draughtsman, focused primarily on copying old masters, were critical building-blocks to his later originality.  The approach of the Academie Royale de Peinture et de Sculpture, founded in 1648, set the standards for art education in Degas’ early years.  Its restrictive approach, according to Lloyd, “produced able draughtsmen, but did little to develop artistic imagination or encourage artists to engage with everyday life.”    The emphasis in the training was on the development of a superior drawing technique, not on originality.  “In short,” Lloyd writes,” to reproduce a work of art was considered to be more important than creating one.”

Degas from the start exhibited a flair for pushing the boundaries of convention.  Even his copies of master paintings revealed him to be actively seeking out alternative interpretations.  He himself said:  “The masters must be copied again and again, and only after having given every indication of being a good copyist can you reasonably be given leave to draw a radish from nature.”

According to Lloyd, Degas exhibited early interest in original methods, introducing the “essence” manner of painting–involving oil paint diluted with turpentine–in the 1870s.  He also had a habit, by the mid-1880s, of adding as many as five to seven strips of paper to his work mid-way through the process, something Lloyd notes that Rubens did in the 17th century “as though there was no physical limit to the boundaries of a composition.”  This was not due to an “initial misjudgment,” according to Lloyd, but because the compositions “grew in an almost organic way:”

“The physical nature of the creative process invests the whole work, therefore, with a kinetic energy of its own.”

Degas was pursuing a new type of art by the 1870s, a type which challenged the traditions of the times and sought to usurp the authorities of the Academie des Beaux-Arts.  Others with a similar aim to depict modern life in their art while experimenting with new styles and techniques included Monet, Renoir, Pissarro, Sisley, and Cezanne.  This was the period of the birth of Impressionism, and Degas participated in Impressionist exhibitions.  His work focused almost exclusively on contemporary subjects including scenes of the battle, horse racing, cafe-concerts, laundresses, and even criminals on trial, writes Lloyd.

Degas’s mastery of precise drawing combined with his capacities for acute observation and “remarkable powers of detachment” equipped him to document modern societal challenges in ways that his contemporaries, such as the novelist, Emile Zola, were doing in other art forms.  Degas’s capacity to depict raw scenes of what then were considered to be socially improper activities tended to shock the art critics of his time.

Like many artists and innovators before and after him, Degas saw new possibilities in everyday realities that others did not.  This book gives one an appreciation for the  conditions and natural endowments that enabled Degas to not only master the conventions of his time but to break with them in order to change contemporary expectations of art.

Illustration: Charcoal and gouache by Black Elephant Blog author

Such challenges and approaches seem just as relevant today, and upcoming posts will consider the how individuals in different fields are able to innovate in “mature organizations” and amid sometimes rigidly defined conventions.  In a turbulent, rapidly transforming world of paradoxically more frequent “rare” and unexpected “black swan” events, knowing more about how “serial innovators” succeed is becoming vital.  So this blog will turn to the book, Serial Innovators:  How Individuals Create and Deliver Breakthrough Innovations in Mature Firms, by Abbie Griffin, Raymond Price, and Bruce Vojak, (Stanford Business Books, 2012), for some meticulously researched findings on this subject.

Seeing What Others Don’t

Illustration: Watercolor, gouache, ink and gesso

Where we left off, in the previous post, “Little Dancer Coincidences,”  was with the notion that “discontinuous discoveries” can result in a shift in our core beliefs. This notion comes from the book, Seeing What Others Don’t:  The Remarkable Ways We Gain Insights, by Gary Klein who, as mentioned previously, is a research psychologist specialized in “adaptative decision-making.” Klein studied 120 cases, drawn from the media, books, and interviews, involving stories of how people “unexpectedly made radical shifts in their stories and beliefs about how things work.”   From these cases, Klein was able to organize his research into five different strategies for how people gain insights, including: Connections Coincidences Curiosities Contradictions, and Creative Desperation According to Klein, all of the 120 cases he examined fit one of these strategies, but most relied on more than one.

Image: New York Times

Klein begins with the strategy of connections, and before proceeding with several fascinating examples, recalls the story told earlier in the book of Martin Chalfie, a biologist at Columbia University who–by virtue of attending a seminar on a topic unrelated to his work–ends up getting the idea for a natural flashlight that would let researchers look inside living organisms to watch their biological processes in action.  At the time he attended the seminar, Chalfie was studying the nervous system of worms.  The seminar covered topics that didn’t interest Chalfie initially, according to Klein; suddenly the seminar speaker described how jellyfish can produce visible light and are capable of bioluminescence.  This led to Chalfie’s insight applicable to his own field.  His insight led to an invention “akin to the invention of the microscope,” writes Klein, because it enabled researchers to see what had previously been invisible.  For his work, Chalfie (seen in the photo to the left above) received a Nobel Prize in 2008.

Image: Wikipedia

Like Chalfie, certain people make connections between unrelated matters that their close colleagues don’t.  Klein also tells the story of how the Japanese Admiral Isoroku Yamamoto (April 4, 1884- April 18, 1943) saw the implications of the British attack on the First Squadron of the Italian Navy early in World War II–before the United States had entered the conflict–then sheltered in the Bay of Taranto.  Since the bay was only 40 feet deep, the Italians believed their fleet was safe from airborne torpedoes.  The British, however, had devised adjustments to their torpedoes, including adding wooden fins to them, so that they wouldn’t dive so deeply once they entered the water.  For Yamamoto, the successful British attack at Taranto produced the “insight that the American naval fleet “safely” anchored at Pearl Harbor might also be a sitting duck,” writes Klein.  Yamamoto refined his ideas until “they became the blueprint for the Japanese attack on Pearl Harbor on December 7, 1941” (although he himself was opposed to Japan’s decision to go to war with the U.S.); ironically, his other insight was that Japan would lose the war with the United States. Yamamoto studied in the U.S., and had two postings in Washington, D.C. as naval attache; he had insights about the U.S. that his colleagues did not. He was resented by his more militaristic colleagues for his views.

Organizations generally block the pathways of connections (and other strategies) needed for such insights to occur, according to Klein.  This is because organizations are primarily concerned with avoiding errors.  Ironically, this risk-aversion makes people inside organizations reluctant to speak up about their concerns, leading organizations to “miss early warning signals and a chance to head off problems.”  Such problems are common in many fields, including science, according to Klein. Promoting forces that can countervail risk-aversion sometimes requires designating “insight advocates,” writes Klein, even though he admits he is dubious that any organization would sustain them or “any other attempt” to strengthen the forces for insight creation.  Another method he suggests is to create an alternative reporting channel so that people can publish work that doesn’t go “through routine editing” and thus would “escape the filters.”  But, he thinks this method “may work better in theory than in practice.”

A key problem for many organizations is not related to having or noticing insights, but instead it is “about acting on them.” Organizations that are less innovative because they are stifling insights, he says, “should be less successful” than they could be.    The deleterious effect of the defect-exposing Six Sigma program on U.S. corporations is an example of how an all-out focus on eliminating errors gets in the way of innovation, says Klein.  Clearly it is not a simple matter to balance the needs for efficiency and innovation within the same organization, particularly a “mature” organization. Klein concludes that the examples he gives are, for him, a “collective celebration of our capacity for gaining insights; a corrective to the gloomy picture offered by the heuristics-and-biases-community.”  He continues: “Insights help us escape the confinements of perfection, which traps us in a compulsion to avoid errors and in a fixation on the original plan or vision.”

Klein ends up recommending “habits of mind that lead to insights” and help us spot connections and coincidences, curiosities and inconsistencies.  The more successful we perceive ourselves being because of our beliefs, “the harder it is to give them {our beliefs} up.”  The habits of mind Klein has covered in his book may “combat mental rigidity,” he writes. “They are forces for making discoveries that take us beyond our comfortable beliefs.  They disrupt our thinking.” There is a “magic” that occurs when we have an insight, Klein concludes, and it “stems from the force for noticing connections, coincidences, and curiosities; the force for detecting contradictions; and the force of creativity unleashed by desperation.” So, while there is no blueprint for insight creation in Klein’s book, the many examples he cites are compelling reminders of the crucial role that insights play in stimulating new directions in any endeavor.

It seems, then, that insights can be both the source of surprises as well as help spur readiness for surprises.  They can be the needed “black swans” to deal with inevitable “black swan events.”  A take-away from this book:  There may be no ten-step  list to creating insights but understanding how to create favorable conditions to disrupt our thinking–so as to stimulate new connections and ideas–seems like useful knowledge in a world of inevitable surprises. Ostriches with their heads in the sand may not do as well as those who see what others don’t.

Little Dancer Coincidences

Illustration: Watercolor, gouache, pencil, and gesso by Black Elephant Blog author

Earlier this week I had a chance to see the glass-walled exhibit containing the wax figurine sculpted by the artist Edgar Degas in the late 19th century out of bric-a-brac and old paintbrushes and wire laying around in his studio.  While at the National Gallery of Art in Washington D.C., I learned that this sculpture caused quite an uproar in the art world at the time.  Depicting a real-life 14 year-old aspiring ballerina of limited means, Degas captured the tensions over the haves and have-nots of his era.  (These young ballerinas were often called “opera rats” , and regularly exploited by unscrupulous individuals (whom Degas also frequently painted) who hung around the theater scene of the time.)

Breaking with conventions of the age, which did not include making sculptures out of trash and using real fabric to dress a figurine, Degas did something all successful artists do:  he forced people to change their perspectives on issues they would rather ignore or take for granted.

Image: From National Gallery of Art website

By coincidence, later this week, in a class, I was told that Degas did not use any measurement techniques to do his figure drawings.  We were told to draw something without looking at the paper on which we were drawing.  This was new to me:  but the teacher said to the class, “Your intellect gets in the way of your ability to see” if you study what you are doing.  This was fascinating; I had just read Seeing What Others Don’t: The Remarkable Ways We Gain Insight, by psychologist and developer of “naturalistic decision-making,” Dr. Gary Klein.  Too much focus, he writes, on eliminating errors prevents us from having insights.

Most places we work focus on preventing mistakes and not on fostering insights.  Klein explains, mistakes embarrass organizations, and it’s easier to measure reduction of mistakes than it is to measure increasing production of insights.  (The enthusiasm over Six Sigma’s statistical approach to eliminating errors has just about killed off any potential for insights in the organizations that rely on it, he says, for instance.)  How natural is it not to make mistakes, and what are the downsides?

According to Klein, a risk-averse environment leads to a checklist mentality.  He notes that: “A checklist mentality is contrary to a playful, exploratory, curiosity-driven mentality.”  Of course, we want people with our lives in their hands–pilots, surgeons, and others–to use a checklist if this assures they won’t forget to close the doors before take-off or that they remember to remove a surgical tool in our brain.  And organizations everywhere play it safe by tabulating how many of their employees had the required training in this or that–a form of accountability and insurance, if not guarantees that errors won’t be made.

Apparently controversies, such as the ones that swirled around Degas’s “Little Dancer,” are necessary for helping us, eventually, to reframe our perspectives.   And this reframing does not involve minor adjustments or “adding more details,” according to Klein; the changes involved are not incremental.  Instead shifts occur that change our core beliefs.  Such shifts are “discontinuous discoveries,” he writes, giving many of his own examples accrued during years of study in his quest to learn where insights come from.

These shifts transform us in several ways, changing how we “understand, act, see, feel, and desire.”  They transform our thinking, and give us a different viewpoint, thus changing how we act and even “our notions of what we can do.”

It seems possible, in an age of digital hyperconnectivity and empowered individuals, etc, that integrating improved understanding (and insights) of how to develop and convey appealing narratives already has become something separating winners and losers in the battles for attention, “hearts-and-minds” and other contests of our age.  Perhaps this always was true but is amplified by today’s unprecedentedly interdependent world.

Discontinuous Discoveries

Image: Gouache and gesso by Black Elephant Blog author

Have you ever wondered how well suited organizations today are for a world of inevitable surprises, including “black swans”, “black elephants,” and other outsized dangers and opportunities?  Most organizations are embedded in complex systems where even the tiniest decisions–such as how many times their drivers back up their trucks in a day–are monitored, measured, and evaluated.  Managers and employees are judged on how well they meet project goals established in the past, never mind what may be the new circumstances today.  While infinitely easier to manage–and, frankly, possibly the only way we know how at this point–such organizations are no longer fit-for-purpose in the world we’re living in, according to a great many people in touch with these realities.

“The world is not just rapidly changing; it’s being reshaped,” says Dov Seidman, author of the book “How” and C.E.O. of LRN, in a column today by New York Times columnist Thomas Friedman.   According to Seidman, “It’s all happened faster than we’ve reshaped ourselves and developed the necessary norms, behaviors, laws and institutions to adapt.”

What are some of the implications of such a mismatch?  As explored in the last post here on this blog, the breakthrough insights that led Alan Turing and his team at Bletchley Park to decode German wartime communications in World War II–the subject of the film, “The Imitation Game,” currently probably in a theater near you–emerged from a series of interactions that were not planned, expected, or inevitable.  (Thus this crucial breakthrough could not be “managed” by any conventional meaning of that term.)  Yet, from this series of connections, coincidences, and hunches emerged the glimmerings of a solution that turned around a war, shortening it possibly by a couple of years, and saving an estimated 14 million lives.   In a thought experiment, consider whether the Bletchley team would have succeeded if its laboratory was housed inside the institution to which it reported.  (It turns out that Bletchley Park was chosen primarily because of its proximity to a rail line connecting it to Oxford and Cambridge from whose universities the needed codebreakers were expected to come.) In today’s world, 75 years later,  are we leaving prospects for our well-being (whether on local or global scales) to chance by relying on outmoded ways of thinking about organizing knowledge?

Fortunately, in pockets here and there, it seems that we have learned, and continue to learn, much more about how innovation and insight creation work, and how their results can be harvested and applied to real-world problems.  Over at the University of Illinois, for instance, under the leadership of Dr. Bruce Vojak, co-author with Abbie Griffin and Raymond Price of Serial Innovators: How Individuals Create and Deliver Breakthrough Innovations in Mature Firms (2012), an online discussion group called the “epistemology of innovation” has been underway for some time.  In a series of essays, Vojak has made the case for the “non-linear” nature both of breakthrough innovation and innovators, as in this essay (cited with permission).

Image: From “On the Epistemology of Innovation: How Breakthrough Innovators Connect the Dots,” Number 23, February 1, 2014, accessed at http://www.ideals.illinois.edu/handle/2142/27667 Copyright Bruce A. Vojak, 2014

Vojak’s essay distinguishes between non-linear and linear systems and includes this helpful chart for ready contrast.  He notes that breakthrough innovation is a “messy, complex process that does not follow nearly defined paths.” He continues:

“While a finite set of certain activities must be conducted as the innovation process unfolds (such as identifying the best problem to address, understanding the problem deeply, and synthesizing what is known into an innovative product concept), these activities typically are attended to repeatedly, in only a general order initially and with little or no predictability thereafter.”  Vojak concludes that the underlying nature of “innovative discovery” can be described mathematically by using chaos theory.  “Non-linear systems abound in nature,” he writes, “and also play a key role in engineered systems, such as the conversion of an audible signal to a much higher frequency, enabling its transmission in a communication system.”  He notes that breakthrough innovation, like a non-linear system (as he describes in the table above right), exhibits “identical extreme sensitivity to initial conditions, as well as the other characteristics of non-linear systems.”

Gary Klein, a cognitive psychologist, dives into similar questions in his book, Seeing What Others Don’t:  The Remarkable Ways We Gain Insights (2013).  One of the insights to come out of his work is that many organizations typically emphasize critical thinking at the expense of generating insights.  Yet, effectiveness (and performance improvement) depend on both.  “Having insights is a different matter from preventing mistakes,” he writes, and yet many people feel that their organizations stifle their attempts to do a good job (by having insights!).  In his book, Klein relates the results of his research into how insights get triggered, what interferes with insights, and how organizations can foster insights.

It turns out, Klein concludes, that:

“Insight is the opposite of predictable. Insights are disruptive. They come without warning, take forms that are unexpected, and open up unimagined opportunity. Executives may believe that they want insights and innovations but are most receptive to new ideas that fit within existing practices and maintain predictability.”

There is a “magic of insights, ” according to Klein.  This magic “stems from the force for noticing connections, coincidences, and curiosities; the force for detecting contradictions; and the force of creativity unleashed by desperation.”  It’s a tall order for organizations accustomed to delivering consistency and predictability.  But what are some implications of applying linear expectations to a non-linear world?

Future posts will take a closer look at what Vojak and his associates call the “epistemology of innovation” and the conditions  and habits of mind which Klein has discovered are forces for disrupting our thinking in ways which–in a disruptive world–are becoming more important in every endeavor.  As ever, links to relative material are welcome and will be included on this blog.

Black Elephants and the Magic of Insights

Illustration: Watercolor, gouache, ink, gesso and coffee grounds by Black Elephant Blog author

If you’ve had a chance to see the new film, “The Imitation Game”, about the brilliant but sadly socially outcast British mathematician Alan Turing, you’ve probably been powerfully reminded–through its artistic rendering of a true story–of the critical roles which serendipity, hunches, and chance encounters have played in devising solutions to the most challenging problems of any age.  (Spoiler alert:  If you haven’t seen the movie, and wish to be surprised when you do see  it, perhaps it is best not to read further.)

In the film, Turing and his teammates–a collection of unusually gifted mathematicians, including one woman– at Bletchley Park in England literally were racing against the clock to figure out how to decode German wartime communications during World War II.  Their efforts centered on the invention by Turing of a decoding machine (basically a prototype computer) but, despite hours of hard work and all their smarts, the team was about to be shut down by uncomprehending bosses under pressure to deliver results.  (The film has received mixed reviews–such as this one–due to its mix of imagined and actual events, and its alleged failure to convey that the Turing effort was part of a much larger effort underway at Bletchley.)

Image: Allstar/The Economist

Without giving away the storyline (the general outline of which is, however, a matter of historical record), it is in a moment of relaxation away from their secret laboratory,  bantering with friends who were supporting the war effort themselves but not privy to any of the Turing team’s information, that a chain of interactions leads to a breakthrough insight.  In the film, a casual comment by someone who is not on the Turing team has an instantaneous effect.  Her hunch becomes Turing’s insight and he and the rest of the team, up to then stymied in their task, had to act immediately.

This insight turns out be the what the team needed to successfully break the Enigma code.  Their success is credited by historians with turning around Britain’s fortunes in the war.  They also estimate that the code-breakers helped shorten the war by two years and saved approximately 14 million lives.

This film subtly highlights  some of the necessary ingredients of breakthrough thinking:  talent, expertise, hard work, team work,  intensity, diversity, false starts, time pressures, clear purpose, and random encounters with ideas from disparate sources outside the immediate field of inquiry.  While perhaps failing to give sufficient credit to Turing’s bosses (per some of the critics), the film also hints at why so many traditional organizations are so poor at facilitating this sort of thinking.  Whatever the gap between the historical reality and the movie, it is worth pondering:  What are some of the implications of a mismatch between the outsized global issues of our time and the incapacity of most organizations to nurture the modern equivalents of Bletchley Parks?  How can talent and good judgment be assembled most effectively to deal with the important, as well as urgent, “Black Elephants” of our times?

Most of us by now have heard of the Black Swan concept but the Black Elephant concept is not well known.  For this writer, it came into being when encountered in an op-ed by New York Times columnist, Thomas Friedman, in late 2014.  As he explains, a “black elephant” is a “cross between a ‘black swan’ (an unlikely, unexpected event with enormous ramifications) and the ‘elephant in the room’ (a problem that is visible to everyone, yet no one still wants to address it) even though we know that one day it will have vast, black-swan-like consequences.”

At a time of mounting challenges (including but extending well beyond the environmental issues cited in the Friedman piece) that are too big to fit into anyone’s inbox, or even anyone’s organization–where speed, as in the case of Bletchley Park, is of essence and stakes are high–the concept of black elephants seems a timely one.

The focus here on the roots of surprise inquires into how insights and breakthroughs come about.  The current age is no different from past ones, such as the example illustrated in The Imitation Game, in needing to aggregate, cull, and distill insights that can be acted upon in a timely way.  With more challenges filled with potential for highly improbable (but, therefore, according to Dr. Hand’s “laws of improbability,” practically inevitable) outcomes, however, the need for insights may be multiplied in present circumstances.

With high stakes involved in multiple arenas, this blog’s inquiry into the roots of surprise will next explore the findings of experimental psychologist and expert in “adaptive decision-making,” Dr. Gary Klein, in his fairly new book, Seeing What Others Don’t:  The Remarkable Ways We Gain Insights (2013).  Klein notes that generally we know very little about how insights are formed or what blocks them.  He too thinks it’s important to know more about where insights come from, so his book is meant to fill some of our knowledge gaps about the magic of insights.  In an upcoming post, I’ll feature some highlights from this book, and link to related material as I come across it.

Monkeys Typing Shakespeare

Illustration: Watercolor, gouache, and ink by Black Elephant Blog author.

Around the world overnight, we rang in the New Year with expressions of wishes for good fortune for all in 2015!  While we are considering here the roots of surprise (even, for fun, a “zoology” of surprise), the start of a New Year is an auspicious (another good luck phrase) occasion on which to consider the chances (there we go again) of things going extraordinarily well or badly.  For this, I’ve been turning to a new book by an eminent British mathematician and statistician who, it seems to me, has done the reading public a great service by translating his insights into language we non-mathematicians can (usually) understand!  (I am composing a blog post to record what I am learning, and not to review the book.  I also have received emails from friends who are looking forward to learning more about this book and reading it themselves.)  Future posts will come back to the ideas presented in this book.

Extraordinarily improbable events occur every day, according to Dr. David Hand, author of The Improbability Principle: Why Coincidences, Miracles, and Rare Events Happen Every Day (2014). (You can watch Dr. Hand explaining the Improbability Principle at a 2014 meeting of the Royal Statistical Society in London at this YouTube video here.)  It is clear from the video that the statisticians see the present moment in history with the deluge of data (including “big” data) as opportune for members of their profession to intersect with the formulation of public policy.  And based on my reading of Hand’s new book so far, I’d have to agree!

Hand traces the history of the study of probability, noting early on in the book the work of Emile Borel, an eminent French mathematician (1871-1956), who held that “Events with a sufficiently small probability never occur.”  Borel cited, according to Hand, the classic example of monkeys who, randomly hitting the keys of a typewriter, happen by chance to produce the complete works of Shakespeare.  Borel explained:

“Such is the sort of event which, though its impossibility may not be rationally demonstrable, is, however, so unlikely that no sensible person will hesitate to declare it actually impossible.  If someone affirmed having observed such an event we would be sure that he is deceiving us or has himself been the victim of a fraud.”

As Hand observes, on first glance it seems like “Borel’s law” contradicts “the improbability principle” which is the subject of his book. The Improbability Principle asserts that “extremely improbable events are commonplace.”   But Borel is referring to “very small probabilities” on human scales, says Hand, and indeed Borel clarifies his “single law of chance” by noting that “at least, we must act, in all circumstances as if they were impossible.”

By contrast, the Improbability Principle explains why highly unlikely events keep on happening.  Hand says, “that is, not only are they not impossible, but we see such events again and again.”  Can both these assertions be right, he asks?  Hand maintains that we can resolve this apparent contradiction by considering different strands of the improbability principle, including the “law of truly large numbers,” the “law of near enough,” the “law of selections”, and others.

When one goes through this process and understands the strands, Hand writes, the principle “tells us that the universe is in fact constructed so that these coincidences are unavoidable:  the extraordinarily unlikely must happen; events of vanishingly small probability will occur.”

Why read such a book at all?  I’ll hazard (another term from the world of risk!) a guess:

In the view of this blogger, in our interdependent, highly-interconnected world, where dense networks create a world of connections that change the meaning of “human scale” relative at least to what Borel understood it to be nearly a century ago, understanding how “rare” events, coincidences, and extraordinarily unlikely events occur has become vital for human (and other forms of life’s) security.   But, grasping that uncertainty is inherent in reality and that, indeed, even in the present, we can only have an approximate understanding of reality, does not come naturally, for some reason.

Spoiler Alert:  The rest of the post will discuss the different types of probability as Hand presents them in his book.  Future posts will delve into other aspects presented in this work. ###

And there is a reason for this mismatch of expectations, explains Hand.  Rarely addressed in our usual day-to-day settings but deftly discussed in this book is the gradual move in the last century or so –in science at least–beyond reliance upon  “deterministic” principles long said to explain the behavior of natural systems.  These principles, it has been assumed until recently, adequately explained the underlying causes and effects of events and outcomes, at least since the natural laws of physics began to be investigated in the 17th century.  The early proponents of the concept of scientifically testing ideas were onto something revolutionary for the times!  But they were limited in their understanding by what the tools and techniques of the day enabled them to observe:  this influenced the types of questions they asked, of course, and led to overconfidence about mankind’s abilities to master nature’s mysteries within the bounds of existing knowledge.

The “Baconian Revolution” first introduced the idea of the scientific method, writes Hand.  This method held that the way to understand the natural world is to collect data, conduct experiments, take observations, and use these as test beds through which to evaluate proposed explanations for what’s going on.  Before that, stories and superstitions held sway.  “But explanations that have not been or cannot be tested have no real force…,”  according to Hand.  “They serve the purpose of reassuring or placating those who are unwilling or unable to make the effort to dig deeper but they don’t lead to understanding.”

The first scientists (“natural philosophers” as they were called then) sought to devise laws that describe how nature works.  Hand notes that these laws are “shorthand summaries” encapsulating “what observations shows about how the universe behaves.”  They are “abstractions,” he notes.  An example is Newton’s Second Law of Motion, which holds that the “acceleration of a body is proportion to the force acting on it.”  The power of such laws is behind humanity’s progress in science and technology, Hand observes.

For a long time and even as recently as the 1930s, scientists and philosophers such as Karl Popper, held that the “rule that extreme improbabilities have to be neglected…agrees with the demand for scientific inquiry.”  Those tiny chances of extraordinarily rare events had to be swept under the rug to allow progress, or so it was (and still is) thought.  In addition, the idea of things happening for which we have no explanation is an intensely uncomfortable one, Hand writes, as humans have an innate need to know why things happen and “to establish the causal connections, and to understand the rules that lie behind what we observe.”  This is a basic human need related to safety and security:  if there “are no causes…illnesses, accidents, and failures couldn’t be avoided.  We’d live in a constant state of fear, waiting the unpredictable disaster just around the corner.”

Over the centuries, it was impossible to miss the inexplicable coincidences and other extraordinarily unlikely events, creating fertile conditions for prophets and fortune tellers, writes Hand–people who tap into the notion that there is some “mysterious force or being behind what happens, often acting with malicious intent.”  These notions have led to different explanations for otherwise unexplained events, including superstitions, prophecies, gods, miracles, and parapsychological explanations, he writes.

Yet, the notion that there is any real causal relationship between, for instance, sighting black cats and falling down, stems from misperceiving patterns.  Hand explains that the goal of science is to distinguish between those patterns that do represent a “real underlying cause-and-effect relationship” and those that don’t.  “Patterns we spot but that are mere accidents, without any underlying cause, have often formed the basis of superstitions.  (Animals also demonstrate this development of “superstitions,” he notes.)  But:

“Even if one event follows another surprisingly often, it doesn’t necessarily mean that the first causes the second.  Statisticians have a sound-bite for this:  correlation does not imply causation…Although the aim of a prophecy is to remove uncertainty about the future, uncertainty in the form of randomness is frequently the mechanism used to generate prophecies.”

The deterministic laws that evolved from the 17th to the 20th centuries were “mathematical equations…that told us how natural objects would behave,” writes Hand.  “There was nothing in the universe that was uncertain or unpredictable, at least in principle, according to science.”  And the immense technological progress of mankind “built on those ideas showed that they were largely correct.”   Thus came into being the ubiquitous view of nature as “the clockwork universe”–a universe ticking along a well-defined path, Hand writes. Ignorance could be eradicated by science.

Later in the 20th century, however, science began to expose gaps that it could not explain.  A huge shift in perception began slowly to take hold at least on the margins of science:

“It seemed as if the universe was not deterministic after all, but that randomness and chance lay at its very foundations.”  Randomness and chance are entirely probable in this universe, Hand explains, and can be understood through the improbability principle which is formed upon the basic laws of probability.

Types of Probability There are different kinds, and definitions, of probability, according to Hand.  Informal definitions even reveal the multi-facetness of probability: both,  “the extent to which an event is likely to happen” and “the strength of belief than an event is likely to happen.”  And Hand tells us that both can be represented by the same mathematics:  probabilities are numbers lying in the range from 0 to 1 with 0 meaning impossible and 1 meaning certain.  There are many other definitions, but none captures “probability” in its entirety.  This is not really a problem, says, Hand, because it is very natural to need “multiple views of an object to understand it properly.”

The three most widely used interpretations of probability are the frequentist, subjective and classical interpretations:

The frequentist interpretation of probability is based on the tendency of physical systems to produce roughly constant relative frequencies when situations are repeated.  For example:  the tendency for a coin to come up heads about half the time it’s tossed, or the 4 (or any other) face to show on a die about one-sixth of the time.  As we learn from reading Hand, there is a lot to think about in the word “roughly” above!  Complete accuracy is impossible event when measuring, as frequentist probability does, properties of the “external world.”

Subjective probability is very different. Instead of representing an aspect of the external world, subjective probability is the confidence an individual has that an event will occur, explains Hand.  This relates to your beliefs, whether about the probability of a coin turning up heads in a coin flip or your beliefs about the person tossing the coin (who might have rigged the process).

“Instead of being a property of the external world, the subjective view has it that probability is an internal property of your mind. Each person will have their own subjective probability for each event.”  Hand notes that another eminent mathematician therefore claimed that probability did not exist because it is a “property of how we think about the world.” Nonetheless, Hand notes that various methods exist for measuring subjective probability, including asking people to bet on an outcome–knowing that the results will depend on what they think.

Only recently have we humans come to understand the significance of these fundamental different notions of probability, Hand says, with steps newly taken to distinguish between  epistemological probability and “aleatory” probability. (For more on this, it will be necessary to read the book!)

The classical interpretation of probability is based on options of symmetry, Hand writes, giving the example of how natural it is to think of probability as distributed equally across the six faces of a die. “This interpretation is very convenient for games of chance, based on symmetrical randomization tools such as dice and coins,” he writes.  But life is not like a die, he says:  “it’s less clear how we might apply classical probability to situations in normal life which lack such obvious symmetries.”

There are other interpretations of probability, which Hand goes on to introduce.  What’s key, and still to be addressed in this blog  in future posts on the roots of surprise, is to understand how probability is calculated in the case of interdependent events, such as make up the natural and manmade systems of our world.  What is the significance of these relatively recent discoveries of the inevitability of improbability?