Illustration: Watercolor, gouache, and ink by Black Elephant Blog author.
Around the world overnight, we rang in the New Year with expressions of wishes for good fortune for all in 2015! While we are considering here the roots of surprise (even, for fun, a “zoology” of surprise), the start of a New Year is an auspicious (another good luck phrase) occasion on which to consider the chances (there we go again) of things going extraordinarily well or badly. For this, I’ve been turning to a new book by an eminent British mathematician and statistician who, it seems to me, has done the reading public a great service by translating his insights into language we non-mathematicians can (usually) understand! (I am composing a blog post to record what I am learning, and not to review the book. I also have received emails from friends who are looking forward to learning more about this book and reading it themselves.) Future posts will come back to the ideas presented in this book.
Extraordinarily improbable events occur every day, according to Dr. David Hand, author of The Improbability Principle: Why Coincidences, Miracles, and Rare Events Happen Every Day (2014). (You can watch Dr. Hand explaining the Improbability Principle at a 2014 meeting of the Royal Statistical Society in London at this YouTube video here.) It is clear from the video that the statisticians see the present moment in history with the deluge of data (including “big” data) as opportune for members of their profession to intersect with the formulation of public policy. And based on my reading of Hand’s new book so far, I’d have to agree!
Hand traces the history of the study of probability, noting early on in the book the work of Emile Borel, an eminent French mathematician (1871-1956), who held that “Events with a sufficiently small probability never occur.” Borel cited, according to Hand, the classic example of monkeys who, randomly hitting the keys of a typewriter, happen by chance to produce the complete works of Shakespeare. Borel explained:
“Such is the sort of event which, though its impossibility may not be rationally demonstrable, is, however, so unlikely that no sensible person will hesitate to declare it actually impossible. If someone affirmed having observed such an event we would be sure that he is deceiving us or has himself been the victim of a fraud.”
As Hand observes, on first glance it seems like “Borel’s law” contradicts “the improbability principle” which is the subject of his book. The Improbability Principle asserts that “extremely improbable events are commonplace.” But Borel is referring to “very small probabilities” on human scales, says Hand, and indeed Borel clarifies his “single law of chance” by noting that “at least, we must act, in all circumstances as if they were impossible.”
By contrast, the Improbability Principle explains why highly unlikely events keep on happening. Hand says, “that is, not only are they not impossible, but we see such events again and again.” Can both these assertions be right, he asks? Hand maintains that we can resolve this apparent contradiction by considering different strands of the improbability principle, including the “law of truly large numbers,” the “law of near enough,” the “law of selections”, and others.
When one goes through this process and understands the strands, Hand writes, the principle “tells us that the universe is in fact constructed so that these coincidences are unavoidable: the extraordinarily unlikely must happen; events of vanishingly small probability will occur.”
Why read such a book at all? I’ll hazard (another term from the world of risk!) a guess:
In the view of this blogger, in our interdependent, highly-interconnected world, where dense networks create a world of connections that change the meaning of “human scale” relative at least to what Borel understood it to be nearly a century ago, understanding how “rare” events, coincidences, and extraordinarily unlikely events occur has become vital for human (and other forms of life’s) security. But, grasping that uncertainty is inherent in reality and that, indeed, even in the present, we can only have an approximate understanding of reality, does not come naturally, for some reason.
Spoiler Alert: The rest of the post will discuss the different types of probability as Hand presents them in his book. Future posts will delve into other aspects presented in this work. ###
And there is a reason for this mismatch of expectations, explains Hand. Rarely addressed in our usual day-to-day settings but deftly discussed in this book is the gradual move in the last century or so –in science at least–beyond reliance upon “deterministic” principles long said to explain the behavior of natural systems. These principles, it has been assumed until recently, adequately explained the underlying causes and effects of events and outcomes, at least since the natural laws of physics began to be investigated in the 17th century. The early proponents of the concept of scientifically testing ideas were onto something revolutionary for the times! But they were limited in their understanding by what the tools and techniques of the day enabled them to observe: this influenced the types of questions they asked, of course, and led to overconfidence about mankind’s abilities to master nature’s mysteries within the bounds of existing knowledge.
The “Baconian Revolution” first introduced the idea of the scientific method, writes Hand. This method held that the way to understand the natural world is to collect data, conduct experiments, take observations, and use these as test beds through which to evaluate proposed explanations for what’s going on. Before that, stories and superstitions held sway. “But explanations that have not been or cannot be tested have no real force…,” according to Hand. “They serve the purpose of reassuring or placating those who are unwilling or unable to make the effort to dig deeper but they don’t lead to understanding.”
The first scientists (“natural philosophers” as they were called then) sought to devise laws that describe how nature works. Hand notes that these laws are “shorthand summaries” encapsulating “what observations shows about how the universe behaves.” They are “abstractions,” he notes. An example is Newton’s Second Law of Motion, which holds that the “acceleration of a body is proportion to the force acting on it.” The power of such laws is behind humanity’s progress in science and technology, Hand observes.
For a long time and even as recently as the 1930s, scientists and philosophers such as Karl Popper, held that the “rule that extreme improbabilities have to be neglected…agrees with the demand for scientific inquiry.” Those tiny chances of extraordinarily rare events had to be swept under the rug to allow progress, or so it was (and still is) thought. In addition, the idea of things happening for which we have no explanation is an intensely uncomfortable one, Hand writes, as humans have an innate need to know why things happen and “to establish the causal connections, and to understand the rules that lie behind what we observe.” This is a basic human need related to safety and security: if there “are no causes…illnesses, accidents, and failures couldn’t be avoided. We’d live in a constant state of fear, waiting the unpredictable disaster just around the corner.”
Over the centuries, it was impossible to miss the inexplicable coincidences and other extraordinarily unlikely events, creating fertile conditions for prophets and fortune tellers, writes Hand–people who tap into the notion that there is some “mysterious force or being behind what happens, often acting with malicious intent.” These notions have led to different explanations for otherwise unexplained events, including superstitions, prophecies, gods, miracles, and parapsychological explanations, he writes.
Yet, the notion that there is any real causal relationship between, for instance, sighting black cats and falling down, stems from misperceiving patterns. Hand explains that the goal of science is to distinguish between those patterns that do represent a “real underlying cause-and-effect relationship” and those that don’t. “Patterns we spot but that are mere accidents, without any underlying cause, have often formed the basis of superstitions. (Animals also demonstrate this development of “superstitions,” he notes.) But:
“Even if one event follows another surprisingly often, it doesn’t necessarily mean that the first causes the second. Statisticians have a sound-bite for this: correlation does not imply causation…Although the aim of a prophecy is to remove uncertainty about the future, uncertainty in the form of randomness is frequently the mechanism used to generate prophecies.”
The deterministic laws that evolved from the 17th to the 20th centuries were “mathematical equations…that told us how natural objects would behave,” writes Hand. “There was nothing in the universe that was uncertain or unpredictable, at least in principle, according to science.” And the immense technological progress of mankind “built on those ideas showed that they were largely correct.” Thus came into being the ubiquitous view of nature as “the clockwork universe”–a universe ticking along a well-defined path, Hand writes. Ignorance could be eradicated by science.
Later in the 20th century, however, science began to expose gaps that it could not explain. A huge shift in perception began slowly to take hold at least on the margins of science:
“It seemed as if the universe was not deterministic after all, but that randomness and chance lay at its very foundations.” Randomness and chance are entirely probable in this universe, Hand explains, and can be understood through the improbability principle which is formed upon the basic laws of probability.
Types of Probability There are different kinds, and definitions, of probability, according to Hand. Informal definitions even reveal the multi-facetness of probability: both, “the extent to which an event is likely to happen” and “the strength of belief than an event is likely to happen.” And Hand tells us that both can be represented by the same mathematics: probabilities are numbers lying in the range from 0 to 1 with 0 meaning impossible and 1 meaning certain. There are many other definitions, but none captures “probability” in its entirety. This is not really a problem, says, Hand, because it is very natural to need “multiple views of an object to understand it properly.”
The three most widely used interpretations of probability are the frequentist, subjective and classical interpretations:
The frequentist interpretation of probability is based on the tendency of physical systems to produce roughly constant relative frequencies when situations are repeated. For example: the tendency for a coin to come up heads about half the time it’s tossed, or the 4 (or any other) face to show on a die about one-sixth of the time. As we learn from reading Hand, there is a lot to think about in the word “roughly” above! Complete accuracy is impossible event when measuring, as frequentist probability does, properties of the “external world.”
Subjective probability is very different. Instead of representing an aspect of the external world, subjective probability is the confidence an individual has that an event will occur, explains Hand. This relates to your beliefs, whether about the probability of a coin turning up heads in a coin flip or your beliefs about the person tossing the coin (who might have rigged the process).
“Instead of being a property of the external world, the subjective view has it that probability is an internal property of your mind. Each person will have their own subjective probability for each event.” Hand notes that another eminent mathematician therefore claimed that probability did not exist because it is a “property of how we think about the world.” Nonetheless, Hand notes that various methods exist for measuring subjective probability, including asking people to bet on an outcome–knowing that the results will depend on what they think.
Only recently have we humans come to understand the significance of these fundamental different notions of probability, Hand says, with steps newly taken to distinguish between epistemological probability and “aleatory” probability. (For more on this, it will be necessary to read the book!)
The classical interpretation of probability is based on options of symmetry, Hand writes, giving the example of how natural it is to think of probability as distributed equally across the six faces of a die. “This interpretation is very convenient for games of chance, based on symmetrical randomization tools such as dice and coins,” he writes. But life is not like a die, he says: “it’s less clear how we might apply classical probability to situations in normal life which lack such obvious symmetries.”
There are other interpretations of probability, which Hand goes on to introduce. What’s key, and still to be addressed in this blog in future posts on the roots of surprise, is to understand how probability is calculated in the case of interdependent events, such as make up the natural and manmade systems of our world. What is the significance of these relatively recent discoveries of the inevitability of improbability?
Black Elephant Blog 