From Stephen Carey, A BEGINNER'S GUIDE TO SCIENTIFIC METHOD

Chapter Four: Extraordinary Claims and Anecdotal Evidence



EXTRAORDINARY
CLAIMS

In Chapters 2 and 3 we examined the methods by which causal links and explanations are tested. On occasion, however, the focus of a scientific investigation will be neither. People make extraordinary claims about things they have experienced and things they can do, and these can be tested by methods similar to those discussed in Chapters 2 and 3. Precisely why such claims should be of interest to science is something we will consider in a moment. But first let's try to get a better sense of what the extraordinary involves by looking at a number of extraordinary claims.

Some people claim to be able to see colorful "auras" emanating from the human body and to be able to discern things about one's personality by careful study of these auric emanations. Others claim to have been contacted by extraterrestrials or to have seen alien space craft - unidentified flying objects or UFOs - hovering in the night sky. Astrologers claim to be able to predict things about your future based on the position of the planets at the time of your birth. Similar claims are made by people who read palms, tea leaves, and tarot cards. Many people claim to have psychic ability of one sort or another: to be able to see the future, read the minds of others, or manipulate objects by sheer mind power. Some claim to have seen ghosts, poltergeists, or assorted cryptozoological creatures-- everything from bigfoot to the Loch Ness monster. Many claim to have lived past lives or to have left their bodies during near death encounters. Others claim to have communicated with the spirits of long-dead people.

On occasion, scientists make extraordinary claims. In March of 1989 two scientists, Stanley Pons and Martin Fleishmann, announced that they had created nuclear fusion at room temperature in a test tube; what makes this so extraordinary is that nuclear fusion usually takes place only at hundreds of millions of degrees. Some linguists and psychologists claim to have taught chimpanzees and gorillas to "speak" using American Sign Language. Around the turn of the century, a group of reputable French physicists claimed to have observed a new kind of electromagnetic radiation that they called "N-rays". This last example differs from the first two in that within five years of being discovered N-rays were conclusively shown not to exist. By contrast, a few physicists and chemists continue to believe that cold fusion may be possible, though the vast majority disagree; and though experimenters have worked with chimpanzees and gorillas for over thirty years now, their findings remain controversial-some critics contend that the experimental subjects are only mimicking speech behavior, occasionally giving the right signs in response to questions and the like but not really using language, at least in the sense that humans do.

Many extraordinary claims involve healing and medicine. Some dentists claim
we are being poisoned by our fillings. Iridologists claim to be able to diagnose
illness by examining nothing more than the iris of the human eye. Faith healers
claim to heal all sorts of illness and disability by prayer and the laying on of hands.
Psychic surgeons claim they can perform operations without the use of anesthetic
or surgical instruments. The list of extraordinary things people claim to have seen
and done is nearly endless.

All these claims have in common a pair of features. First, they are controversial in that though there is some evidence for the truth of each it is somewhat sketchy. Second, all appear to be at odds with some aspect of our current scientific understanding of the natural world. Suppose, for example, someone claims to be able to levitate. This claim is controversial precisely in that though there is actually some evidence for levitation - photographs and the apparently sincere testimony of people who claim to have levitated - it is limited. Moreover, if levitation is indeed possible then our current understanding of how and where gravity operates will have to be revised.

Or consider the claim made by many psychics to be able to divine the future. The evidence for such an ability is scant: in most cases a few clear and correct predictions accompanied by lots of vague and downright wrong ones. But if some people can actually see what has yet to happen, we must rethink our current view about the nature of causation. Common sense, if nothing else, suggests that if A is the cause of B then A must occur before B. Yet if the future can be seen, effects can be established long before their causes come into existence. Thus if the future can be foretold, something somewhere is wrong with our current view of causation.

Extraordinary claims, then, are both controversial and, if correct, revolutionary in calling for revision of our current understanding of some aspect of the natural world. This latter fact accounts, from a scientific point of view, for much of the interest in extraordinary claims. If we can establish the truth of an extraordinary claim, we have a good indication of where our current scientific understanding of things is either incomplete or wrong. How such claims might be established is our next topic.

TESTING
EXTRAORDINARY
CLAIMS

People known as water witches or dowsers claim they can detect underground water with a simple forked wooden branch. They loosely grasp one fork in each hand and point the branch straight ahead, parallel to the ground. When they approach a source of water the dowsing rod, as the forked stick is called, will point in the direction of the water much as a compass needle will point toward magnetic north. Many successful dowsers claim to be able to pinpoint sources of water for well drilling or even to have found water where conventional geologists failed.

This meets both of our criteria for the extraordinary claim. First, the actual evidence for dowsing is limited. We must rely on the testimony of dowsers and their clients about past performances. Moreover, if a dowser points to a location and a well is drilled that actually hits water, it does not prove that the dowser actually located it with the dowsing rod. Perhaps it was a coincidence, or maybe there were visual clues to aid the dowser-patches of greenery near the chosen location or something similar. And we have no sense of dowsers' success rates other than what they and their clients report. How often are they mistaken? Second, if dowsing actually works our current understanding of the various forces and interactions operating in the world today will need to be revised and supplemented. Nothing modern science tells us about ourselves and nature suggests that a simple wooden branch in the hands of a person will respond to a source of water.

Our challenge, then, is to devise an experiment that will give us decisive evidence one way or the other about the dowser's claimed ability to find water with nothing more than a wooden branch. In designing an experiment we need to borrow some ideas from our discussion in Chapter 3 of the requirements for a good test of an explanation. We said that a good test involves a predicted result that will occur if the explanation is correct but not if it is mistaken. We also noted that the predicted result must be independent of the evidence the explanation is invoked to explain. In testing for a claimed extraordinary ability, we are not primarily concerned with an explanation for it but rather with whether a person can do the extraordinary thing. Thus in our test we require nothing like an independent prediction. Instead we will try to devise a set of experimental conditions by which we can predict the following:

1. The subject or subjects in question will actually be able to do the extraordinary thing they claim to be able to do.

2. The subject or subjects will not be able to perform if they do not have the ability they claim to have.

Consider now what an experimental test of dowsing must be like to satisfy criteria 1 and 2.

Criterion 1 will be satisfied if we arrive at a set of experimental conditions under which the dowser clearly ought to be able to perform. A good rule of thumb to follow in setting up tests of extraordinary claims so that they satisfy criterion 1 is to consult the experimental subject or subjects prior to designing the experiment. We want to set up conditions under which the experimental subjects will agree in advance that they ought to be able to perform; otherwise failure may be taken to show only that the experiment is hostile to the ability we are attempting to test. But if our subjects concur that the experiment approximates conditions they should be able to perform under, such excuses lose much of their force. If they claim to be able to perform under a given set of conditions, it is hard to take their protests seriously if they complain about the nature of the test after they fail it.

Criterion 2 will be satisfied if we are careful to set up conditions under which there could be no way for the dowser to find water short of detecting it with the dowsing rod. In effect we want to try to rule out the possibility of cheating, coincidence, inadvertent cuing on our part, visual or audio clues as to where the water is, and the like. If we succeed in imposing controls sufficiently tight to rule out these possibilities, success by the dowser can be taken to vindicate his or her claimed extraordinary ability.

Now that we have a sense of what a good experiment ought to involve, let's try our hand at actually designing one. Imagine that we contact one of the country's best known and most successful dowsers and he or she agrees to take part in our experiment. We propose to place before our dowser ten identical large ceramic jars with covers, one containing water and the others empty, arranged in a straight line equidistant from one another. The dowser can approach but not touch any jar. Our subject agrees that he or she should be able to find the single jar with water. If the dowser is successful, a second test will be performed with the jars rearranged (the subject will, of course, leave the room while the jars are being rearranged). As an additional precaution, no one who knows the location of the jar containing water will be allowed in the room while the dowser is being tested.

Does our imagined experiment satisfy criteria 1 and 2? Once again, here they are:

1. The subject or subjects in question will actually be able to do the extraordinary thing they claim to be able to do.

2. The subject or subjects will not be able to perform if they do not have the ability they claim to have.

Criterion 1 is satisfied provided our subject agrees that the conditions we have built into our experiment are conditions under which he or she can detect water. In addition, it does seem highly improbable that a dowser could pick the right jar with no actual ability to dowse, provided all ten jars are identical and provided no one who knows the location of the jar containing water is in a position to give an inadvertent clue to the dowser. Under these conditions, chances are one in ten that the dowser will select the correct jar. However, such odds are not that small. Hence our insistence that the dowser submit to a retest if he or she succeeds in the first trial: the chances of a single dowser selecting the right jar in two successive trials is considerably smaller--one in a hundred. Thus with the retest provision and the precautions we take to eliminate inadvertent cuing, our test satisfies criterion 2; the odds are slim that the dowser would succeed because of lucky guesses.

One feature of our test deserves additional comment. We have been careful to arrive at a prediction that sets a clear line of demarcation between success and failure. If our dowser can find the jar containing water in two successive trials, he or she is successful; anything less constitutes failure. In designing controlled tests it is important to avoid predictions that blur the line between success and failure. Imagine, for example, that we had decided to test our dowser by burying containers of water a few feet below the surface of a vacant lot. The dowser would then be instructed to place markers where he or she believed the containers to be located. Suppose the dowser placed markers within three or four feet of one of the containers. Does this constitute a hit or a miss? Just how far off must a marker be before we consider it a miss? Or suppose markers are placed at ten locations when only five containers were buried and that seven of the markers are within a few feet of one or the other of the containers. How do we evaluate these results? Has our dowser succeeded or failed?

The line between success and failure can be very difficult to draw when a prediction involves some sort of subjective impression on the part of the experimental subject. Imagine, for example, that we were to test a purported telepath - someone who claims to be able to read the thoughts of another. As part of our experiment we instruct the telepath to sketch a simple picture that someone in another room is concentrating on. Suppose the person in the other room is looking at a postcard of a small sailboat moored at a marina and that the telepath produces a simple drawing that includes a vertical straight line and a narrow triangular shape that might correspond to a boat hull or sail. Unfortunately, however, several of the drawing's details conform clearly to nothing we can discern on the postcard. Is the telepath's impression accurate or inaccurate? Presuming we can decide what constitutes a detail or feature of the picture on the card, how many features or details must the telepath get right to be a clear indication of success?

To take another example, imagine that a tarot card reader gives a personality analysis of someone unknown to the reader based on the position and order of the cards. The reading might indicate that the person in question tends to be optimistic despite occasional moments of depression or pessimism, or makes friends easily, or displays clear leadership ability. How do we evaluate such claims? The problem here is not only with the generality of the predictions but with the lack of a clear basis for judging them. We must first arrive at an accurate personality profile of the person; but presuming we could do this, what objective basis do we have for comparing our profile with that of the tarot card reader? No doubt any two sets of subjective impressions about a person's character will contain some words and phrases in common. How much similarity is required to put some stock in the analysis of the tarot card reader?

In designing a test, then, it is crucial that we arrive at a prediction that clearly spells out the difference between success and failure. If in evaluating the results of a test we are unable to say precisely whether our subject has succeeded or failed, the test has very little point. It certainly fails to satisfy criterion 2, for we have not specified a result that we could rate as highly unlikely if our subject or subjects did not have the ability in question. Fortunately, however, the prediction in our dowsing test seems to be clear and unequivocal: success and failure are clearly spelled out. So let's return briefly to the dowsing test we have proposed and look at the conclusion we are justified in drawing, depending on the results we get.

If our dowser can consistently find water under these experimental conditions, it is highly unlikely it could be accounted for by anything other than a genuine ability to detect water with a dowsing rod. Of course, if we achieve this result our work will have just begun: we must consider what might explain how the movement of a wooden branch is influenced by the presence of water. Where, that is, are the gaps in our understanding of things that have kept us from understanding this remarkable phenomenon?

But if our dowser consistently fails, chances are high that his or her successes under noncontrolled conditions are no indication of an extraordinary ability. Note here a difficulty that plagues most tests of the extraordinary. Suppose our dowser does indeed fail. Does this show conclusively that dowsing doesn't work? No; we have only shown that our particular dowser does not have this extraordinary ability. Furthermore, if we test (say) ten dowsers and none pass, believers in dowsing might begin to wonder if our test makes some auxiliary assumption that is false. If dowsers in general cannot perform under the conditions of our experiment, so the argument would go, perhaps the experiment is flawed in some way neither we nor the dowsers understand. Perhaps, for example, there is an undetected source of ground water underneath the location of our experiment that interferes with a dowser's ability to function.

To make matters worse, even if we could satisfy ourselves that our experiment is not defective and that we are making no false auxiliary assumptions, we would only be in a position to conclude that none of the dowsers we tested are the genuine article. After all, instances of dowsing have been reported for hundreds of years; the earliest record of a successful dowsing dates to 1586 in Spain. Moreover, over the years thousands and thousands of people have attested to the success of dowsers. What, then, are we to make of negative experimental results given the vast body of historical and contemporary evidence that there is something to dowsing? This brings us to our next topic: anecdotal evidence.

ANECDOTAL
EVIDENCE FOR THE
EXTRAORDINARY

The strongest evidence for an extraordinary claim is that provided by careful controlled testing. But it is not the only kind. Anecdotal evidence - historical reports of extraordinary happenings and reports by people who claim to have witnessed or to have done something extraordinary-at- least suggest that extraordinary things do happen. But how do we evaluate such evidence, particularly where it constitutes the sum total of evidence for something extraordinary? Presuming we are open-minded and willing to think critically, what should our attitude be toward extraordinary claims backed largely by anecdotal evidence? Skeptical disbelief? Tentative belief, at least in those cases where the anecdotal evidence is considerable? A kind of scientific agnosticism? An agnostic is a person who claims it is impossible to know whether there is a God or not. We might appropriate this term for our purposes by slightly modifying its standard meaning: a person is an agnostic with respect to a particular claim if he or she b elieves that the evidence for the claim warrants neither belief nor disbelief. An agnostic, in this sense, maintains that one ought to remain intellectually open to the possibilities suggested by extraordinary claims.

No doubt agnosticism would at first appear to be the fairest, most objective, and most critically responsible of our three possible responses. Yet frequently it is not. Often scientific agnosticism about anecdotal evidence for the extraordinary amounts to nothing more than a failure to think critically about it. Even where the anecdotal evidence is considerable, skeptical disbelief is warranted more often than not; ironically, the kind of open-mindedness valued by the agnostic often requires disbelief, not agnosticism. To get at the reasons for this, we must explore a bit further the notions of skeptical disbelief and agnosticism. Consider, to begin, the following case.

A famous psychic contends he can bend keys telekinetically - that is, by simply willing them to bend. Hundreds of people claim to have witnessed our psychic perform this extraordinary feat. Typically, he holds an ordinary house or car key in one hand, concentrates his thoughts on it, and before the eyes of witnesses it actually seems to bend! Unfortunately, our psychic refuses to be tested under controlled conditions on the grounds that he finds it impossible to perform in the presence of experimenters who are understandably skeptical. Some things, claims our psychic, are not meant to be tested. As with most extraordinary claims, however, there is some evidence that this one may not be true. First, our psychic apparently cannot perform under tightly controlled conditions. Why not? Consider the fact that an equally famous magician can do everything our psychic can do, but by out-and-out trickery - nothing psychic is involved at all. It is well known that people can easily be deceived by a skilled magician; we've all seen them make people float in thin air and produce all manner of objects out of nothing. Surely a skilled magician could trick us into believing we have seen a key bend. Of course, under tightly controlled conditions a magician might find it considerably more difficult to accomplish such deceptions.

So what are we to make of our psychic and his extraordinary claim? Is he genuine or a fraud? Based on the evidence we have examined so far, it may seem that this question is impossible to answer and that an open-minded person should withhold judgment pending further evidence. Agnosticism might well seem the proper attitude. But there is a kind of evidence we have not yet considered, a kind that stands in the background of nearly every extraordinary claim.

As we pointed out earlier, extraordinary claims are generally at odds with our current understanding of some aspect of the natural world. A principle that seems well established in nature is that one event cannot influence another without some intervening mechanism or medium. The flow of blood in the human body resists the pull of gravity in part because of the pumping action of the heart. Magnets influence the movement of ferromagnetic particles via an intervening medium, their surrounding magnetic fields. There are in fact no known instances of what is sometimes called "action at a distance"- actions or events causally related to antecedent but remote actions or events wherein there is no intervening medium or mechanism. A variant of this principle seems to hold for human action as well. If I want to bring about a change in the world external to my mind, I must do more than think about it. In general it is well established that a person's mind cannot effect a change in the physical world without the intervention of some physical energy or force. If, say, I want to move an object from one spot to another, simply willing the object to move is insufficient to accomplish my purpose. I must figure out some waysome sequence of actions-that can lead to the goal I will myself to accomplish.

Now, it may turn out that the "no action at a distance," principle is false, that we will eventually discover some phenomenon that involves action at a distance. Imagine, then, that someone claims to be able to do something or to have witnessed something that could only be achieved by action at a distance. Clearly we should require extraordinarily rigorous evidence in its favor before accepting such a claim, for the enormous body of scientific investigation accumulated over centuries is, indirectly at any rate, evidence against it.

What does all this suggest about the extraordinary claims made by our psychic?
The slogan "extraordinary events require extraordinary evidence" sums it up nicely.
If telekinesis exists - if it is indeed possible to cause remote physical events to happen by force of will - then either action at a distance is possible or there is some
subtle medium or mechanism at work that has so far eluded our detection. Because
so much is at stake, we are entirely justified in demanding extraordinarily decisive
evidence for our psychic's claim to influence objects telekinetically. In the absence
of such evidence - evidence of the sort provided by carefully controlled testing,
not by anecdotal testimony or even by the apparently sincere avowals of our psychic - we have good reason to doubt his extraordinary ability. If he can do what he
claims, we must take seriously the notion that forces or processes are at work in
nature that have so far escaped our detection; we must begin thinking about revisions to our current understanding of things.

We began this section with an important question. Given that we are open-minded and willing to think critically, what should our attitude be toward anecdotal evidence for the extraordinary? Tentative belief where the evidence is considerable? Agnosticism? Skeptical disbelief? Where evidence for the extraordinary is entirely anecdotal, our attitude should be one of skeptical disbelief tempered with a willingness to revise our position only in light of well- established experimental evidence. This is because a great deal is likely to be at stake: if the extraordinary can be established, we must set about revising or perhaps even overturning some part of what we currently believe about the nature of things. And since we are likely to have a considerable body of evidence supporting the beliefs in question, we have good reason to retain our skepticism about things extraordinary. This is not to say that extraordinary claims cannot be established, but to suggest that we should remain skeptical until presented with evidence for the extraordinary of sufficient strength and quality to warrant a careful rethinking of the evidence for those beliefs we are being asked to set aside.

The rationale behind this kind of enlightened skepticism is eloquently summed up by Richard Dawkins, professor of zoology at Oxford University:

Some things that have never been reliably seen are, nevertheless, believable insofar as they do not call in question everything else that we know. I have seen no good evidence for the theory that plesiosaurs live today in Loch Ness, but my world view would not be shattered if one were found. I should just be surprised (and delighted), because no plesiosaur fossils are known for the last 60 million years and that seems a long time for a small relic population to survive. But no great scientific principles are at stake. It is simply a matter of fact. On the other hand, science has amassed a good understanding of how the universe ticks, an understanding that works well for an enormous range of phenomena, and certain allegations would be incompatible, or at least very hard to reconcile, with this understanding. For example, this is true of the allegation, sometimes made on spurious biblical grounds, that the universe was created only about 6,000 years ago. This theory is not just unauthenticated. It is incompatible, not only with orthodox biology and geology, but with the physical theory of radioactivity and with cosmology (heavenly bodies more than 6,000 light years away shouldn't be visible if nothing older than 6,000 years exists; the Milky Way shouldn't be detectable, nor should any of the 100,000 million other galaxies whose existence modern cosmology acknowledges).

There have been times in the history of science when the whole of orthodox science has been rightly thrown over because of a single awkward fact. It would be arrogant to assert that such overthrows will never happen again. But we naturally, and rightly, demand a higher standard of authentication before accepting a fact that would turn a major and successful scientific edifice upside down, than before accepting a fact which, even if surprising, is readily accommodated by existing science. For a plesiosaur in Loch Ness, I would accept the evidence of my own eyes. If I saw a man levitating himself, before rejecting the whole of physics I would suspect that I was the victim of a hallucination or a conjuring trick. There is a continuum, from theories that probably are not true but easily could be, to theories that could only be true at the cost of overthrowing large edifices of successful orthodox science.'

The kind of enlightened skepticism exemplified in this passage is likely to be misunderstood. In response to a demand for rigorous evidence, believers in the extraordinary would probably offer something like the following:

Well, what do you expect from a mainstream scientist? Such scientists are conservative in the worst sense of the word. When presented with new ideas that challenge their pet theories, they just reject the new ideas out of hand rather than even consider the possibility that their accepted theories may be wrong. Were scientists to acknowledge the extraordinary, they would have to admit they don't know everything and that there may be things science just cannot explain. So the mainstream scientific community has a vested interest in denying the existence of things extraordinary.

Setting aside for the time being the question of whether there are things that science cannot explain, there is something profoundly wrong in this rejoinder. The idea that science is somehow uncomfortable with or embarrassed by the extraordinary is quite the reverse of the actual case. A great number of major advances in the history of science have their roots in attempts at understanding something extraordinary. After all, such things serve to suggest something amiss in the current scientific picture of how the world works. Indeed, science will continue to develop only so long as there remain phenomena that challenge received explanations. Thus, far from finding the extraordinary embarrassing, science has a vested interest in discovering and establishing instances of extraordinary phenomena. An extraordinary claim is of great scientific interest precisely because it does not fit into our current understanding of things and consequently points us in the direction where research needs to be undertaken. So, the initial reaction of the competent scientist when faced with a report of the extraordinary ought to be the kind of enlightened skepticism we have been discussing. The scientist's first task is to try to show that the purportedly extraordinary event has a more or less conventional explanation. Only when this attempt fails can we be confident that the event points toward an area that requires further investigation.

DESCRI PTION AND
EXPLANATION

Reports of the extraordinary frequently blur the distinction between description and proposed explanation. It may seem that when people report extraordinary events they are simply describing what occurred - no more, no less. But in fact considerably more is likely to be contained in such reports. Imagine, for example, that someone were to report waking in the middle of the night to discover what appeared to be his long-departed grandmother standing at the foot of the bed. He might subsequently claim:

1. 1 saw the ghost of my dead grandmother.

But what, precisely, is factual in statement 1? What can we be confident actually happened? That the person had an extraordinary experience is clear. Beyond that it is hard to say. Consider two rival accounts of what may have happened:

2. He had a vivid lifelike dream in which his grandmother appeared.

3. Somebody played an elaborate and vicious prank on him in the middle of the night.

All three statements implicitly contain explanations of the event, but each presupposes a very different truth: (1) that what he saw was actually a ghost, (2) that what he "'saw" was part of a dream, and (3) that what he saw was real but was a hoax, not a ghost.

Similarly, many anecdotal reports of the extraordinary contain much more than a simple, objective description of the experience. Such reports often blend fact with untested explanation and are what we might call explanation laden. For example, "The flying saucer hovered over the horizon and then accelerated away at a fantastic rate" tells us a couple of things about the person who makes the statement. First, the person had an undeniably extraordinary experience; second, the person believes the proper explanation for the experience is the actual existence of an intelligently controlled spacecraft.

In evaluating such a report, we must do our best to separate the descriptive wheat from the explanatory chaff. If we are able to subtract the explanation-laden portions of a report of the extraordinary, we may be able to arrive at a clear sense of what actually was experienced and thus what needs to be explained. For example, suppose we could establish that the person making the flying saucer report actually saw a bright light near the horizon, looked away to call to a friend, looked back again, and saw only a dim, twinkling light at some distance from the original light. Having gotten clear on this much, we would at least be in a position to think about rival explanations more plausible than the one implicit in the initial description of the event.

I once spoke with a person who claimed to have lived in a haunted house. He recalled that every few nights he would hear a knocking at the front door, but there was never anyone there when he opened it. We agreed that a more accurate description of the experience would contain only the salient facts: on several occasions he heard a series of sounds very much like knocking at a door that seemed to come from the area of the house near the front door. He also added that he was never near the door when he heard the noise. Once we focused on this new, more objective description, several plausible explanations immediately came to mind: a tree or bush knocking against the house or some other activity outside or even inside that sounds from a distance like knocking. Now, we may never discover what really happened on those nights when he heard a "knocking" at the door. At the very least, however, we know which parts of the story are fact and which are speculation. And this is the real value of carefully distinguishing between the descriptive and explanatory elements of an extraordinary claim.

[Now read the first exercise below and the solution to it provided by the author which follows.]

EXERCISES

Exercises 1-10 involve extraordinary claims. Design a good test for each. Try to devise a set Of experimental conditions under which

a. The subject or subjects in question will actually be able to do the extraordinary thing they claim to be able to do.
b. The subject or subjects will not be able to perform if they do not have the ability they claim to have.
c. The difference between successful and failed performance is clearly measurable.

In designing your test, think about any auxiliary assumptions the test requires that may be questionable. Are you assuming something to be true that if false might enable your experimental subjects to explain away a failure to perform under the conditions of the test? If you uncover any such assumptions, try to modify your design so that it avoids them. (Note: A solution to Exercise I is provided on page 111.)

1. The ability to influence physical objects or events by thought alone is called telekinesis or psychokinesis. One extraordinary claim of people who say they have telekinetic power is an ability to influence the outcome of apparently random events. So, for example, by concentrating on a particular number a person trained in telekinetic manipulation might influence the throw of a pair of dice or the spin of a roulette wheel.

2. Graphologists claim to be able to discern a great deal about a person's character and personality simply by analyzing the person's handwriting. So, for example, if you don't care enough to dot your I's it shows that you tend to ignore details, and an illegible signature indicates a desire to escape notice. Similarly, claim many graphologists, people who print rather than write may be trying to conceal their personality from others.

3. Reflexology is the technique of applying pressure with the thumbs and fingers to specific areas of the feet as a means of alleviating various ailments. Reflexologists contend that the body and all its organs and glands are mapped" on the foot. For example, the big toe is said to represent the head, the little toe the eyes and ears, and the inner portion along the side of each foot the spine. Some reflexologists claim to be able to diagnose ailments by looking for painful spots on the foot. Throat and tonsil problems, for example, frequently result in pain on the bottom of the big toe whereas pain on the bottom of the other toes often indicates sinus problems.

A Solution to Exercise I

The first thing we need to do in designing our experiment is to set up conditions under which subjects cannot perform unless they have telekinetic ability. The problem with things like dice and roulette wheels is that we will get some result whether or not telekinesis is operating. Suppose a subject were to try to make a die show the same number on every throw. Whether or not the subject has telekinetic ability, the die is bound to show the selected number about one time out of six because there are six sides on a die. No matter how well our subject does, we will be unable to rule out coincidence. What if the die comes up with the given number just a little more or less than one time in six? It would be hard to evaluate. Just how much more than a sixth of the time provides evidence of telekinesis? Much better would be an experimental result that occurs only if our subject or subjects have telekinetic ability.

[NK's comment: I disagree with the author about the difficulty of evaluating the results of the dice throwing experiments. It is possible, using confidence intervals, to be very precise about which results could happen by chance and which are very unlikely to. However, the method proposed below does have some advantages.]

With this in mind, we might begin by contacting people who claim to have telekinetic power, convincing them to take part in our experiment and finding out about the conditions under which they agree they can and cannot perform. To simplify things, let's assume we have selected one likely candidate to be tested. Assume also that the subject is not too specific about how and where telekinesis works but believes he or she should be able to perform under these conditions:

A small, spherical foam ball is sealed in a glass container on a well-anchored, immovable surface. The experimental subject sits with hands in lap a few feet from the surface and is instructed to move the ball by telekinesis.

It seems highly unlikely that the ball will move if the subject does not have telekinetic ability, so success would provide strong evidence for telekinesis. If the subject succeeds and we have any doubts about the outcome, we can easily repeat the whole procedure.

But what if the subject fails? Our experiment involves several auxiliary assumptions, any one of which, if false, could explain the failure: (1) telekinesis does not work with foam, (2) telekinesis cannot move an object this massive, (3) telekinetic force cannot penetrate glass, or (4) telekinetic force falls off over distance and the distance between subject and object was too great. By slightly manipulating the experimental conditions we could probably eliminate the need for any of these assumptions. We might, for example, try objects made of different materials and masses, move the subject closer, or leave the objects to be manipulated out in the open. (In this last case, we must take precautions to ensure that the subject cannot blow on the object if it is very light.)

Notes

1. Dawkins, Richard. The Blind Watchmaker. London: W. W. Norton, 1986, pp. 292-293. Reprinted by permission of the publisher.

2. Tolbert, Charles R. "Scientist, Astrologer in Horoscope showdown." Oregonian,
July 2, 1992. Reprinted by permission of the author.