Friday, April 20, 2012

The Psychology of Scientific Misinterpretations: The Big Bang Theory, Fred Zain, Cognitive Bias, and The Texas Sharpshooter Fallacy

I am a huge fan of the comedy hit series “The Big Bang Theory”. Please don’t fret, this blog article IS about forensic science, …..with emphasis on DNA. If I were to focus upon everything I know about theoretical physics, this article would scarcely occupy more than ¼ page, ….even on a good day.

Refer to the final Big Bang Theory episode of Season 2, The Monopolar Expedition, ….with Part 2 of the storyline continuing into the first episode of Season 3.

Dr. Sheldon Cooper, a theoretical physicist, is chosen to go on a summer long expedition to the magnetic North Pole. On this adventure, he takes friends/colleagues: Leonard, Howard and Raj. When the four scientists return from the expedition, Sheldon is beside himself with satisfaction. Apparently, the group was able to collect exciting data that PROVED String Theory (On a good day, I might actually have a vague notion of this theory). Sheldon wasted no time in sending out notifications to everyone on his e-mail contact list-Sheldon’s long anticipated Nobel Prize in Physics was all but locked up.

Leonard, Howard and Raj approached Sheldon with a tragic reality check. They FIRST pointed out what an insufferable jerk he was--during the first portion of the expedition, …..when virtually NO USEFUL DATA had fallen into their lap. Sheldon’s comically remorseful friends confessed that their NOBEL PRIZE DATA, …..was nothing more than the result of an electric can opener—secretly being turned on and off.

Sheldon was informed that—due to the relentless aggravation of his North Pole bunk mates—it was a matter of 1) Pleasing Sheldon by temporarily falsifying some scientific data, ….or 2) Expel their expedition leader out into the cold, as a ‘snack’ for the polar bears. Sheldon, ….understandably hurt and humiliated, ….had no choice but to immediately author a follow-up e-mail, forfeiting his bragging rights to the next Nobel Prize in Physics.

Let us now analyze what happened here. What do you suppose Sheldon hoped to achieve at the North Pole? Fame, fortune, ….and of course most important, the scientist expected to prove himself CORRECT, ….a genius among all of the 'common folk' on our planet. What were his three colleagues trying to achieve with their falsified data? They simply wished to maintain a grasp upon their rapidly deteriorating sanity, ….while avoiding possible prosecution for dispatching their colleague to the local carnivores.

But why didn’t the scientists simply keep their lips sealed, publish their exciting breakthrough discovery, ….and let Sheldon enjoy his Nobel Prize? Oh come on, ….everyone out there MUST see the devastation associated with that notion! The alleged ‘breakthrough’ would have been FALSE! It would have defied Sheldon’s chief purpose—to be CORRECT. Further still, …. all four likeable scientists certainly would have been exposed, …as LIARS, ….as the contradictory data rolled in from subsequent research efforts. There is no comedy in that!

The primary problem with the scientists in this ticklish story, ….is that they entered their icy northern adventure, ….with an AGENDA. Sheldon’s agenda was “I will ascend to greatness!” When the North Pole efforts were not accommodating his agenda, ….he became insufferably cranky. Consequently, Sheldon became prone to a common scientific pitfall--known as ‘Observer Bias’.

The other three scientists simply wanted to be part of something cool (a subtle North Pole pun), …..or at least enjoy a once-in-a-lifetime trip and the pursuit of some nifty career possibilities. At first, …they were enjoying the excitement of the endeavor. However, Sheldon spoiled it all with his relentless frustration and poor behavior. So, ….the three friends developed a new AGENDA, ……LIE, …..and confess to Sheldon at a later time.

What is Observer Bias? How does this scientific disease fit in with the Monopolar Expedition, …or better yet, ….with forensic DNA misinterpretations? Upon consulting an internet-based psychology glossary, you will find that: Observer bias occurs when the observers (or researcher team) know the goals of the study or the hypotheses and allow this knowledge to influence their observations during the study

Stop briefly, …..and visualize a forensic analyst who KNOWS a suspect’s DNA profile, BEFORE all of the DNA from the evidence has been typed. Why do you suppose it is a horrible idea for this DNA analyst to permit a 'casual peek’ at this info BEFORE the experimental part of the crime lab work is complete?

Sheldon’s observer bias at the North Pole raised its hideous head when the scientist EXPECTED some sort of magnetic/electrical fluctuations, …..and embraced the data—without question—as soon as the phenomenon was observed. Alternatively, perhaps Sheldon should have OBSERVED how annoyed his friends were with his endless whining, and noted that the electrical fluctuations magically *materialized* right at the peak of their irritation. Even a low level of suspicion on Sheldon’s part might have encouraged him to investigate deeper and expose the devious plot perpetrated by his colleagues.

Before we discuss DNA, let us also touch upon Confirmation Bias. My psychology glossary defines this as follows: Confirmation bias is the tendency to look for information that confirms our existing preconceptions, making it more likely to ignore or neglect data that disconfirms our beliefs.

Sheldon so devoutly wished to uncover data confirming his existing preconceptions, he never noticed how the data seemed to resemble what you might achieve with an ordinary electric can opener. Upon observing the exciting fluctuations, Sheldon did nothing more than continue to collect as much CONFIRMATORY DATA as possible. Perhaps he could have crafted an entirely new set of experiments with potential to debunk his own hypothesis. Perhaps Sheldon could have slipped out into the cold while his friends were resting, and run a few supplemental tests—by himself. If Sheldon had taken such initiatives, he might have prevented the humiliation of retracting his Nobel Prize-bragging e-mail.

Again, having an AGENDA, ….an obsession with confirming one’s scientific hypothesis, … a DANGEROUS thing.

I will discuss the effect of Confirmation Bias on forensic DNA interpretations a bit later. However, if you want to jump ahead to a detailed examination of how ‘cognitive bias’ can affect forensic investigations, look no further than a recent PBS Frontline broadcast on this topic. This televised news article, entitled "The Real CSI", aired on Tuesday, April 17, 2012.

Let us now drive home the concept of cognitive bias with examples. Few that I stumbled across on the internet were more hilarious than the following:

She loves me, and there are so many ways that she has shown it. When we signed the divorce papers in her lawyer’s office, she wore my favorite color. When she slapped me at the bar and called me a “handsome pig,” she used the word “handsome” when she didn’t have to. When I called her and she said never to call her again, she first asked me how I was doing and whether my life had changed. When I suggested that we should have children in order to keep our marriage together, she laughed. If she can laugh with me, if she wants to know how I am doing and whether my life has changed, and if she calls me “handsome” and wears my favorite color on special occasions, then I know she really loves me.

I recently stumbled across an outstanding blog site, hosted by journalist, TV producer, media director, David McRaney, entitled "You Are Not So Smart--A Celebration of Self Delusion". In his September 11, 2010 posting, Mr. McRaney summarizes what is traditionally referred to as “The Texas Sharpshooter Fallacy”. I take some exception to this—as I know some Texans—many of my friends live in Texas. Furthermore, it is quite clear that numerous unaccounted for ‘sharpshooters’ are running a-muck in New Mexico, Arizona, and countless other states. McRaney begins by pointing out the following astounding facts:

“Abraham Lincoln and John F. Kennedy were both presidents of the United States, elected 100 years apart.

Both were shot and killed by assassins who were known by three names with 15 letters, John Wilkes Booth and Lee Harvey Oswald, and neither killer would make it to trial.

Spooky, huh? It gets better.

Lincoln had a secretary named Kennedy, and Kennedy had a secretary named Lincoln.

They were both killed on a Friday while sitting next to their wives, Lincoln in the Ford Theater, Kennedy in a Lincoln made by Ford.

Both men were succeeded by a man named Johnson – Andrew for Lincoln and Lyndon for Kennedy. Andrew was born in 1808. Lyndon in 1908.

What are the odds?”

Before you go rushing off, ….shouting from the mountaintops that something truly incredible or sinister has been uncovered here, …..please listen up. McRaney continues:

“When you are befuddled by the Lincoln and Kennedy connections, you neglect to notice Kennedy was Catholic and Lincoln was born Baptist. Kennedy was killed with a rifle, Lincoln with a pistol. Kennedy was shot in Texas, Lincoln in Washington D.C. Kennedy had lustrous auburn hair, while Lincoln wore a haberdasher’s wet dream.”

The ‘CELEBRATION OF SELF-DELUSION’ point of McRaney’s blog article is the Texas, ….(or insert state name of your choice here) Sharpshooter Fallacy—which is derived from an old joke describing a backwoods gentleman who test fires several bullets at an old barn wall. A thought dawns upon the man, ….he grins widely, ….and rushes off to grab some red paint and some white paint.

Not long after that, he leads various friends and neighbors out to the grassy knoll next to his barn (sorry, I couldn’t resist—what with the Kennedy theme already in place). His associates are genuinely amazed at all of the painted targets on the barn wall, ….each with a bullet hole PERFECTLY located in the center of the target. The backwoods gentleman enjoys all of the comments and admiring attention. His delight and contentment persists right up to the point at which one savvy neighbor walks over to the barn and notices that the paint is still a bit sticky, ….and sees that some of the paint has been sloshed into some of the bullet holes, …..and drips of paint are running down the INSIDE wall!

Speaking of sharpshooting liars, say “Hello” to Fred Salem Zain. From 1979 through 1989, this guy was a self-anointed forensic expert with the West Virginia Department of Public Safety Crime Lab. Zain falsified test results in as many as 134 cases. He testified in countless rape and murder cases about analysis he had never performed and scientific data that simply did not exist. Zain presented himself brilliantly in the courtroom. Judges, juries, prosecutors, and defense attorneys had no reason to doubt his testimony. Over the years, the man rose to the position of Chief of Serology.

In 1989, Zain’s fabulous reputation with the prosecutors in West Virginia landed him a promising opportunity as the Chief of Physical Evidence for the Bexar County Medical Examiner's Office in San Antonio, Texas. In his new job, Fred Zain dazzled Bexar County prosecutors with his SHARPSHOOTING testimony—centering on fabricated results. Little or no effort was ever attempted to disqualify this brazen charlatan as an expert witness. Curiously, Zain was not even properly qualified to be doing forensic lab work in the first place. Examination of college transcripts eventually revealed that Zain was a mediocre scholar who had failed organic chemistry. Apparently, no one ever bothered to look at his transcripts prior to elevating the man from position to position.

Fred Zain was well aware that he was operating in a legal system that relied almost entirely on expert testimony from prosecution witnesses. For the last twenty years, this tilted landscape has been gradually shifting toward a more balanced position.

Zain’s inevitable downfall came as consequence of the State of West Virginia v. Glen Woodall. In 1987, Woodall was sentenced to a prison term of 203 to 335 years. At Woodall’s trial, Zain testified that semen samples recovered from the victims were consistent with the defendant. In 1992, DNA testing cleared Woodall of any guilt. Woodall sued the State of West Virginia for false imprisonment, and received a $1 million settlement. This ultimately led to an extraordinary investigation of the entire body of Zain's work—as ordered by the West Virginia Supreme Court.

As a consequence of Fred Zain’s appalling tactics, nine men have since been exonerated in West Virginia. $6.5 million have been paid out by the state of West Virginia as restitution for the wrongful convictions. In Bexar County, Texas cases, Zain testified to conducting tests that his lab was incapable of doing in the first place. Bexar County paid out more than $1.1 million for wrongful convictions at the hands of Zain. The disgraced man died of colon cancer in 2002.

It is important to note that Fred Zain represents a radical example—an individual who became hopelessly intoxicated with his perception of power. The man fell prey to an exaggerated degree of cognitive bias, lost objectivity, and an obliterated sense of humanity. Although there have been few scientists resembling Fred Zain, there are countless others who—to a much lesser degree—struggle to overcome observer effects, falter with their sense of objectivity, and arrive at flawed decisions.

I am personally familiar with a closed case involving analysis of DNA from a commonly ‘handled’ object. That object was located at a crime scene. The investigators *hoped* to find trace DNA from the defendant on that object. The DNA pattern—a mixture from at least two individuals— was found on a sample from the item, as shown below:

Locus 1                               14, 15, 16
Locus 2                               28, 28
Locus 3                               14, 15, 16
Locus 4                               10, 13, 14
Locus 5                               10, 10
Locus 6                               6. 10, 11
Locus 7                              10, 13
Locus 8                              12, 13, 13.2
Locus 9                               5, 6, 8
Gender Locus                    X, Y

The defendant’s DNA profile was as follows:

Locus 1                               16, 18
Locus 2                               28, 30.2
Locus 3                               14, 15
Locus 4                               14, 14
Locus 5                               7, 11
Locus 6                               6. 10
Locus 7                              10, 11
Locus 8                              12.2, 15.2
Locus 9                               5, 9
Gender Locus                    X, Y

The analyst observed that 10 of the defendant’s 18 DNA markers were indeed consistent with the DNA mixture observed on the object from the crime scene. Based on this observation (or observation bias?) the analyst reported that the defendant could not be excluded as a possible contributor to the DNA mixture profile found on the object. Furthermore, the analyst reported that the DNA mixture profile was 570 times more likely to be comprised of DNA from the defendant and an unknown individual, ….rather than from TWO entirely unknown individuals.

Were these conclusions and the corresponding statistics appropriate, ….or a product of cognitive bias? The answer to this question came many months later.

Long before the trial, an associate of the defendant mentioned to the investigators that he recalled being near the object in question, ….days before the crime, ….and *might* have actually touched it. This associate was NOT a suspect. He had a verified alibi for the entire day during which the crime was committed. Regardless of the alibi, the police rightfully collected a biological sample from this associate, and had it processed for his DNA profile.

The associate’s DNA profile was reported as follows:

Locus 1                               14, 16
Locus 2                               28, 28
Locus 3                               15, 16
Locus 4                               10, 13
Locus 5                               10, 10
Locus 6                               6. 10
Locus 7                              10, 13
Locus 8                              13, 13.2
Locus 9                               6, 8
Gender Locus                    X, Y

Virtually ALL of the associate’s DNA markers were present and accounted for within the DNA mixture profile found on the crime scene object. This provided overwhelming support to the associate’s statement to police that he *might* have touched the object.

It is vital to note that only six DNA markers could NOT have come from the associate. Considering that the defendant possesses ONLY THREE of those six DNA markers, it was suddenly quite apparent that an as yet unidentified person (not the defendant) had contributed the remaining DNA component to the mixture.

In subsequent reports, the “…570 times more likely…” analyst understandably avoided further references to the potential importance of the DNA mixture found on this particular crime scene object, ….hmmm, ….fascinating. Perhaps you have heard the old adage, "There are liars, .....DAMN LIARS, .....and statisticians."

The actual DNA results observed by forensic analysts resemble what some of us have seen on an EKG printout. While EKG stands for Elektrokardiogram (the Dutch/German version of the term), DNA data comes to crime lab analysts in the form of an electropherogram printout. Each DNA marker is visualized by the analyst as a ‘heartbeat-like’ peak, which rises above a low, squiggly baseline of ‘background noise’.

When analyzing evidence for DNA, the crime lab occasionally finds an extremely limited quantity of DNA available for typing. With such a limited DNA yield, the data printout often provides only a partial profile, rather than a FULL DNA profile. This is due to the fact that various DNA peaks are simply too small. Thus, the question for the OBSERVING scientist becomes, …. “Just exactly how tall does a REAL DNA peak need to be?”

The DNA peaks are measured in ‘RFUs’. All crime labs are compelled to conduct studies on their DNA profiling instruments/systems to determine an RFU threshold at which each potential DNA marker peak can be trusted as REAL DNA, … opposed to a meaningless artifact. Even when there is ONLY a modest amount of DNA, RFU levels for each DNA marker might be in the 100’s, 1000’s, or as much as SEVERAL 1000 RFUs.

Again, when the amount of DNA on a crime scene object is profoundly limited, the peaks might be all the way down near the threshold level for reliable interpretation. Under these circumstances—keeping in mind the ASTONISHING sensitivity of modern day forensic DNA detection systems—accurate, unbiased interpretations are of paramount importance.

Let us say that a crime lab establishes an ‘Analytical Threshold’ at 50 RFUs. The lab might set this threshold due to the fact that, when their instruments are running WITHOUT DNA, the analyst routinely observes a background noise level that fluctuates between perhaps 5 RFUs and 15 RFUs. Within these random events of static/noise, some unexplained spikes—similar to sudden gusts of wind—might actually reach 20 RFUs, 25 RFUs, ….and on rare occasions, much higher. To understand, the occurrence of random noise, spikes, artifacts, etc. one might attempt to learn more about Chaos Theory (Warning: Expect a migrane!).

Larger artifact spikes—often occurring during collection of DNA data from evidence samples—might be attributed to effects that include, but are not limited to: stutter peaks, nonspecific amplification products, pull-up peaks, dye blobs, electrophoretic spikes, or products of static/electrical interferences. Hmm, ....this is reminiscent of  Sheldon's electric can opener-induced fluctuations.

Based on the common observance of such random annoyances, forensic DNA analysts are commanded by crime lab management NOT to bother with analyzing anything below the 50 RFU analytical threshold. Such peaks are likely to be nothing more than spurious, elevated background noise.

Let us also suppose that a crime lab sets a ‘Stochastic Threshold’ of 100 RFUs. The term ‘stochastic’ is defined as ‘random’. Thus, any DNA data observed by an analyst BELOW 100 RFUs is substantially prone to stochastic effects. This means that—due to random, poorly-characterized effects—data below 100 RFUs might be noteworthy. However, the data may be ONLY PART of the DNA landscape at those various loci. The MISSING data is a consequence of the insufficient quantity of DNA present on the original sample collected from the item. Such absences of DNA data points are called 'Allelic Drop-Out Effects'.

The unfortunate reality that only part of the DNA data can be visualized, ….with an unknown portion of the data missing, ….creates a breeding ground for cognitive bias, observer effects, and catastrophic misinterpretations. This reality may represent the single most profound flaw in today's forensic DNA analysis laboratories.

Properly setting crime lab DNA threshold levels makes it quite simple for the analysts. When a tiny, insignificant peak is—let us say—only 32 RFUs, the analysts are commanded to ignore what is most likely nothing more than background noise, ….DO NOT ANALYZE. It is also simple for the analyst when a couple of peaks are—let us say—109 RFUs and 132 RFUs. Although these are substantially small peaks, they DO represent interpretable data.

The real challenge arrives when peaks are between the two thresholds. What the analyst MUST NEVER DO, is resort to anything resembling the following approach to interpretations:

“For the most part, I will ignore those peaks below 50 RFUs. Meanwhile, I will embrace all of the peaks rising above 100 RFUs. However, let me get back to you at a later time with my interpretation of any peaks measuring between 50 RFUs and 99 RFUs. This will give me time to carefully OBSERVE the DNA profiles from various known suspects. I will note which individuals happen to demonstrate consistency patterns with those questionable peaks. In fact, ….once I repeat that comparative exercise numerous times, I might even go back to some 47, 48, and 49 RFU peaks and take those into consideration as ‘something seems to be there’ or ‘that might be close enough to call it GOOD’—depending, of course—on what the suspect DNA profiles look like in the first place.”

Noting which individuals "happen to demonstrate consistency" with the evidentiary DNA fosters a fallacy that parallels 'noting consistencies' between the ENTIRELY UNRELATED Kennedy and Lincoln assassinations.

Ladies and gentlemen, such an interpretive approach to any investigation effectively characterizes the very essence of cognitive bias, backwoods sharp-shooting, and ….JUNK SCIENCE.

Michael J. Spence, Ph.D.

April 20, 2012