Fallible DNA evidence can mean prison or freedom
YOU are the juror: would you trust DNA evidence? Most people regard it as near infallible- it produces the right result or no result, exonerating the innocent and securing convictions where other evidence fails.
But DNA is not as objective as you might think. In the first of a two-part investigation, New Scientist reveals that much of the DNA analysis now conducted in crime labs can suffer from worrying subjectivity and bias. We asked forensic analysts to interpret a sample of real DNA evidence and found that they reached opposing conclusions about whether the suspect matched it or not. Our subsequent survey of labs around the world also shows that there are significant inconsistencies in the guidelines on how to interpret a sample. The findings suggest that the difference between prison and freedom could often rest on the opinions of a single individual.
The introduction of DNA evidence to the courtroom in the mid 1980s revolutionised forensic science, resulting in thousands of convictions and exonerating 255 wrongly convicted people so far in the US alone. The reason for more than 50 per cent of these wrongful convictions was unvalidated or improper forensic testing, including incorrect hair, blood or fingerprint analysis.
“It’s not unreasonable to hold up DNA as a way that the rest of forensic science should be done,” says William Thompson of the University of California, Irvine, and an occasional expert witness on DNA. “It is better validated, and often more carefully done and more rigorously interpreted than many areas of forensic science.”
That’s not the same as saying DNA is perfect, however. In a growing number of cases, DNA samples taken from crime scenes produce partial profiles, partly because smaller samples are collected. “Labs are trying to get more samples and they’re trying to [get results from] lower and lower amounts of DNA,” says John Butler, head of the US National Institute of Standards and Technology’s genetics group, which aims to improve standards in DNA testing.
A standard DNA profile consists of a series of peaks relating to the number of repeating stretches of DNA found in certain genetic sequences, or alleles (see diagram). The repeats occur at specific locations on the chromosomes, called loci, and there are two alleles at each locus- one inherited from each parent. The number of repeats in each allele varies widely between individuals, allowing a person to be identified this way. Labs in the US typically look at 13 loci, while UK labs tend to look at 10.
Yet in partial profiles, alleles may fail to show up, a phenomenon called “drop-out”. False peaks in the profile created by imperfections in the analysis machine may also be mistaken for alleles. This is called “drop-in”.
It gets more complicated when several people’s DNA is mixed (see “Mix and mismatch”). Butler has reviewed more than 5000 DNA samples from 14 US labs and found that mixing is a common occurrence: 34 per cent of the samples he studied included DNA from two people, while 11 per cent were three or four-person mixtures.
Interpreting alleles in a mixed or partial sample is where the subjective opinion of an analyst could play a part. To test this, New Scientist teamed up with Itiel Dror, a neuroscientist at University College London and head of Cognitive Consultants International, and Greg Hampikian of Boise State University in Idaho.
We took a mixed sample of DNA evidence from an actual crime scene- a gang rape committed in Georgia, US- which helped to convict a man called Kerry Robinson, who is currently in prison. We presented it, and Robinson’s DNA profile, to 17 experienced analysts working in the same accredited government lab in the US, without any contextual information that might bias their judgement.
In the original case, two analysts from the Georgia Bureau of Investigation concluded that Robinson “could not be excluded” from the crime scene sample, based on his DNA profile. (A second man convicted of the same crime also testified that Robinson was an assailant, in return for a lesser jail term.) Each of our 17 analysts independently examined the profiles from the DNA mixture, the victim’s profile and those of two other suspects and was asked to judge whether the suspects’ profiles could be “excluded”, “cannot be excluded” or whether the results were “inconclusive”.
If DNA analysis were totally objective, then all 17 analysts should reach the same conclusion. However, we found that just one agreed with the original judgement that Robinson “cannot be excluded”. Four analysts said the evidence was inconclusive and 12 said he could be excluded.
“Fingerprinting and other forensic disciplines have now accepted that subjectivity and context may affect their judgement and decisions,” says Dror. “It is now time that DNA analysts accept that under certain conditions, subjectivity and even bias may affect their work.” Dror presented the results at the Green Mountain DNA conference in Burlington, Vermont, last month.
Christine Funk, an attorney in the Office of the Public Defender for the State of Minnesota, says the results of New Scientist’s survey have profound implications for criminal justice. “The difference between prison and freedom rests in the hands of the scientist assigned the case,” she says.
Eric Buel, director of Vermont Forensic Laboratory in Waterbury agrees that there is a problem, although he doesn’t think it will apply to every lab. “I would be a little bit concerned if one person excludes, and one person includes him. At the end of the day, we all should come to about the same answer on this stuff.” Both he and Butler suggest that inconsistencies in analysts’ training may be partly to blame.
The problem of subjective interpretation could be further exacerbated by differences in procedure between labs. According to a second survey conducted by New Scientist, many crime labs set their own thresholds for how high a peak must be to demonstrate the presence of an allele, and these can be inconsistently applied.
New Scientist sent a questionnaire to crime labs in the US, Canada, UK and Australia. Of the 19 that replied, we found that four labs routinely allow analysts to use their discretion when interpreting peaks whose height is below their statistical cut-off. A further two said that although it wasn’t routine, there were circumstances when analysts could use their discretion. Fifteen labs said that they did not have a minimum requirement below which someone would be excluded from a mixture.
So what can be done? This year, the Scientific Working Group on DNA Analysis Methods (SWGDAM), which issues guidance to US labs performing forensic DNA analysis, published new recommendations regarding the interpretation of forensic DNA. These include a suggestion that labs develop strict criteria for deciding what denotes the presence of an allele, and what amount of DNA constitutes the minimum for a profile to be constructed. Labs should also document and define any assumptions used in the analysis of a mixture. “The bottom line is that you want to be as consistent and accurate as possible,” says Butler, who chaired the SWGDAM committee.
It seems lab managers would welcome consistent rules. Forensic lab directors at the 19 labs we surveyed also provided their views about how their analysis is currently done: 15 either agreed or strongly agreed that interpretation procedures should be based on national standards, and 11 agreed or strongly agreed that decisions over alleles should not be based on analyst opinion.
Labs must also take steps to avoid bias. Butler says that some labs continue to insist upon seeing suspect profiles before analysing evidence from the crime scene, which could lead to biased decision-making (see “Crime Scene Investigation: Impartiality”). Analysts also often know too much about a suspect and other evidence to be impartial, and public labs often have close ties to police. “Crime labs, including DNA labs, should not be under the control of a law enforcement agency,” says one US analyst, who wished to remain anonymous. “We are scientists, not cops or prosecutors.”
In our survey using the Georgia sample, respondents were blinded to contextual information about the case. Larry Mueller of the University of California in Irvine says we may have seen different results if the data had been presented to them by police officers or prosecution lawyers. “The difference between you giving them the data and saying ‘what do you make of it?’ and the local district attorney giving them the data and saying: ‘We’ve arrested someone, is his profile in here?’ is huge,” he says.
Bruce Budowle, a former head of the FBI’s DNA lab, would also like to see labs employ a second analyst to review initial conclusions, and all of this data be made available to defence teams.
Eighteen of the labs that we surveyed said they already conduct independent reviews. However, in the majority of cases, the reviewer is allowed to see the first analyst’s conclusions, as well as the original data. “Technical peer reviews are a good step, but I can point to several examples where peer-reviewers just rubber-stamp the cases,” says a different US analyst, who works in a private DNA lab that carries out case work for the police. In the case of a disagreement, 15 of the labs said a supervisor would be called in to make the final call, but only two labs said that this disagreement would be documented in their final report.
Still, when done correctly, DNA remains a powerful tool for fighting crime, says Budowle. “Are there cases done wrongly? Absolutely, there have been cases,” he says. “Are they the vast majority? No, it’s a small number.”
Buel adds that some DNA labs are beginning to accept that improvements need to be made. “DNA analysis is still relatively young. It may take us a while to make sure we’re all playing by the same set of rules,” he says. “At the end of the day, I think we’re all interested in making sure we get the right person. DNA is headed in the right direction, but as in any field, you can make it better.”
Mix and mismatch
Mixed-up DNA from crime scenes already causes headaches for analysts; now it seems it can even be difficult to tell how many people’s DNA is present in a sample.
Dan Krane of Wright State University in Dayton, Ohio, and his team took 959 full DNA profiles and modelled all the possible three and four-person mixtures that could arise from them. They found that 3 per cent of three-person mixtures could be mistaken for those of two people, and more than 70 per cent of four-person mixtures could be mistaken for two or three-person mixtures (Journal of Forensic Sciences, DOI: 10.1520/JFS2004475).
“If you can’t determine how many contributors there were, it is ludicrous to suggest that you can tease apart who those contributors were or what their DNA profiles were,” says Krane.
Crime scene investigation: Impartiality
Sarah Beazley woke up with a hangover and her clothes in disarray. She had attended a party the night before, where she admitted to getting high on drugs and alcohol. Although she couldn’t remember the details, she suspected that she may have been sexually assaulted. When Sarah (not her real name) went to the police, they found traces of saliva on her breasts.
Sarah’s story- a real case in California- provides a good example of how DNA analysts often use subjective judgements to interpret evidence, depending on the case information they have been told. As in many cases, the DNA profile from the saliva sample (shown below) was incomplete: the identifying peaks could only be obtained for three sites, or loci, on the DNA. On a complete sample, US labs would look at 13 loci. The number of repeating DNA sequences at each of these loci varies between people, and so analysts can eliminate or implicate suspects if their peak profiles line up with those in samples taken from a crime scene.
In Sarah’s case, there were four possible suspects: here given the pseudonyms Tom, Dick, Harry and Sally. Each provided DNA swabs, and their number of repeats at each relevant locus was recorded.
To test how analysts would interpret these samples, William Thompson of the University of California in Irvine and Dan Krane of Wright State University in Dayton, Ohio, who have both acted as DNA expert witnesses, presented the information to around 1000 analysts at various forensics meetings. Thomson would vary the identity of the suspect and contextual information about the case. He found this produced a variety of answers (Law, Probability and Risk, DOI: 10.1093/lpr/mgp013).
In the first instance, analysts were showed only Tom’s sample, and told that he was the suspect. Thomson also said that there were concerns that Tom’s profile couldn’t account for all of the peaks seen in the saliva sample: his profile doesn’t quite match up at locus 1 or locus 3. “At that point, several analysts interrupted to say that I did not know what I was talking about,” says Thompson. The interruptors explained that the non-matching peaks were just random fluctuations caused by, say, a flaw in the analysis machine. Conclusion: it could be Tom’s saliva.
However, when a second group of analysts were presented with just Dick’s profile and told that he was the suspect, they concluded that the “flaws” dismissed by the first group were true peaks. So Dick could be a match too.
Thompson then presented the case to a third group of analysts, this time showing only Harry as the suspect. He heard a similarly subjective story. Harry didn’t match at locus 1, but some information could have been lost from the saliva sample, the analysts said. And some of the repeats at locus 3 must be hidden by an artefact or other DNA sample in the saliva.
Finally, Krane presented Sally to a fourth group. Even though one of her alleles at both locus 2 and locus 3 did not match, analysts insisted that she could not be excluded, in part because the evidence sample could be a mixture of more than one person’s DNA.
Although these informal experiments do not necessarily reflect how labs interpret evidence on a day-to-day basis, “about a third of the cases that I’m asked to review have elements of the Tom, Dick, Harry and Sally story,” says Krane.
So who was convicted in the real case? The answer is Tom.
Additional research by Charlotte King and Caitlin Stier