Option 02-1

The high cost of false negative and false positive results

Jun 20, 2019 3:45:58 PM

In the world of diagnostic testing, limit of detection (sensitivity) matters a whole lot

The Theranos debacle has revealed a number of things about the business of diagnostics, not the least of which I have called “the LDT loophole”. https://singleraoncology.com/theranos-the-fda-and-the-laboratory-developed-test-ldt-loophole/ This post certainly got several laboratory director friends of mine a bit irritated, knowing with how much integrity they applied their skills in the context of bringing up a laboratory-developed test. While finishing up John Carreyrou’s well-researched book Bad Blood: Secrets and Lies of a Silicon Valley Startup, some of the most damning information comes out at the end.

Information in this book is revealing, such as the number of tests whose results had to be cancelled, the patients informed and retested, on top of all the legal costs of this fraud. Other information such as the amount of voting shares Elizabeth Holmes had, which rendered her Board of Directors powerless. And information about her ability to insinuate herself to people important for her ends, and then persuade them to join her cause; and for others, to be ruthless and Machiavellian in her grip of power. One friend of mine asked an open question: what if Ms. Holmes used her skills of persuasion to go into sales instead of becoming a CEO?

And yet the book fell short in several ways, not the least of which is in the technical details. Mr. Carreyrou has a reporter’s nose for names, places and financial fact-checking, but do not read the book for learning much about their technology nor their methods other than looking up a manufacturer of gluing equipment in New Jersey to figure out what kind of liquid-handling technology Theranos adapted for the Edison instrument. There is the Seimens ADVIA reference, but only as an aside, in terms of how that system was ‘hacked’ in order to use diluted samples. It goes without saying that this is clearly not a normal situation with diagnostic equipment.

Another way the book fell short was in capturing the human side; Dr. Ian Channing’s death by suicide seemed clinical in its description, and the physicians in Arizona who were doubtful of the accuracy of the Theranos test were also treated as interchangeable parts. The harm to patients due to Theranos was real, and these stories are found elsewhere.

The Dropout podcast on Theranos

In March the US news network ABC Nightline published a series of 45-minute podcast episodes called ‘The Dropout’ a result of three years of investigative journalism, digging into the personal stories the book did not capture or described only in passing. Two exclusive video interviews of patients from the podcast were excerpted here, and in their own words describe the feeling of getting false-positive results. One was a breast cancer survivor Sheri Ackert, describing her experience when she received her Theranos results where a hormone estradiol was measured to be 313.5, indicating a return of her cancer; another healthcare entrepreneur Pallav Sharda received results indicating he was pre-diabetic. Upon conventional testing, both these cases turned out to be false-positives.

Also telling is the story (and the visual component of her video is compelling) of one of the Theranos whistleblowers Erika Cheng, who states how the tests “kept failing over and over and how they handled it blew me away: they took out datapoints” when the controls they were running did not pass their own internal tests.

The Theranos example may be a rarity, in terms of both the size of the fraud (Theranos went through some $900M in investor funds before dissolving) and the secrecy around its technology. It is common for technology to be developed in secret (it is a very competitive business after all) but the technology is eventually evaluated and examined via peer-review in reputable journals before the product is launched, typically as a Research Use Only device, on its way to FDA clearance.

The Theranos debacle illustrates two cases of false-positive results; the anguish and alarm that patients feel after a diagnostic test comes back positive is real. There are also genuine costs of over-diagnosis and over-treatment that can occur at great expense. Yet of the million tests (!) that Theranos provided over the course of the years they were in business (yes it was that many tests that had to be recalled and rerun) you do not hear of the stories of the false-negatives, the individuals who were given a clean bill of health by a Theranos test but may have had a life-threatening condition.

In the cases of these false-negatives, individuals were given test results that indicate a normal condition. What about other breast cancer survivors whose estradiol results were indicative of cancer recurrence (and thus need to be put on another course of therapy ASAP), or someone with a serious infection that went undiagnosed? 

It is both false-negative results as well as false-positive results that are of paramount importance.

Is what lies beneath an iceberg?

In diagnostic testing the Limit of Detection (LoD) is also termed sensitivity, and this a key parameter. Clinical laboratory testing has strict protocols to evaluate new tests that come to market under CLIA (here is a review paper of the kind of parameters for an LDT), including linearity, analytical sensitivity, precision (i.e. measuring variation over time), specificity (including interfering substances) and accuracy (measurement via comparative, alternate methods).

Determination of what is ‘normal’ has been honed over many decades and is naturally tied to the practice of medicine. In the realm of circulating tumor DNA (ctDNA), its biology is abnormal, and thanks to its inherent heterogeneity different cancers shed ctDNA at different rates. While there is a trend associating stage of cancer to amount of ctDNA, there are examples of patients with advanced cancers who have little ctDNA present, and other patients with early stage cancers with an abundance.

It is difficult to overstate the importance of LoD – what you don’t see you cannot act upon. A colleague at Sysmex Inostics put together the following chart that shows the LoD’s of five different liquid biopsy tests offered to the market for NSCLC: Competitor R (real-time PCR-based), Competitor F, G and I (all NGS-based), and Sysmex-Inostics OncoBEAM Lung (digital PCR-based). At a confidence interval of 95% (and where indicated 90%), the Competitor R test is 1.7% - 6.7% MAF, Competitor F's LoD is not reported (although it seems to be reliable down to 0.5%), both Competitor G and I have an LoD of 0.3%, and Sysmex-Inostics (based on digital PCR) is 0.04%.

Screen Shot 2019-06-20 at 4.56.09 PM


You read that correctly – 0.04% - or four parts in ten thousand. If there are ~3000 genomic equivalents in 2 mL of plasma, that is (an average of) 1.2 molecules detected in 3000 genomic equivalents at a 95% confidence interval. At this level of single-molecule detection, you are approaching the limitations of stochastic sampling.

Recent publications illustrate the importance of very low-frequency ctDNA mutations. For example, here is a metastatic colorectal cancer paper from 2017 demonstrating concordance between tissue- and Sysmex-Inostics BEAMing technology; of the 50 samples testing positive for RAS mutations, four were less than 0.3% allele frequency. Here is a recent review (April 2019) looking at ctDNA in colorectal cancer (CRC) titled “Circulating Tumor DNA Analysis in Colorectal Cancer: From Dream to Reality”, a nice overview of the current state of ctDNA testing in colorectal cancer, and specifically comparing RAS mutations (both KRAS and BRAF).

The informative figures from this review are here, and it is to be noted that Sysmex-Inostics has commercialized the Hopkins-developed Safe-SeqS methodology described in this 2011 PNAS paper entitled “Detection and quantification of rare mutations with massively parallel sequencing”. Rebranded as SafeSEQ and offered as a laboratory-developed test out of Sysmex CLIA-qualified laboratories in Baltimore MD and Hamburg Germany, tests for Breast Cancer, Colorectal Cancer and Head and Neck Cancer are available.

ctDNA measurements of KRAS for cancer monitoring


 A logical application for application of cell-free tumor DNA analysis in addition to treatment selection would be monitoring for cancer recurrence, or for monitoring therapy resistance such as the EGFR T790M mutation referenced earlier. In both cases, for monitoring for cancer recurrence or therapy resistance time is critical – you want to detect subtle mutations (as mentioned before, as little as one mutation in a background of several thousand wild-type sequences) as soon as they appear.

In this 2018 Annals of Oncology paper, called “Repeated mutKRAS ctDNA measurements represent a novel and promising tool for early response prediction and therapy monitoring in advanced pancreatic cancer”, a group at the Technical University of Munich used a KRAS BEAMing test to compare mutant KRAS ctDNA as a biomarker for adverse prognosis compared with three currently-accepted protein-based biomarkers for advanced pancreatic cancer patients receiving chemotherapy. Analyzing 284 samples from 54 patients, the abstract states the following:

“During therapy, changes in mutKRAS ctDNA levels were more rapid and pronounced than changes in protein-based tumor markers. A decrease in mutKRAS ctDNA levels during therapy was an early indicator of response to therapy, while there was no significant correlation between kinetics of CA 19-9, CEA or CYFRA 21-1 and response to chemotherapy during the first four weeks of treatment.”

To learn more about our SafeSEQ technology and how it can be used in your biomarker screening, click here.



No Comments Yet

Let us know what you think