Gaming the system – why clinical trial results falter in the real world

@LizSzabo tweeted out this abstract from ASCO reporting that adverse events from Keytruda therapy were much higher than expected from clinical trial results. My retweet added a snarky remark about how pharma has gotten pretty good at picking its patients for trials and that this result should be no surprise at all. Let me expand on this a bit more than is possible in 140 characters.

To some extent, a dropoff in performance from clinical trial to real-world experience is inevitable. Much of the dropoff is due to human factors. When organizing a trial you recruit leading practitioners at prestigious hospitals. The expectation is that these practitioners (called thought leaders by marketing departments) will give favorable and enthusiastic endorsements of your product. After all, they want to be associated with successful new treatments as a way of advancing their own careers.

Careerism aside, these folks have the capability and motivation to provide the best care possible. You can also count on them to implement best practices for using your product as specified in excruciating detail in your trial protocol. But you don’t just count on this – you provide pre-trial training, and you periodically send out trial managers to audit sites, confirm that they are adhering to protocol, and provide remedial training – or terminate the site – if they are not.

There’s nothing the least bit nefarious about this. If investigators don’t adhere to the protocol, then your data risk being contaminated by random noise and becoming worthless.

But the fact is that once the product is launched, it will be used by practitioners who are less skilled, less conscientious and less well-trained. These human factors will result in poorer performance There is just no avoiding this. Well there is – but it would require incorporating robustness analyses (like robust parameter design) into clinical trial protocols. This approach works great for making cars and airplanes that don’t fail even when operated incorrectly. I’m not sure the clinical world is ready for it. Yet.

But there are more insidious ways to stack the odds in your favor. Like politicians picking their voters through gerrymandering, drug companies pick their patients. Their principal means for doing this are through site selection and inclusion and exclusion criteria.

Let’s take a look.

The KEYSTONE-010 trial for treatment of non-small cell lung carcinoma was critical in showing the superiority of Keytruda over standard docetaxel therapy. It is a reasonable comparator for the ASCO abstract, which also reports on NSCLC patients.

Average age for KEYSTONE-010 was 63; for ASCO it was 69. Right there you would expect more complications and problems. Older patients never do as well as younger ones.

I don’t have the exclusion and inclusion criteria for the ASCO abstract, but they are available for KEYSTONE. Here are some highlights:

• ECOG score (a general measure of health – 0 is healthy, 5 is dead) must be 0 or 1 for inclusion. This criterion excludes the sickest patients.
• Organ function must be normal. Excludes patients with kidney or liver problems.
• Excluded if expected to require other neoplastic therapy, or had received cytotoxic therapy within 3 weeks (for chemotoxins) or 6 months (for radiation therapy). This excludes patients with fast-progressing disease.
• No prior malignancies
• No brain metastases. Excludes the most hopeless cases with advanced disease.
• No active infections. Excludes patients whose immune function has been compromised by prior chemotherapy or other diseases. Since Keytruda works by activating the immune system, this excludes a class of patients least likely to respond.
• Patients with any “history or current evidence of any condition, therapy, or laboratory abnormality that might confound the results of the trial”. A catchall for exclusion.
• Taking any CYP3a4 inhibitors. Many small molecule drugs interact with this liver enzyme – some are activated, others are deactivated (this is why you are not supposed to drink grapefruit juice, a CYP inhibitor, with many meds). There is no reason to expect monoclonal antibody drugs like Keytruda to interact with cytochromes. But the strongest inhibitors of CYP3a4 are antiviral and antifungal medications. By excluding these patients, you also exclude AIDS patients and others likely to have compromised immunity.

All of these criteria are supportable. But every single one of them works to exclude patients most likely to suffer adverse events and least likely to benefit. They work as a sorting mechanism to enroll the most promising candidates and exclude the least promising.

You might expect then, since some classes of patients have been excluded for certain conditions, that these conditions would constitute a contraindication for therapy. But you would be wrong. Here is the list of contraindications for Keytruda, reproduced in full from the package insert:

In trials, doctors were explicitly instructed to exclude the weakest patients, those with the worst prognoses, those with weakened immune systems.

But in real life, what doctor is going to say to a patient “We have this new breakthrough drug but I’m not going to give it to you because you are too sick and too likely to die anyway”? You’d have to be a monster.

And you’d have to be averse to getting paid. Unlike most doctors, oncologists routinely act as pharmacists, selling IV drugs (like Keytruda) at a markup.Given the astronomical costs of these drugs, even the 6% Medicare markup is significant.

Health journalists routinely pounce on the catnip of press releases trumpeting breakthrough clinical trial results. The better sort also pay attention to post-marketing studies. These are rarely as dramatic, but are far more indicative of real-world performance. And of just how hard it is to make medicines that really help people. We need more of this kind of journalism.