My Failure Resume, redux

Well, it is time to update my resume. It has been a year, I have failed at more things. I’ve read more failure resumes, and I like some of the newer ideas, for example, listing your NON-skills. I’ve added mine.

One idea for brave souls willing to try, is to submit both your Regular CV / resume AND your Failure resume to your next job interview. Here are 1 page versions of mine (regular / failure).

And, wouldn’t you know, the most popular post on this blog, after 3 years of weekly writing on aspirational topics in informatics?

My original Failure Resume. Go figure.

I love some of the writing out there on Failure Resumes:

  • Stanford Engineering: “come to terms with the mistakes … made along the way and … extract important lessons”
  • Forbes: Of 10 job applications, received 0 responses to traditional resume, but 8 responses for a traditional resume PLUS a failure resume.
  • Inc.com: why to encourage your employees to make a failure resume.
  • Even Einstein struggled: a scientific paper on how describing Einstein’s struggles to science students increased students’ hopefulness and engagement with science class.
  • ScienceAlert.com: A CV of failures is an entertaining and instructive read

CMIO’s take? I’ll be teaching an Informatics Leadership course soon, and will expect all our participants to write a one-page Failure Resume. Join us!

EHR v Covid-19: MDPOA (power of attorney) & Advance Care Plans and the Covid BUMP

Full open-access paper here. https://preprints.jmir.org/preprint/21385/accepted

What I love about working in an academic health center is the luxury of being surrounded by people smarter, and more hardworking than I am. Here are Drs. Portz and Lum analyzing our data on the capture of patient’s Advance Care Plans – ACPs (including the Medical Durable Power of Attorney MDPOA) and other documents online via our patient portal.

We believe we are among the first in the country to offer the ability for patients to complete this online and designate a medical decision-maker in the event of their incapacity. Furthermore, we now accept photos of documents (easy and convenient via our patient portal app integrated with a smartphone camera — hooray modern tools for modern medicine) into the patient chart, and can see signatures, names, contact information, and details of MDPOAs, Living Wills and other ACPs.

And, during the anxieties of the pandemic, we had a significant uptick in patients completing the MDPOA and uploading images.

CMIO’s take? Another publication for our smart colleagues — good. Better patient care — great.

EHR v Covid-19. Taiwan, EHR and effective pandemic response leads to economic growth

image from Statnews

https://www.statnews.com/2020/06/30/taiwan-lessons-fighting-covid-19-using-electronic-health-records/

It is fascinating, inspiring (and disappointing) to see effective responses to the Covid pandemic from other countries. Great partnerships and effective connection of governmental leadership, industrial production, and healthcare information can combine to combat the pandemic.

Taiwan has had only 446 cases and 7 deaths, for 24 million residents, since the start of the pandemic, despite their proximity to, and the frequent travel and many flights to and from China.

Dialing in to an Aging Parents Telehealth Visit… Why aren’t more of us Doing it? (Guest Blog: Glenn Sommerfeld)

I forgot about my father’s memory and neurology clinic visit even though I had promised to go down to Denver with both of my parents to help them navigate the complex world of healthcare four months before.  A lot changed in those four months, most notably COVID-19 swept across the world and made its way into the US.  The pandemic placed my aging parents at a greater risk if they contracted the virus while traveling from Fraser, Colorado to Denver and my work schedule was beyond capacity as I added Federal and State COVID-19 reporting coordination to an already full project portfolio.  How could a take a day and a half off work?  How could my parents stay safe?

Telehealth and Rural (Mountain) Living

I decided to move on from my first health care job in neurophysiological monitoring to acute care in 2011.  I also wanted to move to the mountains of Colorado.  My parents already moved from Colorado Springs to Fraser, just outside of Winter Park, Colorado.  Yampa Valley Medical Center brought me on as a quality analyst before they were part of the UCHealth system.  After moving to Steamboat, I realized how remote and isolated Steamboat Springs, Colorado was from Denver and the other “Front Range” cities in Colorado.  Here are some fun facts about driving from Steamboat for medical care:

  • Steamboat Springs to University of Colorado Hospital and the Anschutz Campus
    • 169 miles
    • 3 Hours and 10 minutes if traffic is good
    • One major mountain pass (or two if Eisenhower Tunnel is closed)
  • Steamboat to Poudre Valley Hospital
    • 159 Miles
    • 3 Hours and 21 minutes if traffic is good
    • Two major mountain passes or the choice to leave Colorado, go to Wyoming and drive back into Colorado so you only have to deal with one major mountain pass (adding on 30 more miles)

Many specialists come up to mountain communities on a rotational basis.  However, this may be once a month and possibly less frequent.  Telehealth is the obvious stop-gap for patients in rural and mountain communities that need specialized care.  A barrier to telehealth visits as Dr. Lin has mentioned in his blog has mostly been the providers.  However, with social distancing and with CMS lifting restrictions on reimbursement for telehealth, providers quickly adopted telehealth to keep revenue streams flowing for their practices.

Telehealth and Telemedicine Expansion and Deregulation

Telehealth and telemedicine rules and regulations relaxed at the start of the COVID-19 pandemic.  Now is the time to figure out how else to utilize technology to improve healthcare delivery.  Now is the time for innovation and policy reform.  So, how can telehealth help patient advocates and family members?  Could it be the answer for me and my dad’s visit?  Will it work for others in an urban setting or family members that are geographically separated?

Being a Patient Advocate Remotely

Before the pandemic, I had planned on taking a day off of work to drive down to Denver to accompany my father to an appointment at a neurology clinic.  This appointment transitioned to a telehealth visit following the outbreak.  I considered making the two-hour drive from Steamboat Springs to Fraser to be with him for the appointment.  After all, I would generate a net gain of two and a half hours from not having to drive all the way to Denver.  In a moment of clairvoyance, however, I decided to find out if I could join remotely.  After working with a few key stakeholders at UCHealth, we discovered that if my father gave me access to his My Health Connection account, I could join the same way he would for the remote visit.  This access also allowed me to review my father’s medications as the provider discussed them with my mom and dad and access the summary notes from the visit, so I could discuss treatment options with him and my mother at a later time.

The Visit (that’s me at the bottom, by the menu bar)

It was strange to know that I would be on a video call with my parents, but to be on the phone with them as well, ensuring that they could log on.  My wife and I have discussed the shift in caring for both sets of aging parents, but this was the first time I needed to support them on multiple fronts.  First working with them on technology and second being a health advocate.  The visits felt distant, yet at the same time normal.  The medical assistant greeted us virtually and started the intake process.  Dr. Zachary Macchi jumped onto the call about five minutes in and reviewed history and started the evaluation.  About twenty minutes into the call, Dr. Samantha Holden was able to join as well.  In the span of twenty minutes a total of six people (including my father) were working together.  Had we all gone down to Denver together, this may have been the same outcome.  However, Dr. Macchi joined the call first to help Dr. Holden.  He stated right away that she would be able to join us, but had other commitments.  My guess is that if we were in a traditional setting, we would have waited an extra 20 minutes but telehealth gave the flexibility for coverage.  Telehealth has its limitations.  My father had difficulty following the motor skills test.  We were unsure if it is his motor function or his ability to follow a two dimensional image in the three dimensional world.  For this and other reasons, everyone agreed on an in person visit three months following the virtual visit.

Just the first step… what are the next.

This visit made me realize the opportunity for telehealth in the patient advocacy realm.  While telehealth offers a convenience for the patient, it certainly helps with obstacles that patient advocates face.  I am lucky to live just a few hours drive from my parents.  If I lived outside of Colorado, I doubt I would be as involved in their care.  However, we now have the tools to improve care coordination between family members.  Our first step needs to be promoting the technology to allow for remote patient advocacy.  However, we could take it even further.  What if we could have an MA set up a camera during an in-clinic visit so the advocate (or family member) could join the visit if they lived too far away to join in person?  What are the other ways to utilize telehealth for family members and patient advocates?  Will CMS go back to restricting reimbursement for telehealth?  Time will tell for these questions, but we need the health care community to (dare I say) advocate for telehealth and the access it can bring for patient advocates.

cid:image005.png@01D645B2.ED7150D0
Guest Blogger Glenn Sommerfeld (thank you!)

EHR v Covid-19. What age groups do well with Video Visits?

Time for more data surfing! UCHealth’s overall visit volume (including in-person and video visits and scheduled phone visits) have recovered about 80-90% of pre-pandemic levels.

Today, we’re looking at visit volumes among different age groups of patients. Keep in mind, UCHealth is primarily an adult hospital. Our partner, Childrens Hospital of Colorado, sees most of the pediatric population regionally. We do have some pediatric practices, and of course our extensive family medicine primary care practices also see pediatric patients. This will explain the low volume of pediatric visits below. On the other end, only 3.9% of UCHealth patients are over age 85.

So, what happened to visit volume with each of these age groups?

Turns out, the curve for EVERY age group is similar! Green is age 40-65 and about 1/3 (our largest fraction) of our patient population. Fuchsia is 65-85 and our second largest, purple is 18-40, orange is under 18, red is over 85. The curves start at different points, but follow the same trajectories. That divot on the right side is Memorial day, clinics closed, so 4/5 of the weekly volume that week.

Here is the Home telehealth Video Visit volume! Some interesting findings here. You notice that fuchsia and purple switched places, meaning that a much higher proportion of 18-40 year old patients chose Video Visits compared to 65-85 year old patients. All the other curves stayed in their relative positions. Furthermore, EVERY age group had a proportional bump up in video visits, even those over 85! Finally, the video visit curve is falling back, about to 50% of the peak (so far). It will be interesting to track this in the coming month or 2 and see where we end up, after in-person visits are fully ramped up again.

CMIO’s take? Who knows? Another example to show that we are going to bed with a cliff-hanger every night. I wonder what happens next. The good news: I’m feeling good about having a better handle, even after a few short months, of what Covid-19 can throw at us. Ain’t data cool?

EHR v Covid-19. Where’s Covid now? And, patient care is already looking different!

Covid-19 RNA positive tests at UCHealth in purple

We are well into our fourth month of this pandemic. Looking at our graph, purple shows influenza B peaking in December, influenza A peaking in February, and leaving aside an artifactual spike in mid March, when we started co-testing for major respiratory viruses at the same time we started testing fro Covid-19 in earnest, all other viruses have dissipated. Then you see this impressive bump in Covid-19 illness, peaking in mid April, in our organization. Keep in mind, this is just POSITIVE tests for Covid-19 RNA in patients seen at UCHealth. Because we care for 1.9 million patients in Colorado, though, it is a reasonably large population sample. Furthermore, Covid-19 tests were SCARCE prior to mid March, and numerous patients were likely developing Covid symptoms in February (see below).

Along left edge, top to bottom lines: In-person visits, online patient messages, phone calls, video visits, scheduled phone visits

So, how has this affected our visits and our telehealth efforts? Purple shows you the dramatic dip with in-person outpatient visits, and the gradual climb back toward baseline. Then there is the green line of home telehealth video visits, going from nearly nothing to about 20,000 weekly in early to mid March, with gradual falling off in the past 8 weeks and it seems we might stabilize near 10,000 visits weekly. This is still about 100x the volume of video visits prior to the pandemic.

Then there are the other trend lines that are interesting: Red is the ongoing volume of Patient messages before and during the pandemic. Leaving aside the bump in mid May (not sure why: perhaps related to a system-broadcast), our baseline of 22,000 messages per week increased to 30,000, about 33% increase in volume, starting to rise on Feb 22. This pre-dated by THREE WEEKS the steep decline of in-person visits and the upswing of telehealth visits on Mar 14, and the Colorado Stay at Home order of Mar 26.

Even more interesting: telephone volume in blue, saw a tiny bump on Mar 14, but then was unchanged during the entire period. By contrast, in fuchsia Scheduled telephone visits (billable as of mid March per CMS rules), appeared in early April.

In one graph, you can see: online patient messaging demand scaling up, phone calls being static, scheduled phone calls appearing when billable, on top of the change for in-person and video visits.

Some hidden factors at work here: UCHealth set up a Covid-19 nurse advice line; those calls are not visible on any line in this graph, and those hard-working nurses took tens of thousands of calls from Coloradoans (not just UCHealth patients).

So, this data dilettante has to ask, could an increase in online patient messaging (regardless of content of message) be another possible leading indicator for future pandemic surges? We can’t be sure if these messages were about general anxiety, Covid symptoms, or perhaps completely unrelated, but it is suspicious that there is a sustained increase in volume of messages by 30%+ since mid-March. On the other hand, why isn’t online message volume falling, like home telehealth visits are falling, now that clinics are opening up in-person appointments? Stay tuned!

The open question now is: what will CMS (Centers for Medicare/Medicaid Services) do with paying for Video visits and scheduled Telephone visits? Will those payments stop or scale back? This will certainly affect all health systems still heavily relying on Fee for Service, until the rise of Value Based Care (insurance plans paying for Quality instead of Volume) takes over.

CMIO’s take? These are unprecedented times, and patient behavior and health system behavior is fascinating. A tiny RNA virus has changed the way (phone, online, in-person) patients and healthcare providers interact. What comes next?

EHR v Covid-19. Leading Indicators: work-in-progress, VERSION 2

From nounproject.com. Leading indicators may seem like astrology, and tarot cards. BUT there is science too!

Thanks to those of you who caught my non-displaying graph images, I’m reposting now converting my original PNG to JPG. Please let me know if you can see these and follow the reasoning below! (edited 6/15, CTL)

—–

Thanks to Brendan Drew, one of our data scientists, who is diving into the analysis of Leading Indicators, for the graphs and reasoning below. If I can twist his arm for more graphs, will pass them along.

If you recall, I discussed this recently: the idea that, our future is uncertain. Even though we have survived the first wave of the Covid-19 pandemic, we are concerned about possible future waves. How might we prepare?

If you don’t know this about me already, I find “making the sausage” in informatics and data science fascinating. Here are some intermediate steps we are taking beyond my “data dilettante” days as we search for signal in the noise.

These are all COVID-19 new codes. Firstly, note that ORANGE line R68.89 , orange shows up WAY before March. Turns out, this is not only “suspected Covid-19” it is also “Other Symptoms and Signs” previously in the ICD10 dictionary. So, that is a terrible signal. Then, RED line Z20.828 “Close exposure to COVID-19” is also “Exposure to influenza”. Hmm. Then, BLUE line B34.2 “Coronavirus Infection” is also “Coronavirus, unspecified.” Also Hmm. Only GREEN line U07.1 “Coronavirus identified” is highly specific for COVID-19 in the graph.

So, how do we make sense of this?

First, we take ONLY hospital patient codes for CONFIRMED (BLUE) versus SUSPECTED Covid (ORANGE), and we see that the BLUE CONFIRMED line shows two peaks, whereas ORANGE, there is no real signal there at all. GREEN is adjusted for Market Share based on 2019 data for that zip code (we are trying to localize prediction to the Zip code level).

Now, we compare zip codes. Blue line is 80011, Aurora near University of Colorado Hospital, a relative hot spot in Denver Metro region, and orange is 80634, the hot spot near Greeley hospital, and we see a temporal difference in the onset and peak of Greeley being earlier than Aurora. Interesting.

Here is where it gets tantalizing, and we have to hold back our excitement: Pair up the outpatient symptom data with the inpatient hospitalization rate for Confirmed Covid. Here it is for Aurora, x-axis lined up by date:

Those of us who cannot contain our excitement will see a visual rise in RED (outpatient symptoms suspicious of COVID, like fever, cough, shortness of breath), in the 80011 zip code increasing about 2 weeks BEFORE the corresponding rise in COVID-19 cases at University of Colorado Hospital in Aurora (also 80011). We WIN! Right?

Also, here’s the corresponding graph for Greeley:

This is a bit messier: what is that symptom peak in February? There is no corresponding COVID hospitalization peak in Feb/Mar. BUT, the symptom peak in mid March DOES correspond to a rise and peak in late March, and all of April.

My theory: mid February was probably Influenza A, and we did NOT track hospitalizations on our graph for that, AND the COVID confirmed codes did not get implemented until mid March, and maybe NOT attached in retrospect to patients who MIGHT have had COVID, but were admitted BEFORE those codes went into effect. This is harder than it looks!

Are you looking for a final answer? SORRY! We are still cranking away at this. Even though we humans have frontal lobes that CANNOT WAIT to see patterns (even where there is no pattern!), we have to resist that urge. AND, how do you teach an algorithm (even if there IS a pattern here), to tell us: YES you should pay attention to THIS rise in the data, but THAT ONE is just random noise.

For example, imagine the 80011 graph prints out one day at a time, moving to the right. At what point, would you tell the algorithm to alert us: YES it is TIME TO BRING IN MORE DOCS AND STAFF FOR THE NEXT SURGE.

Would it be: March 15, when there is an uptick? But there are lots of upticks just like that. March 22, a week later, when the line is DOUBLE of the average from 0.0007 to 0.0014?

AND, worse yet, UCHealth is only one of 5 health systems in Metro Denver and across the state of Colorado. Will cases come to US or to other health systems? What will the peak be? Will it be a tiny peak? (Hey, CT, why did you call all of us in here for these dozen patients?) Will it be a HUGE peak (Hey, CT, you didn’t raise enough of an alarm, there still aren’t enough of us).

Finally, signal to noise MIGHT be easier for the summer months when Influenza is done, but what about the fall when Influenza B and many other viruses are back in action? What about seasonal allergies during spring and summer that might kick off cough and shortness of breath?

CMIO’s take? Figuring out Leading Indicators is HARD. If YOU have this figured out, let us know. We’re still working on it. But the math and the figuring-it-out is pretty fascinating in the mean time.

All y’all EHR-using folks don’t know how good you have it.

The author, Chief Medical Information Officer of 3 million paper medical records.

For fun, I’ve set my Zoom background with an actual vintage 1997 photo I took of the medical records room in the basement of University of Colorado Hospital on Ninth Avenue in Denver (back when giants walked the earth). This aisle featured 6 stacked rows of medical record charts AND piles of paper record folders ON TOP since we were out of room (not shown). This was one of 29 aisles of records in the Records Room, holding ONLY the latest 3 years of records: the rest were retained (for 27 years) in a downtown warehouse.

Fun fact: we turned down lots of innovation partnerships and offers of free services because the medical information locked in those paper records was too difficult to pull out:

  • We have a Pulmonary Function mobile van parked out front: send us all your patients who currently smoke and we will screen their lung function for free!
  • Hey, our insurance company will pay you a bonus payment if you can prove all of the patients who have had a previous heart attack are taking aspirin! (true story, a clinic trying to prove this using paper medical records and clerical staff paid more gathering the data than they received in bonus money)
  • Quick: the mobile mammogram bus is coming next week: let’s call all our patients who are due for mammography screening!
  • We have a new diabetes educator visiting for a couple weeks! Can we contact all our patients with diabetes to come for a free visit?
  • Uh, oh! The medication Bextra is being recalled by the manufacturer; quick: call all our patients taking that medication! (True story: 1/2 of our clinics were able to run a report on our EHR at the time and call affected patients immediately; the other half, still relying on paper records, had to say… “well, when the patient calls for a prescription refill in a few months, THEN we’ll tell them…”)

Fortunately, it is simple in our current EHR to run ad-hoc reports to do all this now. Whew! And, we can do predictive analytics on this data to save lives that would have blown my mind back then.

Here’s another flashback:

Don’t tell the post office; this is where all their bins went

THIS is the Medical Records intake room, back when we were ONE hospital, 40 clinics (we’re now 12 hospitals, 800 clinics). On average, 6 vertical feet of paper, received EVERY DAY. Fifty medical records staff, filing, sorting, pulling, sending, receiving, creating new charts. And, still, we were 2 WEEKS behind on filing.

We had over 20 transcription services, all local, receiving tiny tape-recorder dictaphone tapes, transported by COURIER from the doctors dictating. As an aside, some of us remember hearing doctors mumbling their ultra-fast, only partly understandable dictations walking the halls between patients. On average, outpatient transcriptions took about 2 weeks to complete and print out, mail, and file back into the record. Inpatient daily transcriptions were ordered STAT for 3x the cost and typed same day, arrived by urgent courier in the late evening and taped into the paper chart.

I am proud of my doctor handwriting

For the record, here’s a paper progress note I wrote in 1999 on “non-carbon paper” sending the original copy to Hospital Medical Records, and then keeping the yellow copy in a “shadow chart”: a duplicate set of medical records kept in our “off-site clinic” because … we could not count on Hospital Medical Records to pull the relevant charts for clinic patients scheduled each day.

Don’t even get me started on our appointment scheduling system. “Oh yes, thanks for calling! So you’re looking for Dr. Lin’s next available appointment? Sorry, nothing for the next 3 weeks. Oh, you’d like to see the next available doctor? =sigh= OK I’ll pull down the other twelve 3-ring binders, one for each doctor, and see who might have an open spot.”

Are you keeping track? 50 medical records staff at the hospital to maintain Main Medical Records, and 1-2 additional medical records staff at EVERY clinic (about 40 clinics) to keep a shadow chart. Because we don’t trust each other to keep track and deliver records on time!

Hello, dolly

Oh, and meet this guy. In 1997, our medical information (see: x-ray films, paper medical records, dictaphone tapes) moved at the speed of rush-hour traffic on Colfax Avenue. Seven miles each way, 12 leased buildings throughout metro Denver. Two round trips every day.

With all this person-power and effort, the result? On a typical clinic day, I would see about 18 internal medicine patients. Main medical records would successfully deliver charts for about 9 patients. Our clinic’s shadow chart system would deliver charts to my exam room for about 6 additional patients, leaving, on average THREE patients with NO CHART. Just a piece of non-carbon paper, with handwritten vital signs and a list of patient-reported allergies that day. Mind you, there was no such thing as a clinical computer system at the time. As a result:

“Hi Doc! It is great to see you! What did my cardiologist tell you about me when he saw me 2 weeks ago and did all those tests? He said that I should come talk to you about his report.”

Um. I don’t have any of your records today. I see your blood pressure looks good and that you report no allergies to medicines though.”

“What?! I made this appointment to go over his report! That visit was 2 weeks ago!”

“Yes. Um. What condition, exactly, do you have? Why did we send you to my cardiology colleague? What do you remember that he told YOU? Can you help me out here?”

“This is disappointing. You mean you really have nothing on me? Do you at least have the blood test results or the echo result?”

“Um, no. I’m really sorry about this. Okay, tell you what, no charge for today, my apologies for wasting your time and I will call you later this week after I call and yell at my medical records people and maybe get your chart and see what it says.”

“Whatever. You guys should really get your act together. Okay, can you at least go ahead and refill those 3 medicines that you prescribed for me from last year? I’m about out.”

(Excitedly taking out prescription pad) “Sure, I’m happy to! Do you happen to remember the names of the medications and the doses and what they’re for?”

Let’s not even talk about loading up a 2-foot-tall stack of medical records in our arms, walking out to the car, throwing them in the trunk, driving home and dictating late into the night, and hopefully remembering to bring them back into the office the next day.

And, if there was an urgent need for a particular medical record? We would routinely have a couple staff members wandering the clinic, from office to office, desk to desk asking: “Do you have the chart for Peterson, Mary, or Smith, Joseph, or Samuels, Jane?” and thus not answering the phone, or rooming patients…

Of course, by contrast, with our current EHR, tap-tap-tap: instant access to any patient record.

Yesterday, for example, my patient met her oncologist to discuss a new diagnosis of metastatic cancer. Today, I was able to read her consulting note, review the pathology from a recent biopsy, refresh my education about peritoneal carcinomatosis in an EHR-linked online textbook, secure-chat and then phone call with the oncologist about prognosis and treatment options, set up a video visit with the patient and her family, and have a have a well-informed, thoughtful conversation about her next steps.

This speed and coordination would not have been possible in the era of paper charts.

Not as cool as Jimmy Fallon’s Thank you Notes

Wait! One more thing! Remember the good old days when we received faxed blood test results and then had to notify patients by writing a STACK of folded post cards? I faced a stack of these EVERY EVENING at the end of clinic. Please don’t ask me how many times a patient brought back a post card saying: “Um, this looks pretty important, but, I think you meant to send this to a different Peter Smith. I haven’t had a blood test in awhile.”

Yikes.

Our patient Portal, we call My Health Connection: we release test results to the patient online, and then send comments with our interpretations, arriving to the patient’s inbox instantly. Comment from my patient? “It feels like I have my doctor in my pocket. So cool.”

CMIO’s take? All y’all don’t know how good you have it.

On the other hand, are you old, like me? Do you remember those days?

On the third hand, in another decade, I hope folks will look back to TODAY and marvel how much better the future is.

EHR vs Covid-19. Antibody tests: Yikes.

Antibodies

If you’re here to understand some of the challenges of antibody testing with Covid-19, read on. Be warned: math ahead. What’s the TL;DR?

  • We don’t know whether having antibodies indicate that a person is IMMUNE to future re-infection now or later with Covid-19.
  • We don’t know whether having antibodies mean that a person is NO LONGER infectious to others with Covid-19.
  • We’re going to discuss Sensitivity, Specificity, Positive Predictive Value, Negative Predictive Value and say that MOST antibody tests out there may show Sensitivity and Specificity in the 80 or 90% range (seems good!)
  • BUT because the Prevalence of the disease is unknown and likely low (single digits or teens maybe), the Positive Predictive Value is likely to be TERRIBLE, meaning a positive result might just be … meaningless or WRONG MOST OF THE TIME (OMG).

Some of you know that my father is a statistician, so he is likely to read this uncomfortably and have lots of concerns about the accuracy of my statements. You may also know the quote popularized by Mark Twain:

There are three kinds of lies: Lies, Damn Lies, and … Statistics

But, here goes anyway; the point is important, it is bothering me, and I want people to know at least what little I understand.

Let’s say that an antibody test is 95% sensitive (meaning, for patients who really had COVID-19, it shows Positive for 95% of them), and 92% specific (meaning, for patients with NO Covid-19 prior infection, it shows Negative for 92% of them). Seems like a good test, if you look at it from an omniscient being’s point of view: you already KNOW who has and doesn’t have the disease, and you’re just waiting to snicker at how well the tests turn out.

The trouble is when you turn things around the other way, from a patient’s point of view. You SHOULD NOT CARE what sensitivity and specificity are. You SHOULD CARE what Positive Predictive Value and Negative Predictive Value are.

Okay, now some of you are having hot flashes, or shaking chills, or whatever your reaction was to taking Statistics in high school or college or medical school (or all 3). Imagine also, that your father also knows most of the people teaching your classes because of his professional network, and you’re worried that your grade on this test will reflect poorly not only on you, the son of a statistician, but on your father, your family, your entire lineage. Good, now you’re getting me.

Negative Predictive Value (NPV) is the likelihood that if your test is Negative, it is correct, and you don’t have antibodies.

Positive Predictive Value (PPV) is the likelihood that if your test is Positive, you DID have the disease and now have the antibodies.

Okay, here’s the setup, AND I AM NOT CLAIMING THESE ARE REAL STATS, this is just an exercise. Let’s hypothesize:

  • We will test 100,000 people
  • The prevalence of disease is 3% (3 of each 100 have the disease in our population)
  • Sensitivity of our antibody test is 95%
  • Specificity of our antibody test is 92%

See the table as we calculate this:

Test ResultCOVID past or presentNo CovidTotal
Positive2,8507,76010,610
Negative15089,24089,390
Total3,00097,000100,000
Antibody test Scenario 1

The NPV equals “true negative / (true negative + false negative)”, or 89,240/(150+89,240) or 99.8%. In a population with very few Covid-19 infected patients with antibodies, you’re going to be right MOST OF THE TIME, to find “no antibodies” in most patients. So far so good.

The PPV equals “true positive / (true positive + false positive)”, or 2,850/(2,850+7,760) or 26.8%. What?

This means that PPV, or chance that a POSITIVE antibody result is CORRECT is 26%. So, if you take an antibody test in this population and your result is POSITIVE, then there is a 74% chance THAT TEST IS INCORRECT. Can you imagine? “Here is your result, Sir, your antibody test is Positive, but 3/4 of the time that is wrong.”

So the test we’re describing above, with the above assumptions, is helpful when the result is NEGATIVE (right 99% of the time) but NO HELP AT ALL (wrong 74% of the time) if the test is POSITIVE. Got it?

Okay, lets try a second scenario. Let’s hypothesize:

  • We will test 100,000 people (same)
  • The prevalence of disease is 3% (3 of each 100 have the disease in our population) (same)
  • Sensitivity of our antibody test is 95% (same)
  • Specificity of our antibody test is 99.5% (DIFFERENT)

Here is our new table:

Test ResultCOVID past or presentNo COVIDTotal
Positive2,8504853,335
Negative15096,51596,665
Total3,00097,000100,000
Antibody test Scenario 2

NPV is the same: still 99.8% accurate. A negative is pretty good.

PPV is now: 2,850/(2,850+485) = 85.4%.

Therefore! Pushing this antibody test’s performance up to 99.5% specific makes a HUGE reduction in the number of False Positives, and makes it so that a Positive test for a patient is going to be right 85% of the time! Not perfect, but way better than 26%.

See what I mean? Moving from Sensitivity and Specificity to NPV and PPV make a really big difference when it comes to thinking “should I get this test” and “can I trust the result?” Maybe don’t rush right into getting your test until you chat with your doctor about how well it performs, what it might mean, and truly how useful these tests are.

Right now, for example, at UCHealth, we are only recommending testing patients who wish to donate plasma for our research study to infuse antibody-rich plasma into critically ill Covid-19 patients. Over time, as we learn more, we’ll expand testing to more patients (soon).

Thanks to Ed Ashwood, Medical Director, Clinical Lab, University of Colorado Hospital from whom I “borrowed” much of this example.

CMIO’s take? Whew! Statistics is hard. Who knew that Dad was right about how important Statistics is? Please look on your fellow statistics geek friends with kindness, they’re making our world a safer place. And, be careful what you ask from an antibody test.