If you’re here to understand some of the challenges of antibody testing with Covid-19, read on. Be warned: math ahead. What’s the TL;DR?
We don’t know whether having antibodies indicate that a person is IMMUNE to future re-infection now or later with Covid-19.
We don’t know whether having antibodies mean that a person is NO LONGER infectious to others with Covid-19.
We’re going to discuss Sensitivity, Specificity, Positive Predictive Value, Negative Predictive Value and say that MOST antibody tests out there may show Sensitivity and Specificity in the 80 or 90% range (seems good!)
BUT because the Prevalence of the disease is unknown and likely low (single digits or teens maybe), the Positive Predictive Value is likely to be TERRIBLE, meaning a positive result might just be … meaningless or WRONG MOST OF THE TIME (OMG).
Some of you know that my father is a statistician, so he is likely to read this uncomfortably and have lots of concerns about the accuracy of my statements. You may also know the quote popularized by Mark Twain:
There are three kinds of lies: Lies, Damn Lies, and … Statistics
But, here goes anyway; the point is important, it is bothering me, and I want people to know at least what little I understand.
Let’s say that an antibody test is 95% sensitive (meaning, for patients who really had COVID-19, it shows Positive for 95% of them), and 92% specific (meaning, for patients with NO Covid-19 prior infection, it shows Negative for 92% of them). Seems like a good test, if you look at it from an omniscient being’s point of view: you already KNOW who has and doesn’t have the disease, and you’re just waiting to snicker at how well the tests turn out.
The trouble is when you turn things around the other way, from a patient’s point of view. You SHOULD NOT CARE what sensitivity and specificity are. You SHOULD CARE what Positive Predictive Value and Negative Predictive Value are.
Okay, now some of you are having hot flashes, or shaking chills, or whatever your reaction was to taking Statistics in high school or college or medical school (or all 3). Imagine also, that your father also knows most of the people teaching your classes because of his professional network, and you’re worried that your grade on this test will reflect poorly not only on you, the son of a statistician, but on your father, your family, your entire lineage. Good, now you’re getting me.
Negative Predictive Value (NPV) is the likelihood that if your test is Negative, it is correct, and you don’t have antibodies.
Positive Predictive Value (PPV) is the likelihood that if your test is Positive, you DID have the disease and now have the antibodies.
Okay, here’s the setup, AND I AM NOT CLAIMING THESE ARE REAL STATS, this is just an exercise. Let’s hypothesize:
We will test 100,000 people
The prevalence of disease is 3% (3 of each 100 have the disease in our population)
Sensitivity of our antibody test is 95%
Specificity of our antibody test is 92%
See the table as we calculate this:
COVID past or present
Antibody test Scenario 1
The NPV equals “true negative / (true negative + false negative)”, or 89,240/(150+89,240) or 99.8%. In a population with very few Covid-19 infected patients with antibodies, you’re going to be right MOST OF THE TIME, to find “no antibodies” in most patients. So far so good.
The PPV equals “true positive / (true positive + false positive)”, or 2,850/(2,850+7,760) or 26.8%. What?
This means that PPV, or chance that a POSITIVE antibody result is CORRECT is 26%. So, if you take an antibody test in this population and your result is POSITIVE, then there is a 74% chance THAT TEST IS INCORRECT. Can you imagine? “Here is your result, Sir, your antibody test is Positive, but 3/4 of the time that is wrong.”
So the test we’re describing above, with the above assumptions, is helpful when the result is NEGATIVE (right 99% of the time) but NO HELP AT ALL (wrong 74% of the time) if the test is POSITIVE. Got it?
Okay, lets try a second scenario. Let’s hypothesize:
We will test 100,000 people (same)
The prevalence of disease is 3% (3 of each 100 have the disease in our population) (same)
Sensitivity of our antibody test is 95% (same)
Specificity of our antibody test is 99.5% (DIFFERENT)
Here is our new table:
COVID past or present
Antibody test Scenario 2
NPV is the same: still 99.8% accurate. A negative is pretty good.
PPV is now: 2,850/(2,850+485) = 85.4%.
Therefore! Pushing this antibody test’s performance up to 99.5% specific makes a HUGE reduction in the number of False Positives, and makes it so that a Positive test for a patient is going to be right 85% of the time! Not perfect, but way better than 26%.
See what I mean? Moving from Sensitivity and Specificity to NPV and PPV make a really big difference when it comes to thinking “should I get this test” and “can I trust the result?” Maybe don’t rush right into getting your test until you chat with your doctor about how well it performs, what it might mean, and truly how useful these tests are.
Right now, for example, at UCHealth, we are only recommending testing patients who wish to donate plasma for our research study to infuse antibody-rich plasma into critically ill Covid-19 patients. Over time, as we learn more, we’ll expand testing to more patients (soon).
Thanks to Ed Ashwood, Medical Director, Clinical Lab, University of Colorado Hospital from whom I “borrowed” much of this example.
CMIO’s take? Whew! Statistics is hard. Who knew that Dad was right about how important Statistics is? Please look on your fellow statistics geek friends with kindness, they’re making our world a safer place. And, be careful what you ask from an antibody test.
Virtual meetings are draining, and I’m on them up to 8 hours a day, even busier now with all the EHR modifications, keeping up with policy changes, what Covid-testing is available, how we admit, treat, discharge, follow, track patients.
At the ends of long hours, long days, long weeks, our nerves are frayed.
I’ve observed that interactions between people have everything to do with the interpersonal skills of the individuals. Sometimes the conversation does NOT go well. Whether it is by email (worst for crucial conversations), by phone (slightly less bad), by online video meeting (slightly less bad) or in person (best, when possible), it is certainly worsened by the pandemic situation.
I’ve been taking a Story Skills Workshop (by Seth Godin and Bernadette Jiwa) that recently concluded. I have to say that I’ve learned quite a lot, and not what I was expecting to learn. I highly, highly recommend it. Seth and Bernadette offer a series of online lessons, released over time. There are about 6 expert coaches, and the instruction is to sign up for an interest group or ‘accountability group’. You’re given a story structure (the 5 C’s: Context, Catalyst, Complication, Change, Consequence) and then specific lessons to write and polish specific elements of your own story in this framework. The cool part is the instruction to ‘first write your own story, and then go comment on at least 5 others.’
I learned that it is possible, in an online-only course, to develop a sense of community and collegiality in a short 30 days.
I learned that it is crucial to be gentle in first contact with others online. For example, when giving feedback on others’ stories, DO NOT start right in with ‘why don’t you add more Emotion to that moment in your story?’ You’ll learn (as did I) that conversation either stops or becomes defensive. Remember that online conversations carry ZERO nonverbal: no Kind tone of voice, no Friendly posture. All you see are the words, and it is automatic to imagine them coming from a frowning critic with crossed arms, shaking his brutish head. [Pause for self-reflection amongst my blog-readers, as well as from myself…]
Instead, try something my theater-trained son taught me:
‘I like… I wish… What if …’
My highly emotionally intelligent son
Framing any response this way allows your recipient to hear something positive, then a neutrally posed concern, followed by a tentative suggestion. Having been on both sides of such a well-formed critique, I can say: it is EASY to write, doesn’t take longer, and on the receiving end FEELS COMPLETELY DIFFERENT. It FEELS like a close friend, reaching a hand over to pull you up to a higher step.
FOR EXAMPLE: Take one of my story-critiques of a co-participant in the story workshop, not done well on my part: “Why don’t you add more emotion to your story? It reads like a timeline, but nothing about what you felt, or how that impacted you.” I thought I was clever, to point out one of the main points of that week’s lesson. What I received was… no response. Hmm.
Rephrasing the reply using this framework, when I replied to a different participant’s story, sounded like this: “Hi, Joe! I liked your story, especially the unexpected part about running away from home at 16. I wish I could be there at that moment when you made the decision, everything boiling-over, and then a crucial moment. What if you paused in your story and told us what you were thinking and feeling right then? I would be riveted.” Guess what? We had a great online conversation after that, and he re-wrote his story, and I WAS RIVETED. Win-win.
CMIO’s take? Story telling: cool. Gentle, effective feedback: cooler. Don’t we all need to get better at this?
If you’re not getting away enough from all things Pandemic, here’s a nice long-exposure photo from my iPhone 7. Yes, a CMIO with an iPhone 7. And I still love it. At least I have a smartphone, unlike one of my informaticist colleagues.
Forest bathing is a thing in Japan and increasingly worldwide, and perhaps we could learn a thing or two. OR, try Norway’s Slow TV (YouTube, almost 10 hours! Surprising how compelling it is, try it full screen), as highlighted by CBS Sunday Morning (8 minutes, YouTube). Don’t miss it!
Thanks to my excellent colleagues at UCHealth, Echo Vogel, Hillary Duffy and Duane Pearson, co-conspirators to spend an hour on a Zoom webinar to review Patient Experiences with online Video Visits. We are all on a rapid learning curve. Come spend some time with us as we review what we’ve learned.
For all the talk about Covid-19 testing, and testing for respiratory viruses, keeping mind, that the UCHealth (and most health system) labs do SO MUCH MORE than virus testing. Hemoglobin, for example. Hemoglobin is the measure of how much oxygen-carrying protein we carry in our blood, a way to detect anemia, blood loss. Look at the dip in the orange hemoglobin line in early March. Our testing volume throughout UCHealth fell in HALF. As we all reconfigured our hospitals to prepare for pandemic, we also stopped all surgeries except for emergencies. And the various colored lines at the bottom are all our previously described viral tests (see previous post). How puny and miniscule.
Okay, so wait a minute, lets look back at those virus tests: what’s this? Remember that purple was Influenza B, Green was Influenza A, yellow was RSV (respiratory syncytial virus). Then there are 3 viruses (including Rhinovirus) that SPIKE in March! OMG! None of those are Covid-19.
CT Lin is a GENIUS! He has discovered the TRUE CAUSE of the pandemic: out-of-control mutated winter viruses! AND, there are THREE of them, all spiking at once! Triple infections for everybody!
Well? What do you have to say for yourselves? You gonna buy it? Pause and think. Hmm.
Okay, you’re all smarter than me. Yes, you’re right. When we started Covid-19 testing, we also threw in RVP (Respiratory Virus Panel) testing on the same swab. This graph shows, not the “testing positive” results above, but TOTAL VOLUME OF TESTING, and RSV testing increased 700% in a month, due to co-testing on Covid-suspected patients. Aha!
So, layer in the Covid-19 positive results, and we see this, temporary bumps in 3 viruses (due to all the co-testing of Covid-suspect patients: blue, dark-blue, red), and a persistent and growing number of Covid-19 positive patients (purple). And then, we have cut back the co-testing since then. AND we near the end of the season for those other viruses anyway. Okay, this makes more sense.
Hooray! Let’s celebrate, we made some sense of the data!
Of course it is not a good sign that each week we detect more Covid-19 positive patients. It is either: more patients are showing up for testing, OR we have more test kits, OR we have relaxed the standards (all symptomatic patients can be tested now, instead of just health-care workers and critically ill patients), OR the exponential spread of the pandemic is not over, OR something else we haven’t thought of yet. Wouldn’t you like to know which one it is? Yes, so would I.
CMIO’s take? I hate living in the middle of the pandemic, when every night, we go to sleep in the middle of a cliff-hanger. I hated those season-ending episodes in those shows I loved so much. Here’s hoping our heroes solve the cliff-hanger in time for next season!
It is time we looked back at the past month to update our data. Since we initiated social distancing in mid March, and then the Colorado Stay-at-Home order in late March and subsequent mask use, what has happened to Covid-19, and what about symptoms? other viruses, the geographic spread? Hold on to your hats. This is all UCHealth data from our Slicer de-identified data tool.
Virtual visits are up, and stable at about 4000 visits a day. In person visits are way down from baseline in early March. Overall visit volume is way down, as it is for most health organizations right now. As Colorado starts to ease restrictions from “Stay at Home” to “Safer at Home”, we will see what comes next. Regardless, there will be repercussions from this some time to come.
2. Covid RNA testing is up (not enough), supplies for testing are short nationally, and unlikely to change quickly. The state is ramping up, and we all agree that widespread community testing is critical to safely opening up restrictions. Furthermore, antibody testing, although everyone’s working hard on this, is still not ready for prime time; the accuracy of the tests (false positive, false negative rates) are still too high to be trusted.
3. Cough, Fever, Shortness of Breath, Diarrhea is all decreasing as we reach the end of the worst virus seasons. What does this mean about Covid? Of course, mixed in, is the decrease in visits overall. This curve is worth watching over time as the state “opens up.”
4. Where is the spread across 12 hospitals. University of Colorado Hospital on the far right, Greeley Medical Center on the far left. The higher bumps in the middle are for Poudre Valley, Medical Center of the Rockies, Memorial North, Memorial Central. Some of our hospitals are seeing very little. Interesting regional variation. We are also intentionally protecting some of our hospitals to be Covid-free as much as possible, in case of future surge demand.
5. What about ICD10 diagnoses by docs? You may know that CMS published an new set of ICD10 codes (international classification of disease, version 10) that contain Covid-19: including: Confirmed Covid, Suspected Covid, Exposure to Covid, etc. Grouping them into Confirmed versus Suspected, here’s what you get in our visits since March (orange: suspected, blue: confirmed). This is a reflection of the scarcity of testing. Again, interesting to see where this goes. NOTE: The last down-tick on the orange may be due to counting a partial-week in the data.
So what do we learn?
A. We need more testing to figure out what the community prevalence is.
B. Virtual visits, even scalable at 20x previous volume, will not save the day.
C. Social distancing and stay-at-home are followed quickly by a dramatic reduction of Covid-like symptoms seen in clinics. BUT, be careful of conclusions: social distancing could also have affected the USUAL winter virus spread, OR patients are not choosing to be seen, OR the virus is mutating into a less severe form, OR distancing policies might really be working. Wouldn’t you like to know which it is?
D. ICD10 codes are being used in our clinics! But the trends don’t mean anything yet. It might be promising that “suspect Covid” codes are growing more slowly.
F. Covid is going strong into the late Spring (wasn’t it supposed to evaporate magically likes SARS CoV-1? That outbreak in 2003 largely evaporated by late May, early June). Covid-19 (more accurately, SARS CoV-2) is going strong and is likely to behave quite differently, not a good sign for us humans.
PREVIEW OF NEXT POST: How does testing for OTHER VIRUSES compare to COVID testing results in the late winter/spring? and what does Hemoglobin testing teach us about Covid-19?
CMIO’s take? The lessons from Pandemic are numerous and they just keep coming. Be sure to take a break, and take care of yourself. Don’t worry, I’ll be back soon.
Here’s a Sunday post: Musing about miso soup and it’s role in my battle with the pandemic. If you’re here for data, clinical informatics, and health system thinking, you can leave now.
When I was the age of my youngest niece (okay, my only niece), I remember telling my mother who had just served me a bowl of home-made soup, with a sprinkling of goldfish crackers:
THIS SOUP IS SO YUMMY. I think it must be the goldfish I put in there. Goldfish are the perfect food, and I think I’m going to make a soup just from goldfish crackers and hot water. Mom, can I have some hot water?
Of course I wasn’t watching my mother’s face at the time, I was so sure of my world-shattering upcoming invention: Delicious hot water-and-goldfish soup.
I stood by, as she boiled some hot water for me on the stove. I had carefully selected a big handful of goldfish, and was cautiously resisting eating them, KNOWING that the soup was going to be TOTALLY WORTH THE WAIT. I was bouncing with excitement.
At last, hot water, in a cup on the kitchen table. =Plt-phtl-tlthtpl-plthth-ppth!= I slid all my goldfish from my sweaty palm into the cup, gave it a big stir,
…paused for a drawn-out moment to let the flavors swirl…
And took a big sip.
What did I think? It gradually dawned on me, as my face twisted into a surprised grimace, that the soup was not good … AT ALL. Just a bunch of semi-soggy flavor-less crackers and hot water.
I tried to fix it: added salt and pepper. Even some “Accent” (packaged MSG salt; hey it was the 1970’s! anything goes). Nope. Nope. And Nope.
I finally looked at my mom, who was silently watching, smiling and shaking her head at me. It was a lesson, I suppose, that her son had to learn for himself. Good soup was just not going to be that easy.
FLASH-FORWARD, present day.
I have always been obsessed with soup. Almost every restaurant we go to (or, used to go to), I would scour the soup offerings for candidates. I did let go of my soup-inventing dreams, but have lately been punching out Instant Pot – powered soups like Rosemary Cauliflower and Ginger Carrot, to some pretty good family reviews. But of course, they realize it’s Dad cooking, so the critic-grading-scale is set pretty low and forgiving to start.
Night-times are for miso soup, though. I’m a night owl and do some of my best thinking and working at night, and give me 1 teaspoon of Marukame Boy brand Miso paste, a cup from the hot-water pot, a sprinkling of scallions (pre-sliced and saved in a container in the fridge), and maybe some … GOLDFISH CRACKERS from a huge Costco bag. Heaven.
And, what do you know? Miso soup, consumed daily is supposed to have ability to FIGHT INFECTIONS! Hey! Who needs vaccines or treatments? Daily miso soup for EVERYONE, that’s the ticket. Okay, whatever, no.
Turns out, if my pre-teen self had just known SOMETHING about miso paste, I might have been a chef instead. Happy Sunday, everyone. Hi, Natalie!