Online symptom checkers lacked diagnostic and triage accuracy in simulated cases
The symptom checkers varied in provision of appropriate triage advice from 78% to 33%, and varied in accurate diagnosis, with the best sites listing the correct diagnosis first about 50% of the time and the least accurate diagnosing correctly 5% to 20% of the time.
Online symptom checkers provided the correct diagnosis only about a third of the time and were generally risk averse, according to a recent evaluation of the tools.
Researchers used 45 standardized patient vignettes to evaluate 23 symptom checkers that were free, publicly available, and in English. The vignettes were equally divided among those for which the correct diagnosis required emergent care, those for which nonemergent care was reasonable, and those for which self-care was reasonable. Results were published by The BMJ on July 8.
The symptom checkers provided the correct diagnosis first 34% of the time (95% CI, 31% to 37%), and 58% of the time (95% CI, 55% to 62%) they offered the correct diagnosis within the top 20 possible diagnoses. Appropriate triage advice was provided 57% of the time overall (95% CI, 52% to 61%), with the greatest accuracy for emergent cases, followed by nonemergent and then self-care cases (80%, 55%, and 33% respectively).
The symptom checkers varied in provision of appropriate triage advice from 78% (HMS Family Health Guide) to 33% (Mayo Clinic). Accurate triage advice was most likely to come from symptom checkers owned by provider groups and physician associations, followed by private companies, and then health plans or governments, the study found. The researchers concluded that the symptom checkers' triage advice was generally risk averse but also noted that when the checker suggested several possible care sites, the evaluators used the highest acuity site listed.
The checkers also varied in their rate of accurate diagnosis, with the best sites listing the correct diagnosis first about 50% of the time and the least accurate diagnosing correctly 5% to 20% of the time. The correct diagnosis made the checkers' top 3 between 29% and 71% of the time. Physicians should take from these results that the advice patients receive from symptom checkers may often be inaccurate, although the tools may still be useful “if the alternative is not seeking any advice or simply using an internet search engine,” the study authors said.
To determine whether symptom checkers are valuable, a number of other questions in addition to accuracy need to be investigated, according to an accompanying editorial. It's not known how patients respond to the advice they get from symptom checkers or how the language real patients use to describe their symptoms compares to that in a clinical vignette. If supported by randomized trials, symptom checkers may eventually become part of a “multichannel” triage strategy, the editorialist said.