When it comes to urgent medical problems like ankle injuries or suspected strep, virtual MDs may be no match for the real thing, a new U.S. study suggests.
Researchers got 67 volunteers to test out how well eight popular virtual visit companies diagnosed these problems and four other common medical issues: sore throat, sinus infection, low back pain, and urinary tract infection.
About one in four patients got the wrong diagnosis or none at all from the virtual visits, the study found.
Get Data Sheet, Fortune’s technology newsletter.
What’s more, virtual doctors followed standard protocols for diagnosing and treating these problems only 54% of the time.
“One of the more surprising findings of the study was the universally low rate of testing when it was needed,” said lead researcher Dr. Adam Schoenfeld, of the University of California, San Francisco.
“We don’t know why, but it may reflect the challenges of ordering or following up on tests performed near where the patient lives but far from where the doctor is, or concern about the costs to the patient of additional testing,” Schoenfeld added by email.
Virtual visits using videoconferences, phone calls, and web chats are becoming a more common way for patients to seek urgent care because it can save the inconvenience of a clinic visit or provide access to care when people can’t get an appointment with their regular doctor.
What the health care business model should look like
Some insurers are starting to pay for virtual visits in certain situations, making this option more viable for patients who worry about costs.
For the current study, Schoenfeld and colleagues trained volunteers to act as if they had common acute medical problems and then sent them to virtual doctors provided by companies including Ameridoc, Amwell, Consult a Doctor, Doctor on Demand, MDAligne, MDLIVE, MeMD, and NowClinic.
Altogether, the volunteers completed 599 virtual visits in 2013 and 2014.
The companies varied in how well they followed treatment guidelines, with standard care given anywhere from 34 to 66% of the time across the eight websites, the researchers report in JAMA Internal Medicine.
Mode of communication—such as web chat or videoconference—didn’t appear to influence how often treatment guidelines were followed.
Virtual doctors got complete histories and did thorough exams anywhere from 52% to 82% of the time.
Virtual visits resulted in correct diagnoses anywhere from 65% to 94% of the time.
Often, virtual doctors failed to order urine tests needed to assess urinary tract infections, or to request images needed to diagnoses ankle pain, for example, and antibiotics were often prescribed inappropriately.
One limitation of the study is that the researchers only looked at virtual visits, so they couldn’t compare these online doctors’ visits to what might have happened with in-person clinical exams.
HealthTap teams with Quest Diagnostics to let its virtual doctors order lab tests
Still, it’s possible that at least some of the variation in quality of care was the result of the remote visits, said Dr. Jeffrey Linder, a researcher at Brigham and Women’s Hospital and Harvard Medical School who co-authored an editorial accompanying the study.
“There is a built-in barrier to getting testing, which led to worse care for ankle pain and recurrent urinary tract infections—for which the doctors should have ordered a test—and better care for low back pain—for which doctors should not have ordered a test,” Linder said by email.
In an ideal world, patients would be able to have occasional virtual visits with their primary care providers, who know their medical histories, said Dr. David Levine, co-author of the editorial and also a researcher at Brigham and Women’s and Harvard.
“Although virtual urgent care and in-person urgent care have not been compared head-to-head, virtual urgent care has its downsides—indirect physical exam, difficult access to testing, and unclear follow-up,” Levine said by email. “While the quality of care is not perfect anywhere, a patient’s primary care doctor should be a person’s first point of contact.”
