The program, which began last month, allows an AI system to verify patient prescription histories, walk them through clinical questions, and send approved renewals directly to pharmacies. State officials say the program is a way to reduce costs, prevent medication lapses, and ease the burden on doctors, particularly in rural areas where clinicians are already stretched thin. However, physician groups say they were worried about the risks that come with a lack of human oversight in even minor clinical decisions.

“Any time clinical decisions are made without appropriate physician oversight, patients are put at risk. Medicine isn’t just about arriving at an answer; it’s about judgment, context, and accountability,” John Whyte, CEO of the American Medical Association, told Fortune.

AI gets clincial authority

The state’s commerce department has waived certain rules for a year-long trial, which is run in partnership with health-tech startup Doctronic. Margaret Busse, Executive Director at Utah Department of Commerce, told Fortune that the waiver was part of a “regulatory mitigation” program designed to safely test innovative AI tools.

“We really hope that this can shed light on where AI can be used responsibly in low-risk interactions in medicine that can help drive down costs and increase access. Because we all know we have a crisis…we desperately need technological solutions to help drive those costs down,” she said.

Doctronic, which was founded in 2023 by co-founders Matt Pavelle and Dr. Adam Oskowitz, said the system could be used by tens of thousands of patients in Utah during its first year. If it’s proven to be safe, the co-founders said they are hoping to scale the operation to include new prescriptions in low-risk scenarios, such as antibiotic prescriptions. They say the system can help speed up the prescription process and help patients, doctors, and pharmacists.

“The whole process from starting a conversation with the AI to having a prescription waiting at a pharmacy for you can take less than 30 minutes, and frankly, almost always takes less than 30 minutes. So it’s super efficient,” Pavelle, one of the co-founders and co-CEO of Doctronic, said.

To ensure safety, the phased rollout requires that the first 250 renewals per medication category are reviewed by a doctor before the AI can process them independently. After this, 10% of subsequent interactions will be randomly sampled for safety, Adam Oskowitz, Doctronic AI co-founder, said. Pavelle added that the company also has doctors available if the patient or pharmacist has questions.

Oversight concerns

Doctronic says the system matches human clinician decisions 99.2% of the time and carries a unique malpractice insurance policy covering the AI, ensuring it is held to the same legal standards as a doctor. The pilot is also limited to 190 commonly prescribed medications, excluding drugs for pain management, ADHD, and injectables for safety reasons.

However, Whyte argued that even a small error rate can translate into real harm at scale.

“Accuracy claims do not replace clinical judgment…physicians are uniquely qualified to understand a patient’s individual health context and to validate AI outputs to ensure accuracy and patient safety,” he said. “Even so-called routine care requires human judgment, context, and the ability to recognize when something isn’t routine at all.”

Regulators are struggling to keep pace

AI is poised to revolutionize healthcare, but the sector is moving much more cautiously than some others when it comes to adopting the technology. Part of this is down to regulatory hurdles: medical regulation is managed at the state-level and is typically tougher than other industries due to the stakes of errors when it comes to patient care.

Physicians are hoping that AI can transform routine healthcare work by increasing access, reducing costs, and automating repetitive tasks. However, the technology has been moving faster than state regulations can accommodate, and lawmakers have been struggling to walk the line between innovation and patient safety.

“AI can be a powerful tool to support physicians, especially with administrative tasks or decision support; and we have barely scratched the surface in terms of AI’s potential,” Whyte said. “But physicians must remain responsible for clinical decisions.”