The AI Questions Pharmacy Teams Keep Asking

When I was working in a health system just a handful of years ago, the AI capabilities we’re discussing today weren’t available. Now pharmacy leaders are evaluating technology that can read referral packets, investigate benefits, and submit prior authorizations. There’s a lot to get your arms around, especially for our customers at large health systems.

What’s driving that change, and what does it mean for your team? Plenful CTO GK Brar joined me for our recent Ask the CTO webinar to answer the questions we keep hearing from pharmacy teams: How does AI learn our workflows? What happens when it gets something wrong? Who stays in control? And how do we evaluate whether a vendor can actually deliver?

Read on for key takeaways from the webinar and clips from the best parts of our conversation.

What can AI do now that it couldn’t a few years ago?

GK started by acknowledging how quickly the technology has moved. “If you’re feeling anxious, uneasy, feeling left behind, that is completely normal.”

He described a shift from models primarily used to complete text or summarize information toward systems that can work through a task and take action. In pharmacy, that means connecting steps such as extracting information from a referral, checking what’s missing, and using the results to prepare an authorization.

It helps to distinguish the model from the system around it. Foundation models provide broad capabilities. Putting those capabilities to work in pharmacy requires specialized knowledge, access to the right information, and controls over what the system can do.

An agent also needs context. Knowing how to complete a prior authorization form doesn’t tell it how your organization handles a particular payor requirement or which decisions need a pharmacist’s review. Institutional memory supplies that context.

How does AI learn once embedded into our pharmacy workflow?

Plenful’s platform uses two layers of memory.

Organizational memory captures your team’s decisions, workflows, and preferences. Ecosystem memory supplies broader industry knowledge, such as payer policies, formularies, and authorization forms.

Those layers help the system apply relevant knowledge to the task in front of it. They also allow it to incorporate approved lessons from previous work.

“If you fix our agent once, if you correct a mistake that our model made today, it should learn from it,” GK said.

Consider a Prolia reauthorization where a payer requires two recent DEXA scans, but the requirement isn’t explicit on the authorization form. A team submits one scan and learns about the additional requirement through a denial.

Once an expert validates that requirement, the system can check for both scans during intake on future applicable cases. Missing documentation can be requested before the authorization is submitted.

The lesson from a denial can improve what happens at the beginning of the next case, including when a different person handles it.

At Access Infusion Care, that work began with understanding how coordinators reviewed and classified incoming documents. Their input helped us tailor the models to their existing workflows.

How do I prevent AI from learning from bad or wrong data?

Your pharmacy staff has people with varying levels of experience. A correction might be wrong, or a decision might apply only to one unusual situation. We shouldn’t assume every action deserves to become standard practice.

As new orders come in, Plenful's platform flags critical tasks for expert review in an easy-to-use workqueue. Your clinical experts act, giving feedback to the platform, which determines inputs that form the ongoing  organizational memory.

As GK explained:

“The expert isn't just in the loop, they’re completely in control.”

That includes the ability to remove a rule. Payor policies change. Your organization’s preferences change. The system guardrails can be reviewed and optimized as more information and expert actions are collected.

For pharmacy leaders, this means deciding who has authority to approve what the system learns and making sure those people can see how that knowledge is being used. Your experienced staff have an ongoing role in directing the technology.

‍

How do we know when an answer needs closer review?

The possibility of AI generating an incorrect answer deserves governance and scrutiny. Staff need to recognize uncertainty and understand when their clinical judgment is required.

Plenful uses confidence scores to help identify answers that need attention. GK explained that those scores consider factors such as whether the necessary information is available and whether the system has encountered a similar situation that an expert has reviewed.

An unfamiliar question with missing documentation will trigger a task for expert review and action. The user also needs visibility into the information supporting a proposed answer and a way to correct it.

For prior authorizations, the agent draws on patient documentation and institutional memory to pre-fill the correct PA form . Staff then review  and focus their attention on lower-confidence areas  where their expertise adds the most value. This workflow improvement increases capacity by 4x, compared to prior workflow without AI assistance.

Those are useful specifics to bring to an AI committee. Ask to see what happens when information is missing, when the system encounters an unfamiliar case, and when a pharmacist disagrees with its recommendation. How is that feedback loop stored? Does their expertise fuel the institutional memory?

AI within the intake authorization workflow increases capacity by 4x?

Intake authorization involves several connected steps, each dependent on the quality of the information gathered before it.

The workflow starts with a dense referral packet that may arrive by fax (most prevalent for infusion orders),containing handwritten notes andscattered information across multiple pages. Document processing combines text recognition with an understanding of the document to locate and organize the information needed downstream.

Benefit investigation then adds coverage requirements, formulary options, and site-of-care considerations. Organizational preferences and previous approved decisions can help inform the options presented to staff.

The authorization step uses what those earlier stages gathered to prepare the appropriate form, submit it, and follow the outcome.

The outcome can also improve earlier steps. GK described an authorization rejected because a BMI was missing. Once that requirement is reviewed and approved for the applicable scenario, the system can flag the missing information during future intake.

When evaluating automation, look at how information moves through the whole process. A faster individual step may still leave your team doing substantial work before and after it.

What should we ask an AI vendor?

A vendor should be able to demonstrate how its technology handles your work, including the exceptions. I’d focus on four areas.

Learning over time. Ask the vendor to demonstrate how a correction becomes an approved learning and affects future work. GK suggested asking, “How does performance month 3, month 6 look different from month 1?”

Governance. Have the vendor show you what your experts can approve, change, and remove. Find out where human review happens and how staff inspect the basis for a recommendation.

Pharmacy expertise. Bring a scenario your team regularly struggles with. Ask how the system handles an incomplete referral or a payer requirement that doesn’t appear on the form. The answer should reflect an understanding of the work and its exceptions.

Cost efficiency. Ask which models the vendor uses for different tasks and why. GK discussed specialization as part of Plenful’s approach to accuracy, speed, and cost. Buyers should understand how those choices affect the workflow they’re purchasing.

I’d also ask how the platform could support other areas of your operation. Data and approved knowledge may have value across prior authorization, auditing, and reconciliation. A vendor should be able to explain where those connections exist.

Where should a pharmacy team start?

Look at administrative work that already consumes a substantial amount of your team’s time. Prior authorization, claim auditing, 340B workflows, and refund reconciliation are places to consider. With the 340B Rebate Model Pilot going live January 1, rebate submission and reconciliation is about to join that list, which is why we launched Rebate Intelligence to automate the rebate lifecycle from claim to payment.

At NASP, I moderated a panel with pharmacy leaders from Dartmouth-Hitchcock, UNC Health, and Ohio State on their experiences adopting AI, from call routing to prior authorization. Their approaches to testing, staff involvement, and governance offer a useful starting point for peers considering similar work.

Choose a specific problem. Involve the people who do the work and the experts who understand its exceptions. Establish how you’ll assess whether the technology is reducing rework and giving staff time back.

Pharmacy teams have plenty of questions about AI, and they deserve answers detailed enough to act on. Seeing how a system handles your own workflow is a good next step.

Have an AI question we didn’t answer? Connect with me and the Plenful team to discuss what you’re evaluating and where AI could help in your pharmacy.