When healthcare leaders start thinking about AI, the first question is usually which model is safe to use, whether that’s ChatGPT, Copilot, or Claude. But learning how to implement AI in healthcare safely has much less to do with the model you pick and much more to do with the process you put around it.
That was one of the main takeaways from a recent episode of The Doc Lounge Podcast featuring Technology Rivers CEO Ghazenfer Mansoor. Host Stacey Doyle, Senior Director of Marketing at Pacific Companies, brought Ghazenfer on to cut through the AI hype.
He started Technology Rivers in 2015 after consulting with the Veterans Health Administration, where he first worked with healthcare data standards like HL7. Since then, his team has built around 50 healthcare web and mobile apps, and about half of them are HIPAA compliant. The conversation covered how to keep patient data safe, why AI makes things up, where AI is already helping doctors, and what physicians should look for in a development team.
Watch the full episode: Healthcare Software: The Essential Guide to Building with AI and HIPAA
Start With Data Privacy: What HIPAA Requires for AI
HIPAA was first written for paper records kept in an office. As healthcare moved to web apps and then to mobile phones that could be lost anywhere, the rules had to expand to protect patient data in new places. AI is the next big shift, and many organizations are not ready for it.
People are putting patient data into AI tools without thinking twice, and data entered into these tools can be used to train the AI. That is where the real risk lies.
The first safeguard is a Business Associate Agreement (BAA). No patient data should go into a large language model unless the AI provider has signed one. A signed BAA legally requires the provider to follow rules such as not keeping your data and not using it for training. For a deeper look at these requirements, see our guide to HIPAA compliant AI applications.
The second safeguard is how the system is built. Technology Rivers uses an approach called Retrieval-Augmented Generation (RAG), where sensitive patient data stays in a separate, private database. Before the AI searches anything, the data is protected and filtered down to only the records that are relevant. This way, the AI only sees a small, limited piece of the data rather than everything you have. Audit logs add another layer by recording who accessed the data, when, and from where. For some projects, a private AI model is a better fit than a public one.
Knowing how to implement AI in healthcare safely comes down to these safeguards, not to picking the “right” model. As Ghazenfer puts it, “you just have to know what you’re doing.” That idea is also the foundation of Technology Rivers’ Secure AI Knowledge Platform.
AI Will Make Things Up Unless You Build It Not To
Keeping data safe is only half the job. Ghazenfer estimates that 80 to 90 percent of AI projects fail, and one major reason is the belief that AI works like magic, where you hand it data and get the right answers back. (We look at more reasons in Why Healthcare AI Projects Fail.)
AI tools almost never admit when they don’t know something. Instead, they give an answer anyway, which is what’s known as an AI hallucination.
“The default mode of AI is: you ask a question, it will give you an answer even if it has to make it up. In healthcare, that’s not what you want.”
The way to reduce AI hallucinations in healthcare is to connect the AI to your own data, using the same RAG approach that protects privacy. This only works if the data is clean, though. If your data is poor, the results will be poor too. Lab results, clinical notes, and other internal records need to be cleaned up and organized around your processes before an AI model uses them.
Two other common beliefs about AI also lead projects off track. The first is that AI is plug and play. Connecting ChatGPT to your app without any changes just gives you the same thing ChatGPT already does. A useful healthcare tool needs a clear role, specific instructions, and your own data behind it. The second belief is that AI is cheap. The cheapest approach is to drop all your data into a public AI tool and see what comes out, but by that point, the data has already been handed over. Doing it properly costs more because most of the work goes into preparing the data.
Where AI Is Already Helping Patients and Physicians
When AI is built on clean data with the right safeguards, it can make a real difference. Ghazenfer explains what that looks like through two projects from the Technology Rivers portfolio.
The first is a personal health platform. Users upload their lab results, DEXA scans, and fitness data, and the AI reviews everything in depth and suggests recommendations. This review used to be done by hand, which took a lot of time and added cost. Doctors still check the AI’s output, but the process is now much faster, and the AI can pick up details that might otherwise be missed.
The second is a consultation tool that lets doctors get input from other doctors. A physician shares a case and first receives an AI recommendation based on PubMed and other sources. Then other physicians in their network add their own recommendations. The value comes from combining outside research with the experience of trusted colleagues.
Both projects follow the same principle. AI is there to help doctors review cases and improve their recommendations, not to send advice directly to patients. A physician always stays involved.
From People-Powered to System-Powered Healthcare Businesses
AI and technology can also change how healthcare businesses operate, especially service businesses such as home care, autism care, and assisted living.
These businesses often have lower valuations because they usually compete on service quality, sales, or staff. Those strengths matter, but they are hard to scale. Technology that competitors don’t have can change that, and it is what helps a business grow much faster. Ghazenfer describes the goal as building a system-powered business rather than a people-powered one, which is where the potential for 10x growth comes from.
This doesn’t mean replacing all of your existing software. It means looking at your processes from start to finish and finding the bottlenecks and manual tasks, such as documentation, triage, scheduling, note taking, and patient follow-up calls. Improving those areas makes the business more efficient, which leads to more revenue and more profit. It also gives the business proprietary technology that sets it apart from competitors.
How Physicians Should Vet a Healthcare AI Development Partner
Many physicians are building health tech products, often alongside their practice. For them, knowing how to implement AI in healthcare safely starts with hiring the right development team, and it isn’t enough to hear “yes, we do HIPAA,” because almost every vendor says it.
A better approach is to ask the vendor how they verify that an app is HIPAA compliant. That means looking at their checklist and understanding how their requirements team, project managers, developers, and quality assurance (QA) testers each handle HIPAA. Our Healthcare Software Partner Due Diligence Guide walks through these questions in more detail. Gaps often start in the requirements.
A document might only say the app “needs to be HIPAA compliant,” and a QA tester may read that as simply needing to be secure. But security is only one of many things HIPAA requires. At Technology Rivers, every HIPAA project automatically includes about ten requirements, such as audit logs, authentication, authorization, and Face ID login where it applies, even if no one writes them down. This works because everyone on the team goes through HIPAA training, whatever their role. Beyond that, it helps to look at the vendor’s past work and talk to their customers.
Even a fully compliant app still has to keep its users, and that is often the harder part. Building an app is easy compared with making sure people keep using it every day. Ghazenfer has worked in mobile since 2000, before the iPhone and Android existed, and wrote Beyond the Download: How to Build Mobile Apps That People Love, Use, and Share Every Day to address this challenge. The book shares 32 growth strategies and three case studies, along with sections on AI and healthcare.
Agentic AI and the Next Wave of Healthcare Workflow Automation
Over the next few years, Ghazenfer expects healthcare workflow automation to be one of the biggest areas of AI innovation, with AI agents arriving quickly in hospitals and practices.
To understand agentic AI in healthcare, it helps to know three terms that often get mixed up. Automation follows a fixed set of steps. An AI agent follows rules that you create for it. Agentic AI goes further: it learns your system over time, makes decisions on your behalf, and gives recommendations instead of only following instructions.
For most organizations, standard AI models can simply be bought rather than built. The real advantage comes from creating agents for workflows that are unique to your business, which sets you apart from competitors. Technology Rivers builds these through its AI Agent Development services.
How to Implement AI in Healthcare Safely: Process Beats the Model
Every problem Ghazenfer covers in this episode comes back to the same mistake, which is treating AI as something you plug in rather than a system you design. Patient data gets exposed when no one checks whether the AI provider has signed a BAA. AI makes things up when no one has cleaned the data or limited what the model can see. Development projects fall short when “HIPAA compliant” is a single line in a document instead of a checklist every team member follows.
The organizations getting real value from AI, whether through better patient tools, more efficient operations, or custom agents, are the ones that understand the process before they start building. If you’re figuring out how to implement AI in healthcare safely, that process is the place to start.
