She had spent fifteen years learning to read the subtle shadows on chest X-rays. Now, an AI agent flagged abnormalities in seconds with 96% accuracy. When she asked how the system arrived at its conclusions and what would happen if it failed, no one could answer.
This scene plays out in healthcare organisations every day. Leaders chase accuracy scores while overlooking the metrics that actually determine whether AI agents can be trusted with patient care.
This problem runs deeper than most realise. In a recent Technology Rivers webinar on health AI agents, a panel of experts, CEO Ghazenfer Mansoor, data scientist Anna Shahinyan, regulatory specialist Megan Kane, and AI strategist Archana Puthran revealed why organisations keep getting this wrong, and what responsible AI in healthcare actually demands.
The Confusion Costing Organisations Millions
Before discussing safety metrics, the panel confronted a more fundamental issue: most healthcare organisations don’t understand what they’re building in the first place.
The confusion is widespread. Leaders deploy AI agents when they need simple automation. They build multi-agent systems when a single agent would suffice. They automate processes that should remain in human hands.
Ghazenfer Mansoor cut through the noise with a clear framework: Automation just follows the rules. It works whenever the path is predictable. While AI agents can also follow the rules, they also understand the goal, make decisions and take actions on your behalf. Instead of following a script, they respond based on the user’s input.
The distinction matters because getting it wrong means building the wrong solution from the ground up. In healthcare, that mistake carries consequences measured in patient outcomes and wasted resources.
Understanding these differences is the first step toward measuring what actually matters, which brings us to the metrics themselves.
Why Strategy Precedes Everything
Archana Puthran, drawing from years of experience on both payer and provider sides of healthcare, made clear that AI agent safety metrics mean nothing without strategic alignment:
If your vision and strategy don’t align, from the agents’ perspective, you’ll end up operationalising and solving the wrong things. Healthcare is a high-risk field that involves dealing with people, caring for them, and trusting agents to make decisions.
Her point reshapes how organisations should approach measurement. The first question isn’t about how accurate our AI is, is it? But instead, are we solving the right problem? Without that clarity, even perfect accuracy scores mask fundamental failures.
With strategic alignment established, organisations can turn to the three pillars that define whether an AI agent is truly safe for clinical deployment.
Pillar One: Clinical Performance That Goes Beyond Accuracy
Anna Shahinyan, whose expertise spans medical imaging and data readiness, identified what clinical teams actually need to measure:
First of all, it’s sensitivity and specificity, then accuracy: is the clinical advice correct? That is one aspect. The other one goes to consistency.
There is significant instability in LLMs and all AI solutions we have. We always need reproducible results.
She illustrated what reproducibility means in practice: The same CT image should have the same diagnosis and report. The same action should be taken each day in all types of sites and diagnostic centres.
This standard transforms how organisations must approach AI risk evaluation in healthcare. A system that performs brilliantly in a controlled demo but delivers inconsistent results across different sites isn’t safe, regardless of what the accuracy metrics claim.
Anna raised another metric that teams often overlook until it’s too late: whether we can defend this in court? That is another aspect.
Defensibility isn’t merely legal protection. It reveals whether an organisation truly understands what its AI does and why. Systems that can’t be explained can’t be trusted, and in healthcare, unexplainable decisions put both patients and institutions at risk.
Pillar Two: Operational Reality Over Theoretical Promise
Clinical accuracy loses meaning when AI creates new bottlenecks or shifts the burden rather than reducing it. Anna emphasised the importance of measuring real-world impact.
The most important and underestimated at the beginning is the operational efficiency of the hospitals. How does it really help, and how do we reduce the human-in-the-loop? If the confidence level of our agentic solutions is low and each case is marked for review, we need a human review, which reduces operational efficiency and wastes money.
The metrics that reveal operational truth include time saved per task, reduction in human review required, cost per patient interaction, and system behaviour during failures. If deploying an AI agent means clinicians spend more time managing technology than caring for patients, the implementation has failed, no matter what the laboratory results promised.
This operational lens redefines AI risk evaluation for healthcare organisations: success isn’t measured by what a system achieves under ideal conditions, but by how it performs when conditions are far from ideal.
Pillar Three: Trust Built Through Transparency
Megan Kane, whose background spans quality and regulatory compliance, addressed the mistake that derails more AI projects than any technical failure:
She recommends that teams think about security, privacy and compliance as core design requirements, rather than a final check mark or assessment to do after you’ve developed your system.
This principle applies with equal force to vendor relationships. Organisations that rush to partner with AI vendors often discover too late that they’ve inherited compliance gaps they can’t close.
Megan offered specific criteria for evaluation.
- It’s one thing for them to say they’re HIPAA-compliant, but are they actually showing you their BAAs or other data processing agreements?
- Are they sharing their retention and deletion policies?
- Do you have visibility into their audit logs?
The warning carries weight, and as our society moves towards greater expectations for transparency from AI models, you don’t want to find yourself bound to a vendor who isn’t willing to provide that transparency from the start.
For organisations navigating these requirements, Technology Rivers has developed a comprehensive HIPAA-compliant development checklist that systematically addresses these concerns.
Matching Solutions to Risk Levels
With the three pillars established, a practical question emerges: how do organisations decide which type of solution fits which problem?
Anna Shahinyan offered a clear framework: use standard automation when complexity and variability are low. In contrast, AI agents should be used when complexity and variability are high.
Archana Puthran expanded this into specific guidance for healthcare contexts:
When you have to decide yes or no, zero one, white or black, that kind of deterministic decision making, and rule-based decision making, that is traditional automation. For agents, you have complex reasoning, autonomous decision-making, and orchestration across various domains within the healthcare ecosystem.
But the most important thing is the human in the loop. I assign the highest risk, ambiguous input and complex situations which require empathy and judgment to humans in the loop.
She grounded this in a real example: Let’s say I take all the medical information in, connecting it with all other policy-related documents, putting it all together and extracting what is required, particularly important to that surgeon who’s reading the medical records. I would employ automation plus agents. But the decision-making on exactly what treatment should be offered to this patient would be the human in the loop. In this case, the surgeon.
This framework provides concrete guidance that aligns with the risk level and preserves human judgment where the stakes are highest.
Protecting Data While Enabling Innovation
Every healthcare organisation faces a tension between experimentation and protection. Powerful AI capabilities require data, but exposing patient information creates unacceptable risk.
Ghazenfer explained how RAG resolves this tension:
Whenever you have sensitive data or private internal data that you don’t want to be exposed to LLMs, keep your internal data in your internal database, let’s say a vector database. Even when you still use LLM for certain things, filter the results based on your own data.
The architecture provides multiple layers of protection. RAG would limit what your model sees by giving it only the data it needs to see at that moment, he continued. You still need anonymisation in many cases during development, or testing, and when you don’t want the real data exposed. You can, for example, say that a 60-year-old male has X, Y, and Z without providing any specific PHI.
The final layer, sandboxing, enables experimentation in isolated environments. If you combine all of these approaches, Ghazenfer concluded, that would reduce your risk, protect the patient data, and allow teams to innovate faster without waiting for the perfect data set.
Organisations seeking to implement these approaches can explore Technology Rivers’ AI and machine learning development services, which incorporate these protection layers by design.
Governance as a System, Not a Department
Where does governance live? This question surfaces in every organisation implementing AI, and Megan Kane’s answer challenged common assumptions:
Where people get it wrong is treating governance as a single person, a single job title, or even a standalone department, rather than seeing it as operations or project management. It’s a broad decision-making system. It’s not a single job.
She identified four perspectives that must converge for effective AI safety compliance metrics and governance:
- You need at least one clinical operations person who really understands patient safety, the actual clinical workflow, how users will be trained, and the human factors risks you might encounter.
- You need someone who understands regulations and legal compliance: they know the classification structures, what regulators actually expect, the contracts, and your data privacy rules.
- The third would be IT security infrastructure: they own identity and access management, understand networking, encryption, logging, and version control.
- And lastly, somebody in data science analytics: they own the development of that model, its validation and versioning, and monitoring of the performance.
These perspectives must unite for critical decisions: approving scope changes, reviewing risk assessments, monitoring post-market performance, and investigating incidents. Without all four at the table, blind spots become failures.
The Human Element That Technology Cannot Replace
The conversation took an unexpected turn when Archana Puthran introduced Ubuntu, an African philosophy emphasising collective strength:
Ubuntu, Ohana, comes from an ancient culture that emphasises the power of the collective, of being a family and supporting each other, and of harnessing your strengths rather than focusing on your weaknesses.
She connected this philosophy directly to AI implementation, addressing the fear that pervades every organisation adopting these technologies.
When building these agents, build specialists, not generalists. In building these specialised agents, you are almost stepping on the toes of someone who’s been doing that job manually. But when you create that Ubuntu culture, the messaging there is human in the loop, which says you, the human, are brilliant and will be doing more of solving those ambiguous, high-risk, complex decision-making where you need empathy, the human touch, the judgment.
The reframing matters because fear undermines adoption. When you implement this Ubuntu culture in an organisation, you reduce the fear factor.
When you talk about the elephant in the room, everybody becomes scared that AI is gonna take my job. Your role is to address it right away, assign the human to the work they’re designed to do, and elevate them.
Staff trust, override patterns, and adoption rates all reveal whether an organisation has positioned AI as empowerment rather than replacement. These human metrics deserve as much attention as any technical measure.
Scaling Safely with Multi-Agent Systems
As implementations grow more sophisticated, safety metrics must evolve. Anna addressed the unique challenges of multi-agent systems:
Agentic systems are generally safer than single AI models because they check each other’s inputs and outputs. But back to traceability, we need versioning and to log everything. This will also help us understand and debug the solutions.
She highlighted a risk that remains underaddressed across the industry: data bias. It can be ethnic, gender, vendor, or any other bias. But we still have it.
Monitoring for bias isn’t optional compliance work. It’s a core safety metric demanding continuous attention across all patient populations and data sources.
Integration That Enhances Rather Than Disrupts
Organisations worry that AI agents will create new problems within existing workflows.
Ghazenfer offered practical guidance:
The key is to treat agents the same way you treat any other service in your organisation. The moment you start treating them differently, your expectations change too.
He recommended modern integration approaches called MCP, which he calls API for AI. You can expose your services as MCP, connect with other services using MCP, then create those events and plug into your existing system. You want to keep it loosely coupled, more like a component-based system, so they interface with each other rather than becoming a monolithic application.
Most importantly, he counselled patience: Start with one workflow. Once that is done, you move on to the second one, then the third. Once you have one working, you will have much more buy-in from your team because people will see it as productivity and empowerment, not a replacement.
This gradual approach builds organisational capability to identify and respond to safety issues before they scale into crises. Technology Rivers’ healthcare software development practice applies this incremental methodology to help organisations integrate AI without disrupting the workflows that patients depend on.
The Narrow Path to Trust
Megan Kane’s closing observation crystallised the entire discussion:
Health AI agents can be most successful when they start with a very narrow scope. If you go to a pilot or release 25 different functions that your AI agents can perform out of the gate, and even one of those steps fails, it’s challenging, especially in healthcare, to win back that trust.
The alternative path requires discipline but offers far greater rewards: If you start with only a few core actions and demonstrate a high degree of confidence that you are compliant with regulations, that you have well-documented policies, and that you have the proper security and safety controls in place.
Archana reinforced this for organisations just beginning their journey: Think long-term, break down the work into different domains and create those specialist agents. At least have that in the roadmap, because it will benefit the organisation to have that design in place during the design phase.
From Metrics to Meaning
Ghazenfer Mansoor’s closing words captured what successful healthcare AI safety ultimately requires:
It takes clear workflows, clean data, and systems that let humans and agents work together without friction.
The organisations succeeding with AI agents aren’t chasing the highest accuracy scores. They’re building systems that clinicians trust, regulators approve, and patients benefit from being measured not by what the AI achieves under perfect conditions, but by how safely it operates when reality proves far messier than any test environment.
The radiologist from our opening scene deserves better than impressive accuracy numbers. She deserves a system she can explain to her patients, defend to her peers, and trust with the decisions that matter most. That kind of safety can’t be captured in a single metric. It emerges from the disciplined integration of clinical performance, operational reality, and genuine transparency.
Taking the Next Step
Building safe AI for healthcare requires more than good intentions. It demands the proper framework from the start, one that embeds compliance, governance, and trust into every design decision.
The complete expert discussion explores these themes in greater depth. Watch the full webinar to hear the panel address audience questions about specific implementation challenges, or browse the clip library for focused insights on particular topics.
For organisations ready to move from planning to implementation, Technology Rivers helps healthcare companies map workflows, identify risks, and build AI systems designed for safety from day one. Reach out to start a conversation about your specific use case and challenges.
If you want a deeper primer on why oversight matters in real-world deployments, read When AI Gets It Wrong: Why Humans Still Matter in Healthcare Decisions.
And if you’re designing agentic systems from scratch, you may also want to review Health AI Agents: What Does It Take To Succeed?
