AI and governance: Building trustworthy systems in high stakes healthcare

JOAN PORCARO: Welcome to the WTW Willis Vital Signs podcast program. And to our audience, thank you for joining. My name is Joan PORCARO, and I'm the Senior Vice President Of Risk services for WTW health care. This particular series is called The AI project, What Risk Managers Need to Know.

And this is a one part of many part series on this wonderful topic of artificial or augmented intelligence. And as AI becomes embedded golly into everything from radiology to triage to clinical documentation, risk managers are having to navigate some new and complex challenges around oversight, safety, and accountability. And in today's episode, we're going to focus on one of the critical topics, which is AI governance. And I'm very fortunate today to have two guests with me, and I'm going to ask that they each introduce themselves. So, Jonathan, let's start with you.

JONATHAN DAVIES: Thank you, Joan. Hi, everyone. My name is Jonathan Davies. I am a cyber risk consultant based in Las Vegas, Nevada. I have been with Willis for about seven years. And as a cyber risk consultant, I am fully embedded within our cyber broking team and help out our cyber insurance clients with their various governance, risk and compliance needs.

JOAN PORCARO: Well, thank you Jonathan and welcome. And again, we have a repeat guest Anthony. Anthony, you're up.

ANTHONY WILSON: Hi, good afternoon. My name is Anthony Wilson. I'm talking to you from Southern Arizona. I am also a cyber risk consultant for Willis. And what I particularly focus on is financially quantifying the financial aspects of cyber risk. And AI- related risk for Willis and Willis clients.

JOAN PORCARO: Well, thank you and welcome again. Thank you for joining us for another episode. So I'll throw the question out to either one of you to answer. Let's start with the beginning. What is meant by AI governance?

ANTHONY WILSON: If I could just jump in quickly with a short preamble-- first of all, Jonathan and I are both the security wonks, so a lot of people groan and roll their eyes at the gloom and doom naysayers from security. So first of all, want to acknowledge, what a revolutionary technology, artificial intelligence is and what it is imparting to organizations productivity and efficiency, amongst other things.

That said, as risk professionals, we need to focus on the potential downside to AI or risk. That is the subject that we will be addressing today. The only other thing that I would like to mention is when we talk about AI governance, we also really need to talk about data governance because there are really two sides of the same coin.

And when I think about AI and AI risk, I basically bin it into three buckets, if you will. One is the risk associated with the uplift that artificial intelligence can impart to malicious cyber actors, basically the gain in efficiency, frequency and lethality, if you will, of their malicious cyber activity against organizations. The second is what artificial intelligence can do to help the defenders in defending themselves against malicious cyber actors using artificial intelligence or are not.

And then finally, and this is really the core of what we're going to be discussing today is artificial intelligence technology as an attack source of or origin of potential risk for organizations. And so with that, I'll pause there and see if my colleague Jonathan has any comments.

JONATHAN DAVIES: Thanks, Anthony. So going back to the question, what do we mean by AI governance or governance more broadly. When you strip it down, governance is simply a structure that makes sure the technology you deploy is doing what you intend it to do safely and that someone is accountable when it doesn't. It's the people, policies, and processes that sit around a tool so it's not operating in the dark.

Now I find the easiest way to explain AI governance to a clinical audience is to compare an AI model to a new employee. When you hire someone, you don't just hand them a badge and walk away. You check their credentials, you define their job, you supervise them, you hold them accountable to their level of responsibility, and you keep a manager in the loop.

AI governance is that same discipline but applied to an algorithm, what is this model allowed to do, what data can it touch, Who reviews its output before it affects a patient, and who owns the outcome when it's wrong? So governance isn't a document that sits on a shelf. It's the operating system for how your organization acquires, approves, deploys, and monitors AI from a clinical decision support tool all the way down to a scheduling bot in the back office.

ANTHONY WILSON: And that's a great comment. And the only thing that I would emphasize there that Jonathan pointed out was that governance framework or process could and should embrace both technology to monitor the activity and output of AI technology in an organization's tech stack, but also the humans responsible for that oversight. So AI governance is a partnership of humans and associated technology.

JOAN PORCARO: Those are great comments. And Jonathan, I really want to lean into what you said about the example of the new employee. Governance is really just a very fluid and an active process. It's not just a document that sits on a shelf somewhere. So thank you for that example. I pushed to thinking, if in the act of omission, if you don't have such a program, what's the risks associated with operating your AI models without a formal governance structure?

ANTHONY WILSON: As a former CISO, I can tell you we had a lot of problems with the concept of shadow IT, shadow data, and now shadow AI in the environment that I was responsible for defending. And what is shadow AI?

Like IT and data its partners in crime if you will, it's ungoverned, unsanctioned, unapproved technology operating in your environment, most likely with the best of intentions by an employee, but which may increase the risk to the organization of unintentional data leakage as well as the myriad of other potential problems that AI models could have, such as model drift, hallucinatory behavior biases, or even unethical behavior. I'll pause there if Jonathan has comments.

JONATHAN DAVIES: Thanks, Anthony. So without a formal governance structure, the first thing that shows up usually isn't a dramatic clinical failure. It's loss of visibility. Leaders simply don't where AI is being used or who's using it. And the worst version of that is what Anthony referred to what we call shadow AI.

Imagine a nurse pasting patient information into a free public chatbot to save time. No policy, no oversight, and PFI has just left the building through a vendor nobody vetted. From there, the risks clustered into three buckets clinically, operationally, and ethically. So clinically, ungoverned models hallucinate. And by that I mean they produce confident wrong answers. And they drift over time as the real world changes.

So a tool that was accurate at go live quietly degrades. Operationally, you get overreliance on automation errors, feeding flawed decisions into scheduling, triage or documentation, plus a bigger attack surface for cyber threats. And then finally, ethically, you get bias as Anthony mentioned, bias and discrimination baked into decisions, opaque black box outputs no one can explain and patients who were never told I was involved in their care. The through line is that all of these are happening whether or not you have governance. Governance doesn't create the risk, but it can determine whether you see it coming or find out from a headline.

ANTHONY WILSON: And something I'd just like to add more specific to health care, several of us, Joan included, were at Willis Dallas health conference back in June. And one of the issues we were discussing was the use of AI scribes in clinical settings, essentially, use of ambient AI to produce not only documentation of a clinician's interview of a patient, but to also generate diagnostic output, as well as even proposed treatment options for the patient resulting from that interview.

And the issue here is that busy clinicians, let's say in an emergency room or some other setting where they're really short on time if that doctor, that nurse, if they don't properly review the output from that AI Scribe before signing their name to it, they are possibly opening up not only themselves, but their hospital to possible malpractice lawsuits. And, that's a somewhat extreme case, but it emphasizes the importance of the human in the loop concept of human review of AI tools output. And we'll talk about that I think a little bit later in the session.

JOAN PORCARO: Great call outs. I think most of us may be aware that AI has been around for a longer while than we might realize, maybe from the 1950s. But this new generation of apps that we're starting to see just-- new ones come about almost daily brings forth a lot of opportunities, but also some new risks.

And I think we're still maybe early enough days where there's certain novelty of the applications and it's going to be really important to keep those guardrails in place, which leads me to the question on what makes this a little more challenging, maybe a little worse for health care? What's the risk of ungoverned AI?

ANTHONY WILSON: Well, in essence, it's due to the fact that the health care industry, particularly health providers, exist in a highly regulated industry due to the sensitivity of the personal health information that they're handling for their patients, but also, the Department of Health and Human Services, the HIPAA Act, and other legal frameworks that provide regulatory oversight of the health care industry, in particular, with a focus on protecting patient personal health information or PHI.

There's also the fact and I alluded to this the last question, which is the health care industry is not just handling the production of widgets that are going to be put in a box and mailed off to a customer somewhere. The outputs potentially from AI tools can directly affect the health, safety, and welfare of patients. And so essentially the stakes for the health care industry are much, much higher than almost any other industry.

JONATHAN DAVIES: I agree with Anthony 100%. I mean, on the stakes in most industries, a bad AI output is an inconvenience, whereas in health IT can be a misdiagnosis, a delayed dose, or a safety event. And the data is the most sensitive there is.

We've watched third party disruptions like the Change Health Care and Synovus Attacks, cascade across hundreds of providers, and the Ascension ransomware event take electronic medical records offline for weeks. Now AI sits right on top of that already fragile, highly connected environment and adds new failure modes like hallucinations, model drift, and data poisoning. Anthony is absolutely right on the regulation and liability web you're sitting in.

This is where health is genuinely unique because you already have HIPAA, FDA oversight of software as a medical device, and now AI -specific transparency rules layered on top. Federally, the ONC Health IT transparency requirements pushed disclosure of how predictive decision support tools work though the exact federal expectations are actively shifting right now, which tells you how fast this is moving. Now, states like Colorado, California, and Texas are moving on AI and the EU AI Act treats most clinical AI as high risk with GDPR scale penalties.

Then there's third party liability. Most of your AI arrives through vendors and a model trained on data of unknown origin, or a vendor that quietly changes its model can hand you the liability while you have the least visibility. You can outsource the tool, but you can't outsource the accountability to your patient.

JOAN PORCARO: Well, that's an excellent call out, and I know in a future episode, we're going to just really lean into the idea of contracting with third party vendors, because that does present a whole, other avenue of concerns. But looking ahead a little bit, what elements really make for good governance? What parts and pieces do we need to look at?

JONATHAN DAVIES: If I can jump in there, for me, primarily good governance is visible and it's cross-functional. It consists of the following few things working together. First, a standing AI governance committee that is genuinely multidisciplinary, not just IT You want the CIO and CISO at the table, but also a clinical leader, risk management, legal, compliance, HR, and someone who owns the data.

Secondly, an inventory, a living list of every AI tool in use, whether it's homegrown or from a vendor, what data it touches and what it's used for. You can't govern what you can't see. And then third, clear guardrails and acceptable use policy that says which tools are approved, what data may never go into an AI system, and where a human has to review the output before it reaches a patient.

And critically, for this audience, good governance is what earns clinician trust. A lot of clinicians are skeptical of AI, and for good reason. Governance answers their skepticism directly. It requires that outputs be explainable, that models be validated before and monitored after deployment, and that AI is positioned as support for clinical judgment, never a replacement for it. When a clinician can see why a tool made a recommendation and knows a human stays in the loop and accountable, that's when trust actually forms. Good governance makes the AI a transparent colleague instead of a black box.

ANTHONY WILSON: Yeah, and something else I'd add here-- and I completely agree with Jonathan's comments. Something that I've been personally tracking a bit in-- I recognize the supreme importance of that AI or advanced Technology Committee, whatever the organization calls it, multidisciplinary. Even organizations appointment of a chief AI officers I've seen in some companies and public sector organizations.

But there's a growing number of technical solutions for-- I wouldn't say they're not necessarily AI governance, but they serve that element that Jonathan referred to of providing information, transparency, and insight into how the various components of an AI stack in an organization's environment are operating, whether they're operating within acceptable governance, risk and compliance standards or not-- they're designed to create greater observability monitoring of the AI models, performance, tournament's security even and also from a data loss or DLP perspective.

A couple other elements, the whole process to use an IT term, Identity Access Management has grown increasingly complicated in recent years with the number of non-human identities that IT departments and their IAM or identity and Access Management Systems have to monitor credentialize, authenticate, and authorize for access to different data sets.

And, you've got everything from API tokens to chatbots to-- you name it. There's just a vast number of different non-human identities. Well, AI technologies just increasing the number of non-human identities in an organization's environment and monitoring and administering and regularly reviewing the access that those identities have, which typically in many cases, unlike human identities, aren't regularly reviewed necessarily is extremely complex.

And then final point, and really the scope of today's discussion doesn't allow time for this is the issue of AI sovereignty and sovereign AI, which is something that you're increasingly seeing with some more sensitive industries like health care, defense, et cetera and maybe certain types of critical infrastructure. But you're also seeing it enacted on national scale or sometimes a nation say in partnership with key companies in its region.

For example, you see a lot of work in this respect in the EU, for example, but also some other countries. So AI's sovereignty is the strategic framework by which organizations are consortiums of organizations, maybe even public private partnership create these for lack of a better word, trusted environments.

And AI sovereignty is really the strategic framework for those trusted environments whereas the concept of sovereign AI is actually a technical implementation of the concept of AI sovereignty whereby the group, the consortium, the country what have you, will basically take over control of their AI stack from initial development and training of a model, all the way through deployment, using their own data and their own infrastructure to essentially create that trusted environment.

It's an interesting concept. I don't know necessarily how far the US is along those lines. You tend to see this most particularly in regions or parts of the world where they have particularly strict national data privacy laws like the EU, like some other countries, but it is being adopted by some organizations, including in health in the US.

JOAN PORCARO: A couple of things I just want to draw attention to that you said that really resonated with me as a risk professional. And I would say, first and foremost, the idea that risk is sitting at the table during the early design stages of whatever the governance plan is usually, we may get called in, but sometimes it's later in the process, or only when a problem has arose.

Probably the other two things clients ask me the most about is how to inventory and look at where is AI living in our buildings, in our hospitals. And then finally, just the acceptable use. There may be IT policies about certain technical aspects, but is there a good policy in place for clinical applications and how they are used and when they're used and what's the notification process? Because sometimes physicians may bring an application in. So is there a good solid process for the approval of bringing something in? So I thank you for your comments on that.

And I probably want to go into another question. Not all health systems have many levels of leadership in their AI process. We have small hospitals, that might be single hospitals and then you have large multi-hospital systems. Where's a good place to start on a path to good AI governance?

JONATHAN DAVIES: That's a great question because it can be very easy to fall at the first hurdle when you view governance as such a huge mountain to climb. My advice is start small and start with reality, not with a 50-page framework. The mistake is thinking you need a perfect enterprise grade program before you do anything, and you really don't.

You need to take the first three practical steps and build from there. So step 1, find out where AI already is, inventory, survey your departments, ask your vendors point blank whether their products now use AI. And you usually discover it's already in more places than leadership first realized.

Step 2, triage by risk. A model that influences diagnosis or dosing is in a completely different category to one that drafts a meeting summary. Govern the high risk patient facing uses first, and don't burn energy over controlling the harmless ones. And then step 3, put the basic policy and a small committee in place. So there's a front door, a defined way for someone to propose a new AI tool and get a yes or a no instead of just quietly using it.

If you do want a North Star to follow can always lean on a free public framework so you're not inventing everything from scratch. The NIST AI Risk Management Framework of Governance Measure Manage is free and practical. And then later, ISO 42001 exists if you want something that you can be certified against. The point of starting this way is that it's scalable. A five-person committee meeting monthly with a one page intake form is a legitimate governance program. You mature it over time rather than just waiting for perfect. Anthony.

ANTHONY WILSON: Sure. No. All good points, Jonathan. Something that you mentioned earlier, but I can't also emphasize enough is that whether it's your AI committee or some organizations call it that, maybe their advanced technology committee because it or not, quantum is looming out there, subject of a future podcast, no doubt. But it's important that leadership attend those calls regularly, that it not become an excuse for them to duck out and leave it to the mid-level IT person.

The saddest thing I ever heard from the CEO of a company once when asked about a cybersecurity program was, oh, I've got so many it that handles that for me. OK, that's going to turn out well. So setting the example, showing leaderships emphasis and concern about this issue by being present in those meetings is absolutely a good place to start.

JOAN PORCARO: We're thinking about if you could just mention just maybe one or two best practices for organizations to follow. What would you suggest?

JONATHAN DAVIES: I don't know if I could do one or two, but I can give you a handful. And you stop me when you've had enough. So firstly, good data governance as Anthony mentioned at the start of the recording, AI is only as trustworthy as the date are under it. Clean, well-governed data is a precursor to every successful AI deployment. Get the foundation right before you scale models on top of it.

Next, we've mentioned this a few times. Keep a living AI inventory and risk register. Know every model, its source, its data, and its risk tier. Treat it as a living document rather than a one time audit. Make the committee multidisciplinary and give it a real cadence. Clinical risk, legal compliance, HR, security, and data, meeting at least monthly with decisions published in a known place so the organization can see what's been approved and why.

I mentioned before, validate before and monitor after deployment. By testing models before deployment, also, keep watching for drift and bias afterward, because a model that was safe at go live can quietly degrade over time. We also talked about keep a human in the loop for anything patient facing always requiring human review of AI output before it affects care and demand, explainability. So clinicians can interrogate a recommendation.

Anthony talked about shadow AI earlier. Tackle that head on. Give people an enforceable acceptable use policy plus approved safe tools. If you only ban things without offering a sanctioned alternative, people will root around you. Extend governance to your vendors, really important. Most health AI arrives through third parties, so bake AI specific due diligence, contract terms on data use and accountability and ongoing monitoring into your vendor management processes.

And then finally, plan for failure. Have an incident response plan for when an AI tool gets it wrong, escalation paths, patient communication, and a way to switch the model off. What good looks in one sentence? AI is visible, validated, explainable, supervised by humans, and tied to a named owner. And your people trust it because they can see all of that. Anthony.

ANTHONY WILSON: Yeah. No. All great points, Jonathan. Only thing I would add, and this is something I alluded to earlier in this podcast was training for the users, particularly clinicians who are ultimately going to have to sign their name to the bottom of any document produced by an AI tool in any way relates to patient care, appropriate training, and not just one and done, but regular retraining on how to use the tools and some of the pitfalls.

And this is a particular concern because while not a health professional like Joan, myself, I'm aware of how many health providers are very short staffed, nurses, doctors, other clinical personnel are really rushed for time. And there's that temptation to just give the 15-20 second glance at the ambient AI scribes output on a patient interview and then sign your name to it. And it's that urge needs to be resisted.

And then finally, just because I am the risk quantification wonk on this call is something that is useful for organizations and their management of this new risk factor that AI represents for all of us is, understanding what that risk could cost the organization in terms of liability, regulatory fines, third party lawsuits, et cetera. And so, taking the time, working with your insurance partner or your risk manager to really dig into what are the potential financial ramifications of the AI tooling in your technology stack.

JOAN PORCARO: Oh, excellent. Really good call outs for an organization to not feel too intimidated about a lengthy process, really taking those small steps is a good place to start. So what's one piece of advice you'd like to leave with our audience today?

JONATHAN DAVIES: Well, Joan, if there's one key takeaway, I would like it to be this, that governance is the brake on innovation. It's what lets you accelerate safely. The instinct in a lot of organizations is to treat compliance and innovation as opposites, where every control slows you down. I would argue it's actually the reverse.

The organizations that move fastest with AI are the ones that build the guardrails first, because they can say yes with confidence rather than saying no out of fear. And you don't need to be a data scientist or a lawyer to lead this, which matters because the people listening are managers, nurse leaders, HR, and operational directors. You already how to manage people, manage risk, and ask good questions. AI governance is the same muscle.

So my advice is simple. Start now. Start small, and start with one question in every meeting where a new tool comes up, how are we governing this? If you make that question routine, you're already most of the way there. Protect the patient first and the innovation takes care of itself.

JOAN PORCARO: That is a really excellent point that you brought up. So thank you. So we've come to a close for this episode. And, first I want to give a hearty thank you to our guests. And first, thank you, Jonathan.

JONATHAN DAVIES: Thank you, Joan

JOAN PORCARO: And to Anthony, thank you for joining us again.

ANTHONY WILSON: Yeah, thank you very much, Joan. Always a pleasure to work with you and as always with Jonathan.

JOAN PORCARO: And in closing, I want to thank our audience and those who have tuned in to our discussion. I hope you'll join us again for future discussions in the coming months. And again, thank you for listening to our podcast WTW Willis Vital Signs.

NARRATOR 3: Thank you for joining us for this WTW podcast featuring the latest thinking on the intersection of people, capital, and risk. WTW hope you found the general information provided in this podcast informative and helpful. The information contained herein is not intended to constitute legal or other professional advice, and should not be relied upon in lieu of consultation with your own legal advisors. In the event you would like more information regarding your insurance coverage, please do not hesitate to reach out to us. In North America, WTW offers insurance products through licensed entities, including Willis Towers Watson Northeast, incorporated in the United States, and Willis Canada, incorporated in Canada.