Before Your AI Agent Goes Live: 7 Questions to Ask About Accountability and Your Vendor Contracts

November 1, 2026

AI agents can now do real work on their own. They can route a claim, move a request forward, or start a payment without waiting for a person. That is a big opportunity for healthcare, insurance, and financial services. It also raises a simple question: who answers when an agent gets something wrong? 

 

Washington is starting to ask that question out loud. A new federal AI task force is pressing developers to disclose incidents. The U.S. Treasury Department has said AI companies won't be shielded from liability. But the law still doesn't say whether the developer or the business using the agent answers when something goes wrong. Most businesses don't build their own AI agents. They buy or license them from a technology vendor, such as Salesforce, ServiceNow, or an AI company, and the vendor agreement is where responsibility is supposed to be spelled out. Yet most of those contracts were written before anyone asked this question. 

 

That gap is your opening. Nearly all companies are investing in AI, but according to McKinsey research, only 1% of leaders call their companies mature in how they deploy it. The businesses that settle AI accountability before go-live can move faster, with fewer surprises and more trust from customers and regulators. 

 

Strong AI governance keeps AI agents in check with controls like monitoring, testing, and human oversight. Accountability answers a different question: who's on the hook when something goes wrong. It sits inside Governance, the first of KKC's Four Pillars of AI Readiness: Governance, People, Data, and Process.


Use these seven questions as a checklist before your next agent goes live. 


1. Who inside your company owns this AI agent's decisions? 


Why it matters: When an agent acts, someone at your company needs to answer for it. If ownership is shared by "the team" or "IT," questions from customers, auditors, or regulators can bounce around with no clear answer. 

 

What good looks like: 

 

  • One named person owns each agent, not a committee. 
  • That owner signs off before go-live and after major changes. 
  • Everyone knows who to call when the agent does something unexpected. 
  • Ownership is written down alongside the agent's other governance controls, such as testing and monitoring. 


2. What does your vendor contract say about who is responsible when the agent gets it wrong? 


Why it matters: Since the law hasn't settled AI agent liability, your contract is where responsibility gets decided. If the contract is silent, you may carry more risk than you expect. 

 

What good looks like: 

 

  • The contract clearly states who is responsible for which kinds of errors. 
  • Indemnification terms cover harm caused by the agent's actions, not just software bugs. 
  • Your legal, compliance, and procurement teams have all reviewed the terms together. 


3. Was your contract written for software that acts on its own? 


Why it matters: Most AI vendor contracts were written for tools that suggest. An agent does more than suggest. It takes action. Terms built for a recommendation engine may not fit a system that sends messages, updates records, or approves requests. 

 

What good looks like: 

 

  • The contract describes what the agent is allowed to do on its own. 
  • It covers actions the agent takes, not only the content it produces. 
  • New features or expanded permissions trigger a contract review. 


4. How will you and your vendor report it when something goes wrong? 


Why it matters: The new federal task force is pressing developers to disclose incidents. Your customers and regulators will expect the same openness from you. A clear reporting path turns a problem into a fast, well-handled fix. 

 

What good looks like: 

 

  • The contract sets a timeline for the vendor to tell you about incidents. 
  • You know who tells whom, inside your company and out. 
  • There is a simple internal escalation path that your people have practiced. 


5. Can you see, keep, and explain what the agent did? 


Why it matters: If you can't show what an agent did and why, it is hard to prove who is responsible. Records protect you and help you improve the agent over time. 

 

For example, a healthcare organization might use an agent to help process prior authorization requests. If a patient or provider questions a decision, the organization needs to show what the agent saw and what it did. 

 

What good looks like: 

 

  • You have access to logs and decision records for every agent action. 
  • The contract says your company owns its data and the agent's records. 
  • You keep access to those records even after the contract ends. 


6. Where does a human step in, and who has the final say? 


Why it matters: As operations teams learn to manage AI agents alongside people, clear handoffs keep people in charge of the decisions that matter most. They also make accountability easier to explain. 

 

For example, an insurer might let an agent handle simple claims decisions, with a person reviewing anything above a set amount. A financial services firm might let an agent prepare a loan or payment approval, with a person giving the final yes. 

 

What good looks like: 

 

  • Written thresholds for when the agent must hand off to a person. 
  • People have clear rights to pause, override, or reverse the agent. 
  • Each handoff is recorded, so you can see who decided what. 


7. Does your insurance and risk plan cover what the agent does? 


Why it matters: Once you know who owns the agent and what your contract says, you can see where risk still sits with you. Your insurance and risk plans should reflect that. 

 

What good looks like: 

 

  • You have asked your broker how your current coverage applies to AI agent actions. 
  • Each agent is listed in your risk register with its owner and risk level. 
  • Your risk plan is updated when an agent's role or permissions change. 


Turn These Answers Into AI Readiness 


Accountability is one part of Governance. Strong Governance connects to the other three pillars of AI readiness: People who know their roles, Data you can trust and trace, and Process that defines how work gets done. Together, these pillars shape where your organization stands on the four levels of AI readiness. 

 

These answers will also change over time. As new rules take shape, plan to review your vendor contracts and update your answers to these seven questions. 

 

KKC helps businesses in healthcare, insurance, and financial services reinvent how they work with AI. We build the governance, people, data, and process that make new technology stick. We don't advise from the shore. We get in the water with you. 

 

Ready to see where you stand? Take the AI Readiness Score


Frequently Asked Questions 



Who is liable when an AI agent causes harm? 

The law doesn't say yet. A new federal AI task force is pressing developers to disclose incidents, and the U.S. Treasury Department has said AI companies won't be shielded from liability. Until the rules are clear, your vendor contract and your internal ownership need to spell out who answers for what. 


Should we review existing AI vendor contracts, or only new ones? 

Both. Most AI vendor contracts were written before anyone asked who answers when an agent causes harm. Review existing contracts, especially for agents that take action on their own, and build the seven questions above into every new contract. 


How is AI accountability different from AI governance? 

AI governance is the full set of controls that keep agents in check, such as monitoring, testing, and oversight. AI accountability is one part of governance. It answers a narrower question: who is responsible when an agent's action causes harm, inside your company and in your vendor contracts. 


What is AI readiness, and how can we measure it? 

AI readiness is how prepared your organization actually is to use AI well. KKC measures it across four pillars: Governance, People, Data, and Process. The AI Readiness Score shows where you stand today and where to focus next. 

 


 

This article is for general information and isn't legal advice. Talk with your legal counsel about your specific contracts. 


INSIGHTS

AI Readiness vs. AI Strategy: What Does Your Organization Actually Need?

By Carly Whitte • October 7, 2026
Before deciding where AI should go, you need to understand whether your organization is prepared to support it. That’s the difference between AI readiness and AI strategy.

Explainable AI Is the Adoption Strategy Health Payors Are Missing

By Paul Benvenuto • September 30, 2026
Explainable AI (XAI) means an AI system can give a clear, traceable reason for a specific decision, not just a probability value.

What Happened at Dreamforce 2026?

By Carly Whitte • September 19, 2026
Dreamforce always offers a glimpse into where enterprise technology is heading. This year, the message is especially clear: AI is moving from experimentation into the systems, workflows, and decisions that power everyday business. Here are five developments the Kona Kai team is watching closely: 1. AI agents are taking on more meaningful work Agentforce continues to evolve beyond simple tasks. Salesforce’s newest agents are designed to pursue longer-term goals, learn new skills, collaborate with other agents, and handle more complex work across sales, service, commerce, and workforce operations. 2. Salesforce is building AI specifically for CRM reasoning One of the biggest announcements is Koa, Salesforce’s first CRM reasoning model, developed with NVIDIA. Koa is purpose-built to help agents reason through complex, multistep enterprise workflows using nearly three decades of Salesforce CRM intelligence. 3. AIforce is taking Salesforce beyond the traditional CRM interface Salesforce introduced AIforce , a new live interface layer that essentially makes Salesforce’s data, workflows, business logic, permissions, security, and governance available within the AI tools where people are already working. That means employees and AI agents can interact with Salesforce capabilities through experiences like Claude, Slack, and custom AI interfaces without always having to work directly inside the traditional Salesforce UI. AIforce is launching with Claudeforce, Slackforce, and Agentforce Coworker. For enterprises, that could fundamentally change how employees interact with CRM while making strong data architecture, permissions, and governance even more important. 4. Slack is becoming a true AI workspace Salesforce continues to position Slack as a place where people, agents, enterprise data, and workflows come together. Instead of AI existing as a separate tool, the goal is to bring intelligence and actions directly into the conversations where work is already happening. 5. Governance is becoming part of the AI infrastructure As agents become more autonomous and Salesforce capabilities become available across more interfaces, trust and governance have to evolve with them. It reinforces something we talk about often at Kona Kai: successful AI adoption depends on more than deploying the technology. Organizations need the right governance, people, data, and processes around it. Our biggest takeaway The conversation is shifting from “What can AI do?” to “How do we operationalize it responsibly across the enterprise?” The announcements coming out of Dreamforce show just how quickly the technology is evolving. For organizations, the opportunity now is to make sure their strategy, systems, data, and people are ready to evolve with it.

The AI Skills Gap Isn't Going to Close on Its Own

By Paul Benvenuto • August 19, 2026
AI is changing workforce training from a one-time project into a continuous business capability. For decades, enterprise technology transformations have followed a predictable pattern. A new system is implemented, then employees learn how to use it. Productivity dips for a while, then recovers as the organization adapts. Whether it was a CRM implementation, ERP modernization, a claims platform replacement, or a core banking upgrade, the skills gap eventually disappeared because the technology itself stopped changing. AI is different. Unlike traditional enterprise software, AI capabilities continue to evolve after implementation. New models are released, AI agents become more capable, and workflows change faster than most organizations can retrain employees. The result is a workforce that isn't simply learning a new system, but continuously adapting to one. That fundamentally changes how organizations should think about workforce readiness. Recent research from the World Economic Forum and Microsoft's Work Trend Index suggests many organizations already recognize the challenge. Are enterprises doing enough to prepare for a skills gap that may never close? AI Changes the Rules for Workforce Training Traditional enterprise software had a finish line. Once employees learned the new system, their knowledge remained valuable for years. Training programs could be planned, measured, completed, and archived because the technology itself remained relatively stable. AI doesn't offer that stability. Employees who learned effective prompting techniques six months ago may now be using AI agents. Teams that started with document generation may now be automating entire workflows. Capabilities continue to expand, changing what effective work looks like almost as quickly as organizations can document it. That means workforce readiness can no longer be viewed as a milestone that follows implementation, as it needs to become part of day-to-day operations. The AI Skills Gap Doesn't End After Go-Live The challenge isn't simply that AI is changing jobs. It's that AI itself keeps changing. Foundation models continue to improve. New copilots are released. AI agents take on increasingly sophisticated tasks. Features that didn't exist six months ago become standard workflow tomorrow. Employees aren’t learning one “system” because they need to continuously adapt to new capabilities. Someone who learned the most effective way to use AI six months ago may already be working differently today. Traditional training models weren't designed for that pace of change. AI Is Reshaping the Workforce Faster Than Organizations Can Respond The World Economic Forum's Future of Jobs Report 2025 highlights just how significant this challenge has become.

Trustworthy AI Starts With the Data You Can't See.

By Paul Benvenuto • July 31, 2026
PwC's April 2026 AI Performance Study surveyed 1,217 senior executives across 25 sectors and found something that should reframe how every regulated organization talks about AI investment: nearly three quarters of AI's economic value is being captured by just one fifth of organizations. Not because that top fifth has better models. PwC is specific about the differentiator: those organizations are 1.7 times more likely to have a Responsible AI framework and 1.5 times more likely to have a cross functional AI governance board. Their employees trust AI outputs at twice the rate of everyone else's. The value gap is structural, not a matter of who bought the better tool. That finding lands differently once you connect it to where trust actually comes from. It doesn't come from a more sophisticated model. It comes from knowing where your data originated, who touched it along the way, and what controls sat around it the entire time.  McKinsey's June 2026 research on AI data readiness makes the case that most organizations manage data like a storage problem when they should be managing it like a supply chain. A single PDF can expand into extracted text, tables, images, metadata, sensitivity tags, and quality scores, each one an intermediate artifact that AI systems reuse and recombine downstream. A small error introduced upstream doesn't stay small. It propagates. This matters more in regulated industries than almost anywhere else, because the data causing the most exposure is usually the data getting the least attention. Structured fields get governed. Clinical notes, claim narratives, loan officer comments, and audit trails, the unstructured stuff, usually don't, even though AI systems depend on it heavily. Gartner and IDC both put the share of enterprise data that is unstructured at somewhere around 80 to 90 percent. McKinsey's own research doesn't cite that specific figure, but makes the same underlying point: unstructured content is where AI systems draw the most context, and where governance attention is thinnest. None of this is an argument for waiting until your data is perfect before you deploy anything. PwC's 2026 Digital Trends in Operations Survey argues directly against that instinct: AI can help bridge data gaps, particularly through agents that reason using whatever data is actually available. The real mandate isn't clean data as a prerequisite. It's disciplined governance and iterative improvement running in parallel with deployment, calibrated to how much risk a given use case actually carries. So what does that look like in practice for a CIO or CDO sitting inside a regulated organization right now? A few diagnostic questions worth asking before your next AI initiative launches: Where does data quality actually break down in your pipeline, and does anyone own fixing it? Is lineage visible for the data feeding your highest risk AI use cases, or is it assumed? Where do unstructured assets, like clinical notes, policy documents, and loan files, enter your systems without any governance attached? Have you defined what "good enough" data quality means for each use case, calibrated to its actual risk profile, rather than applying one standard everywhere? Answering those honestly is uncomfortable in most organizations, because the answer is usually "we don't fully know." That's the point. You cannot govern what you cannot see, and you cannot trust an AI output built on a data foundation nobody has actually traced. The organizations in PwC's top 20 percent didn't get there by waiting for perfect data or by buying a better model. They got there by treating governance as a financial performance variable, not a compliance checkbox, and by building the lineage and controls that make trust possible at scale. Kona Kai's data supply chain assessment is built to answer exactly these questions before tool selection, not after. If you're not certain where your organization would land on that list, that uncertainty is worth resolving now. Get in touch to talk through what the assessment covers. Sources: PwC 2026 AI Performance Study, April 13, 2026 (74%/20% figure and 1.7x/1.5x/2x multipliers confirmed directly at pwc.com); McKinsey, AI Data Readiness: The Key to Scaling Impact, June 2026; Gartner and IDC estimates for the 80-90% unstructured data share; PwC 2026 Digital Trends in Operations Survey.

Agentic AI Changes the Job. Your Workflows Haven't Caught Up.

By Paul Benvenuto • July 29, 2026
Every governance and workflow framework most organizations are running today was built for AI that waits for a human to ask it something. Agentic AI doesn't wait. It initiates, executes, and chains actions across systems on its own, and the workflows built around human initiated, human reviewed steps simply don't have

AI Literacy Isn't Training. It's an Operating Model.

By Paul Benvenuto • July 27, 2026
Education was the number one way companies say they adjusted their talent strategy in response to AI. And yet most organizations still treat training as an event. A workshop. A certificate. A box that gets checked once and never revisited.

Policies Don't Create Governance. Enforcement Does.

By Paul Benvenuto • July 20, 2026
Most organizations think they have AI governance because someone in legal drafted a policy and got it signed off. They don't. A policy sitting in a shared drive doesn't know where your AI is actually running. It doesn't flag it when a model drifts. It doesn't do a single thing when an employee routes a client file thro

The Four Pillars of AI Success: Governance, People, Data, and Process

By Paul Benvenuto • July 20, 2026
Governance, people, data, and process are not sequential steps. They are four load-bearing walls, and in regulated industries, a crack in any one of them shows up as risk somewhere else. Here is where each pillar actually breaks down today, and what the data says about the gap between where most organizations sit and w

Why Regulated Industries May Have an Unexpected AI Advantage

By Carly Whitte • July 1, 2026
AI success depends on more than technology. Governance, regulation, and operational oversight are helping organizations turn AI pilots into scalable business capabilities.