What should a health system do when a vendor adds AI?

By Verillian. Published . Updated .

A vendor switching on an AI feature is an operational change to a business associate relationship. Here is what HIPAA asks a health system to check and update.

Two doctors in white coats sit side by side on chairs in a waiting area.

Treat it as a change to a business associate relationship you already have, and work through it in order. Find out whether patient information now reaches the AI feature and, if so, who besides the vendor receives it. Check what the business associate agreement already requires of the vendor, including for anyone the vendor hands the information to.

Update your security risk analysis and evaluation, because HIPAA asks for that in response to operational changes. Then decide whether to turn the feature on, for whom, and on what terms, and write the decision down. None of that requires the feature to be new or exotic. It requires someone to notice that it arrived.

Why does a vendor’s AI feature change anything?

A vendor that creates, receives, maintains, or transmits protected health information for you is already a business associate under the definition at 45 CFR 160.103, and the rules already govern that relationship. What an AI feature can change is the path the information takes.

If the feature sends it to another company’s model, that company may be a business associate too, a subcontractor of your vendor. Under 164.502(e) and 164.308(b), a business associate may permit a subcontractor to handle the information only after obtaining satisfactory assurances that the subcontractor will safeguard it, documented in a written contract or other arrangement.

You are not the party that has to obtain those assurances from the subcontractor. You are the party who has to be able to find out whether the chain exists.

What does the business associate agreement already require?

The required terms are at 164.504(e)(2). Read them as a checklist for the new feature. The contract must establish the permitted and required uses and disclosures and may not authorize any that would violate the rule if you did them.

The vendor must not use or further disclose the information other than as the contract permits or the law requires, and must use appropriate safeguards. It must report any use or disclosure not provided for by the contract, including breaches of unsecured information.

It must, in accordance with 164.502(e)(1)(ii), ensure that any subcontractors that handle the information for it agree to the same restrictions and conditions. And at termination, if feasible, it must return or destroy the information and keep no copies. You also keep the right to terminate if the vendor violates a material term.

Whether your contract permits a particular AI feature depends on the contract’s wording and the vendor’s description of the feature. Contracts written before the feature existed may not say. That is a job for your counsel.

What does the Security Rule ask you to re-evaluate?

Three provisions bear on it. The evaluation standard at 164.308(a)(8) asks for a periodic technical and nontechnical evaluation, based initially on the standards and afterward “in response to environmental or operational changes affecting the security of electronic protected health information.”

The risk analysis at 164.308(a)(1)(ii)(A) calls for an accurate and thorough assessment of the potential risks and vulnerabilities to the confidentiality, integrity, and availability of that information. And the documentation provision at 164.316(b)(2)(iii) asks you to review documentation periodically and update it in response to the same kind of changes.

Whether a vendor turning on an AI feature counts as an operational change affecting the security of the information is a judgment, and we think it is a reasonable one to make, particularly if the feature changes where information goes.

Our full reading of the risk analysis and OCR is in what evidence OCR would expect for AI use in a health system.

What should you ask the vendor?

Put the questions in writing and keep the answers. None of them assumes the feature is risky, and the answers may be reassuring.

  • Does the feature send patient information outside the vendor’s own environment? To whom?
  • Is each recipient bound to the same restrictions and conditions as the vendor, and under what written arrangement?
  • Is any of the information kept, and for how long? Is it used to improve a model, and can that be turned off?
  • What does the vendor log about the feature’s use, and can you see that log?
  • What does the vendor do if it learns of a use or disclosure not provided for by the contract, and how fast does it tell you?
  • What happens to information the feature touched when the contract ends?
  • Can the feature be turned off, or limited to some departments or roles?

What if you cannot get clear answers?

You can leave the feature off, limit it to workflows that involve no patient information, or escalate to the vendor’s leadership. Any of these is a legitimate outcome. “We did not know it was there” is the one that is difficult to defend, because a change you never noticed is a change you never evaluated.

The related question of what happens when the AI feature is a scribe, and who keeps what it produces, is in our post on ambient scribe transcripts.

What should a health system have in place?

  • A way to learn when a vendor adds an AI feature, through a contract clause requiring notice, and a named person who reads vendor release notes for the systems that hold patient information.
  • An intake step for that change, with a short questionnaire like the one above and a decision owner.
  • An up-to-date list of business associates and, where the vendor will tell you, their subcontractors.
  • A risk analysis and evaluation that are updated when the change is made, not at the next annual cycle.
  • A record of the decision, including a decision to leave the feature off.
  • A record of which AI services your own staff use directly, so the vendor’s feature is not the only AI you know about.

What record does Verillian keep?

Each prompt Verillian captures, and the reply, is kept as a signed record on your health system’s own servers, each entry naming the person assigned to the enrolled device, the device and the time. That’s the last item on the list above.

Sources

  • 45 CFR 160.103, definition of business associate.
  • 45 CFR 164.502(e), disclosures to business associates, and 164.504(e), business associate contracts.
  • 45 CFR 164.308, (a)(1)(ii)(A) risk analysis, (a)(8) evaluation, and (b) business associate contracts and other arrangements; and 164.316(b)(2)(iii), updates to documentation.

How Verillian finds the AI already in use

See Verillian govern AI on a real device.