Using generative AI in healthcare settings: Turning off model training is not always enough

Common misconceptions

When entering medical information into generative AI, some assume that simply turning off the training setting avoids any problems. In practice, experts point out that this alone may not be enough.

Are you using AI like this? (Risk assessment required)

[Example 1] Drafting a referral letter with ChatGPT: You enter the course of a patient's symptoms to draft a referral letter. You assume there is no problem because you omitted the patient's name and turned off model training.

[Example 2] Recording a conference with a personal transcription app: You record an in-hospital conference and generate minutes using a microphone-equipped transcription app under a personal subscription.

[Example 3] Entering patient information to draft a nursing care plan: You enter details such as "Male in his 70s, history of heart failure, lives alone, declining ADL..." and ask for nursing care plan ideas.

Any of these uses can create easily overlooked risks.

This article addresses situations where medical institutions use generative AI for work, focusing mainly on cases where staff enter and process patient medical information. Because conclusions can vary depending on specific circumstances, this article provides general information only.

Regulations on medical information

Handling medical information with generative AI generally requires considering the following two areas.

(1) Act on the Protection of Personal Information

First, you must comply with the Act on the Protection of Personal Information (APPI). The APPI is a statutory law, and violations can result in administrative dispositions and penalties in addition to administrative guidance.
It serves as the baseline regulation regardless of how medical information is handled, raising issues such as provision to third parties, cross-border data transfers, and supervision of outsourced contractors. From here, this article focuses on issues related to the safety management guidelines.

(2) Safety Management Guidelines (Three-Ministry Two-Guideline framework)

When handling digitized medical information in medical information systems, institutions must consider guidelines on the safety management of medical information systems in addition to the APPI.

The primary guideline for medical institutions is the Ministry of Health, Labour and Welfare's (MHLW) "Guidelines for Safety Management of Medical Information Systems." A related standard is the "Guidelines for Safety Management of Medical Information by Providers of Information Systems and Services Handling Medical Information," issued jointly by the Ministry of Economy, Trade and Industry (METI) and the Ministry of Internal Affairs and Communications (MIC) (together, these are often called the "Three-Ministry Two-Guideline" framework).

The MHLW safety management guidelines apply to information systems at medical institutions and similar entities that handle medical information. These guidelines are not statutory law, and violating them does not immediately trigger legal penalties. However, authorities may refer to them when interpreting a medical institution's duty of safety management, and compliance is also considered desirable from the perspective of patient safety.

Requirements under the guidelines

The MHLW safety management guidelines set out requirements such as the following:
- Servers must be located in jurisdictions subject to Japanese law (with some exceptions)
- The network connecting the medical institution and the AI must ensure appropriate security (such as TLS)
- Settings must prevent input data from being used for model retraining
- Provided medical information must be reliably deleted after the contract ends

Why contractual guarantees are essential

The Q&A accompanying the guidelines addresses how to handle cases where contracts or similar agreements guarantee that medical information will not be stored (for example, explicitly stating conditions that prohibit storage for training purposes). Rather than relying solely on on-screen settings, institutions are expected to put these guarantees in writing through contracts or internal rules.

The guidelines also state that institutions must establish agreements covering the division of responsibilities with contractors, explanations of specifications and operations, reporting documents on operational status, and the management of subcontracting.

Why individuals cannot easily meet these requirements

When relying solely on standard terms of service without tailored contract terms, it is difficult to fully secure the contractual guarantees described above, such as clear demarcation of responsibility and auditability. For this reason, using generative AI through personal accounts or personally contracted apps and cloud services requires caution.

Applying this to the examples

Among the opening examples, using a standard personally contracted generative AI service to draft a referral letter or a nursing care plan can leave unresolved issues regarding agreements on the demarcation of responsibility.

Transcribing conference recordings with a personal smartphone app or personally contracted cloud service poses even greater challenges, because the hospital or medical institution cannot enter into a formal contract or clearly define the division of responsibility.

Furthermore, these three examples (drafting a referral letter, transcribing a conference, and drafting a nursing care plan) require caution under the APPI as well as the safety management guidelines. The APPI involves multiple legal issues, and what constitutes a problem can vary from case to case. Beyond reviewing general information, institutions should consult experts, including attorneys.

Appropriate ways to respond

Practical options are limited.

Dedicated medical AI services compliant with the Three-Ministry Two-Guideline framework

Guidelines recommend confirming that a service meets the specified security standards, training data controls, and contractual guarantees before using it. Although options are currently limited, the wave of digital transformation (DX) reforms heading toward 2030 should bring more services over the next few years.

On-premises AI installed within the hospital

A configuration that keeps medical information from leaving the institution can reduce the risk of data leaks. At the same time, it requires the organization to handle design and operations in line with operational costs and technical requirements.

For further reading

The issues surrounding generative AI use in healthcare settings are complex, making it difficult to list every potential problem comprehensively. Caution is also necessary under the APPI, and specific legal issues can vary depending on the individual case. In addition to reviewing general information, institutions should consult experts, including attorneys.

The "Guidelines on the Use of Generative AI in the Medical and Healthcare Field (2nd Edition)" by the Medical AI Platform Technology Research Association systematically organizes these issues. It covers everything from the APPI to the requirements and interpretations of the safety management guidelines, as well as practical compliance methods. Anyone considering the use of generative AI in a clinical setting should refer to this document.

https://haip-cip.org/news/20250711/

Summary

When handling medical information, choosing a dedicated medical service compliant with the Three-Ministry Two-Guideline framework or an on-premises AI system is currently the appropriate approach. Casual use carries the risk of violating laws and regulations. Make appropriate choices to protect patient trust.

References:
1) Ministry of Health, Labour and Welfare, "Guidelines for Safety Management of Medical Information Systems"
2) Ministry of Economy, Trade and Industry / Ministry of Internal Affairs and Communications, "Guidelines for Safety Management of Medical Information by Providers of Information Systems and Services Handling Medical Information"
3) Medical AI Platform Technology Research Association, "Guidelines on the Use of Generative AI in the Medical and Healthcare Field (2nd Edition)"

Related:
Key points healthcare professionals should know first when using generative AI

What healthcare professionals should know first when using generative AIshoei05.com