Governing the Data You Send to AI: LLM Data Flows You Can Defend

The arrival of capable AI assistants created a data-governance problem almost overnight. Employees, reasonably seeking productivity, began pasting code, documents, customer details and internal strategy into chat interfaces — often external services — with little visibility into where that data went, whether it was retained, or whether it might train a future model. "Shadow AI" became a genuine risk not because anyone was malicious, but because a powerful new data flow opened faster than governance could keep up. Governing what data reaches which models is now a core part of data protection.

The goal is not to ban AI, which fails, but to make its use safe and defensible.

Know where the data goes

The first step is visibility: understanding which AI tools are in use and what data classes are flowing to them. An external AI service may process your input on infrastructure you do not control, under terms that vary widely on retention and training use. Sending regulated data — personal information, health records, protected financial data — to a service without the right contractual and technical safeguards can be a compliance violation as much as a security one. You cannot govern flows you cannot see, so discovery comes first.

This is precisely where a self-hosted or controlled AI path changes the calculus: when the model runs on infrastructure you control, or connects via approved keys to providers under proper agreements, the data flow becomes something you can actually define and defend rather than something happening in the shadows.

Classify, then route

Sound governance connects directly to data classification. Public and low-sensitivity information may flow freely to approved AI tools; confidential and restricted data needs stricter handling — routed only to approved, contractually covered services, to self-hosted or local models for the most sensitive paths, or blocked entirely. Defining which data classes may go to which AI destinations, and enforcing it, turns an uncontrolled sprawl into a set of deliberate, auditable channels.

The controls that make this real include clear policy on approved tools, technical guardrails that prevent sensitive data from being sent to unapproved destinations, and logging of AI interactions so there is a record of what was shared — the same audit discipline applied to any other sensitive data flow.

The retention and training questions

Two questions should govern any AI destination for sensitive data: is the input retained, and could it be used to train models that others might query? A defensible arrangement answers both clearly — data not retained beyond the request, not used for training, processed under enforceable terms — or keeps the sensitive path on infrastructure where you set those answers yourself. The nightmare scenario, proprietary or regulated data absorbed into a model and later surfaced to someone else, is avoided by controlling these flows rather than hoping.

The organizations getting this right are not the ones banning AI or the ones ignoring the risk. They are the ones treating data-to-AI as a governed flow like any other: discover what is happening, classify the data, route each class to an approved and appropriate destination, keep the most sensitive paths under their own control, and log it all. That is how you get the productivity of AI without quietly leaking the company into someone else's model.

A usage policy people will follow

Policies fail when they are long, vague or purely prohibitive. A good AI usage policy fits on a page and answers the questions employees actually have: which tools are approved, what data may go into each, what must never be entered, how to check AI output before relying on it and whom to ask when unsure. Give examples drawn from real work — summarizing a public article is fine, pasting a customer's contract into an unapproved chatbot is not. Pair the policy with an approved, convenient tool, because people will use whatever is easiest if the sanctioned option is missing.

Questions to ask every AI vendor

Before approving an external AI service, get clear, written answers. Is input data retained, for how long and where? Is it used to train or improve models, and can that be switched off contractually rather than through a setting someone might change? In which regions is data processed, and which subprocessors are involved? Is there a data processing agreement? What security certifications does the vendor hold? Can administrators enforce single sign-on, manage users, view usage logs and delete data? Answers vary widely between consumer and business tiers of the same product, so check the exact plan you will use.

Technical guardrails

Policy alone does not stop mistakes. Data-loss prevention tools can detect sensitive patterns — identification numbers, card numbers, source code, documents labeled confidential — and warn or block before they are sent to an AI service. An AI gateway or proxy between users and models can enforce which models each team may use, redact sensitive fields, apply usage limits and keep logs in one place. Blocking unapproved AI services at the network or browser level reduces shadow use, but only works if the approved alternative is good enough that people do not look for workarounds.

Prompts, outputs and logs are data too

Governance often focuses on what goes into a model and forgets what comes out and what is stored along the way. Conversation histories may contain everything a user pasted, outputs may reproduce confidential material, and logs kept for monitoring become a concentrated store of sensitive information. Apply your classification and retention rules to all of them: restrict who can read conversation logs, set retention periods, include them in deletion processes and protect them as you would the underlying data. A well-governed input path paired with an unprotected log store simply moves the risk.

Source code and coding assistants

Coding assistants are among the most widely adopted AI tools, and they see some of an organization's most valuable assets. Decide which repositories may be used with which assistants, whether code suggestions are filtered for matches with public code under restrictive licenses, and how secrets in code are prevented from reaching the service. Remember that generated code is not automatically secure: it should pass the same reviews, tests and security scans as code written by people, and developers remain responsible for what they merge.

Personal data and legal obligations

When AI tools process personal data, data protection law applies in full. In Europe that means having a lawful basis for the processing, informing people where required, signing data processing agreements with vendors, managing international transfers and, for higher-risk uses, carrying out a data protection impact assessment. The EU AI Act adds further obligations depending on how an AI system is used and the risk category it falls into. Involve your data protection officer or legal counsel early in AI projects; retrofitting compliance after deployment is slower and more expensive.

Fine-tuning and training on company data

Training or fine-tuning a model on internal data raises its own governance questions. Models can memorize and later reproduce fragments of their training data, so anything included may resurface in outputs to anyone who can query the model. Exclude restricted data unless there is a strong reason and strong controls, remove personal data that is not needed, and restrict access to fine-tuned models according to the most sensitive data they were trained on. Keep records of what data went into each model version, so that you can answer questions about it later.

Agents and connected tools

AI systems increasingly act rather than merely answer: they read mailboxes, query databases, call APIs and update records. Each connection extends what data the model can reach and what it can do with it. Give agents the minimum permissions they need, prefer read-only access where possible, require human confirmation for consequential actions such as sending external messages or deleting data, and log every action they take. Treat an agent's credentials like those of a powerful service account, because that is effectively what they are.

Training people

Most AI data leaks are innocent mistakes by people trying to work faster. Short, practical training helps more than warnings: show how to recognize sensitive data, how to use the approved tools, how to remove names and identifiers before asking for help with a document, and why AI output must be checked before it is relied on or shared. Refresh the training as tools change, and make it easy to ask questions without fear of blame, so that people raise uncertainties instead of guessing.

When something goes wrong

Despite controls, someone will eventually paste the wrong document into the wrong tool. Include AI services in your incident response plan. Know how to contact each vendor, how to request deletion of specific data, how to find out whether it was retained or used for training, and how to assess whether personal data notification obligations apply. A quick, calm response — and a culture where people report mistakes promptly — limits the damage far more than discovering the leak months later.

A starting plan

Begin with discovery: find out which AI tools are already in use and for what. Publish a one-page policy linked to your data classification. Approve at least one capable tool with business-grade terms and single sign-on, and make it easy to access. Add guardrails for the most sensitive data, set retention for prompts and logs, and include AI in vendor reviews and incident response. Review the arrangement every few months, because the tools, the terms and the regulations are all changing quickly.