Confidential data entering AI: what happens after the prompt

The risky moment is often ordinary.

Someone pastes a client email into an AI tool and asks for a summary. A manager uploads a draft contract and asks for cleaner wording. An accountant copies a financial statement into a chat window and asks for a plain-language explanation. A staff member uses an approved AI workspace because the business bought the right tier and told the team to use it.

The work may be legitimate, the tool sanctioned, and the account company-managed. The problem starts when the business assumes the data is gone because the tab is closed or because the vendor says the data is excluded from model training.

Once business content enters an AI tool, the vendor's data-handling architecture decides what happens next.

Retention settings, saved chats, memory features, subprocessors, support access, deletion options, and data residency all matter. Microsoft 365 controls, network filtering, and encryption inside the business no longer govern the data after it crosses into the AI vendor's systems.

Shadow AI covered invisible tools. This article covers a different problem: confidential data entering AI tools, including tools the business approved.

What the risk is

Confidential data entering AI means company, client, employee, financial, legal, or regulated information is entered into an AI system where the vendor's terms and technical design control what happens to it.

Approval is only the first question. The data-handling question is more specific:

What data can go into this AI tool, and what happens to that data after it enters?

The answer depends on the tool, the account tier, the contract, and the settings. A free personal account may have one set of terms. A paid individual plan may have another. A team plan may reduce training use but still retain chat history by default. An enterprise contract may offer stronger controls but still require configuration, review, and a retention decision.

For an SMB, the risk usually sits in a few places:

• Saved chats and project workspaces keep client or company content long after the immediate task is done.
• Memory or persistent-context features may store facts from previous conversations and reuse them later.
• Vendor retention defaults may keep prompts, uploads, outputs, and logs longer than the business expects.
• Subprocessors may handle storage, moderation, support, security, analytics, or model infrastructure.
• Data residency may be unclear, especially when a product is sold locally but inference or support happens elsewhere.
• Deleting one client's information may be difficult if it is mixed through months of general chat history.

None of this requires staff to be careless. It happens when the business gives staff a useful tool without deciding which data classes belong in it and without documenting the vendor's data-handling terms.

How it happens in a normal SMB

A small accounting firm pays for a team plan from a well-known AI vendor. The owner chose the team plan because the vendor says customer data from that tier is excluded from public model training. It was a responsible starting point and a clear improvement over staff using personal accounts.

The firm gives five staff members access. They use the AI workspace daily: summarizing client emails, drafting tax memos, reviewing financial statements, preparing plain-language explanations for owners, and drafting responses to CRA correspondence. Staff like it because it saves time and improves first drafts.

The tool becomes part of the firm's normal client work. A senior accountant creates one project for tax planning language. Another keeps a long-running chat for construction clients. The owner uses a saved thread to turn technical notes into client-ready explanations. Nobody is trying to hide anything. The business bought the tool, assigned seats, and told staff to use it.

The missing decision is data handling. The owner checked the training-use headline before buying, but the firm never documented the retention period, chat-history settings, memory features, deletion process, subprocessor list, or data residency position. The chat history stays on by default. Saved projects accumulate months of client information. Staff lack a simple rule for which client data can enter the tool.

The Shadow AI problem from the previous article is under control here. The unresolved question is what the vendor does with the data.

The failure path

The chain looks like this:

Case file Sequence 02 · Confidential data

1. 01

   The business approves an AI tool and gives staff access.

2. 02

   Staff use it for real work because it saves time.

3. 03

   Client emails, financial details, HR material, contracts, or other confidential content enter prompts, uploads, saved chats, or projects.

4. 04

   The vendor's retention, memory, deletion, subprocessor, and residency rules govern that content.

5. 05

   The business has not documented those rules in a way it can explain to a client.

6. 06

   A client, insurer, buyer, regulator, or lawyer asks where the data went and whether it can be deleted.

7. 07

   The business can answer only in partial terms because it never mapped the tool to the data class.

The technical fact is simple: data entered into AI lives in the vendor's systems on the vendor's terms.

Vendor terms vary widely. Many sit in the middle: training use is addressed, while retention, memory, logs, support access, subprocessors, or deletion are still conditional. The business has to know which situation it is in before confidential data goes into the tool.

Business consequence

The first consequence usually shows up in client confidence.

The accounting firm's largest client is a regional construction company that represents about a third of annual billings. The controller calls one Friday before year-end and asks a direct question:

The owner says yes, in a controlled way. The firm uses a paid team plan.

The controller asks the follow-up questions:

• Where does the AI vendor keep our information?
• How long is it retained?
• Is it used for training?
• Can you delete our information from the AI tool if we ask?
• Who at the vendor or its subprocessors can see our financial data?

The owner knows the tool has handled the construction company's work; staff use it daily. She chose the better tier because the vendor said the data was excluded from training. The unanswered items are retention, memory, processing location, subprocessors, and how to delete one client's information without deleting the firm's broader AI work history.

She tells the controller she will get back to her.

Over the weekend, she reads the vendor documentation. The answers are dense and conditional: settings that may be configurable, deletion options tied to whole chats or workspaces, logs that may persist for a period after deletion, a longer-than-expected subprocessor list, and residency language that is hard to match to the client's question.

On Monday, she sends a careful partial answer. The controller thanks her and asks for clearer assurance before the next engagement. The relationship continues with a different tone. The next renewal includes more security and vendor questions than usual, and the construction company asks two other firms how they handle AI data. When the work is reviewed later, the accounting firm is still invited to bid, with the AI answers now part of a broader confidence problem.

The client concern was the firm's inability to answer basic data-handling questions about a tool it used every day.

Other consequences follow:

• Client questionnaires become harder to answer.
• Contract terms around AI, data handling, deletion, or subprocessors become harder to meet.
• Insurance and acquisition diligence ask questions the business cannot answer cleanly.
• Future incidents become harder to scope because client data is spread through saved chats, projects, and vendor logs.
• Staff may believe "no training" means "safe for all client data," which leaves retention and deletion questions unresolved.
• Personal information may create privacy-law obligations if the AI use later becomes part of an incident.

For many SMBs, the first pain is commercial. A client loses confidence, a renewal becomes harder, or a bid tilts toward a firm that can explain its AI data handling with less scrambling.

Controls that interrupt the failure path

The first control is a data rule staff can understand. The business needs to decide which types of information may go into which AI tools.

The table is the staff-facing control. It should tell a person whether the data in front of them can go into the AI tool they are using. If the rule requires staff to interpret vendor terms in the moment, it will fail.

The tier decision needs its own review. A paid team plan may be the right minimum for ordinary business use, but the business should not assume the tier answers every question. Read the terms. Confirm the settings. Write down the answer in business language.

The deletion question deserves special attention. Before confidential client data enters an AI tool, the business should know whether it can delete a specific chat, project, client workspace, uploaded file, or memory entry. If the only practical deletion option is to purge an entire account or workspace, that limitation belongs in the decision record.

Policy rule this creates