When Convenience Meets Confidentiality in the Age of Generative AI
Law firms are confronting a quiet but growing risk: associates using generative AI tools under time pressure, without governance, oversight, or secure infrastructure.
It was a Thursday evening, and Sarah was staring at a blank screen. The senior associate at a midsize law firm had been staffed that morning on a new leveraged credit facility for a private equity client, one of the firm’s most important relationships.
The partner on the deal, Mark, had forwarded the term sheet with a brief note: “Start with Pluto and look for improvements in most recent precedents. Concurrent review by Lender and client. I’ll take a quick look before it goes. Should be straightforward.”
Concurrent review meant the client trusted the firm enough that the lender could see the draft at the same time the client did. It was a vote of confidence in the firm’s institutional knowledge, and in Sarah’s ability to get it right.
The problem was time. Sarah had two other active matters and Mark wanted a draft by Friday mid-morning. She had been hearing that associates at other firms were using AI platforms to streamline document drafting. She decided this was the time to put it to the test.
She knew the firm had a policy restricting the use of external AI tools for client work. She also knew, as most associates did, that the policy was vaguely worded, inconsistently enforced, and that no approved alternative existed that could do what she needed.
These policies existed precisely because the risks were already known, but the gap between the firm’s stated rules and its actual infrastructure left associates to make their own judgments under pressure. Sarah made hers.
What happened next produced two distinct failures: one visible and immediately embarrassing, the other invisible and potentially far more consequential. Together, they illustrate why the legal profession’s embrace of generative AI demands not just individual caution but institutional governance.
This article is a composite of real events. Names, entities, transactions, and identifying details have been anonymized.
Link to The Drafting Shortcut The Drafting Shortcut
Sarah’s plan was methodical, or so she believed. She used only the requested precedent, Pluto, and three other credit agreements from deals involving the same client closed within the last six months. She then fed in the agreed term sheet, asking the platform to produce a draft reflecting the economic and structural terms.
She did this on a consumer AI platform, the kind available to anyone with an email address. The specific platform is not material to the analysis, and the risks described apply broadly across consumer-facing generative AI tools. She logged into the same personal account she used at home, uploaded four executed credit agreements containing her client’s negotiated positions, proprietary deal terms, counterparty names, and financial covenants, and began prompting.
The platform returned a draft that, on its face, looked impressive. It had the right structure, the right defined terms, the right flow. Sarah spent several hours reviewing it against the term sheet and making targeted edits. By early Friday morning, she had what she believed was a competent, client-ready draft.
She emailed it to Mark. He skimmed through the agreement. The blackline against Pluto didn’t show too many changes, just as he would expect. He approved circulation.
What he missed was new language in the assignability section buried in the back of the agreement, language Sarah had mistakenly interpreted as more favorable to the client because it came from a more recent precedent. He also missed a small change to the restricted payments definition, only a few words, but words that imported a regulatory compliance calculation from an unrelated deal and fundamentally altered what the client could do with its cash. The draft was sent simultaneously to the client and the lender on Friday afternoon.
Link to The Call The Call
Sarah headed out for what she believed would be a work-free Friday evening. A couple of post-dinner drinks in at a wine bar, she received that dreaded angry partner call.
“Sarah, it’s Mark. James at the client just called asking if I actually read the agreement. There are provisions in the back that don’t belong, including the RP definition from Lightning. Call Lender, tell them you sent the wrong draft, and fix this tonight. James hasn’t taken it up the ladder yet.”
Mark abruptly hung up.
Sarah collected herself and called James on her way home.
“Did you draft this yourself, Sarah, or did you rely on an AI platform? You can’t seriously think we would restrict RP’s and allow open-ended assignments with no disqualified lender list.”
Sarah was mortified. She tried to suggest she had mistakenly pasted that section into the new agreement, but James wasn’t buying it. He told her to recall the draft from Lender, turn a corrected version before too many questions arose internally, and call him early the next morning with Mark on the line.
The precedent assignment section addressed a unique regulatory issue the client had faced with a portfolio company in a highly regulated industry. It had no relevance to the current transaction. The restricted payments definition required compliance with a regulatory calculation that didn’t need much language to be highly restrictive.
In such case, the provisions reflected fact-specific negotiations that had gone against the client. They were not market terms. They were compromises born of regulatory requirements unique to those prior transactions. And now they had been delivered to all parties as if they represented the client’s standard approach.
Sarah called Mark, admitted the AI use, and agreed to speak again at seven the next morning after the corrected agreement had gone back out.
Link to Two Failures, Not One Two Failures, Not One
One failure could be corrected. The other could not.
The first failure was the draft itself: visible, embarrassing, and immediately remediable. The second was that Sarah had uploaded four confidential, executed credit agreements into a consumer AI platform.
Those documents contained the client’s negotiated positions across multiple transactions, proprietary financial terms, counterparty identities, structural concessions, and regulatory accommodations. This was the kind of information that, if mishandled or disclosed outside a firm’s secure environment, could compromise a client’s negotiating position on future deals or breach confidentiality obligations owed to third parties.
The drafting error could be fixed with a revised document and an apologetic phone call. The confidentiality exposure could not be undone. The information had already left the firm’s secure environment.
Link to Why This Is a Governance Problem, Not an Associate Problem Why This Is a Governance Problem, Not an Associate Problem
It would be tempting to frame this story as a cautionary tale about one associate’s poor judgment. It would be more accurate, and more useful, to understand it as a governance failure that was structurally predictable.
Sarah did not wake up that Thursday morning intending to violate firm policy. She was a senior associate under time pressure on a matter where the partner had signaled the work should be routine. She had access to a tool that promised to compress hours of drafting into minutes. The firm’s policy told her not to use it. The firm’s infrastructure gave her no comparable alternative. The firm’s training had not equipped her to understand, in concrete terms, what the risks actually were. And the firm’s supervisory processes did not catch the result before it went out the door. Every link in that chain is an institutional responsibility.
Associates will use generative AI. This is not a prediction but a description of current reality. A 2024 American Bar Association survey found that a significant and growing percentage of lawyers were already using generative AI in their practice, and anecdotal evidence suggests actual usage far exceeds what firms’ official policies contemplate.
The question is not whether associates will use AI. The question is whether they will use it in a governed environment, with appropriate tools, training, and oversight, or whether they will improvise with consumer platforms in the way Sarah did. A written prohibition without infrastructure is not risk management. It is a gap between stated position and actual exposure.
Why the AI Could Not Know This Was Wrong
Sarah assumed that by feeding the AI precedents from a single client, she was giving it a coherent baseline. What she did not appreciate was that the platform had no independent understanding of what was “standard” and what was “non-standard” in a credit agreement. It could not compare the provisions in her uploaded documents against other deals to flag outliers.
This is not a flaw in the particular platform Sarah used. It is a category limitation of current generative AI technology.
Large language models lack contextual deal history: they cannot know that a particular provision was a concession extracted under unique circumstances. They lack market-wide comparative judgment: they cannot assess whether a clause is standard or idiosyncratic.
They also lack awareness of why deviations exist: they cannot distinguish a provision reflecting a client’s preference from one reflecting a counterparty’s leverage at a specific moment in time. These are not gaps that will be closed by the next software update. They are inherent in the architecture of tools that generate text from patterns rather than from understanding.
When the AI encountered the restrictive assignment provisions and the unusual restricted payments definition, it wove them into the draft because they appeared in the source material and its task was to synthesize. Sarah, for her part, was either too tired or lacked the depth of experience to spot the issues, and her assumption that more recent meant better only reinforced the AI’s errors.
A large language model cannot ask the follow-up question: “Is this provision standard for this client, or was it specific to the circumstances of this deal?” If the human operator does not know enough to ask that question either, the error will propagate unchecked into a draft that looks polished and inspires a confidence it does not deserve.
The Confidentiality Exposure: Understanding the Actual Risks
I received a call from Mark late Friday night. He and Sarah needed to understand exactly what had happened to the information, and they needed to be prepared to answer James’s questions on the morning call. Given the specific platform involved (which I will not name), we were able to walk through its architecture in detail.
What follows is a general explanation of the risks, ordered from most immediate to most technical, which formed the basis of our assessment.
Account History and Synced Devices
The most tangible confidentiality exposure had nothing to do with artificial intelligence. When Sarah used a consumer AI platform with her personal account, the full conversation was stored in her account’s conversation history. That includes every uploaded credit agreement and every AI-generated response.
The history is not confined to the device where the conversation took place. It syncs across every device where she is logged in: her work computer, her personal laptop at home, her phone, and any other active browser session.
Four executed credit agreements were sitting in a conversation thread accessible from Sarah’s home computer. If her partner, roommate, or child opened her laptop and navigated to the platform, the conversation would be right there in the sidebar. Fully readable. No password prompt. No second authentication layer.
This is not a theoretical concern about model architecture. It is a straightforward data security exposure: confidential client documents, in full readable form, persisting indefinitely on personal devices with no firm oversight, no access controls, and no audit trail.
Credential Compromise and Cloud Persistence
If Sarah’s account credentials were ever compromised in a data breach, or if she reused passwords across services, anyone who gained access could read the full conversation and download the uploaded documents. They would know exactly which client, which transactions, and which terms were involved.
Unlike a stolen laptop, which can be remotely wiped, a compromised cloud account provides
access to the entire conversation history. That access persists until the user changes the password and manually deletes the relevant conversations. Both steps require the user to know the compromise has occurred.
Platform Data Retention
Beyond the user’s own account, the platform itself retains data. Consumer AI platforms have published data usage policies that have evolved over time.
Some consumer AI platforms have historically retained conversation data and reserved the right to use it for model improvement, though users could opt out. Enterprise-tier products from major providers typically offer stronger data isolation, contractual commitments around data handling, including no-training guarantees and defined retention periods.
If the platform’s infrastructure were ever compromised in a security breach, the conversation contents could be exposed not through the model’s outputs, but through access to stored logs.
The Model Risk: Lowest Probability, Highest Anxiety
The question Mark feared most was whether the uploaded documents could surface in another user’s AI-generated output. This is the least probable of the four risk layers, though it is the one that generates the most anxiety.
Large language models are trained on vast datasets before they are deployed. A user’s conversation inputs are not used to retrain or update the model in real time. When you upload a document, you are not adding to the model’s permanent knowledge. The model processes your input, generates a response, and moves on.
It would be extraordinarily unlikely, bordering on technically implausible, for another user to prompt the model and receive back a recognizable excerpt from the client’s credit agreement.
This assessment provided some comfort, but not certainty. There is no way to provide an absolute guarantee of zero risk from any of these vectors. Low probability does not mean zero responsibility.
This is precisely why most firms prohibit the use of open AI platforms for confidential work. The risks are layered: confidential information persisting on personal devices through synced account histories, cloud-stored conversations vulnerable to credential compromise, data retained on infrastructure the firm does not control, and the residual possibility of exposure through the platform itself.
A Fortunate Ending
No client suffered financial harm, and the issue was addressed before negotiations substantively progressed.
The lender had not yet substantively reviewed the credit agreement. The firm moved quickly. The lender agreed to delete the circulated draft on the basis that the associate had been working from an incorrect set of precedent templates. A revised draft, prepared the old-fashioned way, was circulated within forty-eight hours.
James was displeased but pragmatic and able to manage the situation internally. The relationship survived, though trust was damaged. Sarah received a stern conversation about firm policies. Mark was confronted with the supervisory obligations that attach to every document that leaves the firm with his name on it.
The principle for every partner is straightforward: if you send it, you wrote it, you own it.
Mark had a friend to call and was able to avoid broadcasting the incident broadly. His firm lacked a dedicated AI practice. I subsequently offered to conduct a CLE and best practices seminar for Mark and his team, in which we worked through several similar hypotheticals. The resolution was more fortunate than the conduct deserved.
Lessons Learned
This episode, condensed and anonymized from real events, illustrates several institutional risks that law firms must confront as generative AI becomes embedded in legal practice.
1. Firms Must Ensure AI-Assisted Drafting Includes Experienced Review
If the input contains deal-specific concessions, the model will treat those concessions as standard. It has no independent judgment about what is market and what is bespoke. Any AI-assisted drafting workflow must include substantive review by someone with sufficient experience to evaluate not just whether the output tracks the term sheet, but whether the provisions below the term sheet level reflect appropriate market positions.
2. Firms Must Build Processes for Curating Precedent Selection
Sarah concentrated the AI’s source material in a way that ensured idiosyncratic provisions would be treated as the norm. Each agreement reflects not just the client’s preferences but also the counterparty’s leverage, the market conditions, and the specific facts of the underlying business. Precedent selection cannot be delegated to associates who may not have the experience to distinguish a preferred position from a situational compromise.
3. Supervisory Systems Must Match the Speed of the Tools
When AI enables associates to produce drafts faster, firms must ensure the time saved is reallocated to review, not absorbed into other matters. The speed AI brings to drafting must not outpace the quality controls designed to catch precisely these errors.
4. Firms Must Close the Gap Between Policy and Infrastructure
A policy that prohibits external AI without providing a viable internal alternative is not a control. It is an invitation for workarounds. Firms need approved tools that actually meet associates’ needs, training that explains risks in concrete terms, and enforcement mechanisms that create accountability.
5. Firms Must Train for Layered Confidentiality Risk
The risk that uploaded documents will surface in another user’s AI output is vanishingly small. The more immediate risks are conversation histories synced across personal devices, cloud-stored account data vulnerable to credential compromise, and the absence of any audit trail. Training must address these concrete exposures, not just the abstract question of model training.
6. Concurrent Delivery Demands the Highest Institutional Standard of Care
When a client authorizes concurrent delivery, it is placing extraordinary trust in the firm as an institution. That trust carries a correspondingly elevated duty of care, one that rests on the firm’s systems and supervision, not solely on the individual associate. It is precisely the wrong context for untested tools or unsupervised AI-assisted workflows.
Looking Ahead
The scenario described here is not unique to credit agreements. Similar risks exist wherever generative AI is used to draft complex legal documents from precedents: merger agreements, employment agreements, regulatory filings, and any context where fact-specific provisions from prior deals can be mistaken for standard terms.
The legal profession’s use of generative AI is going to accelerate. The choice facing firms is not between AI and no AI. It is between governed adoption and improvised adoption. Every firm will end up with one of these two outcomes. The question is which one they choose to build.
This article is a composite of real events. Names, entities, transactions, and identifying details have been anonymized. It is intended for educational purposes and does not constitute legal advice.
Michael Simon Baker is an attorney advising law firms and businesses on AI governance, confidentiality risk, supervision, and the responsible use of generative AI in legal workflows.