Newsletter

I told you to audit the setting. I didn't audit mine

A correction to my July letter on AI and your firm’s data, and six questions to ask about where that data goes. For engineering firm owners.

Updated · First published

For engineering firm owners

A correction to my July letter on AI and your firm's data, and the questions I should have given you instead.

Updated November 2026 from my July letter. What changed: this is a correction, because the July letter made claims I can't stand behind.

A correction card. What I wrote: retention set to zero, every AI call through one gate, search lets nothing out, access details in print. What I can stand behind: not what our records show, not today by choice, a search sends words out, taken out of
Figure: What I wrote in this letter in July, and what I can stand behind today.

In July I wrote you about what happens to your firm's information when your people use AI. The best line in it was this one: you cannot audit a promise, you can audit a setting.

Then I didn't follow it. Here is what that letter got wrong.

I wrote that we had set our AI provider's data retention to zero ourselves. Our own records don't show that it was zero when I wrote it. The audit I told you to run would have caught it.

I wrote that every AI call in the company went through one locked gate. That isn't true today. One gate was a good story. It stopped being our setup, and the letter kept telling it. I also shouldn't have described our setup in public at all.

I wrote that our web search let information in and nothing out. A search sends your words to whoever runs it. That is outbound.

I opened with a story about a public AI model that learned a company's product because someone asked it questions. That mixed up two things. Asking a question doesn't send a model off to read the web about you. What a provider keeps, and whether it trains on it, depends on the terms of the plan you are on.

And I published details of our own access controls. That belongs in an audit, not a newsletter. This letter takes its place in the archive.

So here is the letter I should have written. It isn't about our controls. It's about the questions you should be able to answer about yours.

A flow from your staff to the AI tool, then out to three places: the provider, which may keep it; training, which the contract should cover; and search or plug-ins, which send it on.
Figure: Each box is a question for your vendor or IT lead. No vendor or firm is shown.

Six questions about where your data goes

Ask your vendor or your IT lead, and ask for the document, not the answer.

First, which plan is each person on? This is the big one. When Anthropic changed its consumer terms in 2025, the change covered its Free, Pro and Max plans. Users who allow their chats to be used for training have them kept for five years. The change did not apply to its Team, Enterprise or API customers, or to Amazon Bedrock. Same company, different rules, depending on who paid. Is anyone at your firm putting client files into a personal account?

Second, what does the provider keep, and for how long? "Nothing" is an answer you should see in writing. On Amazon's AI service, for example, retention is a mode you choose, and some models only run if you allow Amazon to keep prompts and answers for up to 30 days for human review.

Third, is the setting the same everywhere it runs? On that same service the setting is set region by region, and a region nobody configured falls back to each model's own default. Ask which regions your tools use, and what each one says.

Fourth, does it train on what you send? Get the contract clause, not the help page.

Fifth, where else does your text go? Web search, plug-ins and the vendor's own subcontractors all receive something. Ask for the list.

Sixth, who reads the setting, and when? A setting is proof only on the day someone reads it. Put a name and a date next to it.

A blank worksheet with the six questions, a line for each answer, and a Yes or No box for whether the answer came from a document.
Figure: One sheet per AI tool. Print it, or copy it into a spreadsheet.

The fair objection is that this is a lot of paperwork for a small firm. For most tools, the answers sit in the vendor's business terms and on an admin page nobody has opened. It's an afternoon of reading. It's the afternoon I skipped in July, and it would have kept me from writing a promise I couldn't show you.

I still believe you cannot audit a promise. I'd add one thing I learned the hard way: you can only audit a setting on the day you read it.

Here is the exercise for this week. List every AI tool your staff used last week, including the ones on personal accounts. For each one, answer the six questions. Then count how many answers came from a document, not from somebody's memory.

Tell me your count through our contact page. I read every one.

Michael Groselle, PE

Founder and CEO

Modern Engineering Solutions

mike@mod-eng.com

(214) 833-6748

mod-eng.com

Talk to an engineer

Tell us about your project

  1. An engineer reads it

    Not a salesperson. Someone who can tell you whether the pathway is real.

  2. You get a first answer

    Your options, the agencies involved and the questions that decide cost and schedule.

  3. You decide

    No commitment to have the first call.

Prefer the phone? (214) 833-6748

Site location, what you are trying to build, and your deadline.

Plans, a site map or photos: up to 5 files, 10 MB each.

Read our privacy policy.