Newsletter

I spent eight rounds making a blind tool smarter. A plain chat beat it, so I closed it

What our plan review tool cost, what it found, and the four checks I would run before your engineering firm builds or buys an AI review tool.

Updated · First published

For engineering firm owners

What our plan review tool cost, what it found, and what I'd check before your firm builds or buys one.

Updated November 2026. This new letter replaces three earlier letters on our plan review tool from August and September. What changed: those letters sold our plan review tool as running, and I closed it on Sept. 14, 2026.

One plan set reviewed three ways. The first tool read the text in the file: 119 to 175 findings, about $10 a run, wrong. The rebuild saw small cropped rectangles: 11 findings, all title blocks, $106.20, and never opened an engineering sheet. A plain
Figure: One plan set reviewed three ways: our first tool, its rebuild, and a plain chat with the whole PDF.

Watch a senior PE review a plan set. The first hour goes to checking. Is every sheet on the index in the file. Does each detail callout land on a real detail. Does every title block carry the same project name. Then he finally reaches the engineering.

Almost none of that first hour needs a license. I still believe that. On hourly work you bill it, and the client wonders why review costs so much. On lump sum you absorb it on every job, and it shows up as your margin.

In August I told you we had moved that hour off the PE's desk with software. Two more letters followed. This one replaces all three, because the tool didn't work. Here is what it cost.

Eight rounds on a tool that could not see

It read the text in the file. A drawing isn't text.

We built our plan review tool in July. It pulled the text out of a PDF and ran our checklist against it. Many CAD exports don't carry their lettering as text. Autodesk's own help says text in its SHX fonts isn't supported in PDF and is kept as drawn geometry. To our tool, a sheet full of notes and dimensions could arrive almost blank, and it filled the gaps with character recognition guesses.

We spent eight rounds making it smarter: better text recovery, a tripwire for missing sheets, warning banners, arithmetic done in code. Each round sounded right. None of them gave it eyes. On one real set, its last three runs returned 119 to 175 findings each, 16 to 24 of them Critical, for about $10 a run. I called the reports completely wrong and useless. It also reported sheets missing that were sitting in the file.

The rebuild could see, through a keyhole

It cost $106.20 and read title blocks.

So we rebuilt it to look at the drawings. It could see, but only small cropped rectangles, never a whole sheet at a size it could read. On the same set it planned 142 investigation sessions. It finished 13 before it hit its $100 spending cap, and all 13 were title blocks. The other 129 never started. It returned 11 findings, all title blocks, and never opened an engineering sheet. The bill was $106.20.

142 squares, one per planned review session. 13 are blue: finished, all title blocks. 129 are grey: never started, because the $100 cap stopped the run. 11 findings, every one from a title block. Bill: $106.20.
Figure: The rebuilt run's planned sessions, from our Sept. 14, 2026 decision record.

In August I told you the cost of a run wasn't a decision variable. At $106.20 for eleven title block findings, it was.

Before the rebuild ever ran, I had tried the plain version. I sent the whole PDF to Claude in a chat, with our checklist as the prompt. No pipeline. It came back with 31 findings for about $10. Of the three, it is the report I would hand an engineer. It had limits too. It read only part of the drawings in full, and it said which part.

Our decision record puts the lesson in one line: the orchestration was the defect, not the model. Both versions chopped the set up, queued it, ranked leads, budgeted spend and cropped regions. Every layer threw away something the model needed to see the set as a set. The $10 run had no layers.

One set is one set. I'm not saying a chat beats software every time. It beat ours on the only test we ran all three ways.

On Sept. 14 I closed the tool to new reviews. Today a reviewer may run our checklist through Claude by hand as a second pair of eyes, never as the QC of record. A person still marks up and signs off. The first hour is moving anyway, through people. Our designers run a checklist built from our reviewers' own markups before a set comes to review, and the reviewer runs the same one.

What I'd take from it

Four checks before you build or buy.

Ask whether it sees the page. Reading the text in a file is not reading the drawing.

Grade it on what it invents, not on how much it finds. My rule since August: if we are not confident, we don't create a comment. Fake comments cost a ton of engineering time.

Make it say what it didn't read. A short list from a tool that skipped most of the set looks exactly like a clean set, and now your PE trusts it. In four healthcare studies, wrong advice from a decision support system raised the risk of a wrong decision by about 26 percent. That's medicine, not engineering, but I'd bet on the same reflex here.

Test the plain version first. Send one whole set to a general model with your own checklist. If your pipeline can't beat that at the same cost, stop building.

Four checks before you build or buy a review tool: does it see the page; grade what it invents, not how much it finds; does it say what it skipped; did it beat the plain version, a whole set sent to a general model with your checklist, at the same co
Figure: Print it, and run it before the demo, not after.

Here's the exercise for this week. Take one set that came back marked up last month. Sort the comments into two piles: the ones that needed a licensed engineer's judgment, and the ones a checklist would have caught. If the second pile is bigger, you are paying PE rates for proofreading. Then ask how you would know if a tool had invented one of them.

Tell me your split through our contact page. I read every one.

Michael Groselle, PE

Founder and CEO

Modern Engineering Solutions

mike@mod-eng.com

(214) 833-6748

mod-eng.com

Talk to an engineer

Tell us about your project

  1. An engineer reads it

    Not a salesperson. Someone who can tell you whether the pathway is real.

  2. You get a first answer

    Your options, the agencies involved and the questions that decide cost and schedule.

  3. You decide

    No commitment to have the first call.

Prefer the phone? (214) 833-6748

Site location, what you are trying to build, and your deadline.

Plans, a site map or photos: up to 5 files, 10 MB each.

Read our privacy policy.