Evaluating AI for Construction: The Honest Buyer’s Guide

The first question: does it act, or does it store?

A system that surfaces your documents faster is a search box with good manners. A system that drafts the RFI response, flags the notice nobody served, catches the duplicate requisition and matches the invoice to the delivery is doing work.

Both are legitimate products. They are not the same product, and they are not the same price. Establish which one you are being shown, in the first ten minutes, by asking what the system DID rather than what it found.

The second question: what can it see?

This is the one that actually predicts whether an AI capability will be useful, and it is almost never asked.

An agent reading your RFI is useful. An agent reading your RFI, the contract clause it turns on, the instruction that preceded it, and the delivery that never arrived is doing a different job — and it is not doing it because of a better model. It is doing it because those four things are on the same record.

AI quality is a function of the data foundation underneath it. That is an architectural fact rather than a release note, and it is why "which model do you use?" is a much less interesting question than it sounds. A superb model reading a fragment produces a confident answer about a fragment.

The third question: can it show its sources?

Ask to see a generated document. Then ask which records each claim came from.

If the answer is vague, you cannot review the output — you can only trust it, and trusting an output you cannot check is not a workflow, it is a hope. Whereas a draft that cites its sources turns the review question from "is this true?" — which takes an evening — into "is this the right record?", which a competent engineer answers in seconds.

And the corollary is the most valuable thing a drafting system produces, which almost nobody markets: the gaps. A draft that says "the specification section this refers to is not in the document set" has found a hole in your project record. That hole was going to hurt you in a dispute whether or not you ever ran the tool. Finding it on a Tuesday is considerably better than finding it in cross-examination.

On accuracy percentages: be suspicious

You will be shown accuracy figures. Treat them the way you would treat a contractor quoting a productivity rate with no project attached: the number is meaningless without the test set, and the test set is never published.

Accuracy against what, exactly? A submittal review has no single correct answer. Two competent engineers will flag different things and both will be right. A headline accuracy figure implies a ground truth that does not exist — so a vendor producing one has either invented it or measured something far narrower than the thing you are buying.

The honest framing is different, and it is not weaker. AI review is a first pass, and its comparison is not against a perfect reviewer. It is against what happens today: a tired engineer reading the fortieth submittal on a Thursday evening and skimming the last twenty pages. Judged that way, the value is obvious — it never gets tired, it reads the whole document every time, and it checks the clause reference against the actual contract rather than from memory.

Human sign-off is not a safety feature. It is the structure.

A document that goes to a client with your company’s name on it was written by your company. An engineer signs it, and if it is wrong, that engineer and that firm carry the consequence — professionally, contractually, and in some jurisdictions personally.

That does not transfer to a model, and it is not a technology limitation waiting to be engineered away. It is the basis on which construction professionals are trusted, insured and paid.

So the reviewer is not in the loop as a safeguard. The reviewer is the AUTHOR, and the tool wrote the first draft. Any vendor telling you their system removes the need for review is describing a risk transfer to you, and pricing it as a feature.

The demo problem, and how to defeat it

Every demo is impressive. Demos are optimised to be impressive in forty minutes, on a dataset that was chosen.

Your project is not a chosen dataset. It is a document set with three revisions of the same drawing, a correspondence thread that changed subject halfway through, a specification with a missing section, and a schedule that has not been updated since March.

So take one real, ugly export from your current system and make every vendor run their product on it, live, in front of you. This single request separates capabilities from demos faster than any evaluation matrix, and the reaction to being asked is itself informative.

The tell

Vendors selling a capability will volunteer the limits. Here is what it does not do. Here is where it needs a human. Here is the case where it gets it wrong.

Vendors selling a demo will not, because acknowledging a limit is off-message. And a system that always produces an answer is worse than one that sometimes says "I could not find this in your records" — because the first one is wrong silently, and the second one is telling you something true about your project.

Ask to be shown a case where it declines. If nobody can show you one, you have learned something.

And the questions to put to any vendor, including us

These five are evergreen, and they are how you find today’s answers rather than reading somebody’s claim about last quarter:

• Does the AI review documents against OUR drawings and contract — and cite its sources?
• Does the platform follow the asset after handover, or does its record end at closeout?
• Does pricing scale with our construction volume, our user count, or neither?
• How long from kickoff to a productive site team — weeks or quarters?
• Where does our data live, and what happens to it if we leave?

The risk nobody discusses, which is not job losses

It is deskilling, and it deserves to be named rather than waved away.

If juniors never assemble a claim by hand, do they learn what a good one looks like well enough to spot a bad draft? That is a real question and the industry has not answered it.

The mitigation is the same discipline that makes the output trustworthy in the first place: read the citations before the prose, treat the gaps as findings, and require the reviewer to be the author rather than an approver. A professional checking sources against records is still doing professional work — arguably more of it, and on the part that matters.