What questions should you ask an AI construction software vendor?

The six questions

Most AI demos are impressive and most AI purchases disappoint, and the gap between those two facts is almost entirely covered by the following:

• Does it ACT, or does it only store and search? A system that surfaces your documents faster is a search box. A system that drafts the RFI response, flags the unserved notice, and catches the duplicate requisition is doing work. Ask which one you are buying.
• Where does the content come from? If a draft is generated from a general model of what construction documents usually say, it is fluent prose about somebody else’s project. If it is assembled from YOUR specification, YOUR diaries, YOUR instructions, it is a draft. These are not the same product.
• Can it cite its sources? Ask to see a generated document and then ask which records each claim came from. If the answer is vague, you cannot review the output — you can only trust it, which is not a thing you should be doing.
• Who signs off, and is that enforced? Not "there is a review step" but "nothing issues without a named human author". Ask to see what happens if someone tries.
• What happens when it is wrong? A system that says "I could not find this in your records" is more valuable than one that always produces an answer. Ask to be shown a case where it declines.
• What does it do with messy data? Every demo runs on a clean dataset. Your project is not one. Ask them to run it on a real, ugly export from your current system, live.

The question behind the questions

All six are really testing one thing: is the vendor selling a capability or a demo? A demo is optimised to look impressive in forty minutes. A capability is optimised to be useful on a Tuesday, on a project that is behind, with a document set that is a mess.

And there is a tell. Vendors selling a capability will volunteer the limits — here is what it does not do, here is where it needs a human, here is the case where it gets it wrong. Vendors selling a demo will not, because acknowledging a limit is off-message.

And one you should ask about the vendor, not the product

Who is accountable for the output, contractually? If a drafted document goes out with your company’s name on it and it is wrong, that is your liability, not theirs — and it should be, because your engineer signed it.

What that means is that the product’s job is not to be right. It is to make it fast and cheap for your engineer to CHECK whether it is right. Any vendor who tells you their system removes the need for review is describing a risk transfer to you, and pricing it as a feature.