Accuracy
Every number below is a production F1 score benchmarked against human-annotated ground truth in MLflow. Not demo runs. Not hand-picked drawings. Our best models exceed 95% F1. Our weakest sit in the 60–70% range. We publish both, because a source that only reports wins isn’t a source worth citing.
F1 balances precision — of the things the model flagged, how many were real — against recall — of the things that were really there, how many it found. A model that finds every duct run while hallucinating false ones has high recall and low precision, and it blows your bid either way. F1 catches both failure directions in one number, and it’s the measure used in the machine-learning literature, so it lets you compare vendors on the same terms.
Accuracy is not one number. It is a number per trade, and the spread across trades in our own stack is more than 30 F1 points. Any vendor quoting you a single blended accuracy figure is averaging their hardest trade into their easiest one.
Measured against human-annotated ground truth. Sorted strongest to weakest.
| Trade / model | F1 | What it covers |
|---|---|---|
| Ceiling & area segmentation | F1 0.997 | Region classification; IoU 0.995. The closest thing to solved in our stack. |
| Structural steel — beam line detection | F1 0.93–0.95 | End-to-end. Beam label classification 99.6%; centerline extraction F1 0.94; brace detection F1 0.92. |
| Architectural — page region classification | F1 0.97 | Sheet-level structure: which region of the page holds the drawing versus schedules and stamps. |
| Mechanical — GRD subtype classification | F1 0.91 | Supply, return, and exhaust diffuser classification once detected. |
| Architectural — door detection | F1 0.88 | Door symbol detection across architectural floor plans. |
| Mechanical — ductwork runs | F1 0.85 | Every straight duct segment — horizontal, vertical, diagonal — with width measurements. |
| Mechanical — HVAC fittings (combined) | F1 0.84 | Elbows, dampers, GRDs, flex ducts combined into one pipeline number. |
| Architectural — wall-type segmentation | F1 0.842 | Production mask F1 on a content-crop validation set; boundary F1 ~0.92 on clean sheets. |
| Mechanical — GRD supply detection | F1 0.80 | Detection of supply diffusers specifically, before subtype classification. |
| Architectural — wall type, end-to-end | F1 ~0.72 | Detection plus label reading. The gap to 0.92 boundary F1 is schedule selection, not segmentation. |
| Mechanical — flex duct detection | F1 0.72 | Flexible duct runs; harder than rigid because the geometry is not straight. |
| Electrical — device detection | F1 0.64 | Our hardest problem. mAP@50 reaches 0.81. Most active model improvement work sits here. |
Separately, we publish the floors we are willing to be held to on work we have not seen before: electrical ≥ 0.90 (fixtures and conduit), architectural ≥ 0.85, mechanical ≥ 0.74. Those are floors, not best cases. Six months ago most of the numbers in this table were 15–25 points lower.
These are the specific ways our models are wrong today. Naming them is how you tell a measured system from a marketed one.
On multi-schedule sheets the reader sometimes locks onto a sibling table — door, fire, or finish — instead of the wall-type legend. When it does, the segmentation is often perfect (boundary F1 0.92–0.98) while the label comes from the wrong vocabulary. Roughly a quarter of pages show this. Moving from a weaker reader to the production reader dropped affected pages from 11 to 4 of 50. Explicit legend-and-schedule disambiguation is the highest-value next fix.
Electrical sheets stack devices, homeruns, circuit tags, and panel schedules in the same visual space at small scale. At F1 0.64 the pipeline finds most devices but precision suffers in dense areas, which is why we report electrical device detection separately from the electrical fixture-and-conduit floor of 0.90.
Flex duct at F1 0.72 is the clearest case: models trained to trace straight segments with consistent width degrade when the run curves. Rigid ductwork at F1 0.85 on the same sheets shows the gap is geometry, not sheet quality.
Sheet quality ranges from crisp CAD exports to fourth-generation scans. This is the distribution problem our synthetic data pipeline exists to cover — thousands of edge-case blueprints per hour with degraded quality, unusual symbol placement, overlapping annotations, and non-standard legends.
On a real hospital set, an AI ductwork takeoff came back at 1,096 linear feet against a senior mechanical estimator’s 3,396 — a 78% quantity match. That sounds like a failure. The estimator still reached a reviewed, bid-ready number in 22 minutes.
The metric that matters is time to reviewed takeoff, not raw quantity match. Correcting a wrong number in place is faster than producing one from a blank sheet. Accuracy determines how much review the estimator does — it does not determine whether the output is useful. At 95% the psychology flips entirely: spot checks instead of full reviews, and the question becomes how fast the bid goes out.
Per-trade F1, the methodology behind it, and the trades where they’re weakest. Most won’t answer. That’s the answer.
Boon Agent starts at $375 one-time for 5M tokens. Founding members — the first 1,000 estimators — get 100K free tokens.