bldg

Benchmarks

What was measured and on which plans. The full tables and the method are in Measured benefits.

These figures were measured with the earlier Python implementation of bldg, on its version of the format, with its benchmark scripts and an earlier local set of test plans. Those scripts are not part of bldg today and the current engine has not re-run them, so the numbers below are not regenerated by it.

Measured benefits

Measured on 55 ground-truth plans: 31 from public floor-plan datasets and 24 synthetic, each stored as a mm document on its image (27 with a calibrated scale, 28 with an assumed one). Method, definitions and environment are in Measured benefits.

Tokens for the same drawing, at full (fine) drawing detail. This is the only depth measured: walls with their joins and cuts, doors with frame / jambs, leaf thickness and swing arc, windows with frame, sash, glass and sill, furniture as catalog symbols, stairs, printed dimensions, type marks, room names with areas. Every baseline is generated from the compiler's fine drawing and decoded back to check that it draws the same primitives (58 of 58 plans for each baseline; the bldg answer recompiled draws them too). Baselines are rounded to 1 mm (0.01 in), coarser than bldg's 0.01 mm, and the pens (style sheet / layer table) are not counted on either side. Tokens: tiktoken o200k_base, minified.

bldg therefore takes 54-73% fewer tokens than these four encodings of the same drawing (they need 2.2-3.7× its tokens). That is the plan alone, not the cost of a whole agent request.

Per element kind (all 58 plans: the 55 plus the three editor examples), the drawing formats take walls 1.1-1.8×, doors 4.0-7.9×, windows 4.8-7.9×, furniture 6.3-10.7×, rooms 2.8-4.3× and annotation 0.8-1.4× the bldg tokens: the gap is in the symbols the compiler draws from a few semantic fields.

The other measurements compare the semantic answer (no drawing; every model-side encoding in the image frame, pixels, 0.1 px), and the encoding under noise:

Against a drawing, bldg is 2.2-3.7× smaller; against a semantic answer of the same elements, the gain is topological correctness, not size or geometric accuracy:

• Topological defects. Shared nodes and openings placed from a wall corner remove almost every open corner and floating door that absolute coordinates produce under the same noise. Repairing absolute coordinates by clustering wall ends does not get there: at its best join distance it still welds corners together or leaves them open, 14.6 per plan. This measures the encoding under synthetic noise, not a model's accuracy.
• Geometric error. bldg does not make corners more accurate. They are as far from the truth as the coordinates that were written (81 mm mean at σ = 6 px), and lint does not change that; the repaired absolute coordinates end up a little closer (71 mm), because averaging a corner's copies averages out part of the noise. Geometric accuracy comes from the extraction model plus constraints (printed dimensions, axes, modules), and those need correct topology to act on.
• Semantic answer size. Against an answer with the same elements and no drawing, the saving is modest, because key names and node ids cost tokens. In OpenAI strict mode, where every key must be present, the output is only about 3% smaller than lean absolute JSON. On these traced datasets types do not save tokens: 360 wall types for 2,095 walls, but 878 opening types for 996 openings, because each traced door and window has its own 0.1 px width; written without types the same plans take 151,710 tokens (−16%). A drawing with standard sizes (every 900 x 2100 door one type) shares them. Types are for BIM, schedules and editing (change a type, every instance follows). The same answer in the plan frame (mm, 0.01 mm) takes 156,055 tokens.

Real-data round trip

Ground truth goes through floorplan/v0 → bldg → compile → floorplan/v0 and is scored with the floorplan/v0 evaluator. 58 plans can be scored: 34 from 5 public floor-plan datasets and 24 synthetic. 4 Wikimedia Commons images have no labels and are excluded. Not every dataset labels every field, so each score's denominator differs. This measures how much the format and compiler lose:

The residuals:

• Hand-traced plans whose walls taper (wall IoU 0.976). Their walls are tapered polygons, which a constant-thickness wall cannot reproduce.
• Project north. The bridge stores a plan straightened (dominant wall direction at 0 deg) and removes skew under 0.5 deg unless that would break it (then it is only turned, the skew kept), so hand-traced walls a fraction of a degree off the axis become exact; scored in the image again, this costs wall IoU in the third decimal at most.
• One Swiss Dwellings window. Its jambs were recorded on the wrong axis by the ground-truth converter.
• A second Swiss Dwellings window (realworld_08 o_win4) sits next to a junction whose walls meet too sharply to be mitred; turned into project north (40°), the junction's square joint covers the window's span and the compiler leaves it out.

The bridge stores each plan as a mm document on its image (the image as the level's underlay: a calibrated scale where the dataset gives one, else the stand-in 10 mm/px marked assumed) and maps every point back through that transform for scoring.

bldg's Boolean-free 3D pieces are checked against a CSG reference on the same geometry (IFC walls with their openings cut by IfcOpenShell's boolean). For 2,086 of the 2,094 compared walls both have the same volume, to within a relative 1e-4; each of the other 8 is listed in Measured benefits with the measured reason (a wall without openings whose outline self-intersects, which the compiler warns about).

Gallery

Rows 1–2: our own synthetic generator (no restrictions). Row 3: adapted from Archilyse AG (Standfest et al.), Swiss Dwellings v3.0.0, doi:10.5281/zenodo.7788422, CC BY 4.0; rendered, converted to bldg, recompiled and re-rendered as SVG and 3D. See NOTICE.

bldg · Editor · Editor guide · Viewer · Format · Benchmarks · Limitations · Terms · Privacy · hello@usebldg.com