Appendix B: Metric to gauge Accuracy & Precision of MMRF
Last updated
Last updated
Figure 12: SIS Metric Components
For this project, we propose a unique metric scoring system tailored for the AI-driven 3D scene generation in civil engineering: Structural Integrity Score (SIS). The SIS is a composite metric that combines accuracy, precision and structural feasibility into a single score ranging from 0 to 100, where higher scores indicate better performance (Fig. 12).
1. Accuracy
Measures the closeness of the AI generated model’s attributes to the real-world structure or intended design.
Calculated using Mean Absolute Percentage Error and scaled to a score between 0 to 40.
2. Precision
Assesses the consistency of AI-generated models of the same structure.
Derived from the Coefficient of Variance (CV) and scaled to a score between 0 to 30.
3. Structural Feasibility
Evaluate the structural soundness of the AI-generated model using simulated load tests.
A model that withstands standard load tests without significant deformation or stress points scores between 0 to 30.
To evaluate MMRF, we will produce 30K image pairs with ground truth for rectified stereo matching. Next, MMRF is trained on all image pairs to gauge the SIS scores for each image-pair. The total score is used to re-evaluate the model and make improvements accordingly.
Example of a prompt: "apply a 10kN force at the midpoint of the beam". ↑
Example of a prompt: "make the brick wall non-collidable" or "increase the bounciness of the rubber ball" ↑
Example of a prompt: "fix the joint between beams A and B" or "allow rotation about the Z-axis" ↑
