Graphyti
  • Table of Contents
  • Problem Statement
  • Literature Review
  • Methodology: Creation and Preprocessing of Datasets
  • Methodology: Introduction to MMRF
  • Methodology: Personalised Physics
  • Expected Outcome
  • Conclusion
  • Bibliography
  • Appendix A: Implementation Timeline
  • Appendix B: Metric to gauge Accuracy & Precision of MMRF
Powered by GitBook
On this page

Appendix B: Metric to gauge Accuracy & Precision of MMRF

PreviousAppendix A: Implementation Timeline

Last updated 1 year ago

Appendix B: Metric to gauge Accuracy & Precision of MMRF

Figure 12: SIS Metric Components

For this project, we propose a unique metric scoring system tailored for the AI-driven 3D scene generation in civil engineering: Structural Integrity Score (SIS). The SIS is a composite metric that combines accuracy, precision and structural feasibility into a single score ranging from 0 to 100, where higher scores indicate better performance (Fig. 12).

1. Accuracy

  • Measures the closeness of the AI generated model’s attributes to the real-world structure or intended design.

  • Calculated using Mean Absolute Percentage Error and scaled to a score between 0 to 40.

2. Precision

  • Assesses the consistency of AI-generated models of the same structure.

  • Derived from the Coefficient of Variance (CV) and scaled to a score between 0 to 30.

3. Structural Feasibility

  • Evaluate the structural soundness of the AI-generated model using simulated load tests.

  • A model that withstands standard load tests without significant deformation or stress points scores between 0 to 30.

To evaluate MMRF, we will produce 30K image pairs with ground truth for rectified stereo matching. Next, MMRF is trained on all image pairs to gauge the SIS scores for each image-pair. The total score is used to re-evaluate the model and make improvements accordingly.

  1. Example of a prompt: "apply a 10kN force at the midpoint of the beam". ↑

  2. Example of a prompt: "make the brick wall non-collidable" or "increase the bounciness of the rubber ball" ↑

  3. Example of a prompt: "fix the joint between beams A and B" or "allow rotation about the Z-axis" ↑