Monte Carlo Integration: Methodology Decision Brief
Résumé
The Core Issue: Before we can integrate Monte Carlo into the Report Template, we’ve identified a fundamental methodological discrepancy between our two calculation engines. They use different statistical models and produce different results (~5-10% variance). We need to resolve which model is canonical before proceeding.
What We Need From You:
- CRITICAL: Which probability model is actuarially correct? (see Section 2)
- Which architecture option aligns with actuarial best practice?
- Which metrics warrant Monte Carlo treatment vs. closed-form?
- Which input uncertainties matter most to model?
- How does this affect our QA verification process?
⚠️ CRITICAL: Methodological Discrepancy Requiring Resolution
Model A: Report Template
Two-Factor with Material-Specific Exposure
Annual Claim Prob = Exposure(material) × Vulnerability(material)
Each material has a different exposure probability based on its damage threshold.
Model B: Monte Carlo Simulator
Single Storm Process, Different Failure Rates
1. Sample λ ~ Triangular(0.16, 0.22, 0.33) 2. Same storms hit both roofs 3. Different failure rates per material
Model A: Material-Specific Exposure
| Material | Threshold | Exposure | Vulnerability | Annual Claim Prob |
|---|---|---|---|---|
| IKO (Class 3) | ~3.8 cm hail | 18% | 70% | 12.6% |
| DECRA (Class 4) | ~5.0 cm hail | 6% | 5% | 0.3% |
Rationale: Class 4 materials only fail when hit by larger hail, which occurs less frequently.
Model B: Single Storm Frequency
| Material | Storm Frequency (λ) | Failure Rate | Annual Claim Prob |
|---|---|---|---|
| IKO (Class 3) | 0.22 | 70% | ~14.3% |
| DECRA (Class 4) | 0.22 | 5% | ~1.1% |
Rationale: Same storms hit both roofs; the difference is purely in material vulnerability.
Numerical Comparison
| Metric | Model A (Report) | Model B (Simulator) | Variance |
|---|---|---|---|
| IKO Annual Claim Prob | 12.6% | ~14.3% | +1.7 pts |
| DECRA Annual Claim Prob | 0.3% | ~1.1% | +0.8 pts |
| Annual Differential | 12.3% | ~13.2% | +0.9 pts |
| 7-Year Policy Prob (IKO) | 61.2% | ~66% | +5 pts |
| 50-Year Claims Avoided | 6.15 | ~6.6 | +0.45 |
| 50-Year Savings | $107,625 | ~$99,000* | -8% |
*Simulator uses $15,000 base cost vs. Report’s $17,500 inflated cost, which partially offsets.
The Fundamental Question
Model A says: “Class 4 materials experience fewer damaging events because only larger storms exceed their threshold.”
Model B says: “All storms hit both materials equally; Class 4 just survives more of them.”
- Is it defensible to say a Class 4 roof has lower exposure (fewer threshold-exceeding events)?
- Or should exposure be location-only, with vulnerability handling all material differences?
- Is there a hybrid approach?
Your recommendation here determines which model becomes canonical before we integrate Monte Carlo.
Why This Matters Now
If we integrate Monte Carlo without resolving this, we’ll have:
- A Report Template using Two-Factor (Model A)
- A Monte Carlo Simulator using storm-level simulation (Model B)
- Two “validated” tools producing different numbers for the same property
This undermines the entire QA methodology of dual independent verification.
Data Availability Consideration: ZRS Format May Dictate the Answer
A critical factor: ZRS data is structured as threshold-based return periods, not raw storm frequencies.
How ZRS data works:
Query: "What is the return period for hail ≥ X cm at [lat/long]?" Response: "Y years" Annual Probability = 1 / Return Period
| Query Threshold | Return Period | Annual Probability | Material |
|---|---|---|---|
| ≥ 3.8 cm | 6.7 years | 15% | Class 3 (IKO) |
| ≥ 5.0 cm | 20 years | 5% | Class 4 (DECRA) |
What this implies:
- ZRS doesn’t provide a single “storm frequency” (λ) — you must specify a threshold
- Model A (Report Template) directly matches how ZRS data is structured
- Model B (Simulator) uses a single λ value that isn’t directly sourced from ZRS
This may resolve the theoretical debate pragmatically: If our data source provides threshold-specific probabilities, then material-specific exposure (Model A) isn’t just defensible — it’s the only approach that correctly uses the data we have.
Clarification: Model A Doesn’t Preclude Monte Carlo
If we adopt Model A as canonical, Monte Carlo is still valuable — we just sample different uncertainties:
| Model B (Current Simulator) | Model A (ZRS-Aligned) |
|---|---|
| Sample λ ~ Triangular(0.16, 0.22, 0.33) | Sample IKO exposure ~ Triangular(12%, 15%, 18%) |
| Same λ for both materials | Sample DECRA exposure ~ Triangular(3%, 5%, 7%) |
| Vulnerability differentiates | Both exposure AND vulnerability differentiate |
Potential Hybrid Approach:
For each simulation: 1. Sample IKO exposure ~ range around ZRS 3.8cm return period 2. Sample DECRA exposure ~ range around ZRS 5.0cm return period 3. Sample vulnerabilities ~ range around IBHS testing values 4. Calculate claims, savings, break-even as before
Integration with Two-Factor Competing Hazard Model
Our current methodology treats age-based and storm-based failures as competing hazards rather than additive.
| Failure Mode | Current | Monte Carlo Approach |
|---|---|---|
| Storm-induced | Annual prob × years | Sample storm arrivals (Poisson), then sample failure per storm (Bernoulli) |
| Age-based | Fixed ESL from Factor Method | Could sample Factor A/E from ranges to get ESL distribution |
The interaction matters: If a storm destroys the roof in year 8, the age-based “clock” resets. Monte Carlo can naturally capture this interaction by simulating year-by-year and tracking state.
- (A) Storm-only simulation — Monte Carlo for storm damage, keep ESL deterministic as a ceiling
- (B) Full competing hazard simulation — Simulate both storm arrivals AND age degradation, track which triggers first
- (C) Separate then combine — Run storm Monte Carlo and age Monte Carlo independently, combine probabilistically
What’s your recommendation on the right level of complexity here?
Architecture Options
Option A: Embed in Report
Port sampling functions directly into report component
✓ Self-contained
✗ Code duplication
Option B: Link to Simulator
Report references/embeds simulator outputs
✓ Single source of truth
✗ Component coupling
Option C: Shared Service
Extract logic into shared utility module
✓ DRY principle
✗ More complexity upfront
Which Metrics Warrant Monte Carlo?
| Metric | Current Calc | Monte Carlo Benefit | Recommendation |
|---|---|---|---|
| Claims Avoided (50-yr) | Δ annual prob × 50 | High — show distribution | ✓ Include |
| Lifecycle Savings ($) | Claims × $17,500 | High — flows from claims | ✓ Include |
| 7-Year Policy Probability | 1-(1-p)⁷ | High — core insurance metric | ✓ Include |
| Break-Even Years | Premium ÷ Annual Savings | Already in Simulator | ✓ Include |
| Risk Reduction % | (LKQ – Upgrade) / LKQ | Medium — could show band | ⚠ Optional |
| GHG Avoided | LKQ_total – Upgrade_total | Medium — depends on claims | ⚠ Optional |
Input Uncertainty Sources
| Input | Current Value | Proposed Range | Uncertainty Source |
|---|---|---|---|
| Hail Exposure (λ) | 18% | 15%–22% | ZRS data confidence interval |
| Vulnerability (LKQ) | 70% | 60%–80% | IBHS testing variance |
| Vulnerability (Upgrade) | 5% | 3%–8% | Installation quality variation |
| Claim Cost | $17,500 | $15,000–$22,000 | Storm severity distribution |
| Inflation | 2% | 1.5%–3% | Economic uncertainty |
QA Implications: Dual Independent Verification
│ REPORT TEMPLATE │ │ REPORT GENERATOR│
│ (Manual Math) │ │ (Coded Math) │
└────────┬────────┘ └────────┬────────┘
│ │
└───────────┬───────────┘
▼
MUST MATCH EXACTLY
Problem: Monte Carlo produces distributions, not point estimates. Two runs of 10,000 simulations will produce similar but not identical results.
Proposed Solution: Validate within statistical tolerance
- Medians should match within ±2%
- Percentile ranges should overlap significantly
- Use seeded random number generator for reproducible QA runs
Output Format Options (Actuarial Audience)
A: Percentile Range
$89,000 – $167,000 (P10–P90)
Median: $107,000
B: Confidence Interval
$107,625 ± $28,000
(90% CI)
C: Probability Statement
90% prob of exceeding $89k
50% prob of exceeding $107k
Summary: Your Input Requested
The model discrepancy is the key blocker. I’d really value your perspective on these areas, roughly in priority order:
- CRITICAL Model Selection: Which probability model is actuarially correct?
- Model A: Material-specific exposure (different thresholds = different event frequencies)
- Model B: Location-only exposure (same storms, different failure rates)
- Or a hybrid approach?
- Competing Hazards: How should Monte Carlo handle the storm vs. age-based interaction?
- Architecture: Which option (A/B/C) would you recommend for methodology auditability?
- Scope: Your view on Monte Carlo for all metrics vs. focusing on high-value ones?
- Uncertainty Inputs: Which ranges do you see as defensible vs. potentially speculative?
- QA Tolerance: What statistical threshold would you recommend for validating agreement?
- Output Format: What would resonate best with an actuarial audience?
Happy to discuss any of these in more detail—the methodology documentation site has all the code and version history if you want to dig into specifics.
Reference Materials
Full technical documentation, code, and methodology versioning available at the project documentation site. Key files:
- Monte Carlo Simulator implementation (page (11).tsx)
- QA Transfer Prompt v5 — Dominant Peril Aligned
- Resiliency Data Architecture Technical Brief