Per-method validation matrix
What this is — and what it is not
This page classifies every analysis the app exposes (source of truth: the
in-app capability sheet, docs/data/capabilities.json). The original campaign
classified 415 distinct method ids (the sheet then listed 422 procedure rows,
7 listed twice across groups). The engine has since grown to 600 procedure
rows (569 full + 31 limited): the 192 new ADV methods (Waves 1–16, 2026-06)
are tiered in their own subsection below — "ADV Waves 1–16 — built and
independently re-validated" — from a dedicated 2026-06-19 adversarial
re-validation campaign. All tiers describe how each method is actually tested
inside this repository's test suite. It is a faithful map of test coverage,
scanned method-by-method against the real tests — nothing here reflects external
peer review, regulatory certification, or independent audit beyond what the cited
test file does.
Read the tiers as a strict ladder of evidence strength. A method is placed in the strongest tier whose evidencing test can be named; when the evidence was ambiguous it was pushed down, not up. Several honest consequences follow:
- "NIST-certified" means a NIST Statistical Reference Datasets (StRD) benchmark test drives this exact engine path and asserts a correct-digit floor against multiple-precision certified values. Only three methods clear that bar today (descriptives, one-way ANOVA, linear regression). One of those tests (Filip, linear regression) documents a real numerical failure of the current solver rather than hiding it — see the note below.
- "Library-faithful" is a genuinely weaker claim than "R-cross-validated": it says the wrapper passes data through to a peer-used library faithfully (proven by a parity or seam test), not that the underlying numbers were checked against an independent system.
- "Unverified" is not an accusation of being wrong — most of these run a thin, well-understood library call. It means the only test we could name is the seam-smoke harness, which proves the handler does not crash and returns a substantive result, and nothing about numerical correctness.
If you need a number to be trustworthy for publication, prefer a method in the top two tiers, or validate your specific case against R/Stata yourself.
The five tiers
| Tier | What the badge asserts | Evidencing test(s) |
|---|---|---|
| NIST-certified | Correct-digit (log-relative-error) floor met against NIST StRD multiple-precision certified values. | test_nist_strd_univariate.py, test_nist_strd_anova.py, test_nist_strd_linreg.py |
| R-cross-validated | Output matches a committed reference produced by R (or an R package: lavaan, spgwr, plm, metafor, survey, mice, …) at a documented tolerance. A row marked R (partial: …) matches R on its primary quantities but has a documented, honest disagreement on a named secondary quantity (see the disagreement inventory). | test_golden_r.py, test_golden_r_sem_lavaan.py, test_rigor_rf_wave*.py, and the academic-validation suite test_acadval_wave1..21_*.py |
| Library-faithful (parity) | Thin wrapper over a peer-used library (statsmodels / scipy / lifelines / semopy / mgwr / shap / arch / pmdarima / PyMC / sklearn) with a parity or seam test proving faithful pass-through. | test_*_parity.py, plus the seam harness where the backend is the named library |
| Internal-rigor-audited | A test_rigor_* test checks formulas/edge-cases against hand-derived or closed-form expectations, but not against an independent statistical system. | test_rigor_*.py (excluding the RF-wave R-cross-checks) |
| Unverified | Only the seam-smoke contract (test_handler_seam_smoke.py) — proves the handler runs and returns substantive output, nothing about correctness. Includes XFAIL seam cases (run but expected to need external state) and methods with no test at all. | test_handler_seam_smoke.py only, or none |
Honesty notes carried over from the tests
- NIST linear-regression / Filip: the engine fits OLS via statsmodels'
default
pinv(SVD pseudo-inverse) solver. On the extremely ill-conditioned 10th-degree Filip polynomial (cond(X) ≈ 1.8×10¹⁵) this loses all precision — the engine returns ~0 correct digits on every coefficient. The test pins this as a documented failure (floor = −1 correct digits) and records that statsmodels' QR solver would recover ~7.9 digits on the same data. This is a real, known weakness of the current solver choice, not loosened away. - NIST ANOVA / SmLs07: the textbook deviation-formula SS computation degrades to ~2–4 correct digits on the 1×10¹² shifted stress dataset. The test floor records that degradation honestly rather than masking it.
- AIC/BIC are excluded from all R comparisons (different additive constants); log-likelihood is compared where both systems agree.
- Survival and Poisson outputs are rounded to 4 dp by the engine, so the R agreement there is verified to within 5×10⁻⁵.
Summary — counts per tier
The matrix below classifies the 415 distinct statistical / data-management /
workflow procedures in the capability sheet. The sheet lists 422 procedure
rows, but 7 ids appear twice across groups — test.manova,
model.linear_regression, model.panel_fe, model.panel_re, model.did,
model.rdd, model.psm — and each is counted once here at its strongest tier
(422 − 7 = 415). Pure UI/IO plumbing (dataset load, cell edit, session, tables,
charts, AI relay) is grouped at the bottom and counted as Unverified for
statistical-correctness purposes — most carry their own functional tests
(test_edits.py, test_session_json.py, test_report_builder.py, etc.) that
prove behaviour but not statistical numbers.
| Tier | Count |
|---|---|
| NIST-certified | 3 |
| R-cross-validated | 220 |
| Library-faithful (parity-tested) | 40 |
| Internal-rigor-audited | 52 |
| Unverified / no statistical gold standard | 100 |
| Total | 415 |
Counts are a deterministic tally of the per-method rows below (scripts/count_matrix_tiers.py). Of the 220 R-cross-validated, 171 match R (or NIST) cleanly and 49 carry a documented caveat — a specific quantity (usually a standard error, or a different-optimizer/grid result) disagrees with R in a bounded, disclosed way while the primary estimates match. The R-cross-validated count rose from 42 → 220 in the 2026-06 academic-validation campaign (21 family waves vs R packages + NIST StRD, each adversarially audited; see reports/academic-validation-blueprint.json). The Unverified bucket is ≈29 statistical methods with only a seam-smoke contract plus ≈71 UI/IO/workflow plumbing ids that have no statistical gold standard (they carry functional tests instead).
A method counted in a higher tier is not re-counted lower. Many R-cross-validated and parity methods also have a
test_rigor_*test and a seam contract; they appear once, at their strongest evidenced tier.The R-cross-validated tier grew from 42 to 214 after the academic-validation campaign (
test_acadval_wave1..21_*.py, committed 2026-06), which drove ~224 engine methods against R across 17 families with baked-in, deterministic R reference fixtures. 172 methods were newly R-validated. Of the 214, 49 carry an honest "R (partial: …)" marker: the engine matches R on the primary quantities (point estimates / closed-form statistics) but a named secondary quantity legitimately differs — a documented backend/algorithm disagreement, not a hidden tolerance loosening. These are listed explicitly per row and in the disagreement inventory below. Methods whose tests showed no clean R agreement on any primary quantity (e.g. Ridge/ElasticNet vs glmnet objective normalization, panel RE/between θ, DCC-GARCH grid-vs-MLE, IRT CML-vs-ML, MGWR, kmeans/MDS/MCA algorithm mismatches) were not upgraded — they stay Library-faithful or Internal-rigor-audited with the disagreement noted.
Documented R disagreements (R-partial methods)
The following methods are R-cross-validated on their primary outputs but carry a documented, honest disagreement on a secondary quantity (each row above states the specifics):
- Standard errors / variance:
model.zip,model.zinb(inflation model),model.tobit/model.truncated,model.fractional(non-robust binomial SE),model.sur/model.3sls(covariance estimator),model.gmm(J-statistic),panel.system_gmm(Windmeijer k),panel.driscoll_kraay,survival.schoenfeld,survival.rmst,survival.logrank_stratified,missing.em(covariance ddof),survey.ttest(design df). - Point estimate / structural component:
model.fgls(intercept),panel.hausman_taylor(time-invariant FGLS),causal.ipw(sklearn-vs-glm PS),latent.irt_2pl(extreme items),latent.irt_grm(slopes),analysis.catpca(eigenvalue),analysis.split_half(meanr estimator),analysis.hierarchical(merge heights). - Test statistic / IC convention:
test.manova(Hotelling-Lawley F-approx),post.contrasts(polynomial),post.model_comparison/model.stepwise(AIC/BIC constant),ts.sarima/ts.ets/ts.holt_winters(log-lik/smoothing),ts.svar(A-matrix),ts.zivot_andrews/ts.toda_yamamoto(lag selection),ts.nardl(order),ts.spectral(PSD factor of 2),model.egarch(parameterisation),test.fisher_exact(sample vs CMLE odds ratio). - Monte-Carlo resampling (observed statistic matches R exactly; resampled
SE/CI/p-value is RNG-dependent):
bootstrap.se,bootstrap.ci,bootstrap.regression,bootstrap.permutation_two_sample,bootstrap.permutation_correlation,test.permutation. - Engine limitation surfaced:
post.predicted_means(categorical grouping returns a documented error).
Full per-method matrix, by family
Legend: NIST = NIST-certified · R = R-cross-validated · LIB = library-faithful parity · RIGOR = internal-rigor-audited · — = unverified (seam-smoke only unless noted).
Descriptives & hypothesis tests
| Method | Tier | Evidencing test |
|---|---|---|
stats.describe | NIST | test_nist_strd_univariate.py (mean, SD ddof=1, lag-1 ACF on 7 StRD sets) |
test.ttest_one_sample | R | test_golden_r.py::ttest_one_sample |
test.ttest_independent | R | test_golden_r.py (pooled + Welch + Cohen's d) |
test.ttest_paired | R | test_golden_r.py::ttest_paired |
test.anova_oneway | NIST | test_nist_strd_anova.py (5 StRD sets) + test_golden_r.py (Tukey HSD) |
test.anova_welch | R | test_acadval_wave11_nonparam.py (vs oneway.test(var.equal=FALSE), F + df1 + df2 + p) |
test.anova_twoway | R | test_acadval_wave12_anova_manova.py (vs car::Anova(type=2), SS/F/p/partial-η² to ~1e-8) |
test.anova_repeated | R | test_acadval_wave12_anova_manova.py (vs afex::aov_ez, F/GG-ε/Mauchly-W/partial-η²) |
test.yuen | R | test_golden_r.py (vs WRS2::yuen) |
test.anova_mixed | R | test_golden_r.py (vs aov(... Error(subj/time))) |
test.dunnett | R | test_golden_r.py (vs multcomp) |
test.cochran_armitage | R | test_golden_r.py |
test.mann_whitney | R | test_golden_r.py (auto/exact/approx branches) |
test.wilcoxon | R | test_golden_r.py (W + p, auto/exact/approx) |
test.kruskal_wallis | R | test_golden_r.py + test_rigor_wave4_medium_a.py |
test.friedman | R | test_golden_r.py::friedman_test |
test.chi_square | R | test_golden_r.py (+ test_rigor_categorical.py) |
test.chi_square_gof | R | test_golden_r.py (+ test_rigor_categorical.py) |
test.fisher_exact | R (partial: odds-ratio sample-vs-CMLE documented-disagreement) | test_acadval_wave11_nonparam.py — exact hypergeometric p-values match fisher.test to 1e-7; engine reports the sample OR, R the conditional-MLE OR |
stats.correlation_matrix | R | test_golden_r.py + test_rigor_parametric.py |
stats.correlation | R | test_golden_r.py (Pearson/Spearman/Kendall, CI) |
stats.partial_correlation | R | test_golden_r.py + test_rigor_parametric.py |
stats.effect_size | R | test_acadval_wave12_anova_manova.py (vs effectsize:: d/η²/ω²/Cramér's V) |
stats.power | LIB | test_rigor_diag.py (vs statsmodels TTestIndPower) |
test.sign_test | R | test_acadval_wave11_nonparam.py (vs DescTools::SignTest, paired + one-sample) |
test.runs_test | R | test_acadval_wave11_nonparam.py (vs tseries::runs.test, Z + p) |
test.mcnemar | R | test_acadval_wave11_nonparam.py (vs mcnemar.test, continuity-corrected χ²) |
test.cochran_q | R | test_acadval_wave11_nonparam.py (vs DescTools::CochranQTest) |
test.jonckheere | R | test_golden_r.py (vs PMCMRplus, tie-corrected variance) |
test.kendall_w | R | test_acadval_wave11_nonparam.py (vs DescTools::KendallW, W + χ² + p) |
test.kendall_tau | R | test_golden_r.py (via correlation_pair Kendall) |
test.moods_median | R | test_acadval_wave11_nonparam.py (vs chisq.test(correct=FALSE); coin variant noted) |
test.permutation | R (partial: MC p-value) | test_acadval_wave11_nonparam.py — observed statistic matches R exactly; permutation p is Monte-Carlo (RNG differs) |
test.ks_2sample | R | test_acadval_wave11_nonparam.py (vs ks.test, D + p) |
stats.kde | RIGOR | test_acadval_wave12_anova_manova.py — scipy density pinned; bandwidth (nrd0/nrd vs scott/silverman) is a documented disagreement with R density(), so not R-cross-validated |
test.manova | R (partial: Hotelling-Lawley F-approx documented-disagreement) | test_acadval_wave12_anova_manova.py (Wilks/Pillai/Roy match stats::manova; Hotelling F-approx + den_df differ) |
analysis.fit_distribution | R | test_golden_r.py (vs MASS::fitdistr + ks.test) |
stats.frequencies | — | seam-smoke only |
Diagnostics & post-estimation
| Method | Tier | Evidencing test |
|---|---|---|
diagnose.normality | R | test_golden_r.py (Shapiro + KS vs R) |
diagnose.homogeneity | R | test_golden_r.py (Levene-median + Bartlett vs R) |
diagnose.multicollinearity | R | test_acadval_wave6_diagnostics.py (VIF vs car::vif to ~5e-10) |
diagnose.vif | R | test_acadval_wave6_diagnostics.py (vs car::vif, per-predictor to ~5e-10) |
diagnose.suggest_test | RIGOR | test_rigor_diag.py::suggest_test |
diagnose.outliers | R | test_acadval_wave6_diagnostics.py (IQR/Z/MAD bounds + exact index sets vs base R) |
diagnose.breusch_pagan | R | test_acadval_wave6_diagnostics.py (vs lmtest::bptest, LM + p to ~5e-10) |
diagnose.white_test | R | test_acadval_wave6_diagnostics.py (vs lmtest::bptest with squares/cross-products) |
diagnose.goldfeld_quandt | R | test_acadval_wave6_diagnostics.py (vs lmtest::gqtest, F + p at matched split; default drop_fraction differs) |
diagnose.breusch_godfrey | R | test_acadval_wave6_diagnostics.py (vs lmtest::bgtest, LM at lags 1/2) |
diagnose.durbin_watson | R | test_acadval_wave6_diagnostics.py (vs lmtest::dwtest, statistic to ~1e-11) |
diagnose.ljung_box | R | test_acadval_wave6_diagnostics.py (vs Box.test(type='Ljung-Box') on residuals) |
diagnose.ramsey_reset | R | test_acadval_wave6_diagnostics.py (vs lmtest::resettest, F + p to ~5e-10) |
diagnose.hosmer_lemeshow | R | test_acadval_wave1_glm.py (vs ResourceSelection::hoslem.test, χ² + df + p) |
diagnose.cooks_distance | R | test_acadval_wave6_diagnostics.py (vs cooks.distance, per-obs + max to ~5e-10) |
diagnose.leverage | R | test_acadval_wave6_diagnostics.py (vs hatvalues, per-obs to ~5e-10) |
diagnose.dfbetas | R | test_acadval_wave6_diagnostics.py (vs dfbetas, full n×k matrix to ~5e-10) |
post.marginal_effects | R | test_acadval_wave1_glm.py (AME + delta-method SE vs margins::margins) |
post.predicted_probabilities | R | test_acadval_wave1_glm.py (vs predict(glm, se.fit=TRUE), prob + link-scale CI) |
post.posthoc | R | test_golden_r.py (Holm + Games-Howell vs R) |
post.effect_sizes | R | test_acadval_wave12_anova_manova.py (d/Glass-δ/Hedges-g/η²/ω²/Cohen-f vs effectsize::) |
post.contrasts | R (partial: polynomial-quadratic documented-disagreement) | test_acadval_wave7_margins.py — pairwise/Helmert estimates+SE match emmeans; Legendre vs contr.poly quadratic differs; CI uses t not Tukey |
post.prediction_intervals | R | test_acadval_wave7_margins.py (vs predict(interval='confidence'/'prediction'), to 1e-4) |
post.coefficient_table | RIGOR | test_standardized_beta.py |
post.model_comparison | R (partial: AIC/BIC +2.0 constant offset) | test_acadval_wave7_margins.py — R²/adj-R²/F match lm; AIC/BIC differ by the documented sigma²-parameter constant |
post.margins_at | R | test_acadval_wave7_margins.py (OLS ME = slope, vs margins/lm coef) |
post.margins_contrasts_ext | R | test_acadval_wave7_margins.py (predicted-outcome difference + delta SE vs R) |
post.margins_interaction | R | test_acadval_wave7_margins.py (conditional ME at p10–p90 vs lm(y~...+x1:x2)) |
post.nested_contrasts | R | test_acadval_wave7_margins.py (Welch t.test per pair, t/df/p/diff to 1e-4) |
post.elasticities | R | test_acadval_wave7_margins.py (full/partial/arc closed-form vs R, to 1e-5) |
post.semi_elasticities | R | test_acadval_wave7_margins.py (semi-elasticities closed-form vs R, to 1e-5) |
post.predicted_means | R (partial: categorical-grouping documented-limitation) | test_acadval_wave7_margins.py — numeric predict path matches R; string grouping returns a documented error |
post.adjusted_predictions | R | test_acadval_wave7_margins.py (vs predict(newdata), pred+SE; CI z-approx within 5e-3) |
post.probability_profile | R | test_acadval_wave1_glm.py (probability curve + link-scale CI vs predict over grid) |
post.treatment_contrasts | R | test_acadval_wave7_margins.py (g-computation ATE/ATT/ATC vs lm β_treat) |
Multivariate, factor & latent variable
| Method | Tier | Evidencing test |
|---|---|---|
analysis.pca | R | test_golden_r.py + test_rigor_factor.py + test_rigor_wave5_low_correctness.py |
analysis.efa | R | test_acadval_wave13_factor.py (vs psych::fa(fm='ml'); loadings sign-aligned ~3e-3, KMO/Bartlett to machine precision) |
analysis.kmeans | RIGOR | test_acadval_wave13_factor.py — structural only; sklearn k-means++ vs R kmeans(Lloyd) is a documented algorithm difference (inertia within ~5%), not R-cross-validated |
analysis.hierarchical | R (partial: merge heights ~1%) | test_acadval_wave13_factor.py — cophenetic correlation matches hclust(ward.D2) exactly; ward merge heights differ by ~1% |
analysis.cronbach_alpha | R | test_golden_r.py::cronbach_alpha |
analysis.mcdonald_omega | R | test_acadval_wave13_factor.py (ω_total vs psych::omega, gap < 2e-6) |
analysis.split_half | R (partial: meanr estimator documented-disagreement) | test_acadval_wave13_factor.py — Guttman matches psych::splitHalf to ~5e-3; engine single odd/even r differs from R's averaged meanr by design |
analysis.parallel_analysis | RIGOR | test_acadval_wave13_factor.py — #factors-retained matches fa.parallel, but R reports factor-residual eigenvalues vs engine correlation-matrix eigenvalues (thresholds not comparable), so not R-cross-validated |
analysis.discriminant | R | test_acadval_wave13_factor.py (vs MASS::lda; means/scalings sign-aligned/accuracy to ~2.5e-4) |
analysis.canonical_correlation | R | test_acadval_wave13_factor.py (canonical r + Wilks λ vs stats::cancor, to ~2e-4) |
latent.cfa | R | test_golden_r_sem_lavaan.py (vs lavaan 0.6.21) |
latent.sem | R | test_golden_r_sem_lavaan.py (vs lavaan) |
latent.path_analysis | R | test_golden_r_sem_lavaan.py (vs lavaan, saturated) |
latent.irt_rasch | R | test_rigor_rf_wave9b_cml_rasch.py (CML Rasch) + test_rigor_wave4_medium_g.py |
latent.irt_2pl | R (partial: extreme-item documented-disagreement) | test_acadval_wave14_irt_sem.py — moderate-item difficulty matches mirt to < 0.175; extreme items (L-BFGS joint MML vs EM-MML) diverge |
latent.irt_3pl | LIB | test_acadval_wave14_irt_sem.py — structural only; engine free-g 3PL vs mirt fixed-g=0.2 is a documented design difference, not R-cross-validated |
latent.irt_grm | R (partial: slopes ~0.4 qualitative) | test_acadval_wave14_irt_sem.py — slopes within ~0.41 of mirt::graded (custom MML vs EM-MML); thresholds ordered |
latent.irt_pcm | RIGOR | test_acadval_wave14_irt_sem.py — ML vs eRm::PCM CML give different (non-comparable) log-likelihoods/step-params; only ordering validated |
latent.irt_rsm | RIGOR | test_acadval_wave14_irt_sem.py — ML vs eRm::RSM CML; only item ordering/threshold-ordering validated |
latent.irt_nrm | RIGOR | test_acadval_wave14_irt_sem.py — mirt nominal parameterisation differs (ak/d vs a/c); structural/category-count only |
latent.dif_analysis | R | test_acadval_wave14_irt_sem.py (MH odds-ratio + ETS-delta vs difR::difMH to < 0.001/0.0003) + test_rigor_advanced.py (Lord χ²) |
latent.ctt_analysis | R | test_acadval_wave14_irt_sem.py (α + item difficulty + corrected item-total r vs CTT::itemAnalysis to < 2e-4) |
latent.irt_fit | RIGOR | test_acadval_wave14_irt_sem.py — engine limited-info approximate RMSEA vs mirt::M2 is a documented design difference (values differ materially); finite-value check only |
analysis.mds | RIGOR | test_acadval_wave13_factor.py — sklearn SMACOF vs cmdscale classical MDS is an algorithm mismatch (coords not comparable); plausible-stress check only |
analysis.correspondence | R | test_acadval_wave13_factor.py (total inertia/χ²/dimension inertias vs R SVD to machine precision) |
analysis.mca | RIGOR | test_acadval_wave13_factor.py — pd.get_dummies vs R model.matrix indicator coding differs (eigenvalues not comparable); structural only |
analysis.finite_mixture | RIGOR | test_acadval_wave14_irt_sem.py — log-lik/means consistent with mclust but BIC sign + component ordering differ; structural/separation only |
analysis.latent_class | R | test_acadval_wave14_irt_sem.py (log-lik < 0.1 + class priors < 0.001 vs poLCA::poLCA) |
analysis.latent_profile | R | test_acadval_wave14_irt_sem.py (log-lik < 0.05 + profile means vs mclust VVI) |
analysis.catreg | R | test_acadval_wave14_irt_sem.py (R²/smc within 0.007 + coef signs vs Gifi::morals) |
analysis.catpca | R (partial: eigenvalue ~0.045 documented-disagreement) | test_acadval_wave14_irt_sem.py — eigenvalue-1 within ~0.055 of Gifi::princals (ALS convergence differs); loadings sign-aligned |
analysis.overals | RIGOR | test_acadval_wave14_irt_sem.py — blocked: Gifi/homals 1.0 export no overals(); engine NL-CCA runs, no R reference |
analysis.optimal_scale_transform | RIGOR | test_acadval_wave14_irt_sem.py — structural only; engine transformed values vs princals per-level quantifications not directly comparable |
analysis.catcorr | R | test_acadval_wave14_irt_sem.py (|r| on optimally-scaled vars vs princals quantifications to < 2e-4) |
Survival, Bayesian, meta-analysis & resampling
| Method | Tier | Evidencing test |
|---|---|---|
survival.kaplan_meier | R | test_golden_r.py (survival probs + log-rank) + test_rigor_survival.py |
survival.cox | R | test_golden_r.py::cox_regression (Efron ties, vs R survival) |
survival.life_table | R | test_acadval_wave15_survival.py (Cutler-Ederer q/p/cumS/Greenwood-SE vs base-R, to 5e-4) |
survival.parametric | R | test_acadval_wave15_survival.py (Weibull/exp/lognormal shape/scale/log-lik vs flexsurv::flexsurvreg) |
survival.aft | R | test_acadval_wave15_survival.py (AFT coef/SE/log-lik vs survival::survreg) |
survival.competing_risks | R | test_rigor_rf_wave8_fine_gray.py (Fine-Gray vs reference) + test_rigor_wave3_survival.py |
survival.frailty | R | test_rigor_rf_wave10_gamma_frailty.py (gamma frailty) + test_rigor_wave3_survival.py |
survival.schoenfeld | R (partial: per-covariate χ² documented-disagreement) | test_acadval_wave15_survival.py — lifelines vs cox.zph per-covariate χ² diverge ~3.5 units (different scaling); conclusion + global agree, engine χ² pinned |
survival.nelson_aalen | R | test_acadval_wave15_survival.py (cumulative hazard at event times vs survfit(type='fh'), ~5e-3) |
survival.logrank_stratified | R (partial: χ² documented-disagreement) | test_acadval_wave15_survival.py — lifelines vs survdiff(+strata) χ² differ ~2 units (Mantel-Cox scoring); both reject, engine χ² pinned |
survival.rmst | R (partial: SE + sign-convention documented-disagreement) | test_acadval_wave15_survival.py — |RMST difference| matches survRM2::rmst2; engine sums per-arm Greenwood SEs (inflated vs survRM2 exact variance) |
survival.time_varying_cox | R | test_acadval_wave15_survival.py (start-stop coef/SE/HR/log-lik vs coxph) |
survival.cox_residuals | R | test_acadval_wave15_survival.py (martingale sum-to-0 + martingale/deviance values vs residuals.coxph, ~5e-3) |
bayes.linear | LIB | test_bayes_parity.py (PyMC/emcee port) |
bayes.logistic | LIB | test_bayes_parity.py + test_rigor_wave4_medium_b.py |
bayes.hierarchical | LIB | test_bayes_parity.py (OLS-shrinkage fallback noted) |
bayes.mcmc_diagnostics | LIB | test_bayes_parity.py (xfail in seam — needs trace) |
bayes.bayes_factor | — | seam-smoke (xfail — needs 2 model specs) |
bayes.prior_posterior | LIB | test_bayes_parity.py |
bayes.ppc | LIB | test_bayes_parity.py |
bayes.credible_interval | LIB | test_bayes_parity.py |
meta.fixed_effects | R | test_acadval_wave16_meta.py (pooled/SE/CI/Q/I² vs metafor::rma(method='FE'), to 1e-4) |
meta.random_effects | R | test_acadval_wave16_meta.py (τ²-DL/pooled/I²/prediction-interval vs metafor::rma(method='DL')) |
meta.egger_test | R | test_golden_r.py (intercept t-test on k−2 df) |
meta.forest_plot | R | test_acadval_wave16_meta.py (per-study CI + pooled vs metafor::forest, to 1e-4) |
bootstrap.se | R (partial: SE Monte-Carlo) | test_acadval_wave16_meta.py — observed mean exact vs R; bootstrap SE within 25% (PCG64 ≠ Mersenne-Twister) |
bootstrap.ci | R (partial: CI Monte-Carlo) | test_acadval_wave16_meta.py — observed exact; percentile CI brackets + width validated, bounds MC-noisy |
bootstrap.regression | R (partial: bootstrap SE Monte-Carlo) | test_acadval_wave16_meta.py — classical OLS estimates + SE exact vs lm; bootstrap SE within 35% |
bootstrap.jackknife | R | test_acadval_wave16_meta.py (deterministic leave-one-out: mean/SE/bias/corrected exact vs R) |
bootstrap.permutation_two_sample | R (partial: p-value Monte-Carlo) | test_acadval_wave16_meta.py — observed mean-difference exact vs R; permutation p MC |
bootstrap.permutation_correlation | R (partial: p-value Monte-Carlo) | test_acadval_wave16_meta.py — observed Pearson r exact vs R; permutation p MC |
bootstrap.cross_validation | LIB | test_acadval_wave16_meta.py — sanity only (CV R² < OLS R²); sklearn-parity primary, no exact R reference |
bootstrap.posterior | RIGOR | test_acadval_wave16_meta.py — Bayesian-bootstrap posterior mean/SD converge to closed-form limits (MC), no exact R-fixture match |
Machine learning & choice/conjoint
| Method | Tier | Evidencing test |
|---|---|---|
ml.random_forest | LIB | test_shap_parity.py helpers + seam (sklearn) |
ml.gradient_boosting | LIB | seam-smoke (sklearn/XGBoost) |
ml.svm | LIB | seam-smoke (sklearn) |
ml.decision_tree | LIB | seam-smoke (sklearn) |
ml.cross_validate | LIB | seam-smoke (sklearn) |
ml.roc_auc | LIB | seam-smoke (sklearn) |
ml.confusion_matrix | LIB | seam-smoke (sklearn) |
ml.classification_metrics | LIB | seam-smoke (sklearn) |
ml.feature_importance | LIB | seam-smoke (sklearn permutation importance) |
ml.shap_values | LIB | test_shap_parity.py (vs shap library; xfail in seam) |
ml.shap_summary | LIB | test_shap_parity.py (vs shap; xfail in seam) |
ml.train_test_predict | LIB | seam-smoke (sklearn) |
ml.mlp_classifier | LIB | test_nn_parity.py (sklearn MLP) |
ml.mlp_regressor | LIB | test_nn_parity.py (sklearn MLP) |
ml.deep_nn | LIB | test_nn_parity.py (Keras port) |
ml.autoencoder | LIB | test_nn_parity.py |
ml.cnn_tabular | LIB | test_nn_parity.py |
ml.nn_hyperparameter | LIB | test_nn_parity.py |
ml.chaid | RIGOR | test_rigor_advanced.py::_chaid_best_split (stub elsewhere) |
ml.quest | — | stub (returns placeholder) |
ml.c50 | — | stub (returns placeholder) |
ml.ensemble_trees | — | stub (returns placeholder) |
ml.tree_export | — | stub (xfail in seam — needs fitted tree) |
ml.compare_trees | — | stub (returns placeholder) |
conjoint.traditional | — | seam-smoke only |
conjoint.cbc | — | seam-smoke only |
conjoint.hb_cbc | RIGOR | test_rigor_advanced.py::hierarchical_bayes_cbc |
conjoint.simulator | — | seam-smoke (xfail — needs part-worths) |
conjoint.importance | — | seam-smoke (xfail — needs part-worths) |
conjoint.design | — | seam-smoke only |
conjoint.segmentation | — | seam-smoke (xfail — needs part-worths) |
Complex-survey & missing data
| Method | Tier | Evidencing test |
|---|---|---|
survey.design | R | test_acadval_wave18_survey.py (sum/mean/min/max weights + strata/cluster counts vs svydesign, exact) |
survey.means | R | test_acadval_wave18_survey.py (weighted means exact + Taylor SE within 2% vs survey::svymean) |
survey.totals | R | test_acadval_wave18_survey.py (totals exact + SE within 2% vs survey::svytotal) |
survey.proportions | R | test_acadval_wave18_survey.py (proportions exact + SE within 5–10% vs svymean on a factor) |
survey.regression | R | test_acadval_wave18_survey.py (WLS coef exact + Binder-sandwich SE within 5% vs svyglm) |
survey.deff | R | test_acadval_wave18_survey.py (weighted mean exact + DEFF within 20% vs deff(svymean)) |
survey.replicate_weights | R | test_acadval_wave18_survey.py (JK1 mean + SE vs manual delete-one R loop, within 0.5%) |
survey.ttest | R (partial: design-df documented-disagreement 8-vs-9) | test_acadval_wave18_survey.py — t-stat/mean-diff/group-means within 1–2% of svyttest; design-df path differs |
survey.chi_square | R | test_acadval_wave18_survey.py (Rao-Scott Pearson/F + p vs svychisq; Satterthwaite df within 10%) |
missing.summary | R | test_acadval_wave19_missing.py (n/total/complete-case + per-variable counts vs naniar::miss_var_summary, exact) |
missing.mcar_test | R | test_golden_r.py + test_acadval_wave19_missing.py (Little 1988 χ²/df/p vs naniar::mcar_test) |
missing.impute_simple | R | test_acadval_wave19_missing.py (mean/median/mode fill exact vs base R) |
missing.impute_knn | RIGOR | test_acadval_wave19_missing.py — engine standardized-Euclidean KNN vs VIM::kNN Gower is a documented disagreement (values differ ~1.9); structural-only |
missing.mice | RIGOR | test_acadval_wave19_missing.py — pooled means near R complete-case mean + FMI∈[0,1], but sklearn-IterativeImputer vs mice PMM is stochastic; no exact R match |
missing.em | R (partial: covariance ddof documented-disagreement) | test_acadval_wave19_missing.py — EM means match norm::em.norm to 5e-4; covariance differs (ddof=1 vs ML 1/n) |
missing.pmm | RIGOR | test_acadval_wave19_missing.py — donor-support + observed-donor properties match R; values are stochastic (partial-imputation limitation noted) |
missing.patterns | R | test_acadval_wave19_missing.py (pattern counts + monotone flag vs mice::md.pattern, exact) |
missing.sensitivity | — | seam-smoke only |
ADV Waves 1–16 — built and independently re-validated (2026-06-19)
The 192 methods below were added this campaign (engine modules adv_wave1..16.py) and then independently re-validated by a 16-agent adversarial workflow that checked each against an EXTERNAL oracle — R (plm/AER/mgcv/car/sandwich/quantreg), scikit-learn/statsmodels computed a different way, closed-form re-derivation, or fresh known-DGP recovery — not the build pipeline's own test. 186/192 confirmed on the first independent pass; 6 genuine defects were found and fixed (marked below), then re-verified against the oracle. Tier reflects the strongest independent oracle that agreed: R = matched an R package; LIB = parity with a peer library (sklearn/statsmodels) computed independently; RIGOR = closed-form / known-DGP recovery (no external statistical-software oracle exists for the method).
| Method | Tier | Evidencing test |
|---|---|---|
bayes.bayesian_cfa | R | test_bayesian_cfa.py — independent re-validation 2026-06-19: Fit R lavaan::cfa(std.lv=TRUE) (ML, factor variance fixed at 1, same identification) on the identical 800x5 data and com… |
bayes.bayesian_irt | R | test_bayesian_irt_2pl.py — independent re-validation 2026-06-19: Fit R mirt (MML 2PL, IRTpars) on the identical simulated 600x10 binary matrix and compared item discriminations and diff… |
bayes.bayesian_lasso | LIB | test_bayesian_lasso.py — independent re-validation 2026-06-19: Validated the load-bearing custom inverse-Gaussian sampler (Michael-Schucany-Haas) against scipy.stats.invgauss via a KS… |
bayes.bayesian_meta | R | test_bayesian_meta_analysis.py — independent re-validation 2026-06-19: Cross-checked the frequentist reference quantities against R metafor::rma on the same 20-study data: DL mu/tau2/se and F… |
bayes.bayesian_poisson | LIB | test_bayesian_poisson.py — independent re-validation 2026-06-19: Ran the RW-Metropolis sampler on a Poisson DGP (n=5000, beta=[1.0,0.5,-0.3]) and compared posterior means to an INDEPEND… |
bayes.bayesian_probit | LIB | test_bayesian_probit.py — independent re-validation 2026-06-19: Re-ran the Gibbs sampler on a fresh probit DGP (n=6000, beta=[-0.4,1.3,-0.9]) and compared the posterior mean to the fre… |
bayes.bayesian_quantile | R | test_bayesian_quantile_regression.py — independent re-validation 2026-06-19: Ran R quantreg::rq at tau=0.1/0.5/0.9 on the same datasets (external oracle) and compared the frequentist quantile slope… |
bayes.bayesian_tobit | R | test_bayesian_tobit.py — independent re-validation 2026-06-19: Ran the Gibbs sampler on a left-censored DGP (n=1500, beta=[0.5,1.2,-0.7], sigma=1, 39% censored at 0) and compared post… |
bayes.bayesian_unit_root | LIB | test_bayesian_unit_root.py — independent re-validation 2026-06-19: Simulated a stationary AR(1) (rho=0.65), fit the engine, and independently fit y_t on [1, y_{t-1}] with statsmodels.OLS;… |
bayes.bma | R | test_bayesian_model_averaging.py — independent re-validation 2026-06-19: Generated data in Python (y=3 x1+1.2 x2+noise, x3/x4 irrelevant), wrote to CSV, and recomputed all 16 subset BICs, softm… |
bayes.bvar_minnesota | R | test_bvar_minnesota.py — independent re-validation 2026-06-19: Generated a stationary VAR(1) in Python, wrote to CSV, fit the engine with lambda1=1e8 (loose prior limit), and compared… |
bayes.bvecm | R | test_bayes_bvecm.py — independent re-validation 2026-06-19: Cross-checked the posterior-mean cointegrating vector, restricted constant, and adjustment loadings against R Johansen M… |
bayes.dic_waic_loo | RIGOR | test_bayesian_information_criteria.py — independent re-validation 2026-06-19: Cross-checked against arviz (az.waic / az.loo, deviance scale) — the reference WAIC/PSIS-LOO implementation by the paper… |
bayes.gibbs_sampler | R | test_gibbs_linear_regression.py — independent re-validation 2026-06-19: Ran R lm() on the same dataset (external oracle) and compared OLS coefficients, SEs and sigma^2 to the Gibbs posterior m… |
bayes.horseshoe | RIGOR | test_horseshoe_regression.py — independent re-validation 2026-06-19: No standard library oracle for the Makalic-Schmidt horseshoe Gibbs sampler, so I ran my OWN fresh sparse DGP (n=180,p=15… |
bayes.importance_sampling | LIB | test_bayes_importance_sampling.py — independent re-validation 2026-06-19: Independently compared IS posterior means/SDs to statsmodels OLS (diffuse prior => posterior ~ OLS) and verified the ana… |
bayes.laplace_approx | RIGOR | test_bayes_laplace_approx.py — independent re-validation 2026-06-19 (defect found + fixed, re-verified): Independently verified the binomial MAP and Tierney-Kadane mean against a fine 2D posterior grid; then computed the TRUE… |
bayes.metropolis_hastings | R | test_metropolis_hastings_regression.py — independent re-validation 2026-06-19: Compared posterior means/SDs and sigma^2 against R lm() OLS (external oracle) on the same data; checked the auto-tuned a… |
bayes.normal_gamma | LIB | test_bayesian_normal_gamma.py — independent re-validation 2026-06-19: Flat prior (precision=0): compared posterior means, 95% credible intervals and sigma2 mean to statsmodels.OLS coefficien… |
bayes.savage_dickey | RIGOR | test_savage_dickey.py — independent re-validation 2026-06-19: Independently re-derived the conjugate Gaussian posterior (Vinv=X'X/sigma2 + I/tau^2, tau=sd(y), sigma2=OLS resid var) a… |
bayes.spike_slab | RIGOR | test_spike_slab.py — independent re-validation 2026-06-19: Independently enumerated all 2^6 models with the exact Zellner g-prior closed-form marginal likelihood (same standardiza… |
bayes.student_t_regression | R | test_student_t_regression.py — independent re-validation 2026-06-19: On the outlier-contaminated dataset (nu=3) maximized the iid Student-t log-likelihood in R via optim(BFGS) over (beta,lo… |
bootstrap.bayesian | R | test_bayesian_bootstrap.py — independent re-validation 2026-06-19: Ran 4000 Dirichlet(1..1)-weighted OLS draws on a known DGP (n=4000, beta=[1,2,-0.5]) and compared the posterior mean to … |
bootstrap.bca | R | test_bca_bootstrap.py — independent re-validation 2026-06-19: Cross-checked against R boot::boot.ci(type='bca') on identical gamma(2,2) data (n=90): compared the acceleration (RNG-fr… |
bootstrap.block | RIGOR | test_block_bootstrap.py — independent re-validation 2026-06-19: No single library oracle for moving/circular block bootstrap; constructed my own known-DGP checks: iid N(0,3) (SE of mea… |
bootstrap.cluster | R | test_cluster_bootstrap.py — independent re-validation 2026-06-19: Cross-checked the engine's reported analytic seClusterRobust (CR1) and seIid against R sandwich::vcovCL(type='HC1',cadju… |
bootstrap.parametric | LIB | test_parametric_bootstrap.py — independent re-validation 2026-06-19: Cross-checked point estimates and classicalSE against statsmodels.OLS (independent oracle, different code path) |
bootstrap.percentile_t | RIGOR | test_percentile_t_bootstrap.py — independent re-validation 2026-06-19: Recomputed the point estimate and classical homoskedastic SE via my own closed-form OLS (X'X)^-1 and confirmed the stude… |
bootstrap.residual | RIGOR | test_residual_bootstrap.py — independent re-validation 2026-06-19: Independent closed-form classical OLS SE (numpy, no library) on large-n homoskedastic data, compared to the engine's boo… |
bootstrap.subsampling | RIGOR | test_subsampling.py — independent re-validation 2026-06-19: Re-derived the full sqrt(b)(theta_b - theta_n) root distribution by exhaustively enumerating all C(5,2)=10 size-2 subsam… |
bootstrap.wild | LIB | test_wild_bootstrap_ols.py — independent re-validation 2026-06-19: Generated genuinely heteroskedastic data (error sd = 0.5+/x/, n=5000) and compared the Rademacher wild-bootstrap SEs (re… |
causal.abadie_kappa | RIGOR | test_abadie_kappa.py — independent re-validation 2026-06-19: Built a 60000-row monotonicity DGP with labeled always/never/complier types; independently computed true complier share,… |
causal.callaway_santanna | R | test_callaway_santanna.py — independent re-validation 2026-06-19: Generated a fixed staggered panel (cohorts 4,7,never; 900 units), wrote CSV, ran engine vs R did::att_gt(control_group='… |
causal.causal_forest | RIGOR | test_causal_forest.py — independent re-validation 2026-06-19: Re-ran both DGPs in tempdir and computed my own oracles: for the randomized-treatment hetero DGP the diff-in-means ATE (… |
causal.changes_in_changes | RIGOR | test_changes_in_changes.py — independent re-validation 2026-06-19: Independently hand-recomputed the Athey-Imbens map on the tiny dataset (y00=[1,2,3,4], y01=[10,20,30,40], y10=[2,4], y11… |
causal.continuous_treatment | RIGOR | test_continuous_treatment_dose_response.py — independent re-validation 2026-06-19: Ran my OWN fresh confounded DGPs (X drives both T and Y; true linear dose effects b=3.0 and b=-2.0) and checked that the… |
causal.double_ml | RIGOR | test_double_ml.py — independent re-validation 2026-06-19: Re-derived two oracles in tempdir on the venv: (1) LINEAR-g case where Frisch-Waugh-Lovell OLS-of-Y-on-[1,D,X] is the ex… |
causal.dr_did | RIGOR | test_dr_did.py — independent re-validation 2026-06-19: In tempdir: (1) hand-computed the 2x2 mean DiD on a fresh dataset = 1.010542, dr_did att = 1.010542 (diff 4e-16), ipw/or… |
causal.honest_did | R (partial) | test_honest_did.py — independent re-validation 2026-06-19 (defect found + fixed, re-verified): Replicated the TWFE entity+time-FE event-study design in statsmodels with HC1 and compared the rel-time-0 coefficient an… |
causal.mccrary | RIGOR | test_mccrary_density.py — independent re-validation 2026-06-19: MC under the null (300 reps, N(0,1), fixed bw=0.5): checked mean/sd of z and rejection rate against the standard normal … |
causal.mte | R | test_marginal_treatment_effect.py — independent re-validation 2026-06-19: No single R function replicates Heckman-Vytlacil local-IV MTE exactly, so I ran my own generalized-Roy DGP with closed-f… |
causal.rdrobust | RIGOR | test_causal_rdrobust.py — independent re-validation 2026-06-19: Generated a sharp-RD CSV (true jump tau=0.75, cubic trend, n=4000) and ran R's rdrobust (CCT 2014) on the exact same dat… |
causal.sun_abraham | R | test_sun_abraham.py — independent re-validation 2026-06-19: Generated a fixed staggered panel (cohorts 4,6,8,never; unequal sizes; noise 0.5), wrote CSV, ran engine vs R feols(y~su… |
causal.synthetic_did | RIGOR | test_synthetic_did.py — independent re-validation 2026-06-19: Re-ran the FE panel DGP in tempdir for tau=+5/-4/0: att=4.955/-4.045/-0.045, unit weights sum to 1.0000 and are non-nega… |
diagnose.harvey_het | RIGOR | test_harvey_multiplicative_het.py — independent re-validation 2026-06-19: Generated multiplicative-het data with Var(e)=exp(gamma*z), gamma=1.2, and confirmed the engine rejects and the auxiliar… |
diagnose.mv_normality | R | test_multivariate_normality.py — independent re-validation 2026-06-19: Recomputed Mardia b1p/b2p and the skewness statistic in R via explicit Mahalanobis quadratic forms and matched the engin… |
diagnose.score_lm | RIGOR | test_score_lm_test.py — independent re-validation 2026-06-19: Recomputed the LM statistic in R: restricted residuals from lm(y |
diagnose.white_im | R | test_white_im_test.py — independent re-validation 2026-06-19: Cross-checked the n*R^2 IM statistic against R lmtest::bptest using the exact auxiliary design the engine uses (squares … |
missing.pattern_mixture | RIGOR | test_pattern_mixture_model.py — independent re-validation 2026-06-19: Hand-recomputed the entire closed form independently: OLS on completers via np.linalg.lstsq, imputed missing mean, overa… |
missing.selection_model | R | test_selection_model_mnar.py — independent re-validation 2026-06-19: Ran R sampleSelection::heckit (2-step) on identical 8000-row data sharing predictors x1,x2,zexcl across selection and ou… |
model.adaptive_lasso | RIGOR | test_adaptive_lasso.py — independent re-validation 2026-06-19: Generated a sparse DGP (p=12, true nonzero at x0,x3,x7 = 3.0,-2.5,4.0, mildly correlated design) and checked the selecte… |
model.additive_quantile | R | test_additive_quantile.py — independent re-validation 2026-06-19: Cross-checked against R quantreg::rq (independent solver) on a linear median DGP: engine additive-spline median curve vs… |
model.bart | RIGOR | test_bart.py — independent re-validation 2026-06-19: Re-ran Friedman DGP (only x1..x5 enter f) in tempdir: BART R2=0.963, RMSE 0.909 vs my independent OLS RMSE 2.516; comput… |
model.bivariate_probit | R | test_bivariate_probit.py — independent re-validation 2026-06-19: Generated fresh-seed biprobit data (rho=0.55, b1=[0.3,0.8], b2=[-0.2,-0.5], n=6000) in Python, ran the engine, then wrot… |
model.choice_based_sampling | R | test_choice_based_sampling_logit.py — independent re-validation 2026-06-19: Built my own population (30000 sets, true beta=[1.2,-0.9]) with a rare oversampled alternative, ran the engine; verified… |
model.clad | R | test_censored_lad.py — independent re-validation 2026-06-19: Re-implemented Powell's iterated-active-set CLAD using statsmodels QuantReg (an IRLS-based LAD solver, totally different… |
model.cloglog | R | test_cloglog_regression.py — independent re-validation 2026-06-19: Cross-checked engine coefficients, std errors, log-likelihood, and AIC against R glm(y~x1+x2, family=binomial(link='clog… |
model.com_poisson | RIGOR | test_com_poisson.py — independent re-validation 2026-06-19: Wrote an independent COM-Poisson sampler (inverse-CDF over the truncated pmf) with known nu in {1.0,2.0,0.5} and known l… |
model.composite_quantile | RIGOR | test_composite_quantile.py — independent re-validation 2026-06-19: Computed the engine's pooled check-loss objective and verified 2000 random slope perturbations (with intercepts re-optim… |
model.conditional_logit | R | test_conditional_logit.py — independent re-validation 2026-06-19: Generated 800-occasion x 4-alt clogit data (beta=[1,-0.8,0.5]) in Python, ran the engine, then fit the SAME data with R … |
model.copula | LIB | test_copula_fit.py — independent re-validation 2026-06-19: On a Gaussian-copula DGP (rho=0.6): compared the fitted gaussian rho against the INDEPENDENT Pearson correlation of the … |
model.cue_gmm | R | test_cue_gmm.py — independent re-validation 2026-06-19: Cross-checked the OVERIDENTIFIED case (1 endog, 3 instruments, n=1500) against R's gmm package gmm(type='cue', vcov='iid… |
model.discrete_time_hazard | R | test_discrete_time_hazard.py — independent re-validation 2026-06-19: Simulated a fresh cloglog discrete-time PH DGP (n=4000,K=6, beta=[0.8,-0.5]), built the person-period expansion EXACTLY … |
model.dpm_density | LIB | test_dp_mixture_density.py — independent re-validation 2026-06-19: Simulated a well-separated 3-component Gaussian mixture (means -6,0,6) and a unimodal Gaussian |
model.el_regression | RIGOR | test_empirical_likelihood_regression.py — independent re-validation 2026-06-19: 400-rep MC: tested the EL-ratio of the TRUE coefficient null (by shifting y) and checked the statistic is chi2(1)-distri… |
model.elastic_net_cv | R | test_model_elastic_net_cv.py — independent re-validation 2026-06-19: Took the engine's lambda_min, exported the standardized design, and ran R glmnet(alpha=0.5, lambda=lambda_min, standardi… |
model.empirical_likelihood | RIGOR | test_empirical_likelihood_mean.py — independent re-validation 2026-06-19: Wrote my OWN bisection-on-dual solver (root of f'(lam)=sum z/(1+lam z)=0, a wholly different algorithm from the engine's… |
model.endogenous_count | LIB | test_endogenous_count.py — independent re-validation 2026-06-19: Hand-rolled the full 2SRI estimator independently (OLS first stage x |
model.expectile_regression | R | test_expectile_regression.py — independent re-validation 2026-06-19: Checked tau=0.5 coefficients equal the OLS closed form; solved the scalar tau-expectile defining equation by independent… |
model.fiml | R | test_model_fiml.py — independent re-validation 2026-06-19: Cross-checked the FIML point estimates against R systemfit (2SLS and 3SLS) on a fixed dataset, and independently re-opti… |
model.finite_mixture_count | RIGOR | test_finite_mixture_count.py — independent re-validation 2026-06-19: Independently recomputed the observed-data mixture log-likelihood from the RETURNED per-class coefficients + mixing weig… |
model.finite_mixture_regression | R | test_finite_mixture_regression.py — independent re-validation 2026-06-19: Generated a well-separated 2-component mixture-of-regressions in Python (b1=[2,3,-1], b2=[-4,-2,4], sd=0.5, pi1=0.65), w… |
model.gam | R | test_gam_regression.py — independent re-validation 2026-06-19 (defect found + fixed, re-verified): Cross-checked against R mgcv::gam (per-term GCV) on y=sin(x1)+x2^2; and ran the engine on a TRULY LINEAR DGP to test the… |
model.gam_glm | R | test_model_gam_glm.py — independent re-validation 2026-06-19: Ran mgcv::gam(y~s(x1)+s(x2), poisson) on the same n=1200 Poisson data; compared deviance, deviance-explained, intercept,… |
model.gaussian_process_regression | LIB | test_gaussian_process_regression.py — independent re-validation 2026-06-19: Constructed two known-DGP cases: (a) near-noiseless y=sin(x)+0.3x -> posterior mean must interpolate; (b) noisy y=sin(t)… |
model.gee | RIGOR | test_gee_population_averaged.py — independent re-validation 2026-06-19: No geepack in R, so used known-DGP recovery: random-intercept Gaussian panel (var(u)=1.44, var(eps)=0.64 => true ICC=0.6… |
model.gel | R | test_generalized_empirical_likelihood.py — independent re-validation 2026-06-19: Cross-checked the OVERIDENTIFIED case (1 endog, 3 instruments, n=4000) against R's gmm package gel(type=c('EL','ET','CUE… |
model.generalized_ordered_logit | R | test_generalized_ordered_logit.py — independent re-validation 2026-06-19: (a) Built a CLEAN GOL DGP with 100% monotone cumulative probs (no sort artifact, true x1 split slopes 0.3/0.8/1.3, x2=-0… |
model.gls | RIGOR | test_general_gls.py — independent re-validation 2026-06-19: Forced structure='ar1', rho=0.5 and independently recomputed the Aitken GLS estimator beta=(X'Sigma^-1 X)^-1 X'Sigma^-1 … |
model.gp_classification | LIB | test_gp_classification.py — independent re-validation 2026-06-19: Generated a circular (radially separable, label-noisy) 2-class DGP that no linear boundary can separate |
model.heteroskedastic_probit | LIB | test_heteroskedastic_probit.py — independent re-validation 2026-06-19: Re-estimated the Harvey log-likelihood from scratch with a DIFFERENT optimizer pipeline (scipy Nelder-Mead -> L-BFGS-B f… |
model.isotonic_regression | LIB | test_isotonic_regression.py — independent re-validation 2026-06-19 (defect found + fixed, re-verified): Cross-checked engine fitted values vs sklearn.IsotonicRegression (increasing and decreasing) on continuous data, and con… |
model.iv_quantile | R | test_iv_quantile.py — independent re-validation 2026-06-19: At tau=0.5 compared the CH IVQR endog coefficient to independent R AER::ivreg 2SLS (theoretical median anchor) and verif… |
model.kernel_conditional_density | RIGOR | test_kernel_conditional_density.py — independent re-validation 2026-06-19: Wrote my own double-kernel CDE (NW weights in x, Gaussian smoothing in y, erf-based CDF inversion) from the Rosenblatt/H… |
model.kernel_regression | RIGOR | test_kernel_regression.py — independent re-validation 2026-06-19: Independently recomputed the Nadaraya-Watson local-constant estimate sum(K*y)/sum(K) at a grid point with my own Gaussia… |
model.knn_regression | LIB | test_knn_regression.py — independent re-validation 2026-06-19: Re-ran the engine on a 200-pt smooth DGP and compared its in-sample R2 and LOO R2/RMSE against an INDEPENDENT sklearn im… |
model.lewbel_iv | R | test_model_lewbel_iv.py — independent re-validation 2026-06-19: Reconstructed the Lewbel generated instrument (centered exog * first-stage residual) exactly as the engine does, exporte… |
model.liml | R | test_liml.py — independent re-validation 2026-06-19: Built a weak/many-instrument case (10 weak instruments, n=500) so LIML kappa is meaningfully >1 and diverges from 2SLS |
model.local_linear | RIGOR | test_local_linear_regression.py — independent re-validation 2026-06-19: Verified the defining degree-1 local-polynomial property (exact reproduction of a straight line) |
model.local_polynomial | RIGOR | test_local_polynomial.py — independent re-validation 2026-06-19: (1) At a mid grid point recomputed the local quadratic fit with an independent np.linalg.solve WLS using an unscaled mon… |
model.mahalanobis_matching | LIB | test_mahalanobis_matching.py — independent re-validation 2026-06-19: Recomputed the matched ATT via scipy.spatial.distance.cdist with metric='mahalanobis', VI=inverse pooled-control covaria… |
model.manski_bounds | RIGOR | test_model_manski_bounds.py — independent re-validation 2026-06-19: On a tiny hand-checkable dataset (treated y=[.6,.8,1.0], control y=[.2,.4], support [0,1]) I computed by hand the Manski… |
model.maximum_score | RIGOR | test_maximum_score.py — independent re-validation 2026-06-19: For k=2, brute-forced the exact maximum-score objective over 40,001 unit directions on the FULL circle [0,2pi) and compa… |
model.minimum_distance | R | test_model_minimum_distance.py — independent re-validation 2026-06-19: Cross-checked the over-identified CMD beta and the overidentification chi-square against R AER::ivreg (2SLS) and its Sar… |
model.mixed_logit | RIGOR | test_mixed_logit.py — independent re-validation 2026-06-19: Constructed my own fresh-seed mixed-logit DGP (random price coef ~ N(-1.0, 1.2^2), fixed qual coef 1.0, Gumbel errors, 3… |
model.mnp_ghk | RIGOR | test_multinomial_probit_ghk.py — independent re-validation 2026-06-19: Generated fresh-seed MNP data (beta=[1.0,-0.7], correlated Sigma, J=3, T=2000) and ran the engine's GHK-MSL |
model.msm_indirect | RIGOR | test_model_msm_indirect.py — independent re-validation 2026-06-19: No standard external library implements this exact AR(1)+noise indirect-inference setup, so I built a strong known-DGP c… |
model.multivariate_probit | RIGOR | test_multivariate_probit.py — independent re-validation 2026-06-19: Generated my own fresh-seed 3-equation MVP data (R off-diag 0.5/0.3/-0.2, slopes 0.8/0.5/-0.6, n=5000), independent of t… |
model.nested_logit | R | test_nested_logit.py — independent re-validation 2026-06-19: Simulated 8000-occasion 2-nest data (beta=[1.2,-0.8], lambda=0.4) in Python, ran the engine, then fit the SAME data with… |
model.nonlinear_gmm | RIGOR | test_nonlinear_gmm.py — independent re-validation 2026-06-19: Simulated Mullahy's multiplicative-error model y=exp(x'b)*eta with endogenous x1 and valid instrument z |
model.oaxaca | LIB | test_oaxaca_blinder.py — independent re-validation 2026-06-19: Ran engine on a 2-group DGP (different betas + different X means) |
model.optimal_iv_gmm | R | test_model_optimal_iv_gmm.py — independent re-validation 2026-06-19: Cross-checked the 2SLS benchmark against R AER::ivreg with HC0 robust SE; independently ran a 300-replication Monte Carl… |
model.overdispersion_test | R | test_overdispersion_test.py — independent re-validation 2026-06-19 (defect found + fixed, re-verified): Cross-checked vs R AER::dispersiontest (trafo=1 NB1 linear, trafo=2 NB2 quadratic) and re-derived the canonical CT auxil… |
model.panel_two_way_re | R | test_panel_two_way_re.py — independent re-validation 2026-06-19: Cross-checked against R plm::plm(model='random', effect='twoways', random.method='walhus') on a balanced N=25,T=15 simul… |
model.partially_linear | RIGOR | test_partially_linear.py — independent re-validation 2026-06-19: Coded a fully independent Robinson double-residual estimator (leave-one-out product-Gaussian NW smoother + residual-on-r… |
model.pmg | R | test_pooled_mean_group.py — independent re-validation 2026-06-19: No standard R/Python oracle for PMG in the available libraries (plm has no PMG) |
model.ps_subclassification | LIB | test_ps_subclassification.py — independent re-validation 2026-06-19: Recomputed ATE and seAte from scratch using statsmodels GLM(Binomial) for the propensity score (a DIFFERENT API than the… |
model.quantile_count | LIB | test_quantile_count.py — independent re-validation 2026-06-19: Generated overdispersed NegBin counts (Gamma-Poisson mixture) with known log-mean b1=0.8,b2=-0.5 |
model.quantile_forest | LIB | test_quantile_forest.py — independent re-validation 2026-06-19: Built an INDEPENDENT Meinshausen leaf-weighted-CDF QRF on a separately-fit sklearn forest and recovered conditional quan… |
model.quantile_panel | R | test_quantile_panel.py — independent re-validation 2026-06-19: Simulated a panel with FE correlated with x (true slope 2.0) |
model.quantile_simultaneous | R | test_quantile_simultaneous.py — independent re-validation 2026-06-19: Cross-checked the per-tau POINT estimates against R quantreg::rq(y~x1, tau=t) on the location-scale DGP |
model.roy_switching | RIGOR | test_endogenous_switching.py — independent re-validation 2026-06-19: Built a fresh-seed Roy DGP (rho1=+0.7, rho0=-0.6, known regime intercepts 2.0/-1.0, n=5000), ran the engine, and indepen… |
model.semiparametric_iv | R | test_model_semiparametric_iv.py — independent re-validation 2026-06-19: Built the identical standardized power-sieve bases P=[xs,xs^2,xs^3] and instrument basis (z1,z2 powers), exported, and r… |
model.sequential_logit | LIB | test_sequential_logit.py — independent re-validation 2026-06-19: Cross-checked each continuation-ratio stage against statsmodels.api.Logit fit on that stage's exact risk set {Y>=r} with… |
model.simex | RIGOR | test_simex.py — independent re-validation 2026-06-19: Built the classical-measurement-error DGP (w=x_true+u, var u=0.2) with true bx=2.0 over 30 seeds |
model.simulated_ml | LIB | test_model_simulated_ml.py — independent re-validation 2026-06-19: Ran my own known-DGP recovery at n=12000 (Halton draws) and independently verified the engine's pooled-probit baseline a… |
model.single_index | RIGOR | test_single_index_model.py — independent re-validation 2026-06-19: No standard R/library oracle for Ichimura SLS, so I ran my OWN fresh-seed known-DGP recoveries with true directions [2,-… |
model.spline_regression | R | test_spline_regression.py — independent re-validation 2026-06-19: Cross-checked against R splines::bs(x,df=6,degree=3) + lm() on the same data (n=150, y=0.5x^2-x+noise) |
model.stochastic_frontier | RIGOR | test_stochastic_frontier.py — independent re-validation 2026-06-19: R 'frontier' package unavailable, so wrote an independent ALS half-normal MLE in (beta, sigma_u, sigma_v) directly (engi… |
model.system_ols | R | test_system_ols.py — independent re-validation 2026-06-19: Cross-checked against R systemfit(method='OLS') on a fresh 2-equation system (n=300, correlated errors rho~0.42) |
model.translog_production | LIB | test_translog_production.py — independent re-validation 2026-06-19: Rebuilt the exact translog design (const, logK, logL, 0.5logK^2, 0.5logL^2, logK*logL) and ran statsmodels.OLS indepen… |
model.two_sample_iv | R | test_two_sample_iv.py — independent re-validation 2026-06-19: Computed full-sample 2SLS via the projection formula independently in Python, and ran R AER::ivreg on a comparable trian… |
model.var_ic_table | R | test_var_ic_table.py — independent re-validation 2026-06-19: Cross-checked the full per-lag AIC/HQ/SC/FPE table and the recommended orders against R vars::VARselect(type='const') on… |
model.wavelet_regression | RIGOR | test_wavelet_regression.py — independent re-validation 2026-06-19: Verified the db4 single-level analysis operator is orthonormal (=> perfect reconstruction) and the filter sum-rules; the… |
panel.anderson_hsiao | R | test_anderson_hsiao.py — independent re-validation 2026-06-19: Built the first-differenced data and the y_{t-2} level instrument BY HAND in R and ran AER::ivreg(DY ~ 0+DYL1+DX / 0+INS… |
panel.baltagi_li | R | test_baltagi_li_ar.py — independent re-validation 2026-06-19: (1) Reproduced the engine's rho independently: within(FE) residuals -> raw pooled AR(1) rho_raw -> applied the documente… |
panel.bias_corrected_fe | RIGOR | test_bias_corrected_fe.py — independent re-validation 2026-06-19: MC over 60 reps per case of pure AR(1) FE panels (N=200, short T=6-8): estimated the ACTUAL within-FE bias (mean within … |
panel.blup | R | test_panel_blup.py — independent re-validation 2026-06-19: Cross-checked the per-entity BLUPs against R nlme::lme(y~x1+x2, random=~1/entity, REML) ranef() on a balanced N=30,T=10 … |
panel.cce | R | test_common_correlated_effects.py — independent re-validation 2026-06-19 (defect found + fixed, re-verified): Two fixed factor-panels (N=50,T=70), wrote CSV, ran engine MG and Pooled vs R plm::pcce(model='mg') and model='p') |
panel.fod_gmm | R | test_fod_gmm.py — independent re-validation 2026-06-19: On the AR(1) dynamic-panel DGP (rho=0.6,beta=0.5) compared the engine to R plm::pgmm (Arellano-Bond first-difference GMM… |
panel.minque | RIGOR | test_minque.py — independent re-validation 2026-06-19: Re-implemented MINQUE(0) from first principles forming the explicit N x N matrices M=I-X(X'X)^-1X', A1=ZZ', A2=I, q_j=r'… |
panel.parks_kmenta | R | test_parks_kmenta.py — independent re-validation 2026-06-19: Re-implemented the full Parks-Kmenta estimator from scratch in numpy (pooled OLS -> panel rho_i -> Prais-Winsten quasi-d… |
panel.random_trend | RIGOR | test_random_trend_panel.py — independent re-validation 2026-06-19: Built the explicit FWL design in R: lm(y ~ entity + entity:period + x - 1) with entity as a factor (full entity intercep… |
post.generated_regressor | RIGOR | test_generated_regressor.py — independent re-validation 2026-06-19: Ran my own 2000-replication Monte-Carlo of the two-step DGP (s1=fstar+noise; y=0.5+3fstar-0.8x3+e) and compared the EM… |
post.potential_outcomes | R | test_potential_outcomes.py — independent re-validation 2026-06-19: Cross-checked ATE/ATT/ATU against R lm(y~Tx1+Tx2) with predict-everyone-treated vs predict-everyone-control averaging |
stats.joint_confidence | R | test_joint_confidence_region.py — independent re-validation 2026-06-19: Cross-checked the joint F statistic against R car::linearHypothesis(lm, c('x1=0','x2=0')) |
survival.cure_model | R | test_cure_model.py — independent re-validation 2026-06-19: Cross-checked against R's dedicated flexsurvcure package: flexsurvcure(Surv(time,event)~1, dist='weibull', mixture=TRUE)… |
survival.piecewise_exponential | R | test_piecewise_exponential.py — independent re-validation 2026-06-19: (1) n_intervals=1: compared engine beta/baseline-hazard to R survreg(dist='exponential') with the AFT->PH conversion |
test.cips | R | test_cips_panel_unit_root.py — independent re-validation 2026-06-19: Cross-checked the CIPS statistic against R plm::cipstest(type='drift' = Pesaran Case II intercept, model='cmg', truncate… |
test.cox_nonnested | R | test_cox_nonnested_test.py — independent re-validation 2026-06-19: Cross-checked the standardized Cox statistics N1, N2 against R lmtest::coxtest on the same shared dataset |
test.encompassing | R | test_encompassing_test.py — independent re-validation 2026-06-19: Cross-checked both directions' artificial-nesting F statistics against R lmtest::encomptest (plm/lmtest) on a shared CSV… |
test.j_test | R | test_davidson_mackinnon_j.py — independent re-validation 2026-06-19: Cross-checked both directions against R lmtest::jtest on a non-nested DGP (Model1=x1,x2 true; Model2=x3,x4 correlated ri… |
test.panel_bplm | R | test_panel_bplm.py — independent re-validation 2026-06-19: Cross-checked BP and Honda statistics against R plm::plmtest (type='bp' and type='honda', effect='individual') on a one-… |
test.panel_cd_bplm | R | test_panel_cd_bplm.py — independent re-validation 2026-06-19: Cross-checked all four statistics against R plm::pcdtest (test='lm','sclm','bcsclm','cd') on the fixed-effects within mo… |
test.panel_poolability | RIGOR | test_panel_poolability.py — independent re-validation 2026-06-19: Cross-checked the Chow/Baltagi F against a fully independent dummy-variable interaction F-test (entity dummies x interce… |
test.panel_serial_correlation | R | test_panel_serial_correlation.py — independent re-validation 2026-06-19: (1) Ran the engine under H0 (iid idiosyncratic errors) over many panels to confirm the auxiliary slope rho converges to … |
test.panel_unit_root_ips | R | test_panel_unit_root_ips.py — independent re-validation 2026-06-19: Independently recomputed entity-0's ADF t-ratio (intercept, p=2) from scratch AND via statsmodels.tsa.stattools.adfuller |
test.vuong | R | test_vuong_test.py — independent re-validation 2026-06-19: Independently recomputed the Vuong V = sqrt(n)*mean(m)/sd_pop(m) and per-model total log-likelihoods in R using lm() res… |
ts.bds_test | LIB | test_bds_test.py — independent re-validation 2026-06-19: Ran the engine on fresh iid Gaussian data (n=400) at max_dim=4 (one dim beyond the build test) and compared statistic+p-… |
ts.bekk_garch | RIGOR | test_bekk_garch.py — independent re-validation 2026-06-19: Took the engine's reported diagA/diagB/intercept(C C') and independently re-ran the diagonal-BEKK recursion H_t=CC'+A ep… |
ts.blanchard_quah | R | test_blanchard_quah.py — independent re-validation 2026-06-19: Cross-checked against R vars::BQ(VAR(...,p=1,type='const')) on the bivariate VAR(1) DGP with a known lower-triangular tr… |
ts.causality_in_variance | RIGOR | test_causality_in_variance.py — independent re-validation 2026-06-19: Verified the lag-sign DIRECTIONAL CONVENTION (the only place a subtle error could hide) by injecting a spillover where s… |
ts.dfm_em | LIB | test_ts_dfm_em.py — independent re-validation 2026-06-19: Simulated a fresh 1-factor DGP (phi=0.75, N=6, T=500), ran the engine and statsmodels DynamicFactor on standardized data… |
ts.dols | R | test_dols_cointegration.py — independent re-validation 2026-06-19: Reconstructed the full DOLS augmented design (levels + 2 leads + contemporaneous + 2 lags of dx) in R, fit by lm, and ap… |
ts.figarch | RIGOR | test_figarch.py — independent re-validation 2026-06-19: Simulated FIGARCH(1,d,1) returns via the ARCH(inf) recursion for true d in {0.45,0.6,0.3} and checked d_hat and the long… |
ts.fmols | RIGOR | test_fmols_cointegration.py — independent re-validation 2026-06-19: No external oracle (cointReg not installed) |
ts.garch_m | LIB | test_garch_in_mean.py — independent re-validation 2026-06-19: Simulated a true GARCH(1,1)-in-mean (mu=0.05, lambda=0.5, omega=0.05, alpha=0.08, beta=0.90), ran the engine AND a direc… |
ts.garch_portmanteau | LIB | test_ts_garch_portmanteau.py — independent re-validation 2026-06-19: Compared the engine's QMLE GARCH(1,1) parameters to the independent arch package (arch_model); compared Ljung-Box on z… |
ts.hamilton_filter | LIB | test_ts_hamilton_filter.py — independent re-validation 2026-06-19: Built the Hamilton design matrix (regress y_t on const + p lags starting h back) independently and ran statsmodels.OLS o… |
ts.har_rv | LIB | test_har_rv.py — independent re-validation 2026-06-19: Built the identical HAR design independently and fit it with statsmodels.api.OLS (different code path than the engine's … |
ts.igarch | RIGOR | test_igarch.py — independent re-validation 2026-06-19: Simulated an IGARCH(1,1) (omega=0.02, alpha=0.10, beta=0.90, persistence=1) |
ts.intervention | LIB | test_intervention_analysis.py — independent re-validation 2026-06-19: Simulated AR(1)+step, refit the IDENTICAL design (const, step dummy, lag1) with statsmodels OLS and compared omega, t, p… |
ts.kalman_smoother | LIB | test_kalman_smoother.py — independent re-validation 2026-06-19: Simulated a local-level DGP (sigma_eta=0.4, sigma_eps=1.0) |
ts.koyck_dl | LIB | test_ts_koyck_dl.py — independent re-validation 2026-06-19: Re-estimated the Koyck-transform regression with statsmodels OLS on a known DGP (lambda=0.6, beta=2.0); recomputed long-… |
ts.local_projection | LIB | test_local_projection.py — independent re-validation 2026-06-19: Simulated y_t=rhoy_{t-1}+thetax_t+e_t (rho=0.6,theta=1,T=2000) and ran the engine alongside statsmodels OLS with cov_t… |
ts.markov_chain | RIGOR | test_markov_chain.py — independent re-validation 2026-06-19: Simulated 300k transitions from a known 3-state P; checked estimated P vs true P; checked stationary dist via an INDEPEN… |
ts.mgarch_ccc | RIGOR | test_mgarch_ccc.py — independent re-validation 2026-06-19: Simulated CCC-GARCH (n=6000, true rho=0.5) and independently re-fit each series with my own arch.arch_model GARCH(1,1) c… |
ts.ms_var | LIB | test_markov_switching_var.py — independent re-validation 2026-06-19: Simulated a 2-regime MS-VAR(1) on 4 fresh seeds (7,99,314,2718) NOT relying on the test's single seed: regime classifica… |
ts.nonlinear_irf | R | test_generalized_irf.py — independent re-validation 2026-06-19: Estimated the VAR(1) coefficients and Sigma independently in R via vars::VAR, then computed the Pesaran-Shin unit-shock … |
ts.particle_filter | LIB | test_ts_particle_filter.py — independent re-validation 2026-06-19: Wrote a fully independent hand-coded Kalman filter using the SAME documented prior (x0~N(y0, data_var+obs_var), predict … |
ts.periodic_var | RIGOR | test_periodic_var.py — independent re-validation 2026-06-19: Simulated a univariate PAR(1) with season-varying phi=[0.2,0.6,-0.3,0.7], c=[0,1,-1,0.5] |
ts.phillips_ouliaris | R | test_phillips_ouliaris.py — independent re-validation 2026-06-19: (1) Cross-checked Z_alpha against R tseries::po.test (demeaned, lshort=FALSE) and urca::ca.po Pu on the same cointegrate… |
ts.quandt_andrews | R | test_ts_quandt_andrews.py — independent re-validation 2026-06-19: Cross-checked against R strucchange::Fstats / sctest (an independent implementation of the sup-F unknown-breakpoint test… |
ts.quantile_var | R | test_quantile_var.py — independent re-validation 2026-06-19: Simulated a fixed VAR(1) (N=2000), wrote CSV, ran the engine per-equation vs R rq(y~La+Lb, tau) at tau=0.25,0.5,0.9 and … |
ts.realized_garch | RIGOR | test_ts_realized_garch.py — independent re-validation 2026-06-19: Simulated the Hansen-Huang-Shek linear Realized GARCH from a fresh seed with different params (beta=0.60,gamma=0.35,phi=… |
ts.spectral_coherence | LIB | test_ts_spectral_coherence.py — independent re-validation 2026-06-19: Ran scipy.signal.coherence (Hann, 50% overlap, detrend='constant') and scipy.signal.csd for phase on the same shared-sin… |
ts.spectral_granger | LIB | test_spectral_granger.py — independent re-validation 2026-06-19: Cross-checked the time-domain Granger F against statsmodels VAR.test_causality on the same series; numerically verified … |
ts.star | R | test_lstar_model.py — independent re-validation 2026-06-19: Cross-checked against R tsDyn::lstar (m=1, thDelay=0) on an identical simulated LSTAR(1) series; compared the threshold … |
ts.stochastic_volatility | RIGOR | test_stochastic_volatility.py — independent re-validation 2026-06-19: Wrote my own SV simulator (AR(1) log-variance, r_t=exp(h_t/2)eps_t) at two parameter sets and ran the engine |
ts.svecm | LIB | test_svecm.py — independent re-validation 2026-06-19: Independent checks: (1) urca::ca.jo Johansen trace test confirms the DGP rank; (2) recomputed BB' and compared to statsm… |
ts.threshold_cointegration | R | test_threshold_cointegration.py — independent re-validation 2026-06-19: Re-implemented the ENTIRE Enders-Granger TAR pipeline independently in R (cointegrating lm, residual diff, Heaviside spl… |
ts.threshold_var | RIGOR | test_threshold_var.py — independent re-validation 2026-06-19: Re-ran the regime-switching DGP (true gamma0=0.3): gamma_hat=0.297, rejects linearity p=0.005, recovered regime coeffici… |
ts.tsay_test | LIB | test_tsay_test.py — independent re-validation 2026-06-19: On fresh quadratic-AR data (y_t=0.4 y_{t-1}+0.3 y_{t-1}^2+e), recomputed the Tsay F from a from-scratch full augmented O… |
ts.tvp_regression | RIGOR | test_tvp_regression.py — independent re-validation 2026-06-19: Re-implemented the prediction-error-decomposition Kalman filter from scratch (random-walk states, diffuse P0=1e6*I, T=I)… |
ts.tvp_var | RIGOR | test_tvp_var.py — independent re-validation 2026-06-19: Simulated AR(1) with a linearly drifting phi (0.1->0.9) on 3 fresh seeds (501,777,1234) and confirmed the smoothed coeff… |
ts.unobserved_components | LIB | test_ts_unobserved_components.py — independent re-validation 2026-06-19: Simulated a fresh local-linear-trend DGP (sigma_level=0.12, sigma_trend=0.015, sigma_irreg=0.35, n=500), ran the engine;… |
ts.var_residual_diag | R | test_var_residual_diagnostics.py — independent re-validation 2026-06-19: Cross-checked the Hosking portmanteau and Doornik-Hansen/JB normality against R vars::serial.test (PT.asymptotic & PT.ad… |
ts.var_subset | LIB | test_subset_var.py — independent re-validation 2026-06-19: (1) Built the restricted EGLS a completely different way via a full Kronecker SUR (block_diag designs, W=Sigma^-1 kron I… |
ts.varhac | R | test_hac_long_run_variance.py — independent re-validation 2026-06-19: Cross-checked the Bartlett (Newey-West) long-run variance at bandwidth=8 against R sandwich::lrvar(y, type='Newey-West',… |
ts.varma | RIGOR | test_varma.py — independent re-validation 2026-06-19: Simulated a VARMA(1,1) (T=4000) with known A,M and checked max recovery error of the estimated AR and MA matrices and th… |
ts.vma | LIB | test_vector_ma.py — independent re-validation 2026-06-19: Fit statsmodels VARMAX(0,1) (independent state-space MLE) on the identical 4000-obs bivariate series and a univariate AR… |
ts.wavelet_coherence | RIGOR | test_ts_wavelet_coherence.py — independent re-validation 2026-06-19: Fed pure sinusoids of known period (16,32,64) into both series and checked the dominant period; verified the Morlet peri… |
Econometrics families — deep subsection (priority niche)
This is the niche the product is positioned for, so it gets the most exacting honesty. The headline: the core panel / GLM / causal-IV estimators are genuinely tested, with the dynamic-panel and GLM workhorses cross-checked against R or hand-derived rigor tests; but a substantial tail of time-series and specialist macro/DSGE methods rests on library-faithful or seam-only coverage, and several carry documented scope limitations in the capability sheet itself (MIDAS single-dataset, DFM no latent-factor extraction, RDD ad-hoc bandwidth, PMG not implemented, event-study no IW correction).
Regression & GLM (the estimation backbone)
| Method | Tier | Evidence / honest caveat |
|---|---|---|
model.linear_regression (OLS) | NIST | test_nist_strd_linreg.py (Norris/Pontius/Longley ≥10 digits; Filip = documented 0-digit pinv failure) + test_golden_r.py |
model.linear_regression (HC3 robust) | R | test_golden_r.py (vs coeftest(vcov=vcovHC("HC3"))) |
model.logistic_regression | R | test_golden_r.py (coef/SE/z/p/llf, rtol 10⁻⁴) |
model.poisson | R | test_golden_r.py (vs R glm Poisson; + exposure/offset case) |
model.gamma | R | test_golden_r.py::gamma_glm (vs glm(Gamma(log))) |
model.tweedie | LIB | seam-smoke (statsmodels Tweedie GLM; insurance_glm.csv, p=1.5 log link) |
model.negative_binomial | R | test_acadval_wave1_glm.py (coef + SE ~3.5e-3 + α=1/θ vs MASS::glm.nb) |
model.probit | R | test_acadval_wave1_glm.py (coef/SE/z/log-lik vs glm(binomial('probit')); OIM-vs-EIM SE gap ~6e-4) |
model.ordered_logit / model.ordered_probit | R | test_acadval_wave2_ordered.py (coef/thresholds/SE/log-lik vs MASS::polr, to ~2e-4) |
model.multinomial_logit | R | test_acadval_wave2_ordered.py (per-category coef/SE/RRR/log-lik vs nnet::multinom, ~5e-4) |
model.zip | R (partial: SE documented-disagreement) | test_acadval_wave1_glm.py — count/inflation coef + log-lik match pscl::zeroinfl; statsmodels SEs run ~10–25% smaller |
model.zinb | R (partial: inflation-model documented-disagreement) | test_acadval_wave1_glm.py — count coef/α/log-lik match pscl::zeroinfl; weakly-identified inflation logit diverges (cx2 ~0.12) |
model.hurdle | R | test_acadval_wave1_glm.py (logit + truncated-Poisson coef/SE/log-lik vs pscl::hurdle) |
model.tobit / model.truncated | R (partial: SE documented-disagreement) | test_acadval_wave2_ordered.py — coef/σ/log-lik match AER::tobit/truncreg; BFGS inv-Hessian SE vs Fisher-info differs |
model.heckman | R | test_acadval_wave2_ordered.py (selection + outcome coef/SE, λ, ρ, σ vs sampleSelection::selection 2-step) |
model.beta_regression | R | test_acadval_wave1_glm.py (mean coef/SE + φ + log-lik vs betareg::betareg) |
model.fractional | R (partial: SE non-robust documented-limitation) | test_acadval_wave1_glm.py — coef identical to glm(quasibinomial); engine reports binomial-model SE, not the Papke-Wooldridge robust QMLE SE |
model.quantile_regression | LIB | seam-smoke (statsmodels QuantReg) |
model.ridge / model.elastic_net | RIGOR | test_acadval_wave3_penalized.py — sklearn vs glmnet penalty-objective normalization is a documented disagreement (coef differ at matched λ); not R-cross-validated |
model.lasso | R | test_acadval_wave3_penalized.py (coef + active set vs glmnet(alpha=1), shared objective, ~1e-3) |
model.robust_regression | R | test_acadval_wave3_penalized.py (coef/SE/Wald-z vs MASS::rlm(psi.huber), ~1e-3) |
model.stepwise | R (partial: AIC/BIC constant offset) | test_acadval_wave3_penalized.py — selected variable set + final coef match MASS::stepAIC; AIC/BIC differ by the documented sigma²-param constant |
model.wls | R | test_acadval_wave3_penalized.py (coef/SE/t/R² vs lm(weights=), ~1e-3) |
model.fgls | R (partial: intercept documented-disagreement) | test_acadval_wave3_penalized.py — slopes + ρ match nlme::gls(corAR1); intercept differs ~0.048 (Cochrane-Orcutt vs ML) |
model.nls | R | test_acadval_wave3_penalized.py (params/SE/RSS vs stats::nls, ~1e-4) |
model.sur / model.3sls | R (partial: SE documented-disagreement) | test_acadval_wave3_penalized.py — coef match systemfit SUR/3SLS to ~1e-3; linearmodels vs systemfit covariance estimator differs (~0.006/0.009) |
model.gmm | R (partial: J-statistic documented-disagreement) | test_acadval_wave3_penalized.py — coef within ~3e-3 of gmm::gmm; J-stat differs ~0.04 (weighting-matrix normalization) |
model.mixed_linear | R | test_acadval_wave4_mixed.py (fixed-effect coef/SE + variance components + REML log-lik vs lme4::lmer) |
model.icc | R | test_acadval_wave4_mixed.py (ICC(1) + variance components vs psych::ICC/irr::icc) |
model.loess | RIGOR | test_acadval_wave4_mixed.py — statsmodels lowess vs stats::loess (Cleveland Fortran) is a documented kernel disagreement (~0.047); shape + bounded-gap only |
analysis.box_tidwell | RIGOR | test_acadval_wave4_mixed.py — engine Wald/LR vs car::boxTidwell score-t are different statistics (documented method disagreement); same conclusion only |
analysis.brant | R | test_acadval_wave4_mixed.py (omnibus + per-variable χ²/df/p vs brant::brant, ~0.009/5e-4) |
ERM family (erm_linear, erm_probit, erm_ordered, erm_interval, erm_diagnostics) | R / RIGOR / — | erm_linear (vs AER::ivreg), erm_probit (vs glm probit), erm_interval (vs survreg(interval2)) all R in test_acadval_wave2_ordered.py; erm_ordered RIGOR (test_rigor_advanced.py); erm_diagnostics xfail |
Panel data
| Method | Tier | Evidence / honest caveat |
|---|---|---|
model.panel_fe (Fixed Effects) | R | test_acadval_wave5_panel.py (within coef + clustered HC1 SE vs plm(within) + vcovHC, machine precision) |
model.panel_re (Random Effects) | LIB | test_acadval_wave5_panel.py — linearmodels vs plm(random) Swamy-Arora θ is a documented disagreement (slope gap ~0.017); engine values pinned, not R-cross-validated |
test.hausman | RIGOR | test_rigor_panel.py::hausman_test |
model.iv_2sls | R | test_golden_r.py::panel_iv (vs R, robust first-stage F) |
panel.arellano_bond (Difference GMM) | R | test_rigor_rf_wave7_dynamic_gmm.py + test_rigor_wave1.py + test_rigor_wave4_medium_d.py |
panel.system_gmm (Blundell-Bond) | R (partial: Windmeijer SE documented-disagreement) | test_acadval_wave5_panel.py — two-step coef match pgmm to ~3e-3; engine k=2 vs pgmm twoways k=9 Windmeijer correction gives ~2× SE |
panel.hausman_taylor | R (partial: time-invariant FGLS documented-disagreement) | test_acadval_wave5_panel.py — time-varying coef = FE estimates (match); approximate HT (no θ quasi-demeaning) differs from plm HT FGLS on time-invariant/endogenous |
panel.between | LIB | test_acadval_wave5_panel.py — linearmodels BetweenOLS vs plm(between) parameterisation is a documented disagreement (coef gap ~0.27); engine pinned |
panel.first_difference | R | test_acadval_wave5_panel.py (slopes + SE vs plm(fd), ~2e-4) |
panel.pcse (Beck-Katz) | RIGOR | test_acadval_wave5_panel.py — engine uses FE base, R vcovBK uses pooled OLS (documented disagreement); validates FE point estimates + finite SE only |
panel.driscoll_kraay | R (partial: SE documented-disagreement) | test_acadval_wave5_panel.py — coef = FE (match); linearmodels Bartlett kernel vs plm::vcovSCC SE differs ~10–30%, engine pinned |
panel.cointegration | R | test_acadval_wave5_panel.py (Kao DF_t* statistic vs R closed-form, machine precision) + test_rigor_wave3_panel.py |
panel.granger | RIGOR | test_acadval_wave5_panel.py — Fisher-combined statistic within ~5% of per-unit grangertest (lag-construction differs); range-checked, not R-cross-validated |
panel.cross_dep | RIGOR | test_acadval_wave5_panel.py — engine Pesaran CD on raw demeaned variable vs pcdtest on FE residuals is a documented methodological disagreement; engine pinned |
panel.ardl (Mean Group / DFE) | RIGOR | test_acadval_wave5_panel.py — per-entity AIC lag selection differs from R (T=10 AIC-formula gap) giving different long-run averages (documented disagreement); PMG still not implemented |
panel.mundlak | RIGOR | test_acadval_wave5_panel.py — within coef = FE (match); Wald χ² differs from plm RE (9.63 vs 40.05, different RE vcov), same FE-vs-RE decision only |
Causal inference
| Method | Tier | Evidence / honest caveat |
|---|---|---|
model.did (Diff-in-Diff) | R | test_acadval_wave20_causal.py (DiD coef + cluster-robust SE + R² vs lm + vcovCL, machine precision) |
model.rdd (Sharp RDD) | R | test_acadval_wave20_causal.py (local-linear triangular-kernel estimate/SE/R² vs lm+vcovHC(HC1) at fixed bandwidth, machine precision) |
model.psm (Propensity matching) | RIGOR | test_acadval_wave20_causal.py — sklearn NN vs Matching::Match greedy algorithm is a documented disagreement (ATT gap ~0.039, different matched sets); engine pinned |
test.chow (structural break) | R | test_acadval_wave20_causal.py (Chow F + df vs manual lm split, machine precision) |
causal.synthetic_control | R | test_acadval_wave20_causal.py (donor weights + pre/post RMSPE vs R BFGS-softmax SCM, ~1e-4) |
causal.event_study | RIGOR | test_acadval_wave20_causal.py — engine drops never-treated units (R keeps them): a documented design limitation for staggered designs; engine-only reference-period + pre-trend checks |
causal.staggered_did (Callaway-Sant'Anna) | RIGOR | test_acadval_wave20_causal.py — engine "not-yet-treated" vs R "never-treated" comparison group is a documented estimator-design disagreement (ATT gap ~0.024, bounded) |
causal.iv_diagnostics | R | test_acadval_wave20_causal.py (first-stage F + Wu-Hausman F vs AER::ivreg/car::linearHypothesis, machine precision) |
causal.ipw (IPW) | R (partial: ATT sklearn-vs-glm documented-disagreement) | test_acadval_wave20_causal.py — PS mean matches glm to 1e-3; ATT within documented 0.020 (gap ~0.007, L-BFGS vs IWLS) |
causal.doubly_robust (AIPW) | R | test_acadval_wave20_causal.py (AIPW ATE vs R closed-form, ~3e-4) |
causal.late | R | test_acadval_wave20_causal.py (LATE point estimate + 2SLS SE vs AER::ivreg, machine precision; delta-method SE differs ~1.2e-4) |
causal.mediation | R | test_acadval_wave20_causal.py (Baron-Kenny a/b/indirect/direct/total + Sobel z vs lm, machine precision) |
causal.sensitivity | R | test_acadval_wave21_causal_rf.py (E-value point/Cohen-d/RR/HC1-SE vs R sensemakr-style, machine precision; E-value-CI z-vs-t differs ~0.0027) |
causal.lee_bounds | R | test_acadval_wave21_causal_rf.py (selection rates/trim/lower+upper bounds vs manual R, machine precision; bootstrap CI MC-noisy) |
causal.rkd | R | test_acadval_wave21_causal_rf.py (piecewise-OLS+HC1 slope-change/SE/t/slopes vs R OLS, machine precision; rdrobust Gaussian-kernel is a documented cross-check only) |
causal.bunching | R | test_acadval_wave21_causal_rf.py (bin width/excess mass/normalised excess vs degree-7 polynomial R, ~1e-4; FP-binning difference only) |
Time series & macro
| Method | Tier | Evidence / honest caveat |
|---|---|---|
ts.sarima | R (partial: log-lik/AIC documented-disagreement) | test_acadval_wave8_ts_core.py — forecasts within ~0.07 of forecast::Arima; statsmodels diffuse-Kalman vs Gardner backcasting gives ~25-unit LL gap |
model.auto_arima | R | test_acadval_wave8_ts_core.py (selected d=1 + AIC vs forecast::auto.arima) + test_auto_arima_parity.py (pmdarima) |
model.arima | R | test_acadval_wave8_ts_core.py (AR1/sigma²/log-lik/forecasts vs forecast::Arima, CSS-ML vs Kalman ~2.5e-6) |
model.var | R | test_acadval_wave8_ts_core.py (per-equation coef + residual covariance vs vars::VAR, to 1e-8) |
model.vecm | R | test_acadval_wave8_ts_core.py (α loadings + β cointegrating vector vs urca::cajorls, ~3e-3/1e-3) |
test.coint_johansen | R | test_acadval_wave8_ts_core.py (trace + max-eigen statistics vs urca::ca.jo, ~0.2/0.02; CV-table differs) |
test.coint_eg | RIGOR | test_acadval_wave8_ts_core.py — statsmodels ADF-on-residuals vs tseries::po.test are different test families (documented disagreement); both reject, conclusion only |
ts.holt_winters | R (partial: smoothing-params documented-disagreement) | test_acadval_wave9_ts_adv.py — forecasts within 0.5 of forecast::hw; statsmodels α collapses toward 0 vs R 0.129 |
ts.ets | R (partial: log-lik documented-disagreement) | test_acadval_wave9_ts_adv.py — model-code ETS(A,A,A) + forecasts within 0.5 of forecast::ets; log-lik differs (initialization) |
ts.state_space | LIB | test_acadval_wave9_ts_adv.py — UnobservedComponents vs stats::StructTS is a documented structural difference (irregular-variance parameterisation); finite-LL + non-zero-variance check only |
ts.svar | R (partial: A-matrix Cholesky-vs-scoring documented-disagreement) | test_acadval_wave8_ts_core.py — Cholesky B-matrix structurally matches; A[2,1] differs ~0.022 from vars::SVAR scoring MLE |
ts.irf / ts.fevd | R | test_acadval_wave8_ts_core.py (orthogonalized IRF + FEVD shares vs vars::irf/vars::fevd, exact to 1e-8) |
ts.markov_switching | LIB | test_acadval_wave9_ts_adv.py — statsmodels vs MSwM EM converge to different local optima (documented disagreement, LL gap ~5); regime/transition-validity only |
ts.tar | R | test_acadval_wave8_ts_core.py (regime AR1 coefficients vs tsDyn::setar, exact to 1e-6 at fixed threshold) |
ts.dcc_garch | LIB | test_acadval_wave9_ts_adv.py + test_arch_parity.py — engine 30×30 grid-search DCC vs rmgarch::dccfit BFGS is a documented design limitation (a≈0.145 vs 0.397); not R-cross-validated |
GARCH family (model.garch/tgarch) | R | test_acadval_wave9_ts_adv.py (ω/α/β/γ + log-lik vs rugarch::ugarchfit sGARCH/gjrGARCH, ~0.005/1.0) + test_arch_parity.py |
model.egarch | R (partial: parameterisation documented-disagreement) | test_acadval_wave9_ts_adv.py — log-lik within 2 of rugarch eGARCH; arch magnitude/sign parameterisation differs |
ts.zivot_andrews | R (partial: statistic lag-selection documented-disagreement) | test_acadval_wave9_ts_adv.py — break index matches urca::ur.za exactly; test statistic differs ~0.13 t-units (lag convention) |
ts.dfgls | LIB | test_acadval_wave9_ts_adv.py + test_arch_parity.py — arch 0-lag vs urca::ur.ers 4-lag is a documented lag-selection disagreement (~2.2 t-units); both reject, not R-cross-validated |
ts.ers | R | test_acadval_wave9_ts_adv.py (P_T statistic + lag + critical values vs urca::ur.ers, faithful port to ~1e-5) |
ts.bai_perron | RIGOR | test_acadval_wave8_ts_core.py — engine mean-shift segmentation vs strucchange::breakpoints sup-F is a documented method difference; detects 1 break at t≈100 only |
ts.cusum | LIB | test_acadval_wave8_ts_core.py — statsmodels vs strucchange::efp standardization differs (documented disagreement); break-detection + boundary-exceedance only |
ts.toda_yamamoto | R (partial: lag-selection documented-disagreement) | test_acadval_wave8_ts_core.py — Wald χ² within ~0.07 of vars+aod::wald.test; engine AIC lag vs R fixed k+dmax |
ts.nardl | R (partial: order-selection ~0.05) | test_acadval_wave9_ts_adv.py — long-run LR± within ~0.05 of manual partial-sum ARDL; symmetry-test conclusion agrees |
ts.bounds_test (ARDL bounds) | R | test_acadval_wave9_ts_adv.py (PSS bounds F-statistic vs ARDL::bounds_f_test, ~0.1) + test_ardl_bounds_seam.py |
panel_unit_root (test.panel_unit_root) | RIGOR | test_acadval_wave10_ts_misc.py — Fisher χ² within ~1.3 of plm::purtest, but per-entity AIC-lag vs fixed lag=1 (documented disagreement); engine pinned, conclusion agrees |
granger_causality (test.granger) | R | test_acadval_wave10_ts_misc.py (per-lag F + p vs lmtest::grangertest, to 1e-4) |
test.phillips_perron | RIGOR | test_acadval_wave10_ts_misc.py — arch 15-lag Newey-West vs tseries::pp.test 4-lag is a documented bandwidth disagreement (~0.31 t-units); engine pinned, conclusion agrees |
ts.diebold_mariano | R | test_acadval_wave9_ts_adv.py (DM statistic vs forecast::dm.test, within 0.1) + test_forecast_backtest.py |
ts.forecast_combine | R | test_acadval_wave10_ts_misc.py (equal/inverse-MSE/OLS/Bates-Granger weights + combined MSE vs R closed-form, ~1e-5) |
model.dfm (diffusion-index forecast) | R | test_golden_r.py (forecast_backtest, AR-benchmark backtest) — not a latent-factor DFM (see caveat) |
ts.spectral | R (partial: PSD factor-of-2 scaling) | test_acadval_wave10_ts_misc.py — dominant frequency/period match stats::spec.pgram exactly; scipy one-sided PSD is 2× the raw R periodogram |
MIDAS family (u_midas, almon_midas, beta_midas, exponential_midas, midas_forecast, midas_compare, nowcasting) | RIGOR / — | u_midas + midas_forecast in test_rigor_wave4_medium_e.py and test_midas_handlers.py; Almon/Beta/Exponential/compare/nowcasting seam-smoke; all carry the single-dataset limitation |
DSGE family (dsge.solve_re, dsge.simulate, dsge.irf, dsge.variance_decomp, dsge.rbc, dsge.nk, dsge.calibrate) | RIGOR / — | solve_linear_rational_expectations (Klein-Sims QZ) in test_rigor_advanced.py; rbc/nk/calibrate seam-smoke; solve_re/simulate/irf/variance_decomp xfail in seam (need structural matrices). Log-linearized first-order only |
Econometrics honest summary
The academic-validation campaign reshaped the econometrics tiers substantially —
the great majority of the estimation backbone is now genuinely R-cross-validated.
NIST-certified: OLS (with the documented Filip pinv failure). R-cross-validated
(clean): OLS-HC3, logistic, Poisson (+ offset), Gamma GLM, probit, ordered
logit/probit, multinomial logit, NB2, hurdle, beta, Heckman, lasso, robust
regression, WLS, NLS, mixed model, ICC, the ERM linear/probit/interval triple,
2SLS-IV, Arellano-Bond difference GMM, panel FE (+ clustered SE), first-difference,
Kao panel cointegration, ARIMA/auto-ARIMA/VAR/VECM, Johansen cointegration, TAR,
IRF/FEVD, sGARCH/GJR-GARCH, ERS, ARDL-bounds, Granger, Diebold-Mariano,
forecast-combination, DiD, RDD, Chow, synthetic control, IV diagnostics, AIPW
doubly-robust, LATE, mediation, sensitivity (E-value), Lee bounds, sharp RKD,
bunching, and the DFM direct-forecast backtest. R-cross-validated (partial —
primary quantity matches R, named secondary quantity is a documented disagreement):
ZIP/ZINB (SEs / inflation model), tobit/truncated (SE), fractional (non-robust SE),
FGLS (intercept), SUR/3SLS (SE), GMM (J-stat), stepwise (AIC offset), System GMM
(Windmeijer), Hausman-Taylor (time-invariant FGLS), Driscoll-Kraay (SE), IPW (PS
backend), SARIMA/ETS/Holt-Winters (log-lik/smoothing), SVAR (A-matrix),
Zivot-Andrews/Toda-Yamamoto (lag selection), NARDL (order), spectral (PSD scaling),
EGARCH (parameterisation). Internal rigor-audited (formula-checked, not
cleanly cross-system): Callaway-Sant'Anna staggered DiD, PSM, event study, LOESS,
Box-Tidwell, panel RE/between/PCSE/cross-dep/Granger/ARDL/Mundlak, panel unit root,
Phillips-Perron, Bai-Perron, U-MIDAS, the DSGE QZ solver. Library-faithful
(parity- or seam-tested): Tweedie GLM, quantile, ridge/elastic-net (glmnet
objective-normalization disagreement), the DCC-GARCH grid-search, DF-GLS, CUSUM,
Markov-switching, state-space, MGWR, K-fold CV. Unverified
(seam-smoke only): the non-u_midas MIDAS variants and the RBC/NK DSGE models.
Three things to keep front-of-mind when relying on the econometrics niche:
- The MIDAS suite and the DFM are functionally constrained (single-dataset MIDAS; DFM needs user-supplied factor columns, no latent extraction). The tests verify what the code does, which is narrower than the textbook method.
- Several causal/time-series methods report a reduced-form or simplified estimand by design (sharp-RKD slope change, bunching excess-mass ratio, TWFE event study without the Sun-Abraham IW correction, Bai-Perron as mean-shift segmentation rather than the sup-F test). These limitations are stated in the capability sheet and not contradicted here.
- The standard-error story for the dynamic-panel and selection estimators is still the weakest link — and it is now measured rather than assumed. PMG is not implemented (errors out); the System-GMM Windmeijer SE, Hausman-Taylor time-invariant FGLS, ZIP/ZINB, tobit/truncated and SUR/3SLS standard errors are each pinned as documented R disagreements above. Point estimates are tested against R throughout; for these methods some inference legitimately differs.
UI / IO / workflow plumbing (statistical-correctness: Unverified)
These are not statistical estimators; they carry their own functional tests but no statistical-number validation, so they are counted Unverified above. Listed here for completeness with their functional test where one exists.
| Family | Methods | Functional test |
|---|---|---|
| Dataset I/O | dataset.* (11), data.* (16) | test_engine.py, test_importers_*.py, test_columnar_*.py |
| Cell editing | edit.* (9) | test_edits.py |
| Formulas | formulas.* (4) | test_formulas.py |
| Variable metadata | variables.* (5) | test_varmeta.py, test_type_conversion.py |
| Transforms | transform.* (8) | test_transform_persist.py, test_transform_safe_eval.py |
| Session / macro | session.* (12) | test_session_json.py, test_repro_codegen.py |
| Tables | table.* (7) | test_report_builder.py |
| Auto/EDA | auto.* (4) | test_perf_parity.py (data_quality), seam-smoke |
| Graphics | graph.* (4) | seam-smoke |
| Reporting | export.* (2), report.* (2) | test_report_builder.py |
| Guided / AI | guided.* (6), ai.* (2) | test_guided.py, test_ai_bridge.py (AI xfail in seam — live relay) |
Reproducing this classification
Every tier above is derived from files in python/tests/. To re-derive:
- NIST:
python -m pytest tests/test_nist_strd_univariate.py tests/test_nist_strd_anova.py tests/test_nist_strd_linreg.py - R-cross-validated:
python -m pytest tests/test_golden_r.py tests/test_golden_r_sem_lavaan.py tests/test_rigor_rf_wave7_dynamic_gmm.py tests/test_rigor_rf_wave8_fine_gray.py tests/test_rigor_rf_wave9b_cml_rasch.py tests/test_rigor_rf_wave10_gamma_frailty.py tests/test_acadval_wave1_glm.py tests/test_acadval_wave2_ordered.py tests/test_acadval_wave3_penalized.py tests/test_acadval_wave4_mixed.py tests/test_acadval_wave5_panel.py tests/test_acadval_wave6_diagnostics.py tests/test_acadval_wave7_margins.py tests/test_acadval_wave8_ts_core.py tests/test_acadval_wave9_ts_adv.py tests/test_acadval_wave10_ts_misc.py tests/test_acadval_wave11_nonparam.py tests/test_acadval_wave12_anova_manova.py tests/test_acadval_wave13_factor.py tests/test_acadval_wave14_irt_sem.py tests/test_acadval_wave15_survival.py tests/test_acadval_wave16_meta.py tests/test_acadval_wave18_survey.py tests/test_acadval_wave19_missing.py tests/test_acadval_wave20_causal.py tests/test_acadval_wave21_causal_rf.py(the academic-validation campaign — committed deterministic R fixtures; the per-method rows note where a method is R (partial)) - Library-faithful: the
test_*_parity.pyfiles, plustests/seam/contracts.pyfor which methods route to a named library backend. - Internal-rigor: the
test_rigor_*.pyfiles. - Seam-smoke (the floor for every UI-reachable method):
python -m pytest tests/test_handler_seam_smoke.py
The exact tolerances and provenance for the R cross-checks live in the companion page Validated Analyses.