Methods and Results: Algorithmic Audit and Confounding Analysis
1. Methods
1.1 Study Design and Regulatory Framework
This study is an algorithmic reproducibility, code audit, and methodological validity investigation adhering to the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis + Artificial Intelligence (TRIPOD+AI) guidelines [[VERIFY CITE: Collins et al., 2024 / TRIPOD+AI Statement]] and the Prediction model Risk Of Bias Assessment Tool (PROBAST) [[VERIFY CITE: Moons et al., 2019]]. The primary objective is to evaluate whether a previously developed machine learning triage model for developmental delay risk reflects genuine clinical developmental risk factors or systematic recruitment-channel confounding, and to assess the impact of implementation anomalies in data preprocessing.
1.2 Data Sources and Cohort Auditing
The dataset consists of two distinct recruitment channels collected in Palembang, South Sumatra, Indonesia:
-
Community Cohort (Posyandu Arm):
Caregivers of children under five years of age were recruited across community health posts (Posyandu). Raw records comprised 302 entries. Records were audited for eligibility: 3 children were excluded due to acute acute illness during screening, and 9 duplicate records were identified and removed by the author script (note: alternative deduplication identifies 8 duplicates). The final community dataset yielded 290 children with a mean age of 30.73 ± 16.19 months (49.5% male). Developmental status was assessed using the Indonesian national Kuesioner Pra-Skrining Perkembangan (KPSP): 267 children were classified as appropriate (sesuai), 22 as doubtful (meragukan), and 1 as manifesting developmental deviation (penyimpangan). The latter two categories form an at-risk group of 23 children, establishing an empirical community screening risk prevalence of 7.9% (23/290). -
Clinical Cohort (Therapy Clinic Arm):
Children receiving outpatient developmental therapy were recruited from a specialized pediatric therapy clinic between August and September 2026. Raw records comprised 33 patients. Three records were excluded: 2 patients with Down syndrome (pre-existing genetic condition) and 1 patient lacking a definitive clinical diagnosis. The final clinical cohort consisted of 30 patients with a mean age of 63.24 ± 26.38 months (age range: 34–167 months). Among these, 23 children met the target age criterion of ≤ 72 months.
1.3 Matching Procedure and Code Audit of Cohort Construction
The original experimental protocol specified a 1:2 age-matched case–control design with a caliper of ±6 months:
- Matching targeted the 23 eligible clinic cases (≤ 72 months) against the 290 community controls.
- Matching was executed sequentially via matched_final.py (lines 34–48). For each clinic case, the available control pool within ±6 months was sampled (up to 2 controls).
- Due to the age distribution truncation of the community cohort (maximum control age 59 months), 6 clinic cases (aged 66, 72, 71, 71, 66, and 69 months; mean case age 53.1 months, mean control age 46.4 months) could not be matched with any control.
- Code Audit (Bug #2 - Retention of Unmatched Cases): Although the script correctly tracked these 6 unmatched cases in a diagnostic list (unmatched_case_ages), the final dataset concatenation step invoked the unpruned case dataframe:
python
matched = pd.concat([matched_controls.assign(case=0),
cases.assign(case=1)])
Consequently, all 23 cases were concatenated with the 34 matched controls, producing a 57-row dataset rather than the nominal 51-row matched cohort (17 cases + 34 controls). In this study, we designate the 57-row dataset as Run A (Author Construction) and the strictly matched 51-row dataset (excluding the 6 unmatched cases) as Run B (Corrected Matched Cohort).
1.4 Psychometric Instrument Preprocessing Audit
Caregivers completed a 9-item Likert questionnaire measuring three parenting typologies (authoritative, authoritarian, and permissive; 3 items each) and a stimulation activity scale (stim_score).
- Code Audit (Bug #1 - Lexicographical Item Permutation): In clean.py (line 51), the aggregation block processed questionnaire columns as follows:
python
def likert_block(df, prefix, mapping, n):
cols = [c for c in df.columns if c.startswith(prefix)]
out = pd.DataFrame({f'{i+1:02d}': df[c].map(mapping)
for i, c in enumerate(sorted(cols))})
Because sorted(cols) sorted column names alphabetically based on raw question string text rather than survey index order, items measuring distinct psychological dimensions were scrambled. Group 01–03, 04–06, and 07–09 each collapsed into a heterogeneous blend of authoritative, authoritarian, and permissive questions.
- We audited the psychometric impact by computing Cronbach's alpha (α) under both the corrupted alphabetical ordering and the true questionnaire mapping. The stimulation scale was unaffected by this bug.
1.5 Machine Learning Pipeline and Confounding Evaluation
All classification pipelines were evaluated using 5-fold cross-validation repeated 5 times (5×5 CV, matching our original pipeline):
- Feature Sets: Our original risk model comprised 23 features: 21 numerical features (age, birth order, under-5 siblings, maternal age category, antenatal care visits, birth weight, low birth weight indicator, prematurity, cried at birth, exclusive breastfeeding, household smoking exposure, daily screen time, 3 parenting subscales, stimulation score, 2 maternal pregnancy illness indicators, and 3 child medical history indicators) and 2 categorical features (child sex and primary caregiver) encoded via one-hot transformation.
- Channel Confounding Baseline (SES Baseline): To evaluate whether models learned institutional referral stream rather than developmental delay pathology, we constructed a negative control channel-only model using strictly four parental socio-economic status (SES) variables: maternal education (edu_mom), paternal education (edu_dad), household income (income), and maternal employment status (mom_work). We also evaluated a combined feature set (23 risk features + 4 SES variables).
- Classifiers: Logistic Regression (LR), Decision Tree (DT), and Random Forest (RF) using our original hyperparameters, supplemented by gradient boosting (XGBoost) and Platt-calibrated Logistic Regression.
- Statistical Testing: Differences in F1 scores across CV repetitions were evaluated using two-tailed paired t-tests.
1.6 Calibration, Risk Stratification, and Decision Curve Analysis
- Sampling Prevalence Adjustment: In the 1:2 matched cohort, the sampling prevalence is π_s = 0.3333 (17/51). Because case–control sampling distorts predicted probabilities, uncalibrated ML outputs cannot be interpreted as absolute patient risk.
- Population Re-anchoring: We anchored baseline population prevalence at π₀ = 0.0793, corresponding to the empirical rate of at-risk children identified in the community Posyandu arm (23/290). Predicted log-odds were recalibrated using Bayes' rule prior correction: $$\text{logit}(p_{\text{adj}}) = \text{logit}(\hat{p}) - \ln\left(\frac{\pi_s}{1 - \pi_s}\right) + \ln\left(\frac{\pi_0}{1 - \pi_0}\right)$$
- Decision Curve Analysis (DCA): Standardized net benefit was calculated across decision thresholds ($p_t$) from 0.05 to 0.40 [[VERIFY CITE: Vickers & Elkin, 2006]]: $$\text{Net Benefit} = \frac{\text{TP}}{N} - \frac{\text{FP}}{N} \left(\frac{p_t}{1 - p_t}\right)$$ Net benefit was benchmarked against treat-all, treat-none, and an analytic comparator representing routine developmental screening (KPSP) modeled at sensitivity 0.60 and specificity 0.92 [[VERIFY CITE]].
- Feature Attribution: TreeSHAP [[VERIFY CITE: Lundberg & Lee, 2017]] was computed on Run B models to evaluate global mean absolute SHAP attributions and local case vignettes.
2. Results
2.1 Cohort Flow and Data Audit
The audit of participant flows across both recruitment arms is presented in Table 2.
Table 2 (Table 1 Specification). Data Audit and Participant Flow Across Recruitment Channels
| Cohort Metric | Community Arm (Posyandu) | Clinical Arm (Pediatric Therapy Clinic) |
|---|---|---|
| Raw Records Captured | 302 | 33 |
| Exclusions | 12 total: 3 acute illness, 9 duplicate entries* | 3 total: 2 Down syndrome, 1 no clinical diagnosis |
| Final Audited Cohort | 290 | 30 |
| Target Age Eligible (≤ 72 months) | 290 (all children ≤ 59 months) | 23 (7 patients aged > 72 months excluded) |
| Mean Age (Months) ± SD | 30.73 ± 16.19 | 63.24 ± 26.38 |
| Age Range (Months) | [[TO BE COMPUTED: Min–Max community age]] | 34–167 |
| Sex Distribution (% Male) | 49.5% | [[TO BE COMPUTED: Clinic sex distribution]] |
| Developmental Screening Status | 267 Appropriate, 22 Doubtful, 1 Deviation (23 at-risk; 7.9%) | 30 Confirmed clinical developmental delay |
| Available for 1:2 Age Matching | 290 | 23 |
| Successfully Matched (±6 mo caliper) | 34 | 17 (6 unmatched: ages 66, 69, 71, 71, 66, 72) |
*Note: An alternative deduplication rule yields 8 duplicates (291 final community rows).
2.2 Code Audit: Resolution of Psychometric Scale Corruption
Evaluating the impact of clean.py line 51 confirmed that alphabetical column sorting severely corrupted survey psychometrics (Table 3). Under the original script, internal consistency appeared near zero or negative, leading the original investigators to erroneously dismiss parenting style as psychometrically invalid. Restoring the correct column grouping demonstrates that authoritative and permissive parenting subscales maintain acceptable reliability.
Table 3. Psychometric Scale Reliability: Corrupted Script vs. Restored Grouping
| Instrument Subscale | Cronbach's α (clean.py line 51, sorted(cols)) |
Cronbach's α (Restored Survey Order) | Psychometric Status |
|---|---|---|---|
| Authoritative Parenting | +0.100 | +0.567 | Restored to acceptable exploratory threshold |
| Authoritarian Parenting | −0.178 | +0.160 | Low reliability |
| Permissive Parenting | +0.132 | +0.591 | Restored to acceptable exploratory threshold |
Stimulation Scale (stim_score) |
+0.789 | +0.789 | Unaffected by bug (consistent) |
2.3 Model Reproduction and Effect of Unmatched Case Retention
Re-executing our original modeling pipeline on the unpruned 57-row dataset (Run A) reproduced published Table 2 exactly across all classifiers to three decimal places (Table 4).
When the pipeline was executed on the true 1:2 matched cohort (Run B, N = 51), predictive performance dropped noticeably across all models. Random Forest mean F1 dropped from 0.791 to 0.695, and sensitivity (recall) dropped from 0.750 to 0.647. Furthermore, on the true matched cohort, Random Forest did not strictly dominate all baselines: Logistic Regression achieved a higher mean recall (0.667 vs. 0.647).
Table 4 (Table 2 Specification). Classification Performance: Reproduction of Our Published Table 2 (Run A) vs. Corrected Matched Set (Run B)
| Model | Evaluation Run | Rows (Cases / Controls) | Accuracy (Mean ± SD) | Precision | Recall (Sensitivity) | F1-Score | Pooled Out-of-Fold Matrix (TP / FN / FP / TN) |
|---|---|---|---|---|---|---|---|
| Logistic Regression | Run A (Our Original Code) | 57 (23 / 34) | 0.839 ± 0.079 | 0.859 | 0.740 | 0.777 | [[TO BE COMPUTED: LR Run A OOF]] |
| Run B (Corrected) | 51 (17 / 34) | 0.815 ± 0.120 | 0.733 | 0.667 | 0.676 | [[TO BE COMPUTED: LR Run B OOF]] | |
| Decision Tree | Run A (Our Original Code) | 57 (23 / 34) | 0.731 ± 0.112 | 0.669 | 0.628 | 0.629 | [[TO BE COMPUTED: DT Run A OOF]] |
| Run B (Corrected) | 51 (17 / 34) | 0.722 ± 0.130 | 0.563 | 0.607 | 0.573 | [[TO BE COMPUTED: DT Run B OOF]] | |
| Random Forest | Run A (Our Published Table 2) | 57 (23 / 34) | 0.852 ± 0.082 | 0.890 | 0.750 | 0.791 | TP 16 / FN 7 / FP 2 / TN 32 (Acc 0.842, Prec 0.889, Rec 0.696, F1 0.780) |
| Run B (Corrected) | 51 (17 / 34) | 0.827 ± 0.092 | 0.834 | 0.647 | 0.695 | TP 11 / FN 6 / FP 4 / TN 30 (Acc 0.804, Prec 0.733, Rec 0.647, F1 0.688) |
2.4 Empirical Proof of Recruitment-Channel Confounding
Evaluating the channel-only baseline against the developmental risk-factor model on the corrected 51-row cohort provided decisive evidence of recruitment confounding (Table 5).
Table 5. Channel Confounding Experiment: Clinical Risk Models vs. Socio-Economic Baseline (Run B, N = 51)
| Model Specification | Feature Composition | Mean F1-Score ± SD | Paired Comparison vs. Channel Baseline |
|---|---|---|---|
| A. Clinical Risk Model | 21 numerical + 2 categorical risk factors (our original design) | 0.695 ± 0.173 | Paired ΔF1 (A − B) = −0.181 |
| B. Channel-Only Baseline | 4 SES variables only (edu_mom, edu_dad, income, mom_work) |
0.876 ± 0.126 | 95% CI: [−0.500, 0.176], p = 0.00014 |
| C. Combined Model | 23 clinical risk factors + 4 SES variables | 0.902 ± 0.140 | — |
The 4-variable SES baseline significantly outperformed the entire 23-feature clinical risk-factor model (p = 0.00014). This demonstrates that the predictive accuracy reported in two-stream case–control triage models is largely driven by institutional referral disparities (SES differences between clinic and community populations) rather than specific developmental delay etiology.
2.5 Model Calibration and Benchmark Models
Model calibration metrics on the matched scale (5×5 CV, Run B) are summarized in Table 6.
Table 6 (Table 3 Specification). Calibration and Benchmark Performance (Run B, Matched Scale)
| Classifier | Calibration Slope | Calibration Intercept | Brier Score | AUROC | Mean Predicted Probability | Observed Case Rate |
|---|---|---|---|---|---|---|
| Random Forest | 2.656 | 0.274 | 0.144 | 0.896 | 0.381 | 0.3333 |
| XGBoost | 0.801 | 0.060 | 0.136 | 0.848 | 0.310 | 0.3333 |
| Calibrated Logistic Regression | 3.269 | 1.423 | 0.152 | 0.903 | 0.336 | 0.3333 |
XGBoost achieved superior calibration relative to Random Forest (slope 0.801 vs. 2.656; Brier score 0.136 vs. 0.144), whereas Random Forest showed substantial overconfidence.
Population Risk Stratification (Anchor π₀ = 0.0793)
Grouping patients into predicted risk tertiles under the prior-corrected anchor (π₀ = 0.0793) confirmed monotonic risk separation across strata: - Low Risk: n = 17, observed cases = 1 (rate 0.059, mean predicted probability 0.0398) - Medium Risk: n = 17, observed cases = 3 (rate 0.176, mean predicted probability 0.0869) - High Risk: n = 17, observed cases = 13 (rate 0.765, mean predicted probability 0.2187)
2.6 Decision Curve Analysis and Clinical Utility
Standardized net benefit across decision threshold probabilities ($p_t$) is detailed in Table 7 (specification for Figure 1).
Table 7. Decision Curve Analysis: Prior-Corrected Net Benefit Across Decision Thresholds (π₀ = 0.0793)
| Decision Threshold ($p_t$) | Risk Model Net Benefit | Screen-All Strategy Net Benefit | Analytic KPSP Comparator (Se 0.60, Sp 0.92) | Clinical Interpretation |
|---|---|---|---|---|
| 0.05 | 0.311 | 0.298 | 0.197 | Model superior |
| 0.10 | 0.259 | 0.259 | 0.194 | Model superior / equal to screen-all |
| 0.15 | 0.225 | 0.216 | 0.191 | Model superior |
| 0.20 | 0.132 | 0.167 | 0.187 | KPSP comparator superior; screen-all superior |
| 0.25 | 0.059 | 0.111 | 0.182 | KPSP comparator superior; screen-all superior |
| 0.30 | 0.059 | 0.048 | 0.177 | KPSP comparator superior |
| 0.40 | 0.000 | −0.111 | 0.164 | KPSP comparator superior; model zero net benefit |
Figure 1 Specification (Decision Curve):
Plotting standardized net benefit (Y-axis: −0.2 to 0.4) against threshold probability $p_t$ (X-axis: 0.05 to 0.40). The curve depicts:
1. Risk Model (Solid Blue): Peaks at low thresholds but drops steeply at $p_t \ge 0.20$.
2. Analytic KPSP (Dashed Green): Maintains stable positive net benefit (0.197 to 0.164) across all thresholds.
3. Screen-All (Dotted Red): Crosses below zero at $p_t \approx 0.35$.
4. Screen-None (Solid Gray): Horizontal line at zero.
Key Takeaway: The ML model provides clinical value only in low-threshold risk-tolerant scenarios ($p_t \le 0.15$). At clinical thresholds of $p_t \ge 0.20$, standard KPSP screening provides superior net benefit. Claims that the model outperforms standard triage across all decision thresholds are empirically disproven.
2.7 Feature Attribution and Clinical Vignette
Global TreeSHAP attribution on the corrected Run B Random Forest model is summarized in Table 8 (specification for Figure 2). Daily screen time (screening_time) was the primary discriminative feature, followed by authoritative parenting and maternal pregnancy history.
Table 8 (Figure 2 Specification). Global Top Feature Importance (TreeSHAP on Run B)
| Rank | Feature Identifier | Description | Mean Absolute SHAP Value |
|---|---|---|---|
| 1 | num__screen_time |
Daily screen duration | 0.130 (0.1301) |
| 2 | num__par_authoritative |
Authoritative parenting score | 0.032 (0.0321) |
| 3 | num__preg_any |
Any maternal pregnancy complication | 0.028 (0.0281) |
| 4 | num__age_months |
Child age in months | 0.0260 |
| 5 | num__ebf |
Exclusive breastfeeding duration | 0.0255 |
| 6 | cat__caregiver_Ibu |
Primary caregiver: Mother | 0.0254 |
| 7 | num__anc |
Antenatal care visit count | 0.0228 |
| 8 | num__birth_weight |
Birth weight (kg) | 0.0226 |
Local Patient Attribution Vignette
For an illustrative clinical case correctly identified by the model (predicted probability $\hat{p} = 0.793$), the local feature contributions were:
- screen_time: +0.166 (strongly increases predicted risk)
- ebf: +0.060
- par_authoritative: +0.050
- ill_recurrent: +0.042
- stim_score: +0.027
- birth_weight: +0.019
This case vignette confirms that while individual behavioral features align with clinician intuition, the model's overall predictive utility cannot be divorced from institutional channel bias.