Manuskrip Opsi A · Asahfikir

Opsi A — 02-methods-results

Methods and Results: Algorithmic Audit and Confounding Analysis

1. Methods

1.1 Study Design and Regulatory Framework

This study is an algorithmic reproducibility, code audit, and methodological validity investigation adhering to the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis + Artificial Intelligence (TRIPOD+AI) guidelines [[VERIFY CITE: Collins et al., 2024 / TRIPOD+AI Statement]] and the Prediction model Risk Of Bias Assessment Tool (PROBAST) [[VERIFY CITE: Moons et al., 2019]]. The primary objective is to evaluate whether a previously developed machine learning triage model for developmental delay risk reflects genuine clinical developmental risk factors or systematic recruitment-channel confounding, and to assess the impact of implementation anomalies in data preprocessing.

1.2 Data Sources and Cohort Auditing

The dataset consists of two distinct recruitment channels collected in Palembang, South Sumatra, Indonesia:

  1. Community Cohort (Posyandu Arm):
    Caregivers of children under five years of age were recruited across community health posts (Posyandu). Raw records comprised 302 entries. Records were audited for eligibility: 3 children were excluded due to acute acute illness during screening, and 9 duplicate records were identified and removed by the author script (note: alternative deduplication identifies 8 duplicates). The final community dataset yielded 290 children with a mean age of 30.73 ± 16.19 months (49.5% male). Developmental status was assessed using the Indonesian national Kuesioner Pra-Skrining Perkembangan (KPSP): 267 children were classified as appropriate (sesuai), 22 as doubtful (meragukan), and 1 as manifesting developmental deviation (penyimpangan). The latter two categories form an at-risk group of 23 children, establishing an empirical community screening risk prevalence of 7.9% (23/290).

  2. Clinical Cohort (Therapy Clinic Arm):
    Children receiving outpatient developmental therapy were recruited from a specialized pediatric therapy clinic between August and September 2026. Raw records comprised 33 patients. Three records were excluded: 2 patients with Down syndrome (pre-existing genetic condition) and 1 patient lacking a definitive clinical diagnosis. The final clinical cohort consisted of 30 patients with a mean age of 63.24 ± 26.38 months (age range: 34–167 months). Among these, 23 children met the target age criterion of ≤ 72 months.

1.3 Matching Procedure and Code Audit of Cohort Construction

The original experimental protocol specified a 1:2 age-matched case–control design with a caliper of ±6 months: - Matching targeted the 23 eligible clinic cases (≤ 72 months) against the 290 community controls. - Matching was executed sequentially via matched_final.py (lines 34–48). For each clinic case, the available control pool within ±6 months was sampled (up to 2 controls). - Due to the age distribution truncation of the community cohort (maximum control age 59 months), 6 clinic cases (aged 66, 72, 71, 71, 66, and 69 months; mean case age 53.1 months, mean control age 46.4 months) could not be matched with any control. - Code Audit (Bug #2 - Retention of Unmatched Cases): Although the script correctly tracked these 6 unmatched cases in a diagnostic list (unmatched_case_ages), the final dataset concatenation step invoked the unpruned case dataframe: python matched = pd.concat([matched_controls.assign(case=0), cases.assign(case=1)]) Consequently, all 23 cases were concatenated with the 34 matched controls, producing a 57-row dataset rather than the nominal 51-row matched cohort (17 cases + 34 controls). In this study, we designate the 57-row dataset as Run A (Author Construction) and the strictly matched 51-row dataset (excluding the 6 unmatched cases) as Run B (Corrected Matched Cohort).

1.4 Psychometric Instrument Preprocessing Audit

Caregivers completed a 9-item Likert questionnaire measuring three parenting typologies (authoritative, authoritarian, and permissive; 3 items each) and a stimulation activity scale (stim_score). - Code Audit (Bug #1 - Lexicographical Item Permutation): In clean.py (line 51), the aggregation block processed questionnaire columns as follows: python def likert_block(df, prefix, mapping, n): cols = [c for c in df.columns if c.startswith(prefix)] out = pd.DataFrame({f'{i+1:02d}': df[c].map(mapping) for i, c in enumerate(sorted(cols))}) Because sorted(cols) sorted column names alphabetically based on raw question string text rather than survey index order, items measuring distinct psychological dimensions were scrambled. Group 01–03, 04–06, and 07–09 each collapsed into a heterogeneous blend of authoritative, authoritarian, and permissive questions. - We audited the psychometric impact by computing Cronbach's alpha (α) under both the corrupted alphabetical ordering and the true questionnaire mapping. The stimulation scale was unaffected by this bug.

1.5 Machine Learning Pipeline and Confounding Evaluation

All classification pipelines were evaluated using 5-fold cross-validation repeated 5 times (5×5 CV, matching our original pipeline): - Feature Sets: Our original risk model comprised 23 features: 21 numerical features (age, birth order, under-5 siblings, maternal age category, antenatal care visits, birth weight, low birth weight indicator, prematurity, cried at birth, exclusive breastfeeding, household smoking exposure, daily screen time, 3 parenting subscales, stimulation score, 2 maternal pregnancy illness indicators, and 3 child medical history indicators) and 2 categorical features (child sex and primary caregiver) encoded via one-hot transformation. - Channel Confounding Baseline (SES Baseline): To evaluate whether models learned institutional referral stream rather than developmental delay pathology, we constructed a negative control channel-only model using strictly four parental socio-economic status (SES) variables: maternal education (edu_mom), paternal education (edu_dad), household income (income), and maternal employment status (mom_work). We also evaluated a combined feature set (23 risk features + 4 SES variables). - Classifiers: Logistic Regression (LR), Decision Tree (DT), and Random Forest (RF) using our original hyperparameters, supplemented by gradient boosting (XGBoost) and Platt-calibrated Logistic Regression. - Statistical Testing: Differences in F1 scores across CV repetitions were evaluated using two-tailed paired t-tests.

1.6 Calibration, Risk Stratification, and Decision Curve Analysis


2. Results

2.1 Cohort Flow and Data Audit

The audit of participant flows across both recruitment arms is presented in Table 2.

Table 2 (Table 1 Specification). Data Audit and Participant Flow Across Recruitment Channels

Cohort Metric Community Arm (Posyandu) Clinical Arm (Pediatric Therapy Clinic)
Raw Records Captured 302 33
Exclusions 12 total: 3 acute illness, 9 duplicate entries* 3 total: 2 Down syndrome, 1 no clinical diagnosis
Final Audited Cohort 290 30
Target Age Eligible (≤ 72 months) 290 (all children ≤ 59 months) 23 (7 patients aged > 72 months excluded)
Mean Age (Months) ± SD 30.73 ± 16.19 63.24 ± 26.38
Age Range (Months) [[TO BE COMPUTED: Min–Max community age]] 34–167
Sex Distribution (% Male) 49.5% [[TO BE COMPUTED: Clinic sex distribution]]
Developmental Screening Status 267 Appropriate, 22 Doubtful, 1 Deviation (23 at-risk; 7.9%) 30 Confirmed clinical developmental delay
Available for 1:2 Age Matching 290 23
Successfully Matched (±6 mo caliper) 34 17 (6 unmatched: ages 66, 69, 71, 71, 66, 72)

*Note: An alternative deduplication rule yields 8 duplicates (291 final community rows).


2.2 Code Audit: Resolution of Psychometric Scale Corruption

Evaluating the impact of clean.py line 51 confirmed that alphabetical column sorting severely corrupted survey psychometrics (Table 3). Under the original script, internal consistency appeared near zero or negative, leading the original investigators to erroneously dismiss parenting style as psychometrically invalid. Restoring the correct column grouping demonstrates that authoritative and permissive parenting subscales maintain acceptable reliability.

Table 3. Psychometric Scale Reliability: Corrupted Script vs. Restored Grouping

Instrument Subscale Cronbach's α (clean.py line 51, sorted(cols)) Cronbach's α (Restored Survey Order) Psychometric Status
Authoritative Parenting +0.100 +0.567 Restored to acceptable exploratory threshold
Authoritarian Parenting −0.178 +0.160 Low reliability
Permissive Parenting +0.132 +0.591 Restored to acceptable exploratory threshold
Stimulation Scale (stim_score) +0.789 +0.789 Unaffected by bug (consistent)

2.3 Model Reproduction and Effect of Unmatched Case Retention

Re-executing our original modeling pipeline on the unpruned 57-row dataset (Run A) reproduced published Table 2 exactly across all classifiers to three decimal places (Table 4).

When the pipeline was executed on the true 1:2 matched cohort (Run B, N = 51), predictive performance dropped noticeably across all models. Random Forest mean F1 dropped from 0.791 to 0.695, and sensitivity (recall) dropped from 0.750 to 0.647. Furthermore, on the true matched cohort, Random Forest did not strictly dominate all baselines: Logistic Regression achieved a higher mean recall (0.667 vs. 0.647).

Table 4 (Table 2 Specification). Classification Performance: Reproduction of Our Published Table 2 (Run A) vs. Corrected Matched Set (Run B)

Model Evaluation Run Rows (Cases / Controls) Accuracy (Mean ± SD) Precision Recall (Sensitivity) F1-Score Pooled Out-of-Fold Matrix (TP / FN / FP / TN)
Logistic Regression Run A (Our Original Code) 57 (23 / 34) 0.839 ± 0.079 0.859 0.740 0.777 [[TO BE COMPUTED: LR Run A OOF]]
Run B (Corrected) 51 (17 / 34) 0.815 ± 0.120 0.733 0.667 0.676 [[TO BE COMPUTED: LR Run B OOF]]
Decision Tree Run A (Our Original Code) 57 (23 / 34) 0.731 ± 0.112 0.669 0.628 0.629 [[TO BE COMPUTED: DT Run A OOF]]
Run B (Corrected) 51 (17 / 34) 0.722 ± 0.130 0.563 0.607 0.573 [[TO BE COMPUTED: DT Run B OOF]]
Random Forest Run A (Our Published Table 2) 57 (23 / 34) 0.852 ± 0.082 0.890 0.750 0.791 TP 16 / FN 7 / FP 2 / TN 32 (Acc 0.842, Prec 0.889, Rec 0.696, F1 0.780)
Run B (Corrected) 51 (17 / 34) 0.827 ± 0.092 0.834 0.647 0.695 TP 11 / FN 6 / FP 4 / TN 30 (Acc 0.804, Prec 0.733, Rec 0.647, F1 0.688)

2.4 Empirical Proof of Recruitment-Channel Confounding

Evaluating the channel-only baseline against the developmental risk-factor model on the corrected 51-row cohort provided decisive evidence of recruitment confounding (Table 5).

Table 5. Channel Confounding Experiment: Clinical Risk Models vs. Socio-Economic Baseline (Run B, N = 51)

Model Specification Feature Composition Mean F1-Score ± SD Paired Comparison vs. Channel Baseline
A. Clinical Risk Model 21 numerical + 2 categorical risk factors (our original design) 0.695 ± 0.173 Paired ΔF1 (A − B) = −0.181
B. Channel-Only Baseline 4 SES variables only (edu_mom, edu_dad, income, mom_work) 0.876 ± 0.126 95% CI: [−0.500, 0.176], p = 0.00014
C. Combined Model 23 clinical risk factors + 4 SES variables 0.902 ± 0.140 —

The 4-variable SES baseline significantly outperformed the entire 23-feature clinical risk-factor model (p = 0.00014). This demonstrates that the predictive accuracy reported in two-stream case–control triage models is largely driven by institutional referral disparities (SES differences between clinic and community populations) rather than specific developmental delay etiology.


2.5 Model Calibration and Benchmark Models

Model calibration metrics on the matched scale (5×5 CV, Run B) are summarized in Table 6.

Table 6 (Table 3 Specification). Calibration and Benchmark Performance (Run B, Matched Scale)

Classifier Calibration Slope Calibration Intercept Brier Score AUROC Mean Predicted Probability Observed Case Rate
Random Forest 2.656 0.274 0.144 0.896 0.381 0.3333
XGBoost 0.801 0.060 0.136 0.848 0.310 0.3333
Calibrated Logistic Regression 3.269 1.423 0.152 0.903 0.336 0.3333

XGBoost achieved superior calibration relative to Random Forest (slope 0.801 vs. 2.656; Brier score 0.136 vs. 0.144), whereas Random Forest showed substantial overconfidence.

Population Risk Stratification (Anchor π₀ = 0.0793)

Grouping patients into predicted risk tertiles under the prior-corrected anchor (π₀ = 0.0793) confirmed monotonic risk separation across strata: - Low Risk: n = 17, observed cases = 1 (rate 0.059, mean predicted probability 0.0398) - Medium Risk: n = 17, observed cases = 3 (rate 0.176, mean predicted probability 0.0869) - High Risk: n = 17, observed cases = 13 (rate 0.765, mean predicted probability 0.2187)


2.6 Decision Curve Analysis and Clinical Utility

Standardized net benefit across decision threshold probabilities ($p_t$) is detailed in Table 7 (specification for Figure 1).

Table 7. Decision Curve Analysis: Prior-Corrected Net Benefit Across Decision Thresholds (π₀ = 0.0793)

Decision Threshold ($p_t$) Risk Model Net Benefit Screen-All Strategy Net Benefit Analytic KPSP Comparator (Se 0.60, Sp 0.92) Clinical Interpretation
0.05 0.311 0.298 0.197 Model superior
0.10 0.259 0.259 0.194 Model superior / equal to screen-all
0.15 0.225 0.216 0.191 Model superior
0.20 0.132 0.167 0.187 KPSP comparator superior; screen-all superior
0.25 0.059 0.111 0.182 KPSP comparator superior; screen-all superior
0.30 0.059 0.048 0.177 KPSP comparator superior
0.40 0.000 −0.111 0.164 KPSP comparator superior; model zero net benefit

Figure 1 Specification (Decision Curve):
Plotting standardized net benefit (Y-axis: −0.2 to 0.4) against threshold probability $p_t$ (X-axis: 0.05 to 0.40). The curve depicts: 1. Risk Model (Solid Blue): Peaks at low thresholds but drops steeply at $p_t \ge 0.20$. 2. Analytic KPSP (Dashed Green): Maintains stable positive net benefit (0.197 to 0.164) across all thresholds. 3. Screen-All (Dotted Red): Crosses below zero at $p_t \approx 0.35$. 4. Screen-None (Solid Gray): Horizontal line at zero.
Key Takeaway: The ML model provides clinical value only in low-threshold risk-tolerant scenarios ($p_t \le 0.15$). At clinical thresholds of $p_t \ge 0.20$, standard KPSP screening provides superior net benefit. Claims that the model outperforms standard triage across all decision thresholds are empirically disproven.


2.7 Feature Attribution and Clinical Vignette

Global TreeSHAP attribution on the corrected Run B Random Forest model is summarized in Table 8 (specification for Figure 2). Daily screen time (screening_time) was the primary discriminative feature, followed by authoritative parenting and maternal pregnancy history.

Table 8 (Figure 2 Specification). Global Top Feature Importance (TreeSHAP on Run B)

Rank Feature Identifier Description Mean Absolute SHAP Value
1 num__screen_time Daily screen duration 0.130 (0.1301)
2 num__par_authoritative Authoritative parenting score 0.032 (0.0321)
3 num__preg_any Any maternal pregnancy complication 0.028 (0.0281)
4 num__age_months Child age in months 0.0260
5 num__ebf Exclusive breastfeeding duration 0.0255
6 cat__caregiver_Ibu Primary caregiver: Mother 0.0254
7 num__anc Antenatal care visit count 0.0228
8 num__birth_weight Birth weight (kg) 0.0226

Local Patient Attribution Vignette

For an illustrative clinical case correctly identified by the model (predicted probability $\hat{p} = 0.793$), the local feature contributions were: - screen_time: +0.166 (strongly increases predicted risk) - ebf: +0.060 - par_authoritative: +0.050 - ill_recurrent: +0.042 - stim_score: +0.027 - birth_weight: +0.019

This case vignette confirms that while individual behavioral features align with clinician intuition, the model's overall predictive utility cannot be divorced from institutional channel bias.