Manuskrip Opsi A · Asahfikir

Opsi A — 03-discussion

Discussion and Methodological Lessons

1. Principal Findings

This validity study provides an empirical demonstration of how dual-channel case–control recruitment and minor code-level implementation defects undermine the integrity of biomedical machine learning (ML) models. By executing an auditable, bit-for-bit reproduction of our own previously published Random Forest (RF) triage model for pediatric developmental delay in Indonesia, we uncovered two critical insights.

First, the apparent predictive performance of the 23-feature clinical risk model is largely attributable to institutional recruitment-channel confounding rather than developmental pathology. A negative-control baseline consisting solely of four parental socio-economic status (SES) variables achieved an F1-score of 0.876 ± 0.126, significantly outperforming the complete 23-feature clinical risk model (F1 = 0.695 ± 0.173, p = 0.00014). In dual-channel designs where cases are recruited from specialized therapy clinics and controls from community health posts (Posyandu), the classifier predominantly learns structural socio-economic divergence between patient referral pathways.

Second, the study demonstrates the disproportionate impact of minor programming oversights in small-sample biomedical ML. Two subtle errors—lexicographical column sorting in survey preprocessing and inadvertent retention of unmatched cases in cohort construction—substantively distorted both the psychometric conclusions and cross-validation benchmarks of the original study. When corrected, the primary classification metrics declined (RF F1 decreased from 0.791 to 0.695; recall decreased from 0.750 to 0.647), and the claim that Random Forest strictly dominated all baseline algorithms was invalidated, with Logistic Regression achieving superior sensitivity (0.667 vs. 0.647). Furthermore, decision curve analysis revealed that the clinical utility of the risk model is confined strictly to low decision thresholds (0.05–0.15) and is dominated by standard routine screening (KPSP) at decision thresholds ≥ 0.20.


2. Anatomy of Two Implementation Bugs

2.1 Bug #1: Lexicographical Item Permutation in Psychometrics

In modern health informatics, survey questionnaires measuring behavioral constructs are frequently ingested into automated dataframes. In our original preprocessing script (clean.py, line 51), we used sorted(cols) to organize questionnaire items. While syntactically benign, Python’s string sort reordered question labels alphabetically by question prompt text rather than preserving survey item order.

As a consequence, the 9 Likert items—originally designed to capture three distinct parenting typologies (authoritative, authoritarian, and permissive; 3 items each)—were scrambled. Each 3-item block collapsed into a composite of divergent behavioral constructs. This generated severe psychometric artifacts, producing near-zero or negative Cronbach’s alpha coefficients (authoritative α = +0.100, authoritarian α = −0.178, permissive α = +0.132). These distorted metrics misled us into concluding that parental behavior was psychometrically unmeasurable in this cohort, demoting parenting style to an exploratory variable.

Restoring the true survey item sequence immediately recovered acceptable internal consistency for authoritative (α = 0.567) and permissive (α = 0.591) styles. This demonstrates how a single automated sorting operation can silently erase a legitimate clinical risk domain from subsequent scientific analysis.

2.2 Bug #2: Data Leakage and Retention of Unmatched Cases

In 1:2 age-matched case–control studies, strict matching criteria are essential to prevent age-related confounding. In matched_final.py (lines 34–48), the matching loop correctly flagged 6 clinical cases (aged 66–72 months) that lacked community controls within the ±6-month caliper (because community controls were truncated at 59 months). However, during final cohort concatenation, the script concatenated the entire unpruned case dataframe (cases.assign(case=1)) with the 34 matched controls.

This unintended leak inflated the dataset from 51 to 57 rows, appending 6 unmatched, older clinical cases without corresponding community controls. Because older clinical cases presented with distinct clinical and exposure profiles, their presence artificially boosted cross-validation discrimination metrics (Run A RF F1 = 0.791). Re-running the pipeline on the strictly matched 51-row dataset (Run B) revealed a marked drop in performance: RF F1 fell by nearly 0.10 to 0.695, and recall dropped from 0.750 to 0.647. Furthermore, on the true matched cohort, Random Forest lost its across-the-board superiority: Logistic Regression attained a higher recall (0.667 vs. 0.647).


3. The Mechanism of Recruitment-Channel Confounding

The most consequential finding of this audit is that multi-feature developmental risk models can be severely confounded by the institutional architecture of participant recruitment. In low- and middle-income countries, families who actively seek specialized pediatric outpatient therapy clinics typically possess higher formal education, higher household income, and greater health literacy compared to the broader population attending routine community Posyandu centers.

When machine learning algorithms are trained to differentiate clinic cases from community controls, they exploit any covariate distribution shift between the two recruitment streams. When evaluated in isolation, 4 parental SES variables (edu_mom, edu_dad, income, mom_work) yielded an F1 score of 0.876 ± 0.126. In contrast, the 23 clinical, perinatal, and environmental risk factors carefully selected by domain experts yielded an F1 score of only 0.695 ± 0.173. The paired difference (ΔF1 = −0.181, p = 0.00014) confirms that the model's discriminative power is dominated by institutional selection artifacts rather than clinical developmental pathology.

This finding exposes a fundamental vulnerability in case–control machine learning: without negative control baselines (e.g., channel-only models), researchers risk mistaking institutional selection mechanisms for true clinical predictive signal.


4. Methodological Implications for Indonesian Health Informatics

Our findings carry direct implications for the growing literature on applied healthcare machine learning in Indonesia and regional biomedical informatics:

  1. Mandatory Channel-Only Negative Controls:
    Studies employing multi-stream recruitment (e.g., hospital cases vs. community controls) must routinely include negative control models trained strictly on recruitment artifacts (demographics, administrative markers, interview timing). If non-clinical administrative or SES features outperform clinical features, the model cannot be deployed as a clinical triage tool [[VERIFY CITE: PROBAST / Moons et al., 2019]].

  2. Prevalence Distortion and Prior Correction:
    Artificially matched 1:2 case–control cohorts enforce an artificial sampling prevalence ($\pi_s = 0.3333$). Deploying raw model probabilities in primary care environments where community prevalence is roughly 7.9% ($\pi_0 = 0.0793$) causes severe probability miscalibration. Calibration curves and prior-odds corrections must be standard reporting requirements.

  3. Clinical Utility Beyond Discrimination (Decision Curve Analysis):
    Reporting accuracy or AUC alone is insufficient for clinical triage tools. As demonstrated in our decision curve analysis, the risk model yielded positive net benefit only at low threshold probabilities ($p_t = 0.05–0.15$). At decision thresholds $p_t \ge 0.20$, the model was strictly inferior to simple analytic KPSP screening. Machine learning tools should only be recommended when they demonstrably exceed routine, low-cost screening tools across relevant decision thresholds [[VERIFY CITE: Vickers & Elkin, 2006]].

  4. Open Code and Pipeline Auditing:
    Subtle bugs like sorted(cols) and unpruned dataframe concatenation do not trigger runtime errors and remain invisible during standard peer review unless executable scripts and data pipelines are fully audited. Journal editors and reviewers must mandate public code repositories for published predictive models.


5. Limitations

Several limitations of the present validity investigation should be emphasized:

  1. Sample Size Constraints:
    The audited matched cohort is modest (N = 51; 17 clinical cases and 34 community controls). While this sample size accurately reflects the empirical dataset used in the original investigation, confidence intervals around cross-validation metrics remain wide.

  2. Analytic Nature of the KPSP Comparator:
    In the clinical arm, developmental delay was confirmed through clinical diagnosis and therapy admission; the formal KPSP screening instrument was not administered to clinic patients. Consequently, the KPSP comparator in our decision curve analysis was modeled analytically using published diagnostic sensitivity (0.60) and specificity (0.92) rather than head-to-head empirical testing.

  3. Assumption of Population Prevalence Anchor ($\pi_0$):
    The baseline prevalence anchor ($\pi_0 = 0.0793$) was derived from the empirical rate of at-risk children in the Posyandu community arm (23/290). This empirical anchor represents an operational assumption that requires formal verification through regional epidemiological surveys in South Sumatra.

  4. Observational and Cross-Sectional Nature:
    Although TreeSHAP identified daily screen time as the predominant behavioral contributor (mean absolute SHAP = 0.130), this association reflects observational correlation and recruitment-stream differences; it cannot be interpreted as a direct causal etiology.