# Framing and Core Structure: Validity Study

## Title Options
- **Option 1 (Recommended):** *Recruitment-Channel Confounding Outweighs Risk Factors in Machine-Learning Case–Control Models of Developmental Delay: A Validity Study*
- **Option 2:** *When Case–Control Machine Learning Learns the Recruitment Channel: A Validity Study of Developmental-Delay Risk Prediction in Indonesia*

**Authors:** Dini Hari Pertiwi, Yarza Aprizal, Eko Setiawan  
**Affiliation:** Institut Teknologi dan Bisnis Palcomtech, Palembang, Indonesia  
**Target Journal:** Jurnal ELTIKOM (Revised Submission / Re-framing)

---

## Abstract (~230 words)

Dual-channel case–control recruitment—pairing clinically confirmed cases with community controls—is widely utilized in low-resource biomedical machine learning (ML) when prospective cohorts are unavailable. This study is a self-audit of our own previously reported Random Forest (RF) triage model for early developmental delay risk screening in Indonesia. Rather than proposing a new predictive model, we conduct an algorithmic reproducibility audit and empirical confounding analysis using our original analysis pipeline. We reproduced the previously reported performance bit-for-bit (RF F1 0.791), revealing that 6 unmatched cases (aged 66–72 months) were inadvertently retained in a nominal 1:2 age-matched cohort (Run A, N = 57). Correcting this data handling defect to strict 1:2 matching (Run B, 17 cases, 34 controls; N = 51) reduced RF F1 to 0.695 ± 0.173 (recall 0.647), with Logistic Regression achieving superior recall (0.667). Crucially, a channel-only baseline comprising solely four parental socio-economic status (SES) variables achieved an F1 of 0.876 ± 0.126, significantly outperforming the entire 23-feature developmental risk model (paired ΔF1 = −0.181, 95% CI [−0.500, 0.176], p = 0.00014). An additional alphabetical column-sorting bug in feature preprocessing was found to have artifactually corrupted parenting subscale reliabilities. Decision curve analysis indicated that the risk model yielded positive net benefit only at low threshold probabilities (0.05–0.15) and was dominated by analytic developmental screening at thresholds ≥ 0.20. Case–control ML classifiers in dual-stream recruitment predominantly learn institutional referral disparities rather than developmental pathology. Negative control channel baselines and code audits are essential prerequisites prior to clinical deployment.

---

## 1. Introduction

Early detection of developmental delay in children under five years of age represents a critical public health priority in low- and middle-income countries (LMICs). In Indonesia, national developmental screening relies on the *Kuesioner Pra-Skrining Perkembangan* (KPSP) administered through the *Stimulasi, Deteksi, dan Intervensi Dini Tumbuh Kembang* (SDIDTK) program at integrated community health posts (*Posyandu*) and primary health centers (*Puskesmas*). However, high patient-to-cadre ratios, inconsistent caregiver attendance, and operational constraints often limit comprehensive developmental monitoring. Consequently, automated machine learning (ML) algorithms leveraging parent-reported perinatal, environmental, and behavioral risk factors have attracted substantial interest as potential community triage aids [[VERIFY CITE]].

Because developmental delay exhibits relatively low community prevalence—typically between 5% and 10% in routine screening—researchers frequently adopt two-stream case–control recruitment strategies to construct sufficiently balanced datasets. In these designs, confirmed cases are recruited from tertiary hospital pediatric clinics or specialized therapy centers, whereas asymptomatic controls are sampled from community *Posyandu* cohorts. While operationally pragmatic, dual-channel designs introduce profound vulnerabilities to recruitment-channel confounding, spectrum bias, and selection artifacts [[VERIFY CITE: Moons et al., PROBAST 2019]]. When demographic and institutional recruitment characteristics differ systematically between study arms, machine learning classifiers risk identifying the institutional pathway rather than the underlying clinical pathology.

Furthermore, machine learning pipelines applied to modest biomedical samples are acutely sensitive to subtle implementation errors. Small programming choices during data cleaning, scale aggregation, and matched subset extraction can silently distort psychometric properties and cross-validation performance without throwing execution exceptions. In the Indonesian medical informatics literature, studies predominantly report standard discrimination metrics (e.g., accuracy, sensitivity, precision) without publishing auditable code pipelines, conducting negative control evaluations, or verifying model calibration against target population priors.

In this paper, we pivot from an initial exploratory triage model to a rigorous validity, reproducibility, and code-audit study. We evaluate an Indonesian developmental delay dataset originally collected to develop a 23-feature Random Forest screening tool. By auditing the original scripts bit-for-bit, we evaluate the degree to which reported findings reflect developmental risk factors versus structural recruitment channel confounding and procedural implementation bugs.

---

## 2. Gap Analysis

The literature on pediatric developmental risk prediction in regional informatics manifests three pervasive blind spots, summarized in Table 1.

### Table 1. Methodological Gap Analysis: Regional ML Literature vs. Our Original Submission vs. Present Validity Study

| Methodological Dimension | Standard Regional Health ML Literature | Our Original Manuscript Submission | Present Validity Study (This Work) |
|---|---|---|---|
| **Primary Research Claim** | Algorithmic deployment: proposing novel classifiers claiming high predictive accuracy for screening/triage. | Algorithmic feasibility: claimed RF triage tool achieved superior discrimination across all evaluation metrics. | **Methodological validity:** auditing whether reported predictive signal reflects clinical risk factors or recruitment confounding. |
| **Recruitment Channel Confounding** | Overlooked: clinic cases are paired with community controls without testing for channel-specific confounding. | Acknowledged descriptively: excluded explicit demographic features from model but did not test channel-only baselines. | **Formally falsified:** empirical testing demonstrates a 4-variable SES baseline significantly outperforms the 23-feature clinical model (p = 0.00014). |
| **Pipeline & Code Integrity** | Code rarely audited or released; data preprocessing steps treated as procedural formalities. | Unaudited execution: alphabetical column sorting scrambled psychometrics; unmatched cases leaked into matched cross-validation. | **Bit-for-bit reproduction & repair:** isolated two specific implementation bugs, demonstrating their exact mathematical impact on conclusions. |
| **Psychometric Scale Verification** | Reliability reported uncritically or dropped if coefficients appear poor without investigating code artifacts. | Dropped parenting scales to exploratory status due to artifactual alphas (+0.10, −0.18, +0.13) from sorting bug. | **Psychometric recovery:** demonstrated that correcting column sorting restores acceptable reliability for authoritative (0.57) and permissive (0.59) styles. |
| **Clinical Utility & Decision Analysis** | Reliance on raw accuracy and AUC; absence of calibration assessment or decision curve analysis (DCA). | Reported CV accuracy and F1 on balanced sample; uncalibrated for population base rates. | **Decision curve analysis:** evaluated threshold-specific net benefit adjusted for community prevalence anchor (π₀ = 0.0793). |

---

## 3. Contributions Formulated as Falsifiable Claims

Rather than presenting an optimistic model deployment narrative, this work formalizes its findings into four explicit, falsifiable claims:

1. **Claim 1 (Exact Reproduction and Cohort Integrity):**  
   Our previously reported Table 2 performance (RF F1 0.791, accuracy 0.852) is an exact mathematical artifact of an uncorrected 57-row pipeline that failed to drop 6 unmatched cases (aged 66–72 months). When the pipeline is corrected to strict 1:2 age-matching (17 cases, 34 controls; N = 51), RF F1 drops to 0.695 ± 0.173, and RF does not outperform baselines across every metric (Logistic Regression achieves higher recall: 0.667 vs. 0.647).  
   *Falsified by:* Demonstrating that Run A results can be derived without the 6 unmatched cases, or that RF strictly dominates LR in recall on the true 51-row set.

2. **Claim 2 (Recruitment-Channel Confounding Dominance):**  
   In dual-channel case–control datasets, non-clinical parental socio-economic status (SES) variables carry stronger discriminative signal than multi-domain clinical risk factors. Specifically, a 4-variable SES baseline (maternal education, paternal education, family income, maternal employment status) achieves an F1 of 0.876 ± 0.126, significantly outperforming the 23-feature developmental risk model (F1 0.695 ± 0.173, paired ΔF1 = −0.181, 95% CI [−0.500, 0.176], p = 0.00014).  
   *Falsified by:* A clinical risk-factor model demonstrating statistically superior or equivalent predictive performance to an SES channel-only baseline under identical cross-validation folds.

3. **Claim 3 (Psychometric Artifact Resolution):**  
   The apparent unreliability of parenting style subscales in our original analysis was entirely an operational artifact of lexicographical column sorting (`sorted(cols)`) during Likert block preprocessing. Restoring questionnaire item ordering improves internal consistency from artifactual values (0.100, −0.178, 0.132) to psychometrically plausible values (authoritative α = 0.567, authoritarian α = 0.160, permissive α = 0.591).  
   *Falsified by:* Demonstrating that the scrambled alphas originated from respondent response inconsistency rather than code column permutation.

4. **Claim 4 (Threshold-Bounded Clinical Net Benefit):**  
   When re-anchored to the community developmental delay prevalence (π₀ = 0.0793), the 23-feature machine learning model provides clinical net benefit over default strategies only at low decision thresholds (0.05–0.15), and is strictly inferior to standard analytic screening (KPSP comparator) at decision thresholds ≥ 0.20.  
   *Falsified by:* Showing that the prior-corrected risk model yields superior net benefit across all decision thresholds compared to standard screening instruments.

---

## 4. Conclusion

This study demonstrates that dual-channel case–control machine learning models in pediatric developmental screening are profoundly vulnerable to learning recruitment streams rather than clinical risk markers. By reproducing published results bit-for-bit from the original analysis codebase, we showed that four socio-economic status indicators substantially outperform an entire 23-feature developmental risk model, achieving an F1 score of 0.876 compared to 0.695 (p = 0.00014). Furthermore, we documented how minor programming oversights—including accidental inclusion of unmatched age outliers and lexicographical sorting of psychometric survey questions—can distort model performance metrics and substantive conclusions regarding risk factor reliability. 

When adjusted for community prevalence, the proposed triage model provides clinical net benefit only within a narrow low-threshold band (0.05–0.15) and is outperformed by standard developmental screening at decision thresholds of 0.20 and above. These findings indicate that machine learning triage tools cannot be validated on two-stream observational samples without formal negative-control baselines, transparent code audits, and prior-corrected decision curve analyses. We urge the medical informatics community in Indonesia and comparable LMIC settings to adopt these validity protocols as standard peer-review criteria before advancing automated screening tools toward clinical application.
