# ECPE scoring stress-test run

Completed: 2026-09-01 20:36:43 UTC. Program 1.0.0. Seed 20260901. Quick mode: FALSE.
Data: 2922 examinees x 28 binary ECPE grammar items; edmdata 1.3.0. Calibration: 2045, audit: 877.
Score reference: rest. Bootstrap 500; sensitivity replicates 100; permutations 500; cross-fitting folds 5.

## Exact scoring specification

C is cross-group concordance with half credit for tied corresponding scores; Dxy = 2*C - 1. Cplus = max(0,Dxy).
For constant items empirical C and Dxy stay NA, while Cplus=0 is used for weighting.
pad1=(correct+1)/(n+1); pad2=(n-n*Cplus+1)/(n+1); raw_weight=(1-log(pad1))*(1-log(pad2)).
Operational weights = raw_weight/min(raw_weight); normalized score = sum(x*weight)/sum(weight). No rounding of n*Cplus.
GitHub source blob: a336d1b3d9964bd5f2e0f90f214e15d540be827d.

## Descriptive results

- Full-sample weighted versus raw Spearman correlation: 0.995772.
- Strict reversals between persons with different raw totals: 0.5228% of 4006254 comparable pairs.
- Maximum full-sample item share: 5.5276%; effective item count 1/sum(share^2): 27.081.
- Median audit score RMSE under calibration bootstrap: 0.002372 (0-1 score units).
- Median audit score RMSE with n=50 calibration persons: 0.016086.
- Median split-half corrected correlation: raw 0.784384, weighted 0.785045; median paired difference 0.000905.

## Interpretation boundaries

The full-sample scores are descriptive. Audit scores always use calibration-only weights. Bootstrap intervals measure calibration uncertainty conditional on these responses and this calibration population; they are not latent-ability or test-retest confidence intervals.
The frozen audit reference is the original calibration fit, not a known true score. Agreement with it measures stability, not scoring accuracy. Neither high raw-score correlation nor high consistency establishes superiority or external validity.
Percentile intervals are pointwise and can be irregular at the Dxy=0 truncation boundary. Permutation tests are one-sided for positive association, with +1 correction and BH adjustment separately by score type. Total-score nulls retain self-inclusion effects by design.
Sample-size draws use no replacement within a draw. Population-shift draws use replacement from raw-score-selected pools; this selection itself changes association. It is not demographic DIF evidence. Contamination changes calibration responses only; 50% random responding is a specified artificial condition, not an ECPE guessing-rate estimate.
Augmented-item and deletion scenarios change test content and the score denominator. Their score shifts cannot be interpreted solely as estimation errors. Duplicate rows in sample_replication.csv add no information: any change isolates the +1 padding's sample-size dependence.
Random split halves are content-overlapping measurements of a multidimensional test. Spearman-Brown values are descriptive under its parallel-form assumptions, not a validated reliability coefficient for ECPE. Q-matrix skills overlap, so skill coverage values do not partition the score.
Top-decile overlap uses fractional membership for boundary ties; rank shifts use average ranks, in percentage points. Exact raw reversals exclude raw-score ties and weighted differences <=1e-12. Cross-fitted scores from different folds use different weights; inspect within-fold metrics before comparing pooled scores.
Constant and negative-Dxy items are retained exactly as in the source. A rare uninformative item still receives the rarity factor. All-zero items can consume denominator weight. These are properties exposed by the tests, not silently changed by this program.

## Validation and provenance

PASS: 120 exhaustive pair/rank comparisons plus scoring, ties, constants, reversal, normalization, ordering and input checks. Hmisc comparison: passed.

Input dimensions, complete 0/1 coding and Q-matrix alignment were checked. settings.R, sessionInfo.txt, package_citation.txt, input_md5.csv and run_state.rds record reproducibility information.
The RDS contains audit bootstrap scores and weight draws, split IDs and RNG states. It does not change or refit the original GitHub program.
Implementation generated by OpenAI ChatGPT from Anders Hellstrom's scoring specification and current source. The scoring-method definition remains the user's; the stress-test design and implementation were generated for this request.

## Sources

- https://tmsalab.github.io/edmdata/reference/items_ecpe.html
- https://tmsalab.github.io/edmdata/reference/qmatrix_ecpe.html
- https://github.com/AndersH3/Scoring/blob/main/item_analysis.r
- https://github.com/harrelfe/Hmisc/blob/master/R/somers2.s

Elapsed time: 39.4 seconds.