E-Commerce Session Analytics for Conversion Decisions: Predictive Performance, Calibration, and Dependence on Outcome-Proximal Metrics

Abstract

E-commerce analytics often frames purchase prediction as a classification problem, yet a high performance metric does not ensure that a model can support a decision at the required time. This study assessed how predictive performance changes when adding groups of variables plausibly available at different stages of a session, with particular attention to an outcome-proximal metric, PageValues. We conducted a secondary analysis of 12,330 anonymous sessions from the Online Shoppers Purchasing Intention dataset, including 1,908 conversions (15.47%). Penalized logistic regression and boosted trees were evaluated through five-fold stratified cross-validation repeated three times. Predictors were organized into four cumulative panels: context (P1), navigation (P2), exit (P3), and complete with PageValues (P4). We assessed ROC AUC, average precision, Brier score, log-loss, calibration, threshold scenarios, and net benefit. With P3, logistic regression achieved a ROC AUC of 0.751 (95% CI: 0.740–0.761) and average precision of 0.325 (0.306–0.344); with P4, these increased to 0.896 (0.888–0.904) and 0.642 (0.615–0.666). For boosted trees, ROC AUC increased from 0.788 to 0.936 and average precision from 0.374 to 0.757. Internal calibration was close to ideal but does not replace external validation. At a 0.20 threshold, P4 reduced the actionable share of sessions from 30.9% to 19.4% compared with P3 and increased positive predictive value from 29.7% to 55.4%. The disproportionate gain after adding PageValues reveals predictive dependence on a feature whose temporal availability cannot be confirmed from the dataset. Responsible business use therefore requires recording when each metric is computed, temporal validation, explicit cost definitions, and intervention testing before claiming conversion improvements.