Predicting late-delivery risk before it affects the customer

What I built

I designed the work as one traceable path from source data to a score another system can consume. Each stage leaves an auditable table, metric, test, or model artifact behind.

1. 01

   Data foundation

   7 raw tables → validated analysis and modelling cohorts

2. 02

   Statistical study

   Adjusted dissatisfaction analysis with seller-clustered uncertainty

3. 03

   Prediction pipeline

   Checkout-time features → baseline and model comparison

4. 04

   Temporal evaluation

   Untouched holdout → six monthly backtests

5. 05

   Decision layer

   Top-k capture → intervention economics

6. 06

   Delivery artifact

   Saved model, metadata, and batch-scoring CLI

›Audit trail and project artifacts

• Cohorts: a row-count audit separates the review-dependent dissatisfaction study from the review-independent delivery model.
• Evaluation: machine-readable outputs store validation comparisons, bootstrap intervals, risk bands, rolling backtests, calibration, drift, and intervention scenarios.
• Deployment artifact: the fitted preprocessing and logistic model are saved together with feature-order metadata and a two-row scoring contract example.
• Documentation: the repository explains reproduction; the model card states intended use, failure modes, monitoring, and retraining expectations.

Two questions, two cohorts

Olist connects marketplace sellers to customers. Every delivered order carries an internal seller handoff deadline and a customer-facing promised delivery date. The project asks two related questions: which missed deadline is most strongly associated with dissatisfaction, and can late customer delivery be ranked from information available at checkout?

Those questions require different populations and are kept separate throughout.

Starting from 99,441 raw orders, the pipeline validates keys and schemas, resolves reviews, aggregates order items, checks timestamp coherence, and restricts the main populations to delivered single-seller orders with interpretable delivery timestamps.

What drives dissatisfaction?

Dissatisfaction is defined as a 1–2-star review. Splitting reviewed orders by the deadlines they met reveals a sharp operational divide.

Dissatisfaction is 8.2% when both deadlines are met and 13.2% when a late seller handoff is recovered before delivery. Once the customer-facing promise is missed, it rises to 62–63% whether the seller handoff was on time or late.

›Show the numbers

Dissatisfaction by fulfilment outcome — Dissatisfaction rate by Fulfilment outcome

Fulfilment outcome	Dissatisfied (1-2 star)
Both deadlines met	8.2%
Late handoff, on-time delivery	13.2%
On-time handoff, late delivery	62.1%
Both deadlines missed	63.1%

After adjusting for order size, price, freight, product category, customer state, and seller-clustered uncertainty across 2,935 sellers, the pattern remains. A late handoff recovered before delivery carries 1.52× the odds of dissatisfaction (95% CI 1.36–1.70). Late customer delivery carries 18.13× (16.49–19.94), and missing both deadlines carries 19.58× (17.32–22.14).

›Statistical design and sensitivity checks

The analysis reports Wilson intervals for outcome rates, a chi-square association test (Cramér's V 0.413), and an adjusted binomial logistic model with standard errors clustered by seller. The model includes order composition, economics, category, geography, and deadline outcome.

The result survives alternative definitions. With only 1-star reviews flagged, late-delivery groups are about 9.2× baseline; with 1–3-star reviews flagged, they remain about 4.6× baseline. The magnitude changes, but the direction does not.

Can risk be ranked at checkout?

The prediction task uses only information available when an order is placed: composition, price and freight, weight, promised window, geography, and cyclical purchase timing. Actual handoff, delivery, and review fields are prohibited by the feature contract.

Models are selected on an intermediate chronological validation period rather than a random split.

The full logistic model reaches 0.199 PR-AUC, compared with 0.071 for the prevalence baseline, 0.069 for the promised-window-only baseline, and 0.164 for gradient boosting.

›Show the numbers

Validation PR-AUC by candidate model — PR-AUC by Validation candidate

Validation candidate	Validation PR-AUC
Prevalence baseline	0.071
Promised-window only	0.069
Full logistic regression	0.199
Gradient boosting	0.164

Logistic regression generalized better than the more flexible model. Gradient boosting reached 0.832 training ROC-AUC but only 0.650 on validation; logistic regression moved from 0.739 to 0.731 and was selected before the final holdout was opened.

›Leakage and uncertainty controls

Preprocessing is fitted on training data only and serialized with the selected model. The latest 20% of eligible orders—18,998 orders from May 24 to August 29, 2018—form the untouched holdout. Reported uncertainty comes from 500 bootstrap resamples: holdout ROC-AUC 0.698–0.733, PR-AUC 0.070–0.094, capture 20.6–26.6%, and lift 2.06–2.66× at 95% confidence.

Does the model hold up over time?

A single holdout can conceal changes inside the period. I therefore added six expanding-window evaluations, each trained only on data available before the month being scored.

The full model has the best PR-AUC in five of six monthly comparisons. In August, the promised-window-only baseline scores 0.167 while the full model falls to 0.087.

›Show the numbers

Late-delivery ranking performance varies by month — Monthly PR-AUC by Evaluation month

Evaluation month	Prevalence baseline	Promised-window only	Full logistic model
2018-03	0.192	0.165	0.359
2018-04	0.046	0.041	0.192
2018-05	0.067	0.078	0.156
2018-06	0.012	0.018	0.092
2018-07	0.034	0.06	0.086
2018-08	0.063	0.167	0.087

The full model's monthly ROC-AUC ranges from 0.603 to 0.794. Its loss to the promise-only baseline in August is not a footnote: it shows that the relationship between checkout features, Olist's promise policy, and late delivery changes over time. Monitoring and recalibration are requirements for any operational use.

›How to read the rolling test

March through August 2018 are evaluated one month at a time. The training window expands through the end of the previous month; no future labels enter fitting or preprocessing. Each month compares the full logistic model against the same prevalence and promised-window-only baselines used during model selection.

What is the model actually learning?

Held-out permutation importance makes the model's dependence on existing policy visible.

Shuffling the promised delivery window reduces ROC-AUC by 0.273. Every other feature changes it by less than 0.004 in the positive direction, while several features have negative held-out importance.

›Show the numbers

Held-out permutation importance — Drop in held-out ROC-AUC when shuffled by Feature

Feature	ROC-AUC drop when shuffled
Promised window	0.273
Purchase month (cos)	0.003
Order price (log)	0.003
Total weight	0.001
Purchase hour (cos)	0.001
Purchase hour (sin)	0.001
Item count	0
Customer state	-0
Purchase day (cos)	-0.001
Purchase day (sin)	-0.001
Freight value	-0.002
Freight ratio	-0.003
Purchase month (sin)	-0.008
Distance (km)	-0.012
Same state	-0.034

The promised window is Olist's own estimate, not an independent measurement of fulfilment capacity. The full feature set clearly beats the promise-only ablation on validation, so there is incremental ranking signal; however, the dominant held-out contribution still reflects Olist's changing promise-setting policy.

Risk generally rises across held-out bands, but calibration is imperfect. The top band averages 13.7% predicted risk versus 8.4% observed risk.

›Show the numbers

Predicted versus observed late-delivery risk — Late-delivery rate by Held-out risk band (1 = lowest)

Held-out risk band (1 = lowest)	Actual late rate	Mean predicted risk
1	0.5%	0.5%
2	0.7%	1.1%
3	1.2%	1.6%
4	1.3%	2.1%
5	2.3%	2.6%
6	3.3%	3.3%
7	5.6%	4.0%
8	6.7%	5.1%
9	5.4%	7.1%
10	8.4%	13.7%

The holdout Brier score is 0.0338 and expected calibration error is 1.21 percentage points. The model is more defensible for ranking limited outreach capacity than for treating raw scores as stable probabilities.

Drift changes the question

Prevalence spikes to 18.9% in March 2018, falls to 1.2% in June, then rises again. Training, validation, and holdout prevalence are 7.85%, 7.07%, and 3.54%, respectively.

›Show the numbers

Monthly late-delivery prevalence — Late-delivery rate by Purchase month

Purchase month	Late-delivery rate
2016-10	0.8%
2017-01	3.0%
2017-02	2.6%
2017-03	4.5%
2017-04	6.4%
2017-05	2.8%
2017-06	3.0%
2017-07	2.8%
2017-08	2.9%
2017-09	4.4%
2017-10	4.1%
2017-11	12.4%
2017-12	7.4%
2018-01	5.7%
2018-02	14.0%
2018-03	18.9%
2018-04	4.5%
2018-05	6.6%
2018-06	1.2%
2018-07	3.4%
2018-08	6.3%

The unusually low June rate coincides with a wider median promised window, but that window narrows again in July and August while lateness rises. The target captures both fulfilment performance and a moving promise policy.

›Monitoring plan

Monthly monitoring should track prevalence, PR-AUC, top-k capture, calibration, feature missingness, and the promised-window distribution. A recent labeled window should be used for recalibration before probabilities drive cost-based decisions, and temporal validation should be repeated before a replacement model is accepted.

When is intervention worth it?

Ranking performance matters only if it supports a decision. The repository turns top-k capture into a replaceable scenario with 25% intervention effectiveness, 30 cost units per late delivery, and 1 unit per intervention.

Under the stated assumptions, targeting the highest-risk 1% produces +20 expected value units. Wider 5%, 10%, and 20% programs produce −162.5, −707.5, and −1,842.5 units.

›Show the numbers

Expected intervention value by targeting capacity — Expected net value (scenario units) by Share of held-out orders targeted

Share of held-out orders targeted	Expected net value
1%	20
5%	-162.5
10%	-707.5
20%	-1,842.5

›How the scenario is calculated

For each capacity, expected prevented late deliveries equal captured late deliveries multiplied by 25%. Avoided cost equals that result multiplied by 30; program cost equals targeted orders multiplied by 1. Net value is avoided cost minus program cost. The output also reports the break-even intervention cost at each capacity.

How the work is made reproducible

The analysis is packaged as software rather than a one-off notebook. The repository makes the data contract, transformations, evaluation, saved artifact, and generated evidence independently inspectable.

• ✓Modular Python package
• ✓Separate analysis and modelling cohorts
• ✓Leakage-safe preprocessing fitted on training data only
• ✓Saved model and adjacent metadata
• ✓Batch-scoring CLI with an explicit input contract
• ✓27 automated tests
• ✓Seven-table integration test
• ✓GitHub Actions continuous integration
• ✓Pinned dependency lockfile
• ✓Checksum-verified data downloader
• ✓Model card covering use, limits, and monitoring
• ✓Machine-readable metrics and output tables

›Reproduce, test, and score

The locked Python 3.12 environment installs the package in editable mode. The olist-delivery entry point rebuilds the processed cohorts, metrics, tables, figures, saved joblib pipeline, and JSON metadata. pytest runs the 27 tests in GitHub Actions on pushes and pull requests. After fitting, olist-score validates an engineered checkout-time CSV against the saved feature contract and writes order-level risk scores.

The downloader verifies checksums for all seven required source files before the pipeline runs. Generated CSV and JSON outputs let the claims on this page be audited without reading plots.

Data-quality appendix: the multi-seller anomaly

Olist records one customer-delivery timestamp per order even when an order is split across multiple sellers and parcels. That makes seller-level handoff and parcel completion ambiguous, so the main analysis is restricted to single-seller orders and the excluded population is audited separately.

The anomaly is substantial: two-seller orders show 46.6% dissatisfaction but only 1.0% recorded lateness, compared with 12.2% and 6.7% for single-seller orders. Their median promised window is only one day longer (25 versus 24 days), and reviews created before recorded delivery are less common (1.1% versus 5.1%), not more.

›Population audit

The modelling flow moves from 99,441 raw orders to 96,478 delivered orders, then checks complete and coherent timelines, usable items, and seller count before reaching 94,988 eligible modelling orders. Only after that are usable reviews required, yielding the 93,306-order dissatisfaction cohort. Multi-seller rows and their diagnostics remain in exported audit tables.

Where the analysis stops

• The dissatisfaction result is observational. Adjustment reduces measured confounding but does not make the 18× association causal.
• Reviews affect only the analysis cohort. Review responders may differ from non-responders; the late-delivery model itself does not require a review.
• Promise policy is embedded in the strongest feature. The model adds validation lift beyond the promise-only baseline, but its signal remains tied to how Olist set delivery expectations.
• Performance and calibration move over time. Monthly backtests, prevalence shifts, and top-band overprediction require monitoring and recalibration.
• Multi-seller attribution is unresolved. One order-level delivery timestamp cannot identify completion of multiple parcels.
• Intervention value is policy-dependent. The economics are illustrative until effectiveness and costs are measured in the intended setting.
• The data is historical. Straight-line distance is not route distance, and 2016–2018 marketplace operations may not represent current conditions.

The work supports retrospective analysis and a capacity-constrained risk-ranking prototype. It is neither a production-ready service nor a causal model.