Articles
One million nets, fifty districts, and a lot of missing data
ICS553 Machine Learning Essentials · Prosit 1 · Ashesi University. MLE Group 1: Germain, Iris, Joseph, Liza. Code and notebook: github.com/ASU-MICS-2028/mle-group-1-prosit-1
The brief
Our first Prosit in Machine Learning Essentials placed us as analytics consultants advising Ghana's National Malaria Elimination Programme. A limited consignment of insecticide-treated bed nets has arrived in the country. Where should those nets go, and how much confidence can we realistically place in the district-level estimates guiding that decision?
There was no single obvious answer. Our evaluation centered on four critical choices: which statistical estimator we selected, how we quantified uncertainty, how we guarded against sampling bias and data leakage, and which specific districts we prioritized. Over three weeks, our deliverables were:
- A fully reproducible Jupyter notebook in a GitHub repository covering data preparation, modeling, and a leakage audit
- A 15-minute Resource Allocation Proposal delivered to a mock Ministry of Health evaluation panel, supported by probability heatmaps
- A one-page individual reflection including an AI usage declaration
We modeled an allocation pool of 1,000,000 nets. While the priority ranking of districts does not depend on this total, the exact number of nets allocated to each district scales with population.
Why the obvious plan fails
The most straightforward approach is to direct nets to whichever districts report the highest absolute number of malaria cases. However, raw case counts reward districts that have functioning clinics, reliable electricity, and staff who diligently record every visit. A remote, under-resourced district whose sole rural clinic runs out of paper logbooks reports few cases and falls to the bottom of the list. When you sort by raw volume, resources follow paperwork rather than disease burden.
To counter this, we established three core principles before building any predictive models:
| Choice | What we picked | Why |
|---|---|---|
| Target Population | People exposed to malaria who do not currently own a bed net | Delivering a fourth bed net to a household that already has three does not save lives. |
| Analysis Unit | Net allocations per district across the 50 northern districts with monthly surveillance records (2014 to 2017) | Matches the geographic resolution of available routine clinical data. |
| Prioritization Metric | Unprotected rate = cases per person × (1 - net coverage), ranked by the upper bound of its 95% confidence interval | Accounts for population scale, existing net ownership, and measurement uncertainty. |
The data we actually had
We worked with three primary datasets: district malaria case totals from Ghana Health Service routine clinical surveillance (2014 to 2017), an anonymized household extract from the 2022 Demographic and Health Surveys (DHS) Malaria Indicator Survey (17,933 households grouped into 618 village clusters), and published regional malaria indicators. We also incorporated GeoBoundaries shapefiles to construct spatial maps.
The project prompt suggested that the district surveillance file would contain climatic variables (rainfall, temperature, elevation) and that the household survey would include child rapid blood-test results. Neither was present in the provided files. Rather than fabricating synthetic features, we restricted our analysis strictly to verified data: our regression relied on administrative regions and indoor residual spraying history rather than climate, while our prevalence bootstrap leveraged official regional blood-test benchmarks.
Historical administrative boundaries presented another hurdle. In 2018, Ghana partitioned the former Northern Region into three separate regions: Northern, North East, and Savannah. Because the 2014 to 2017 surveillance records labeled all these districts under "Northern", we constructed an administrative crosswalk mapping historical district boundaries to the 2021 administrative map (resolving 17 name discrepancies and 5 district splits). This ensured every district was assigned to its modern administrative region and linked to corresponding regional net ownership data.
| Metric | Value |
|---|---|
| Districts with monthly clinical surveillance | 50 |
| Districts with zero routine clinical records | 205 (out of 260) |
| Variance-to-mean ratio in district counts | 77,000x |
| Evaluated configurations in experiment matrix | 168 |
District case counts are right-skewed, and dividing by population does not remove the skew. Source: GHS routine surveillance 2014–17.
The decisions that mattered
Negative Binomial over Poisson
A Poisson distribution assumes that variance equals the mean. In our district surveillance data, the variance was roughly 77,000 times larger than the mean. If you fit a Poisson model where lambda equals the sample mean, it predicts that 99.7% of districts should fall within roughly plus or minus 1,326 cases of 195,640. In reality, only 2 out of 50 districts fell within that narrow interval. A Poisson model calculated the probability of observing Bolgatanga (a district with well-documented caseloads) as virtually zero.
A Poisson model with λ set to the mean expects every district inside the red sliver. Only 2 of 50 are.
The Negative Binomial distribution solves this by incorporating an over-dispersion parameter, alpha, such that variance = mu + alpha * mu^2. Calculating directly from the sample mean and variance gave an initial estimate of alpha around 0.39. Fitting the model yielded an alpha of 0.42 (95% CI 0.26 to 0.57), while the Akaike Information Criterion (AIC) plummeted from 4,419,517 down to 1,309. Introducing administrative region as a covariate brought alpha down to 0.14, demonstrating that regional geographic disparities explained most of the remaining over-dispersion. We included log(population) as an offset term so the model predicted case rates per capita rather than raw population scale.
Poisson cannot spread. The Negative Binomial (α = 0.42) can.
Resample clusters rather than individual households
The Malaria Indicator Survey interviews clusters of approximately 30 neighboring households per village. Because neighbors share local water sources, mosquito breeding habitats, and healthcare facilities, their malaria risk is correlated rather than independent.
When estimating net ownership in the Northern Region, a naive household bootstrap produced a 95% confidence interval width of 0.055. Resampling entire village clusters widened the interval to 0.120, more than doubling the measured uncertainty. In statistical terms, the 1,137 surveyed households carried about as much independent information as 230 unclustered households (reflecting a design effect of nearly 5). Our survey-weighted ownership estimate of 67.7% closely matched the official DHS benchmark of 67.6%, confirming that our weighting adjustments were accurate.
Same data, same estimate. Ignoring village clusters cuts the interval in half. Source: DHS 2022, 2,000 resamples.
Missing data is information in itself
Routine clinical records were available for only 50 districts. The remaining 205 districts (79% of Ghana) had no surveillance data at all. This absence is not random. Routine reporting breaks down in areas with weak transportation, unreliable power grids, and chronic medical staffing shortages, which are often the same regions where malaria transmission is most severe. Regions like Oti (23% child blood prevalence), Western (22%), Bono East (22%), and Ahafo (21%) show high malaria rates in national surveys despite having zero district-level surveillance entries. A district ranking based solely on clinic records completely misses them, so we flagged these regions separately using regional survey benchmarks.
Left: malaria prevalence by region. Right: the only districts with case data. Several high-prevalence regions in the south and centre have none. Click to enlarge.
Where malaria rates are highest, the survey sample is thinnest.
Within the 50 monitored districts, seven district-month entries were blank, and the original summary dataset had silently replaced them with zeroes. We imputed those missing values using each district's own historical median monthly count. This adjustment shifted district rates by only 1% to 5% and caused zero changes to our top-ten priority ranking.
Let uncertainty work in favor of vulnerable districts
Waiting for perfect certainty means directing resources only to well-documented clinics. Ranking districts by their upper confidence bound ensures that measurement doubt works in favor of neglected populations.
Every district's estimated unprotected rate is a confidence interval, not a single static number. Sorting by the sample mean ignores estimation uncertainty, while sorting by the lower bound penalizes districts that suffer from sparse or noisy record-keeping. By ranking districts by the upper bound of their 95% confidence interval, we give thinly documented communities the benefit of the doubt. This decision was a central theme across our team's individual reflections.
| District | Region | Lower Bound | Mean | Upper Bound | Recommended Nets |
|---|---|---|---|---|---|
| Nabdam | Upper East | 1.11 | 1.46 | 1.88 | 21,347 |
| Daffiama-Bussie-Issa | Upper West | 0.99 | 1.28 | 1.60 | 18,412 |
| Wa | Upper West | 1.03 | 1.29 | 1.56 | 60,663 |
| Lambussie-Karni | Upper West | 1.02 | 1.27 | 1.54 | 28,737 |
| Sissala East | Upper West | 0.83 | 1.14 | 1.51 | 28,192 |
| Nadowli-Kaleo | Upper West | 0.82 | 1.02 | 1.26 | 27,554 |
| Pusiga | Upper East | 0.85 | 1.04 | 1.22 | 26,028 |
| Lawra | Upper West | 0.75 | 0.97 | 1.21 | 23,366 |
| Sissala West | Upper West | 0.74 | 0.95 | 1.21 | 20,625 |
| Nandom | Upper West | 0.68 | 0.87 | 1.10 | 17,598 |
Top ten districts ranked by the upper bound of their unprotected malaria rate (confirmed cases per person from 2014 to 2017, multiplied by the proportion of households lacking bed nets). The right column shows each district's population-weighted share of 1,000,000 nets. Sissala East has a lower point estimate than Lambussie-Karni but wider variance, which rightfully keeps it near the top of the priority list. Data sources: GHS routine surveillance 2014 to 2017, DHS 2022, and our 48-month bootstrap model.
Our final proposal placed Nabdam at the top of the priority list, followed by a concentrated cluster across the Upper West: Daffiama-Bussie-Issa, Wa, Lambussie-Karni, and Sissala East.
Burden, uncertainty and the final allocation. Districts dark on both of the first two maps get the most nets. Hatched districts have no case data and cannot be ranked. Click to enlarge.
Keep net coverage out of predictive models
Bed net ownership looked like an obvious input feature for regression models, but including it introduced two severe methodological traps. First, net coverage was measured in 2022 while clinical caseloads spanned 2014 to 2017, meaning the model would use future data to explain historical outcomes. Second, public health campaigns distribute bed nets precisely where malaria burdens are already elevated, making net ownership a consequence of disease rates rather than an independent cause. We deliberately excluded net coverage from our predictive models, incorporating it only during the final resource allocation calculation.
My part: pipeline design and the leakage audit
I maintained our team's GitHub repository, ensuring the code remained fully reproducible: pinning package versions, fixing random seeds, keeping licensed DHS data out of git, and configuring Google Colab workflows so teammates could run experiments without manual file transfers. I managed our notebook iterations as the analysis developed, from exploratory drafts through the experiment matrix and boundary reconciliations.
My primary analytical responsibility was designing the supervised machine learning pipeline and conducting the data leakage audit (notebook sections 7 through 9). I audited the code line by line, implemented scikit-learn Pipeline and ColumnTransformer workflows, validated all transformation statistics, and presented our methodology and findings on slides 7 and 8 during our defense.
Our predictive task was classifying whether a household owned a bed net using features accessible to planners ahead of a campaign: household wealth quintile, number of eligible children, region, and urban versus rural status. We partitioned data prior to calculating any summary statistics: holding out 124 of 618 village clusters without cluster overlap, and stratifying districts across administrative regions (40 training, 10 testing). All imputation, feature scaling, and one-hot encoding operations were encapsulated inside pipeline objects, ensuring preprocessing steps learned parameters strictly from training folds. Grouped five-fold cross-validation yielded an ROC AUC of 0.676 ± 0.027, with a held-out test AUC of 0.660.
To prove the importance of rigorous pipeline design, we intentionally built four leaky variants and scored them against the same test partition:
| Leak Type | Leaky Implementation | Clean Implementation | Leaky AUC | Clean AUC | Performance Gap |
|---|---|---|---|---|---|
| Pooled Preprocessing | Impute and scale across train and test sets simultaneously | Fit preprocessors strictly on training fold | 0.6604 | 0.6602 | +0.0002 |
| Group Leakage | Standard random K-fold across households | GroupKFold preserving village clusters | 0.6775 | 0.6759 | +0.0016 |
| Post-Treatment Feature | Include children_under_net_last_night |
Exclude target-dependent feature | 0.7497 | 0.6602 | +0.0895 |
| Target Encoding | Replace cluster IDs with mean target outcome | One-hot encode region without cluster labels | 0.7207 | 0.6602 | +0.0605 |
Two lessons stood out. First, a child can only sleep under a mosquito net if the household possesses one, so including that feature handed the model the answer, creating nearly +0.09 of artificial AUC. Second, the pooled-preprocessing leak produced an almost undetectable score difference. While tempting to dismiss as harmless, it is flawed in principle; the difference only appeared small because this specific survey had few missing entries. Computing a mean, median, or standard deviation is fundamentally learning a model parameter, which reshaped how I think about data preprocessing.
Our experiment matrix highlighted this dynamic clearly. Out of 64 supervised model configurations, only 4 were completely clean, and not one of the ten highest-scoring models was valid. Every model scoring above 0.70 AUC owed its performance to data leakage. On verified clean features, gradient boosted trees scored 0.004 lower than simple logistic regression.
64 model configurations. Every high score contains a leak.
What actually moved the list
We evaluated 168 configurations across four analytical dimensions: distribution families, uncertainty estimation techniques, supervised model variants, and allocation formulas. The allocation analysis provided the most practical guidance for the panel:
- Five districts showed consistent priority: Wa, Nabdam, Lambussie-Karni, Daffiama-Bussie-Issa, and Pusiga remained in the top ten across more than 80% of our 48 rate-based configurations.
- Statistical fine-tuning had minimal impact: Choosing between mean and upper-bound ranking, varying imputation approaches, or changing interval methods only swapped 1 or 2 districts out of 10. Empirical Bayes shrinkage produced no noticeable effect because all 50 districts had complete 48-month records with case counts in the tens of thousands.
- Bed net coverage was the dominant decision factor: Omitting net coverage from the formula shifted four districts out of the top ten: Lawra, Nadowli-Kaleo, Nandom, and Sissala West dropped out, replaced by Bolgatanga, Builsa South, Garu-Tempane, and Kasena-Nankana West.
Left: overlap with our top ten under other estimators and ranking statistics. Right: five districts make the top ten in more than 80% of configurations. Click to enlarge.
Our key takeaway for the panel was that complex modeling delivers marginal improvements compared to investing in fresh, reliable, district-level net coverage surveys.
Reading official numbers carefully
A core requirement of the brief was evaluating data provenance. The World Health Organization's reported figures of 263 million global cases and 597,000 deaths in 2023 are mathematical model estimates carrying roughly ±25% uncertainty for countries like Ghana. While useful for broad national policy, they lack the granularity needed to allocate resources across individual districts.
Furthermore, Ghana's officially reported clinical cases dropped dramatically from 11.2 million in 2018 to 6.7 million in 2019. This decrease did not reflect a sudden 40% reduction in malaria; rather, the public health service changed diagnostic reporting standards, halting the inclusion of unconfirmed "presumed" cases. A machine learning model trained naively across that reporting shift would learn an artificial clinical breakthrough.
Left: WHO's estimates come with wide intervals. Right: the 2019 drop in reported cases is the end of counting presumed cases, not a fall in malaria.
How we used AI
We utilized Claude Code extensively and documented its role transparently. The assistant helped draft our project plan, generate baseline notebook code and explanatory documentation, structure the experiment matrix, and render charts and slide drafts. It also assisted us in refining conceptual definitions around over-dispersion, Negative Binomial regression, clustered bootstrap methods, missingness mechanisms, and data leakage.
The tool operated locally and interacted solely with aggregate statistical summaries. It examined dataset headers and two sample rows during initial inspection, and nothing more. No raw survey records were ever uploaded or committed. Each team member independently reviewed, validated, and defended their assigned component; my responsibility was the machine learning pipeline and leakage audit.
What I am still unsure about
The reliance on regional net coverage remains my biggest concern. The 2022 survey data was recorded five years after the clinical surveillance period and applies a single uniform coverage rate across every district in a region. Yet that single number decides four of the spots in our top ten. The allocation list is only as reliable as that coverage estimate.
I also continue to think about whether "unprotected disease burden" is the most effective target. Prioritizing "cases averted per distributed net" is arguably more sound from a public health perspective, but calculating it requires localized transmission dynamics and usage compliance data that were not available.
Where this could go next
- Acquire contemporary district-level coverage surveys: Our sensitivity analysis indicates that this is the single most valuable missing dataset. Even a small, recent district-level survey would strengthen allocations more than any modeling adjustment.
- Estimate prevalence for unmonitored districts: Small-area estimation techniques could combine regional survey prevalence with district covariates to provide uncertain, ranked estimates for the 205 unmonitored districts rather than omitting them entirely.
- Incorporate spatial modeling across borders: Because neighboring districts share climate, topography, and mosquito populations, spatial models (such as CAR or BYM random effects) could borrow statistical strength across borders to narrow estimation intervals.
- Optimize for health impact and delivery cost: With richer utilization and transmission data, the optimization objective could transition from simple burden to averted clinical cases, factoring in transportation and supply-chain logistics.
- Refine team research workflows: In future projects, reading data dictionaries thoroughly before writing code will save time. Our largest delay was discovering mismatches between the provided dataset and assignment brief. We also agreed to use shared kanban task boards and daily standups rather than relying on last-minute deadline syntheses.
Run it yourself
Everything is organized within a single notebook, notebooks/prosit1.ipynb, which runs top to bottom in approximately six minutes. It guides the reader from data ingestion through the leakage audit to the final allocation table, with Section 15 containing the full experiment matrix. Generated figures and summary tables are version-controlled in the repository under outputs/, allowing you to explore the findings without running code.
git clone https://github.com/ASU-MICS-2028/mle-group-1-prosit-1.git
cd mle-group-1-prosit-1
uv venv --python 3.12 && uv pip install -r requirements.txt
.venv/bin/jupyter nbconvert --to notebook --execute --inplace notebooks/prosit1.ipynb
The data/ directory is excluded from version control because it contains licensed DHS survey microdata, requiring authorized access to the Ghana MIS dataset to execute raw data ingestion.
Repository: github.com/ASU-MICS-2028/mle-group-1-prosit-1. Special thanks to Dr. Albert Dede, along with Germain, Iris, and Liza.