⚡ Predicting U.S. Power Outage Severity

The Question

Power outages disrupt hospitals, halt transit, and strand millions. When one starts, utilities face an immediate resource-allocation problem: how bad will this be? This project asks whether that question can be answered at the moment an outage begins — using only information available before a single repair crew is dispatched.

The dataset covers 1,534 major outage events across the continental United States from January 2000 to July 2016, compiled from U.S. Department of Energy (DOE) records. Each row is one event, with geographic, demographic, climatic, and cause information recorded at the time of occurrence.

Key Variables

Column	Description
`OUTAGE.DURATION`	Duration of the outage (minutes)
`CUSTOMERS.AFFECTED`	Number of customers impacted
`CAUSE.CATEGORY`	Primary cause — severe weather, equipment failure, etc.
`U.S._STATE`	State where the outage occurred
`CLIMATE.CATEGORY`	Climate conditions at time of outage (normal / cold / warm)
`POPULATION`	State population
`POPPCT_URBAN`	Share of state population in urban areas (%)
`NERC.REGION`	North American Electric Reliability Corporation region

Understanding the Data

The raw dataset arrived as an Excel file with metadata rows, mixed headers, and 57 columns — most of them irrelevant. Getting it into shape required significant preprocessing before a single chart could be drawn.

Cleaning Steps

Column names were extracted from row 4 of the sheet (with units from row 5), 35 irrelevant columns were dropped, and OUTAGE.DURATION and CUSTOMERS.AFFECTED were coerced to numeric with invalid entries treated as missing. Date strings were parsed to datetime objects, and 'NA' string literals replaced with proper NaN values. Missing customer counts were filled with 0, on the assumption that unreported outages had minimal impact.

Feature Engineering

Seven derived features were created to capture patterns not present in the raw columns:

OUTAGE_SEASON — season of occurrence based on start date
IS_WEEKEND — boolean flag for weekend outages
CUSTOMER_DENSITY — ratio of customers to population
URBANIZATION_RATIO — decimal urban population share
POPULATION_DENSITY — weighted blend of urban and rural densities
SEVERITY_CATEGORY — Small (<10K), Medium (10K–100K), Large (>100K)
IS_EXTREME_WEATHER — boolean flag for weather-related causes

Summary Statistics

Metric	Value
Average outage duration	2,625 min (43.75 hours)
Median outage duration	701 min (~11.7 hours)
Median customers affected	30,534
Most common cause	Severe weather
Most affected season	Summer

The gap between mean (2,625 min) and median (701 min) duration signals a heavily right-skewed distribution — a small number of catastrophic outages are pulling the average up dramatically.

Univariate: Duration Distribution

Most outages resolve within 4,000 minutes, but a long tail of extended events — hurricanes, major ice storms, infrastructure failures — extends well beyond that. The dataset skews toward larger events; small, localized outages are underrepresented or unreported.

Distribution of outage durations (minutes). Right-skewed with a long tail of extreme events.

Distribution of outages by severity category. A large share of events are unclassified or unreported at the customer level.

Bivariate: Duration by Cause

Severe weather events produce the longest outages with the highest variability. Intentional attacks tend to be shorter and more predictable. Equipment failures fall in the middle — moderate duration, less variance than weather.

Outage duration by cause category. Severe weather dominates in both duration and spread.

The relationship between customers affected and duration is weak — larger outages don't necessarily last longer. Scope (how many customers) and duration (how long) appear driven by different factors: scope by grid interconnectedness and population density; duration by damage complexity and repair logistics.

Customers affected vs. outage duration. Weak positive correlation — scope and duration are largely independent.

Seasonal distribution of outage events. Summer peaks align with peak demand and storm season.

State-Level Aggregates

State	Avg Customers Affected	Avg Duration (min)
Florida	282,939	4,095
South Carolina	251,913	3,135
Illinois	198,026	1,602
District of Columbia	175,238	4,304
Texas	165,227	2,705

Florida leads in both impact and duration — a predictable result given its exposure to Atlantic hurricanes and subtropical storm systems.

Cause Category Aggregates

Cause	Mean Duration	Median Duration	Mean Customers
Severe Weather	3,884 min	2,460 min	177,206
Fuel Supply Emergency	13,484 min	3,960 min	~0
System Operability	729 min	215 min	137,941
Equipment Failure	1,817 min	221 min	50,968

Why Is Data Missing?

Not all missing data is the same. Some columns are blank because the information wasn't recorded; others are blank because the value itself is informative.

NMAR Analysis

HURRICANE.NAMES is 95.31% missing — but this missingness is not random. Hurricane names are only recorded when the outage was actually caused by a named hurricane. A blank entry means "not a hurricane event," which is itself information. This makes the column NMAR (Not Missing At Random).

To convert it to MAR, we'd need external data — wind speed measurements, barometric readings, official weather service classifications, or storm tracking data for unnamed systems.

Missingness Dependency Testing

To determine what drives missingness in CAUSE.CATEGORY.DETAIL, two permutation tests were run:

Test 1

Question Does CAUSE.CATEGORY.DETAIL missingness depend on CAUSE.CATEGORY?

Observed diff 0.885

P-value 0.000 → Reject H₀. Missingness strongly depends on cause category — some causes naturally have more detailed sub-classifications.

Test 2

Question Does CAUSE.CATEGORY.DETAIL missingness depend on OUTAGE_DAYOFWEEK?

Observed diff 0.122

P-value 0.088 → Fail to reject H₀. No systematic reporting bias based on day of week.

Do Weather Outages Last Longer?

Before building a model, one question demanded a direct answer: do severe weather outages actually last longer than equipment failure outages, or does the difference just look that way in the raw aggregates?

H₀ Mean duration is the same for severe weather and equipment failure outages.

H₁ Severe weather outages last longer on average.

Test Statistic Difference in mean duration (Weather − Equipment)

Significance α = 0.05

Permutations 10,000

Permutation distribution of mean duration differences under the null hypothesis. The observed difference of 2,067 minutes falls well into the tail.

Result: p < 0.001. Severe weather outages last ~34.5 hours longer than equipment failure outages on average. The null hypothesis is rejected. Weather damage typically affects larger areas and requires more complex, coordinated repairs than localized equipment failures.

Can We Predict Severity at the Start?

The prediction task is binary: given information available at the moment an outage begins, classify whether its duration will be Short (below the log-duration threshold of 6) or Long (at or above it). This corresponds roughly to outages under vs. over ~400 minutes.

The constraint is real: only features knowable before resolution are allowed — geography, timing, cause (when immediately apparent), demographics, and climate. Duration and customers affected cannot be used as features since those aren't known until after the fact.

Features Available at Prediction Time

Geographic: state, NERC region
Temporal: month, season, day of week, weekend flag
Cause: cause category, extreme weather flag
Demographic: population, urbanization rate, customer density
Climate: climate category, anomaly level

Early severity classification helps utilities pre-position repair crews, set realistic restoration timelines for customers, and coordinate with emergency services — all within the first minutes of an event.

Starting Point: Decision Tree

The baseline uses a Decision Tree Classifier with 11 features: six quantitative (month, anomaly level, customers affected, urban percentage, customer density, population density) and five nominal (climate category, cause category, severity category, season, extreme weather flag).

Numeric features were imputed with column means; categoricals used most-frequent imputation followed by one-hot encoding.

Decision Tree · Baseline

76%

Overall Accuracy

Class	Precision	Recall	F1
Long	0.80	0.78	0.79
Short	0.71	0.73	0.72

Assessment

Reasonable first result — 76% accuracy with a single tree and no hyperparameter tuning. The model predicts Long outages slightly better (precision 0.80) than Short ones (0.71).

The gap between classes suggests the model is leaning on features that more clearly signal long events — weather cause and seasonal patterns — while struggling with borderline short outages.

Random Forest with Tuning

Two improvements drove the upgrade from Decision Tree to Random Forest: better preprocessing and hyperparameter search over a wide grid.

CUSTOMER_DENSITY received a log transformation to handle its right-skewed distribution. Most other numeric features were standardized with StandardScaler. A 5-fold grid search over 240 combinations of tree count, depth, and split thresholds found the optimal configuration.

Best Hyperparameters

Parameter	Value
`n_estimators`	300
`max_depth`	20
`min_samples_split`	5
`min_samples_leaf`	1

Decision Tree · Baseline

76%

Overall Accuracy

Class	Prec	Rec	F1
Long	0.80	0.78	0.79
Short	0.71	0.73	0.72

Random Forest · Final

80%

Overall Accuracy

Class	Prec	Rec	F1
Long	0.82	0.82	0.82
Short	0.76	0.76	0.76

Accuracy improved +4 points (76% → 80%). Precision on Long outages: +0.02. Precision on Short outages: +0.05. The class gap narrowed — the model became more balanced, not just more accurate on the easy class.

Does Urbanization Affect Model Fairness?

A model that performs well on average can still systematically underperform for specific groups. Here, the question is whether the Random Forest is equally reliable for high-urbanization states (≥50% urban population) versus low-urbanization ones (<50%).

Metric Precision for Long outage predictions

H₀ Precision is equal across high- and low-urbanization areas.

H₁ Precision differs between the two groups.

Observed diff −0.1796 (high-urbanization areas have lower precision)

P-value 0.7018 (from 10,000 permutations)

Result: fail to reject H₀. With p = 0.70, there is no statistically significant evidence of unfairness across urbanization levels. The observed precision gap is well within the range expected by chance. The model achieves fairness parity between urban and rural contexts for predicting long-duration outages.

What We Learned

Three findings held up across every stage of the analysis:

Finding 01

Severe weather dominates. It causes the most widespread outages and the longest ones. Weather-driven events last ~34.5 hours longer than equipment failures on average — a difference confirmed by permutation test at p < 0.001 with 10,000 draws.

Finding 02

Geography and demographics carry signal. State, NERC region, urbanization rate, and population density all contribute to the model. Outage patterns are not uniform across the grid — where an outage happens matters as much as why.

Finding 03

Early classification is feasible. Using only information available at the moment an outage begins, the final Random Forest correctly classifies severity 4 out of 5 times — and does so without systematic bias across urbanization levels.