🌡️ Protostar Challenge: Climate Risk Intelligence
Protostar Challenge
Climate Risk Intelligence
AI systems to analyze extreme weather trends and recommend climate adaptation strategies.
210 weather stations, 1851–2018 · data: CMU-OSPO/protostar-climate-train-to-2018
1900 → 2018: each stripe is one year, average temperature of the 210 cities vs. 1961–1990
- Nov 2Challenge opens
- Nov 13Deadline, 11:59 PM ET
- After the deadlineFinal Score
- ThenWinners
The challenge
The Urban Climate AI Challenge: The 2019–2025 Test
A major Carnegie Mellon climate study (Lai & Dzombak) collected weather records for 210 U.S. cities from ~1900 up through 2018.
Your job is simple: put yourself at the end of 2018 and predict the actual yearly weather for each city from 2019 through 2025, without using any data that reveals how those years turned out. You are welcome to bring in other data too (e.g., weather records up to 2018 from other sources, census, land cover, building stock); the data rule below says what can feed your forecasts. We will compare your predictions against the weather actually recorded at airport and city weather stations during those years. Then, use what you learned to propose one policy idea that helps a city, or the country, cope with a climate risk.
Data rule (honor code): to forecast temperature, precipitation, hottest day and CDD, use only records up to 2018-12-31: the challenge dataset, or other sources cut at the same date. Do not use any 2019–2025 data that responds to the weather as an input to your forecasts: weather observations from any source, and anything driven by them (e.g., streamflow, air quality, energy demand, plant or disease data). Other data (e.g., census, land cover, energy, health) is welcome for your analysis and policy idea; list every external source in your README. Examples are in the FAQ (Data & submission tab).
Dates: November 2 – November 13, 2026.
Teams
Registration is team-based. Each participant can be in one team only.
What your model predicts
For each of the 210 cities, predict a single number for each year from 2019 through 2025 for four metrics:
Average Annual Temperature
°F
annual_avg_tempThe overall yearly average temperature.
Total Annual Precipitation
inches
total_precip_inTotal liquid moisture (rain plus melted snow) recorded for the year.
Hottest Day of the Year
°F
hottest_day_tempThe single highest daily peak temperature recorded that year.
Cooling Degree Days (CDD)
°F-days
cddA single yearly number that adds up how many degrees each day's average temperature rose above 65 °F; used to estimate how much air conditioning a city needs.
How CDD is computed
$$ \text{CDD} = \sum_{d=1}^{D} \max\left(0,\ \frac{T_{\max,d} + T_{\min,d}}{2} - 65\right) $$
- D: the number of days in the year (365 or 366).
- Tmax,d and Tmin,d: that day's highest and lowest temperatures, in °F.
- max(0, …): a day at or below 65 °F adds nothing.
Example: a day with a high of 90 °F and a low of 70 °F averages 80 °F, so it adds 80 − 65 = 15 to the year's CDD.
No probabilities or intervals required: just your model's best guess for each value.
How the forecasts are scored
Each metric is scored with Mean Absolute Error (MAE), the average gap between your prediction and what actually happened (lower is better):
$$ \text{MAE} = \frac{1}{N} \sum_{i=1}^{N} \left| \text{Predicted}_i - \text{Actual}_i \right| $$
The four metrics use different units, so each one is compared against the 30-year-average baseline. The Score says how much better than that baseline your model was:
$$ S_m = 1 - \frac{\text{MAE}_{m,\ \text{your model}}}{\text{MAE}_{m,\ \text{30-year average}}} \qquad \text{Score} = 100 \times \max\left(0,\ \frac{S_{\text{temp}} + S_{\text{precip}} + S_{\text{hot}} + S_{\text{cdd}}}{4}\right) $$
Score = 0: no better than the 30-year average. Score = 100: a perfect forecast. Higher is better.
The two are kept separate on purpose: a model tuned to 2019–2021 does not necessarily predict later years well.
Note: a few rows are not scored: years where the station missed more than 10 days, and the few cities without enough past data for a baseline. Predict every row anyway.
Live leaderboard
Coming soonComing soon: the leaderboard is not open yet. It will appear in a Leaderboard tab on this Space; submissions made before then are kept and will be scored when it opens.
Your Live Score (2019–2021) can take up to 24 hours to appear on the leaderboard. Use it to check how your model is doing as you improve it; the final ranking and the winners are decided after the deadline (see Judging). If your score has not appeared after 24 hours, post in the Space's Community tab.
What to beat
Three simple reference baselines:
- 1Last Year Repeated
every year from 2019 to 2025 repeats the city's last valid value (2018 for most cities).
- 230-Year Average
every year equals the city's average from 1989 to 2018.
- 3Linear Trend
a straight line through the last 30 years, extended out to 2025.
Target: if your model cannot beat the 30-year average, it is probably overfitting to historical noise. Hottest day and precipitation are noisy: matching the 30-year average on them is already reasonable; most of the gain is in temperature and CDD.
CDD target: beat the 30-year average (level 1), then the linear trend (level 2). In a practice run (train up to 2010, predict 2011–2018), the 30-year average scores an MAE of 213 °F-days and the trend line 177 °F-days. Differences under 10 °F-days count as a tie.
Policy idea
Write a short Policy Idea section (half a page at most) in your submission's README.
The task, in one sentence: using what you learned from the data, suggest one idea that could help a city, or the country, cope with a climate risk sustainably. Answer these three questions however you like:
- 1What risk did you notice?
E.g., more cooling days, hotter peak days, wetter or drier years.
- 2What would you do about it, and who would do it?
A city, a state, the federal government, a utility, a school, a business: your choice.
- 3Why is it sustainable?
It helps without making emissions or costs worse in the long run.
Any city, group of cities or the whole U.S.; any risk in the data; bold ideas welcome. Inspiration: EPA Heat Island Reduction, heat.gov.
What you submit
A link to your Hugging Face dataset, containing your forecasts (submission.csv), a README with your method, policy idea and feedback to the ecosystem, and your code.
You can revise and submit as many times as you like until the deadline. Your latest valid submission counts. Details are in the Data & submission tab.
submission.csvYour forecasts: a number for each of the 5,880 rows
README.mdMethod · Policy idea · Feedback to the Ecosystem
code/The notebooks or scripts that produce your forecasts
paste the link; its current version (commit) is recorded
your latest submission counts
Judging
Final winners are selected after the deadline, by a panel of 3–4 judges, on six criteria:
Meaningful use or rigorous attempted reuse of ProtoStar artifacts; useful extension, diagnosis, or recommendation for future researchers. Evidence about discovery and reuse friction is scored under Feedback to the Ecosystem, not here.
Successful combination or portability across repositories, institutions, agencies, environments, or domains, or rigorous evidence showing why integration fails; value and limits of common standards and machine-actionable formats.
Ability for others to reproduce the work using provided materials; for submissions declared in advance as negative results, a well-documented attempt that isolates why reproduction is not possible with the available artifact, its metadata, or its documented environment. Recommendations arising from that finding are scored under Feedback to the Ecosystem, not here.
Useful knowledge, new research capability, improved decision-making, or meaningful public benefit.
Specific, actionable evidence about barriers, failed reuse or reproduction, metadata issues, licensing, documentation, infrastructure, policy, or other opportunities for improvement.
Novel methods, AI approaches, technical solutions, or combinations of resources that advance the goals of the challenge.
Judges consider your Score and method under Scientific, Research, or Societal Value and Technical Creativity, and your policy idea under Scientific, Research, or Societal Value.
Write your Feedback to the Ecosystem in its own section of your README: what got in the way (data, metadata, documentation, licensing, tools) and what would fix it.
If your main finding is that reuse or reproduction did not work, set Negative result: yes at the top of your README; the version recorded at your latest submission counts.
Explain not just what you built, but why it is innovative, who would use it, and whether it could actually work in the real world.
Prizes
$500 in Hugging Face credits
Second place team
$1,000 in Hugging Face credits
First place team
$250 in Hugging Face credits
Third place team
$2,500, shared among the team members
If the top-placing CMU team also places in the top three, it receives that place's Hugging Face credits as well.
Data
One dataset, with daily and yearly weather for 210 U.S. cities up to 2018-12-31:
CMU-OSPO/protostar-climate-train-to-2018
| Subset | Rows | Content |
|---|---|---|
daily |
10,245,754 | Daily tmax, tmin (°F) and prcp (inches) per city, 1851–2018 |
annual_indices |
28,052 | One row per city and year: averages, totals, extremes, cdd, and Miss * counts of missing days |
city_info |
461 | Station ID, city Name, latitude, longitude, and the stations joined for each city |
from datasets import load_dataset
REPO = "CMU-OSPO/protostar-climate-train-to-2018"
daily = load_dataset(REPO, "daily", split="train").to_pandas()
annual = load_dataset(REPO, "annual_indices", split="train").to_pandas()
cities = load_dataset(REPO, "city_info", split="train").to_pandas()
Metric codes and their columns in annual_indices: annual_avg_temp = Avg Temp, total_precip_in = ToT Prcp, hottest_day_temp = Max Tmax, cdd = cdd.
The dataset page describes each subset and its columns.
How to submit
Your submission is a Hugging Face dataset. Build it like any dataset on the Hub, then submit its link.
- Create your dataset by copying the template CMU-OSPO/protostar-climate-submission-template into your Hugging Face account.
- Add your forecasts in the template's
submission.csv: the city, year and metric of all 5,880 rows are already filled in; fill inpredicted_value(format below).code/check_submission.pyin the template runs the same checks as the Space. - Write your
README.md: your method in a few lines, your Policy Idea and your Feedback to the Ecosystem (see the About tab). It is your write-up. - Add your code in a
code/folder (datasets are git repos, so code uploads like any other file). - Make the dataset public and paste its link, e.g.
https://huggingface.co/datasets/your-name/your-dataset, in the Submission tab.
When you submit, the Space checks that the dataset is public, contains submission.csv and README.md, and that submission.csv has a number for every one of the 5,880 rows; it then records its current version (commit). You can update your dataset and submit again until the deadline; the version recorded at your latest submission counts, so submit again after every change you want judged.
Forecast file format
submission.csv, 4 columns, one row per city × year × metric:
city_id,year,metric,predicted_value
USW00094823,2019,annual_avg_temp,52.4
USW00094823,2019,total_precip_in,38.2
USW00094823,2019,hottest_day_temp,91.0
USW00094823,2019,cdd,845.0
USW00094823,2020,annual_avg_temp,53.1
...
USW00024233,2025,cdd,310.0
city_id: the stationIDfromcity_info(e.g., Pittsburgh =USW00094823, Seattle =USW00024233).- Total rows: 210 cities × 7 years × 4 metrics = 5,880 rows.
Starter code
A working model in under 30 lines (scikit-learn):
import pandas as pd
import numpy as np
from datasets import load_dataset
from sklearn.ensemble import RandomForestRegressor
# 1. Load the yearly data (up to 2018)
ds = load_dataset("CMU-OSPO/protostar-climate-train-to-2018", "annual_indices")
train_df = ds["train"].to_pandas()
# Metric codes and their columns in annual_indices:
# annual_avg_temp = "Avg Temp", total_precip_in = "ToT Prcp",
# hottest_day_temp = "Max Tmax", cdd = "cdd"
target = "Avg Temp"
# 2. Build lag features: each city's value 1, 2 and 3 rows (years) earlier
train_df = train_df.sort_values(["ID", "Year"])
for k in (1, 2, 3):
train_df[f"lag_{k}"] = train_df.groupby("ID")[target].shift(k)
features = ["lag_1", "lag_2", "lag_3"]
fit_df = train_df.dropna(subset=features + [target])
model = RandomForestRegressor(n_estimators=100, random_state=42)
model.fit(fit_df[features], fit_df[target])
# 3. Predict 2019-2025 iteratively
# ... generate predictions and save to submission.csv
# 4. Check your error locally against validation years:
def compute_mae(actual, predicted):
return np.mean(np.abs(actual - predicted))
FAQ
Can I use data other than the challenge dataset? Yes, with one line drawn by use. Your forecasts may only use records up to 2018-12-31. After 2018, anything that responds to the weather gives away the answers, even when it is not weather data. The same data is welcome in your analysis and policy idea, where real impacts are exactly what a policy targets.
| Data (2019–2025) | Why it gives away the weather | Leaks | Forecast input? | Analysis / policy? |
|---|---|---|---|---|
| Temperature or rain from any source (NOAA, weather sites, ERA5 reanalysis, satellites) | It is the answer | All four metrics | ❌ | ✅ |
| Baseball home runs per game | Hot air is thinner, so balls fly farther (Callahan et al., 2023, BAMS, doi:10.1175/BAMS-D-22-0235.1) | Hottest day, CDD | ❌ | ✅ |
| River streamflow (USGS) | Rain becomes runoff within days | Precipitation | ❌ | ✅ |
| Ozone exceedance days (EPA) | Ozone forms faster in heat and sun | Hottest day, CDD | ❌ | ✅ |
| Spring leaf-out and bloom dates (USA-NPN) | Plants respond to accumulated warmth | Average temperature | ❌ | ✅ |
| West Nile virus cases (CDC) | Mosquitoes multiply faster in heat | CDD | ❌ | ✅ |
| Electricity demand (EIA), searches for "AC repair" | Air conditioning drives summer peaks | CDD | ❌ | ✅ |
| Census, land cover, tree canopy, building stock | Does not respond to the year's weather | — | ✅ | ✅ |
| Weather records up to 2018 from other sources | Same cutoff as the challenge dataset | — | ✅ | ✅ |
When in doubt: if a 2019–2025 series would be different in a hotter or wetter year, keep it out of your forecasts. List every external source in your README.
Credits
Please cite the data you use (and list any external data in your README):
- Lai, Yuchuan; Dzombak, David (2019). Compiled historical daily temperature and precipitation data for selected 210 U.S. cities. Carnegie Mellon University. https://doi.org/10.1184/R1/7890488.v6
- Lai, Yuchuan; Dzombak, David (2019). Historical changes of annual temperature and precipitation indices at selected 210 U.S. cities. Carnegie Mellon University. https://doi.org/10.1184/R1/7961012.v6
- Lai, Y., & Dzombak, D. A. (2019). Use of historical data to assess regional climate change. Journal of Climate, 32(14), 4299–4320. https://doi.org/10.1175/JCLI-D-18-0630.1
My team
Submit
You must be signed in and in a team. Your submission is the link to your team's public Hugging Face dataset, containing submission.csv (your forecasts) and README.md (your method, policy idea and feedback to the ecosystem) and your code in code/. See the Data & submission tab.
When you submit, the current version (commit) of your dataset is recorded. You can submit again until the deadline. Only your team's latest submission counts, at the version recorded then.