We assessed the seven leading AI weather models against direct station readings across the Indian subcontinent during the Indian Monsoon. The standard benchmark, ECMWF's reanalysis, is itself model output, so comparing forecasts to it often means comparing one model's guess to another's. They can agree while both miss reality. Against real measurements, errors are 15-45% larger, enough to make a forecast several degrees off, hours late, or wrong on cyclone landfall. In these regions, missed monsoon onset or under-forecast heatwaves mean lost harvests, blackouts, and lives at risk.
Read on arXivWe chose seven of the most widely used global AI weather models, the systems leading research labs and operational forecasting centres are training and deploying today. To compare them fairly, we standardised on a single configuration: a grid resolution of about 25 km and a forecast every 6 hours. Aurora is included at its matching 25 km version. GenCast (the only model that produces a spread of possible outcomes rather than a single forecast) is evaluated separately for cyclone tracks, where ensembles matter most.
9 km deterministic forecast, 12-hourly, 10-day lead. The operational gold standard.
50-member ensemble, 18 km grid, 15-day lead.
The mean of the 50-member IFS ensemble.
The standard reference, ERA5 reanalysis, is itself a model output, so grading AI forecasts against it can quietly mask real-world errors. To get an honest picture, we compared each forecast against direct measurements the models rarely see: weather stations on the ground, dense rain-gauge networks, and satellite cloud imagery, alongside reanalysis as the familiar baseline.
6-hourly, 0.25° × 0.25° reanalysis, 2021–2024. The default baseline in the AWP literature. Copernicus CDS.
Hourly point-based surface observations from IMD's weather station network, accessed through the MeteoStat Python API. Used to validate T2, U10, V10, and precipitation during extreme events.
Daily-averaged gridded rainfall from up to 6,995 rain gauges across India. Used for 2022 and 2024 monsoon-season verification.
Geostationary cloud-top products (processed clear-sky cloud fraction at 0.5°), used to validate total cloud cover diagnostics from AIFS. Source: ISRO MOSDAC.
Best-track cyclone trajectory data for Tauktae (2021) and Yaas (2021), used to benchmark deterministic tracks, the 50-member IFS ensemble, and the 32-member GenCast ensemble.
Headline forecast metrics tell one story; observation-grounded validation tells another. The same models that look strong against reanalysis show systematic regional biases when checked against direct measurements, especially during the Indian Monsoon.
All seven models match or exceed traditional numerical forecasts on standard skill metrics for 1 to 10 day forecasts. AIFS shows the most consistent skill across atmospheric variables, with GraphCast and GenCast also strong. These averages mask region-specific biases that only surface when the comparison is against ground observations.
When the same forecasts are graded against weather station readings instead of reanalysis, mean absolute errors are 15 to 45% larger across temperature, wind, and precipitation. The gap widens at longer lead times. The standard benchmark systematically flatters AI weather models in the regions with the sparsest observation networks.
During peak monsoon, day-ahead temperature forecasts show a consistent cold bias over the Indo-Gangetic plain and a warm bias over the Western Ghats. Errors are typically 1 to 2 °C, but reflect structural problems in how the models capture monsoon thermodynamics.
All models capture average daily rainfall reasonably well, but systematically underrepresent the heavy-rain tail above 50 mm/day, which is the rainfall that drives flooding. They overestimate light to moderate rainfall in turn.
Across the cyclones evaluated, AI models systematically underpredict peak wind speeds and disagree on track location. Predictability degrades sharply for the most intense storms, and ensemble spread often fails to contain the observed track within its envelope.
AI weather models look better against reanalysis than against real station readings. But reanalysis is not ground truth. It is another model output with many of the same blind spots.
