Artificial Intelligence weather forecasts in seconds with one computer, 5,000

A team of researchers set out to revolutionize weather forecasting using artificial intelligence. 

So far, forecasts have come out 5,000 times faster and outperformed the current gold standard of weather forecasting models on 74% of data, such as wind speed, wind direction and humidity, according to the Microsoft Aurora model developers.

“Traditional weather and climate models aim to predict how the atmosphere evolves by formulating a set of governing equations using first physical principles (e.g. conservation of mass, momentum, energy, etc.),” Principal Research Manager of the AI Aurora model Paris Perdikaris said. “Then the solutions to these equations are approximated in space and integrated in time using sophisticated numerical algorithms that run on computer clusters.”

AIRLINE HARNESSES ‘NEXT GENERATION OF FORECASTING’ TO CUT DOWN ON WEATHER DELAYS AND CANCELATIONS

Euro (ECMWF) model is gold standard of traditional models

According to Perdikaris, the European Center for Medium-Range Weather Forecast’s (ECMWF) model called the HRES-IFS, which stands for High Resolution Integrated Forecast System, is the gold standard of traditional forecasts. However, producing a 10-day forecast at a 6.8-mile resolution takes about 65 minutes on large sets of computers with hundreds of processors. That’s expensive and slow, he said.

“Traditional numerical weather predictions would require hours to produce the same result using a supercomputer with thousands of CPU cores,” Perdikaris said. “The primary advantage here is Aurora circumvents the cost of running a physics-based simulation, and its evaluation is highly parallelized across compute cores and fully leverages the capabilities of modern GPU (graphics processing unit) hardware.”

Perdikaris said his team looked at AI models like PanguWeather and Google’s GraphCast, which are only trained on a single data set and designed to only create a single specific prediction task like a 10-day global weather forecast with a 6.8-mile resolution. The developer said that these were the first AI models that competed with and sometimes outperformed the gold standard.

However, the team wanted the model to tackle a wide variety of Earth system and science data prediction tasks. Aurora was born.

FORECASTING WITH AI: COMPANY CREATES FIRST WEATHER AND CLIMATE GENERATIVE ARTIFICIAL INTELLIGENCE

“With Aurora, we asked the question of whether we can build a much more general AI system that can tap into the vast expand of Earth system data and excel at a range of prediction tasks,” Perdikaris said.

Building a foundation model to include more weather, climate and atmospheric chemistry solutions

So they developed a “foundation model.” It’s a machine learning program, “that makes decisions in a manner similar to the human brain, by using processes that mimic the way biological neurons work together to identify phenomena, weigh options and arrive at conclusions,” according to IBM.

Aurora was trained in two phases: pretraining and fine-tuning. Pretraining is the most expensive and time-consuming part, Perdikaris said. It took about 32 NVIDIA graphics processing units about 2-and-a-half weeks to ingest and learn from a collection of weather and climate simulation data which included analysis and reanalysis. AI models only need pretraining and fine-tuning once.

HOW AI CAN HELP (AND HURT) CLIMATE CHANGE

“Aurora’s effectiveness lies in its training on more than a million hours of diverse weather and climate simulations, which enables it to develop a comprehensive understanding of atmospheric dynamics,” the developers wrote for a Microsoft blog. “One of the key findings of this study is that pretraining on diverse datasets significantly improves Aurora’s performance compared to training on a single dataset. By incorporating data from climate simulations, reanalysis products, and operational forecasts, Aurora learns a more robust and generalizable representation of atmospheric dynamics.”

Then comes fine-tuning, which improves Aurora’s accuracy at specific tasks like 10-day global weather forecasting or 5-day global air pollution prediction.

“During ‘finetuning’ Aurora…