FOSS4G 2022 general tracks

Cihan Sahin

Mr Cihan Sahin works at ECMWF as an analyst. His work focuses on development of web services both backend and frontend with a special interest on meteorological and climatological data manipulation and preparation.


Sessions

08-26
12:30
30min
A high performing data retrieval system for large and frequently updated geospatial datasets
Cihan Sahin

ECMWF is a research institute and a 24/7 operational service, producing global numerical weather predictions and other data for a broad community of users. To achieve this, the centre operates one of the largest supercomputer facilities and data archives within the meteorological community. ECMWF also operates several services for the EU Copernicus programme to provide data for Climate Change, Atmospheric monitoring and Emergency services.

As part of ECMWF's Open data initiative, more and more meteorological data and web services are freely available to a wider community. ECMWF's web services include an interactive web application to explore and visualize its forecast data, a Web Map Service (WMS) server and many graphical products including geospatial weather diagrams so called Ensemble (ENS) meteograms and vertical profiles.

ENS meteograms and vertical profile diagrams are among the ECMWF's most popular web products and presents ECMWF's multi-dimensional real-time ensemble forecast data for a given position globally. They are freely available through various ECMWF web services, and integrated on ECMWF's GIS based interactive web application. Datasets powering the dynamically generated diagrams are formed from a rolling archive of 10 days data, updated twice a day and each update consists of data around half a Terabyte. An upcoming update on ECMWF's forecasting system will increase the data size by a factor of 3-4 times in the near future. In addition to ECMWF's forecast data, similar services are requested as part of various Copernicus projects producing different datasets.

This talk presents migrating legacy data structure used for ENS meteogram datasets to a more flexible, extensible, and high performing one fit to be used by GIS systems by using Free Open Source Software (FOSS). The new data structure uses Python ecosystem. The data preparation workflow as well as the challenges and the solutions that are taken when dealing large and frequently updated geospatial datasets are presented. Talk will also include early experiments and experiences to offer these datasets as part of OGC's Environmental Data Retrieval (EDR) API.

Use cases & applications
Room Onice