2026-09-03 –, Conference Management Room5
We share GISTDA’s Dragonfly project lessons on national-scale agricultural monitoring. We optimize COG performance to reduce TTFB and ensure data reliability using Great Expectations automated quality gates. Attendees gain actionable DataOps insights for building resilient, high-performance cloud-native ecosystems from real-world operational experience.
Operationalizing satellite imagery for national-scale agricultural monitoring within the GISTDA Dragonfly project taught us two critical lessons: first, "standard" Cloud Optimized GeoTIFF (COG) configurations often fail under the pressure of low-latency production requirements, and second, speed is useless if the data is incorrect. This paper shares our practical journey in building a raster delivery pipeline that prioritizes both extreme performance and data reliability.
The first part of our story focuses on overcoming performance bottlenecks encountered in the field. We discuss how sub-optimal COG parameters—such as misaligned internal tiling or inefficient compression—directly degraded the user experience. By implementing advanced tuning techniques, including strict byte-alignment and optimized ZSTD compression, we achieved reduction in Time to First Byte (TTFB), enabling seamless data access via OGC Environmental Data Retrieval (EDR) APIs.
The second part addresses our most challenging hurdle: "silent failures." From inconsistent spatial resolutions to corrupted metadata, these issues often bypassed traditional checks. To solve this, we integrated Great Expectations (GX) as a mission-critical automated quality gate. We share our experience in designing geospatial-specific "Expectations" to ensure that every 10m pixel and internal structure adheres to rigorous production standards before being served to end-users.
Attendees will gain actionable DataOps insights derived from real-world operational experience, moving beyond basic data conversion to building a resilient, high-performance cloud-native ecosystem.
Key Takeaways for Attendees:
- Deep Understanding: Grasp the critical impact of byte-alignment and internal tiling on cloud storage I/O performance.
- Data Reliability: Learn how to design and enforce "Geospatial Data Contracts" to eliminate silent failures.
- DataOps Roadmap: A strategic guide for transitioning from manual data processing to automated, continuous cloud-native delivery.
Our talk demonstrates a novel integration of Great Expectations into a GDAL/Rasterio based pipeline to enforce scientific data integrity for national-scale COG assets, served via OGC API standards.
I make my conference contribution available under the CC BY 4.0 license. The conference contribution comprises the abstract, the text contribution for the conference proceedings, the presentation materials as well as the video recording and live transmission of the presentation:
Passionate geospatial advocate dedicated to leveraging FOSS4G tools for open-source mapping and building community-driven data solutions. Having actively contributed to FOSS4G Asia in Thailand (2024) and India (2026), I find immense fulfillment in sharing my journey and technical experiences within the community. I believe that collective knowledge-sharing is the key to creating a more open and spatial-aware world.