Hazard datasets are critical and exhausting to work with: shapefiles, oversized polygons, per county distribution. This workshop walks through a pattern that turns this kind of data into a clean, query-optimized GeoParquet asset — using DuckDB, Python, and object storage.
Participants will: Stream NFHL shapefile ZIPs from FEMA directly into a per-source GeoParquet archive — no intermediate disk conversions, through a python GDAL script (explained). Normalize disparate schemas to a unified flood layer using a YAML-driven mapping pattern. Apply a Python subdivide (recursive bbox bisection capped at N vertices per row) to make spatial joins and rendering an order of magnitude faster. Land everything in DuckDB with an RTree index, validate, and benchmark. Explore and analyze the content.
Re-export production-grade GeoParquet: ZSTD compression, Hilbert sort, partitioning by state, with bbox covering for predicate pushdown. Serve the result directly from object storage to a web map. The session is opinionated and surfaces the rough edges: shapefile encoding traps, subdivide trade-offs, GeoParquet writer disagreements, and where DuckDB's spatial extension still leaves gaps. Attendees leave with a working repository, a benchmarked dataset, and a reusable pattern for any messy public geospatial source.
Learning outcomes Build a reproducible ingest → normalize → Geotransform → publish pipeline with DuckDB and Python. Tune GeoParquet output: compression, spatial sorting, partitioning, bbox covering, RTree.
Prerequisites SQL and Python fluency. Laptop with a recent DuckDB (spatial, httpfs) and uv or pip. Optional: an S3 / Cloudflare R2 bucket for the serving section.