The USDA’s Agricultural Research Service (ARS) has an extensive research portfolio dedicated to ensuring a healthy and productive food supply, protecting natural resources, and solving real-world problems impacting agriculture including invasive species and diseases or wildfires. ARS’s scientific computing initiative, SCINet, aims to facilitate and expedite this research, and easy access to open geospatial data and tools are an integral part of that effort. SCINet oversees two high-performance computing (HPC) clusters that are used for a wide variety of computational geospatial research projects, many of which incorporate multiple, large raster datasets to effectively model complex agricultural systems. Although the science questions and spatial scales vary across these research projects, they often have overlapping input datasets (e.g., gridded products of climate, satellite remote sensing, land-use, or socioeconomic indicator data) that can require significant preprocessing before any analysis can be done. The expertise and time required for this preprocessing work, along with the challenge of sourcing and acquiring all relevant data, can be significant barriers to the use of these datasets in ARS’s geospatial research projects.

To address these concerns, we have developed the Geospatial Common Data Library (GeoCDL), which delivers efficient, easy-to-use, and automation-friendly access to a diverse collection of large geospatial datasets on shared computing infrastructure such as HPC clusters. The GeoCDL is a software stack and application programming interface (API) that facilitates geospatial research projects by allowing users to request spatiotemporal subsets of commonly used geospatial datasets using intuitive query parameters that are flexible enough to support many use cases and are also consistent enough to be easily applied across all datasets. In response to each query, the GeoCDL returns a custom, analysis-ready dataset that includes the desired source datasets and variables harmonized to the user-specified coordinate reference system and spatial and temporal extents and resolutions. Thus, users do not need to know how to complete these technically demanding steps themselves, which greatly expedites researchers' ability to perform their geospatial analyses. To further simplify geospatial analysis using open-source tools, we have also developed R and Python packages that seamlessly integrate the GeoCDL’s functionality with common geospatial packages and data structures in those languages.

Other interfaces exist for accessing open geospatial datasets from multiple online sources, but a unique feature of the GeoCDL is its support for local shared storage infrastructure, as commonly available on HPC infrastructure, and its seamless integration of local and cloud-based datasets. Datasets that are available online but do not have subset streaming options, or that could be more efficiently served from local storage, can be staged on shared storage and then, from the user’s perspective, used like any other GeoCDL-supported dataset. This support for local datasets is also advantageous for hosting internal datasets that are often directly generated on the shared HPC infrastructure and are not yet publicly available.

Recent development efforts are focusing on incorporating large language models (LLMs) to augment the GeoCDL project. A locally hosted open-weight LLM API service is already available on SCINet infrastructure, and we are exploring using this service to enhance the user-friendliness of the GeoCDL, such as by translating natural language user queries into dataset, spatial, and temporal parameters for GeoCDL API queries.

This presentation will provide an overview of the architecture, implementation, and available features of the GeoCDL, the supported datasets and how additional datasets are added to the library, the R and Python packages that extend the GeoCDL, example research use cases for the GeoCDL, example LLM query enhancement functionality, and guidance on implementing the open-source code on other infrastructure.