Computer vision is often used to predict spatially-explicit maps of vegetation attributes from imagery collected by small uncrewed aerial vehicles, commonly known as “drones”. These maps, representing attributes such as the functional type of vegetation in a rangeland or different tree species in a forest, can be used to inform land management or ecological research. Despite the fact that drones collect many overlapping images of the same location, most approaches in the literature generate predictions on only a synthesized top-down view of the scene, known as an orthomosaic. This neglects important information about the sides of objects and the fact that combining multiple predictions of each location can increase robustness. Furthermore, the orthomosaic generating process introduces artifacts and decreases quality relative to the input images.

An alternative approach of generating predictions on each individual image addresses these limitations and has been shown in the literature to generate higher quality geospatial predictions. However, this poses technical challenges since it requires that all pixel-level predictions be accurately mapped to geospatial coordinates. Prior approaches make simplifying assumptions, such as assuming the world is planar or can be represented as a 2.5D heightmap with no inter-object occlusions. These assumptions do not hold in forested landscapes, where overhangs are frequent and occlusions need to be modeled, especially when images are taken from an oblique (non-vertical) perspective. As part of the Open Forest Observatory project, we have developed a new open source tool called Geograypher (https://github.com/open-forest-observatory/geograypher) that addresses these issues. Geograypher enables occlusion-aware correspondence generation between aerial images and arbitrarily-complex scenes to support increased prediction accuracy.

Similar to other workflows that generate predictions from drone images, the user must first use photogrammetry to align all of the images together and estimate the structure of the scene. To accurately identify correspondences, we use the detailed 3D model of the scene that nearly all photogrammetry software can produce. This is represented as a mesh data structure, which consists of vertices connected by triangular faces, and is commonly used in computer graphics applications. Photogrammetry also precisely estimates the location and orientation from which each image was taken as well as the calibration parameters of the associated camera. The first step toward using this data is converting both the mesh and the camera representations to a shared, cartesian coordinate system that is zero-centered for numerical stability. Geograypher handles parsing the input data and applying the appropriate geospatial and geometric transformations to hide this complexity from the user. The core functionality is the ability to precisely identify correspondences between a pixel in a given image and the first mesh face hit by the ray observed by that pixel. This is done using computer graphics rendering techniques, which efficiently handle this task in an occlusion-aware manner. We support two rendering backends, one based on OpenGL and the other on the PyTorch3D library.

Using these capabilities, users can complete two main tasks: rendering and aggregation. The first involves rendering geospatial content to the perspective of each image which can be used to train computer vision models from accurate field-derived ground truth data. We support both vector and raster data inputs, and attributes from this 2D content are used to label each face in the mesh. From there, these attributes can be rendered to the perspective of each image. Conversely, if machine learning is used to generate predictions on each image, these pixel-wise predictions can be projected onto the mesh model using the same underlying pixel-to-face capabilities used for the rendering step. These predictions are aggregated per face, by taking the most commonly predicted class across all views observing a face, to label the mesh with predicted attributes. From there, we implement an optimized algorithm to convert the mesh-based class labels into a 2D vector representation which provides a non-overlapping classification map. This can be used directly for downstream tasks, or rasterized to provide a drop-in replacement for orthomosaic-based prediction workflows. All operations have optional computational optimizations such as mesh decimation, reduced rendering resolution, and spatial chunking to enable operation on landscape scale-data with consumer hardware.

This tool is designed to be user-friendly and accessible to a wide variety of geospatial practitioners. Command-line scripts and associated functions support running workflows end to end. There is a documentation website, notebooks summarizing key workflows, and representative example data. We provide a docker image and it is pip installable. Our goal is that Geograypher enables users to unlock the improved accuracy of predictions derived from individual drone images across a variety of vegetation mapping and land use prediction tasks.