FOSS4G 2022 academic track

Approach and concepts
3D-georeferenced historical pictures have a high potential for the analysis of different landscape features such as melting glaciers, the effects of urbanization or natural hazards. Moreover, historical pictures have a higher temporal and spatial resolution than satellite imagery and therefore allow for analyses that go farther back in time. A 3D georeferenced picture can for instance be combined with a digital terrain model (DTM) and other reference data to calculate the exact footprint of the picture and to generate a list of visible toponyms that can be used to find pictures of a specific place or region.

The utilization of historical pictures is unfortunately still difficult: 1. historical pictures need to be digitized 2. collections are often spread across several places in different archives and collections 3. metadata is often not available. In the ongoing open-source project Smapshot (Produit et al. (2018), https://smapshot.heig-vd.ch/) over 150’000 digitized historical pictures have been georeferenced in 3D by more than 700 participants. In the web-platform Smapshot a participant can georeference a picture using monoplotting (Bozzini et al, 2012): ground-control-points (GCP) are digitized both in the historical picture and in a virtual globe that displays recently updated data. These GCP allow for the calculation of the exact position from where the picture has been taken (3D point) and the three angles that define the direction of view: roll, pitch and yaw. Once the position and the direction of view has been calculated a footprint of the picture is generated using a DTM.

Results
In order to make the pictures and the metadata from Smapshot available to the public, an open API for 3D-georeferenced historical pictures has been created. The goal was to offer free access to all the data in the Smapshot database and to allow for different types of queries such as retrieving the footprints of the photos, fetching metadata for a picture (e.g. owner, title, date, x/y/z position and roll, pitch, yaw angles) or retrieving photos that are within a certain range from a specific point.

This API was built in NodeJS (https://nodejs.org/) with a PostgreSQL/PostGIS (https://www.postgresql.org/, https://postgis.net/) database and python code for the georeferencing algorithm. The API is a REST API fully documented using the OpenAPI specification. The API project has been open-sourced and specific test-suites have been put in place to ensure quality and to allow community contribution with confidence.

One challenge for the establishment of this API was standardization: Today there are several standards for the definition of metadata in pictures such as the IIIF (https://iiif.io/) or the Dublin-Core (https://www.dublincore.org/) standards. These standards however have limited support for geospatial data. On the other hand, spatial standards poorly support pictures that are oriented in 3D. The glTF standard (https://www.khronos.org/gltf/) is one example and there is also a recent initiative from the OGC called GeoPose (https://www.ogc.org/projects/groups/geoposeswg) which formalizes a standard to define a 6DoF pose everywhere on Earth including a position and orientation in 3D.

Reasons why it should be considered
3D georeferenced images are increasingly used by several projects that document change over time; e.g. within the field of digital humanities even paintings can be considered for 3D georeferencing and differences between the real world and the painted world can give room for analysis and interpretation. Another use-case is the creation of geovisualization-applications that show the contents of historical pictures in 3D and that enable a user to compare its contents to the real world (e.g. augmented or virtual reality applications)

Furthermore in the context of climate change, pictures and paintings document change and deliver evidence. Image processing techniques can be used to automatically detect features (machine learning) and if several pictures are available for one region (but taken from slightly different viewpoints) 3D features can be generated.

The open API for 3D georeferenced historical pictures makes these types of analyses easier and opens up the data for a larger public. It also becomes possible to implement other solutions that utilize the data directly - e.g. for displaying historical pictures in a 3rd party web page or for implementing machine-learning processes that automatically download pictures and metadata in order to recognize features and places.

The results of the project are also an important input for standardization activities that aim at establishing standards in the context of georeferenced pictures and their metadata.

An important perspective of the project is the establishment of an infrastructure for 3D-georeferenced pictures that can be deployed on a national or international level and that also offers the possibility to push new data (e.g. pictures) in the database.

This work is of interest for researchers who want to utilize and analyze 3D georeferenced historical imagery and for people who want to establish open API’s to give access to data that is relevant for research.

Bibliography
Bozzini, C., Conedera, M., Krebs, P., 2012. A new monoplotting tool to extract georeferenced vector data and orthorectified raster data from oblique non-metric photographs. International Journal of Heritage in the Digital Era 1 (3), 499–518.

Produit, T., Blanc, N., Composto, S., Ingensand, J., Furhoff, L, 2018. Crowdsourcing the georeferencing of historical pictures. Proceedings of the Free and Open Source Software for Geospatial (FOSS4G) Conference. Guimarães, Portugal. 2018-07

Source code : https://github.com/MediaComem/smapshot-api