When handling massive LiDAR datasets on the web, is the traditional gazillion-of-tiles approach still the best choice? Or can structured file storage like SQLite, DuckDB, and Parquet deliver a faster, more scalable format? Solutions for managing massive LiDAR datasets already exist, some even attempt to package 3D Tiles into databases to reduce fragmentation. However, some of these approaches come with limitations: they are often proprietary or lack flexibility for large-scale datasets.

Faced with 1.3 trillion LiDAR points from Croatia’s nationwide airborne LiDAR mapping project, we needed a way to efficiently store, process, and visualise this immense dataset. For web visualisation, Cesium and 3D Tiles offered powerful rendering, but converting raw LiDAR data into millions of tiny files led to a storage and performance nightmare. File fragmentation overwhelmed the filesystem, causing sluggish read/write operations, unreliable backups, and some request overhead when serving tiles online.

To overcome these challenges, we explored alternative structured file storage solutions.
SQLite, a compact embedded database, reduced fragmentation while enabling fast spatial queries. DuckDB, an analytical database optimized for large-scale data, delivered high-speed querying and processing power.
Parquet, a columnar storage format used in big data, provided strong compression and rapid sequential access, making it a promising alternative to millions of fragmented 3D Tiles.

In this presentation, we will share our experience, comparing fragmented 3D Tiles with structured file storage formats. Whether you’re working with 3D Tiles, small raster and vector tiles, or other massive spatial datasets, this session will provide you with insight to a practical alternative of the millions-of-files problem.