In this talk, we describe Element 84’s work to improve natural-language search for geospatial imagery as part of Queryable Earth. Queryable Earth allows users to search for visible features on Earth using plain-language queries, such as “solar panels,” “golf courses,” or more complex change-detection concepts such as “new football fields around Boston”. Our work explores whether multimodal large language models can improve this search experience, compared to traditional image-embedding models like SkyCLIP, by generating richer textual descriptions of imagery and embedding those descriptions for retrieval.

We describe the pipeline we built to caption large collections of geospatial imagery with Antrophic’s Haiku and Opus models, embed those captions, and use the resulting embeddings for natural-language retrieval. This differs from image-to-embedding approaches like SkyCLIP by introducing an explicit LLM generated text description of each image before retrieval. A major focus of the talk is how we evaluated this approach at scale: whether the captions accurately captured important visual features, whether caption-derived embeddings improved search results, and where each model tended to agree, disagree, or fail.

To answer these questions, we developed several complementary evaluation methods. First, we built a curated caption benchmark using imagery that included both representative examples and difficult failure cases. These images were labeled with key concepts, such as “solar panels” or “swimming pools” and could be captioned by different LLM models and prompt configurations. Finally, the captions were graded by another LLM to determine whether the caption captured the important content of the image, based on if it hit the key concepts manually labelled.

Second, we evaluated the embeddings through natural-language retrieval. Using the Queryable Earth API, we ran large sets of search and change-detection queries against different embedding backends, including SkyCLIP, Haiku-derived embeddings, and Opus-derived embeddings. The returned results were then graded by Opus and humans to estimate whether the retrieved imagery actually matched the user’s query.

Third, we performed large-scale embedding similarity analysis across Massachusetts imagery. By comparing cosine similarity between Haiku-Opus, Haiku-SkyCLIP, and Opus-SkyCLIP representations, we identified where models agreed, where they strongly disagreed, and what types of imagery produced the largest errors. These comparisons were visualized in internal dashboards, allowing us to inspect tile-level disagreements and uncover recurring patterns, such as LLMs misclassifying golf courses, airports, water tiles, or dense urban infrastructure.

Finally, we discuss what these results suggest about the strengths and weaknesses of LLM-generated geospatial captions. Opus often produced richer captions that improved natural-language retrieval, while SkyCLIP remained competitive for certain geospatial-specific scene types. We close by discussing possible improvements, including hybrid retrieval approaches, better prompting, additional spatial context, and targeted use of stronger models in areas where cheaper models struggle.