2026-11-03 –, Carr
An LLM that answers spatial questions confidently — but wrongly — is dangerous. We share an open-pattern evaluation harness using verified ground truth and an LLM-as-judge to measure whether Geo-AI actually reasons correctly.
Geospatial AI is having a moment — natural-language interfaces, MCP servers, LLMs writing spatial SQL. But a spatial answer that's confidently wrong is worse than no answer at all, and "looks plausible" is not a measure of correctness. This lightning talk makes a single argument: if you're shipping Geo-AI, you need a way to verify it before you trust it.
In five minutes, I'll walk through a lightweight evaluation pattern for holding an LLM-driven spatial system accountable:
- Verified ground truth — expected answers computed independently (SQL / GeoPandas) and hand-checked, so the benchmark isn't itself a guess.
- An LLM-as-judge — extracts the semantic answer from free-form replies ("256 stops," "there are about 256," "I found 256") so phrasing doesn't masquerade as error.
- Spatial tolerance — because "near" and "within" carry real boundary ambiguity, correctness is scored with a tolerance band, not an exact match.
I'll show one real evaluation run end-to-end — question in, scored result out — plus the per-category accuracy breakdown that reveals where a system is weakest.
No live coding, no architecture deep-dive: one problem, one pattern, and one takeaway you can apply to any LLM-driven geospatial tool — verify, then trust.
Pat Smyth is a full-stack geospatial data engineer with over 12 years of experience designing, deploying, and maintaining large-scale spatial data pipelines, knowledge-graph integrations, and cloud-based geospatial infrastructure.