BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//pretalx//talks.osgeo.org//foss4g-2026//talk//VBGM3W
BEGIN:VTIMEZONE
TZID:JST
BEGIN:STANDARD
DTSTART:20000101T000000
RRULE:FREQ=YEARLY;BYMONTH=1
TZNAME:JST
TZOFFSETFROM:+0900
TZOFFSETTO:+0900
END:STANDARD
END:VTIMEZONE
BEGIN:VEVENT
UID:pretalx-foss4g-2026-VBGM3W@talks.osgeo.org
DTSTART;TZID=JST:20260903T110000
DTEND;TZID=JST:20260903T113000
DESCRIPTION:Retrieval-Augmented Generation (RAG) has become a widely adopte
 d approach for enhancing Large Language Models (LLMs) with external knowle
 dge. Recent studies such as Spatial-RAG (Yu et al.\, 2025) and MapQA (Li e
 t al.\, 2025) have proposed frameworks that integrate spatial data with LL
 Ms for geospatial question answering. However\, most existing work focuses
  on proposing individual frameworks rather than systematically comparing m
 ultiple architectural approaches under controlled conditions. Consequently
 \, practitioners lack empirical guidance on which RAG architecture best su
 its geographic Point of Interest (POI) queries—a common use case in loca
 tion-based services\, urban planning\, and tourism applications.\n\nThis s
 tudy presents a systematic comparison of six RAG architectures for geograp
 hic POI question answering\, evaluated under identical conditions to ident
 ify which architectural approach best suits spatial queries about real-wor
 ld POIs. Specifically\, we investigate the relative effectiveness of deter
 ministic structured processing\, graph-based retrieval\, and vector search
 —and whether these components function as complementary rather than comp
 eting strategies.\n\nWe constructed a POI dataset by extracting 1\,047 poi
 nts of interest from OpenStreetMap (OSM) within the Shibuya station area o
 f Tokyo\, Japan. Each POI was enriched with computed spatial metadata incl
 uding Haversine distance and compass direction (eight cardinal/intercardin
 al directions) from the Shibuya station reference point (35.658034°N\, 13
 9.701636°E). To validate generalization across different urban contexts\,
  we additionally collected POIs from three other major Tokyo districts—S
 hinjuku\, Ikebukuro\, and Tokyo Station areas—totaling approximately 3\,
 600 POIs. All POI embeddings were generated using the multilingual-e5-base
  model (768 dimensions) and stored in ChromaDB\, an open-source vector dat
 abase.\n\nTo address these research questions\, we designed and compared s
 ix RAG architectures\, all sharing the same underlying LLM (Qwen2.5-7B-Ins
 truct\, 4-bit quantized) and embedding model: (1) Vector RAG (Baseline)\, 
 using cosine similarity retrieval with no spatial computation\; (2) Struct
 ured RAG\, introducing four deterministic spatial processing modules—pro
 ximity sorting\, sensitivity analysis across varying search radii\, direct
 ional comparison\, and category aggregation—triggered by question-type c
 lassification and applied in combination rather than as mutually exclusive
  alternatives\, with vector search always contributing as a complementary 
 source\; (3) GraphRAG\, constructing a knowledge graph with 1\,090 nodes a
 nd approximately 82\,000 edges across seven relationship types for relatio
 nal context retrieval\; (4) Hybrid RAG\, combining structured spatial proc
 essing with vector retrieval in a fixed pipeline\; (5) Adaptive RAG\, dyna
 mically selecting between Structured RAG and GraphRAG based on query chara
 cteristics\; and (6) Agentic RAG\, employing a ReAct-style reasoning loop 
 where the LLM autonomously decides which retrieval tools to invoke.\n\nEva
 luation relied on a hierarchical five-level test prompt framework designed
  to capture the diverse reasoning demands inherent in geospatial queries\,
  with a total of 55 prompts (L1: 10\, L2: 15\, L3: 10\, L4: 10\, L5: 10) s
 panning 12 subcategories: L1 Basic Retrieval (factual POI lookup by name o
 r category)\, L2 Spatial Reasoning (proximity ranking\, density estimation
 \, directional comparison)\, L3 Constraint Satisfaction (queries combining
  spatial and categorical filters)\, L4 Decision Support (location selectio
 n and business-oriented recommendations requiring multi-criteria reasoning
 )\, and L5 Advanced Reasoning (sensitivity analysis under varying search r
 adii\, comparative evaluation\, and uncertainty quantification). All syste
 ms were evaluated using identical test prompts and a multi-dimensional sco
 ring function comprising: keyword success rate\, composite score (0–100)
 \, reasoning quality (0–5)\, evidence citation (0–5)\, constraint sati
 sfaction (0–5)\, and uncertainty acknowledgment (0–5). All metrics wer
 e computed via rule-based automatic evaluation using regular expression an
 d keyword matching—not LLM-as-judge—ensuring deterministic and reprodu
 cible scoring.\n\nIn the architecture comparison\, Structured RAG achieved
  89.1% overall accuracy\, outperforming GraphRAG at 76.7% and Adaptive RAG
  at 86.1%. Notably\, Structured RAG outperformed GraphRAG even on graph-sp
 ecific test cases designed to favor relational retrieval (86.9% vs. 80.2%)
 \, demonstrating that coordinate-based deterministic computation renders e
 xplicit graph edge representations largely redundant for spatial relations
 hip reasoning. Performance differences were most pronounced at L2 and L5\,
  where deterministic spatial computation provided precise distance and dir
 ectional data that graph traversal could not match.\n\nIn the multi-area g
 eneralization evaluation across four Tokyo districts\, Hybrid RAG achieved
  the highest keyword success rate of 96.9%\, followed by Adaptive RAG (96.
 9%)\, Graph RAG (100.0% keyword but lower composite)\, and Agentic RAG (89
 .2%). Multi-dimensional composite scoring revealed a persistent quality ga
 p across all systems: Hybrid RAG scored 67.1/100 in composite quality desp
 ite near-perfect keyword accuracy\, with the gap widening at L4 (Decision 
 Support) and L5 (Advanced Reasoning)\, where keyword matching alone failed
  to capture reasoning quality.\n\nWe attribute the consistent superiority 
 of Structured RAG over GraphRAG to the inherent computability of spatial r
 elationships: distance and direction between POIs are directly calculable 
 from coordinates\, making explicit graph edges redundant for the majority 
 of geographic queries. The quality gap observed in Hybrid RAG—near-perfe
 ct keyword success yet composite scores well below that level—indicates 
 that retrieving relevant POIs is necessary but insufficient\; generating w
 ell-reasoned answers at L4–L5 requires structured context that guides th
 e LLM's reasoning process\, not merely correct entity retrieval. This find
 ing empirically demonstrates that single-metric evaluation based on keywor
 d accuracy is insufficient for geospatial QA\, underscoring the importance
  of hierarchical\, multi-dimensional evaluation frameworks.\n\nOur systema
 tic comparison yields three principal findings for the geospatial open-sou
 rce community: (1) deterministic structured processing using coordinate-ba
 sed computation consistently outperforms graph-based retrieval for POI-cen
 tric spatial queries\; (2) structured processing and vector search are com
 plementary—not competing—retrieval strategies\, and combining them yie
 lds the best results\; (3) controlled multi-architecture comparison under 
 identical evaluation conditions is essential for advancing geospatial RAG 
 research. All experiments were conducted using exclusively open-source sof
 tware (ChromaDB\, NetworkX\, Hugging Face Transformers) and open data (Ope
 nStreetMap)\, ensuring full reproducibility. Future work will extend this 
 evaluation framework to larger-scale multilingual POI datasets beyond the 
 Tokyo metropolitan area and investigate adaptive weighting between structu
 red and vector retrieval components.
DTSTAMP:20260717T225742Z
LOCATION:Cosmos2
SUMMARY:A Systematic Comparison of RAG Architectures for Geographic POI Que
 stion Answering Using OpenStreetMap Data - Noboru Otsuka
URL:https://talks.osgeo.org/foss4g-2026/talk/VBGM3W/
END:VEVENT
END:VCALENDAR