Retrieval-Augmented Generation (RAG) has become a widely adopted approach for enhancing Large Language Models (LLMs) with external knowledge. Recent studies such as Spatial-RAG (Yu et al., 2025) and MapQA (Li et al., 2025) have proposed frameworks that integrate spatial data with LLMs for geospatial question answering. However, most existing work focuses on proposing individual frameworks rather than systematically comparing multiple architectural approaches under controlled conditions. Consequently, practitioners lack empirical guidance on which RAG architecture best suits geographic Point of Interest (POI) queries—a common use case in location-based services, urban planning, and tourism applications.

This study presents a systematic comparison of six RAG architectures for geographic POI question answering, evaluated under identical conditions to identify which architectural approach best suits spatial queries about real-world POIs. Specifically, we investigate the relative effectiveness of deterministic structured processing, graph-based retrieval, and vector search—and whether these components function as complementary rather than competing strategies.

We constructed a POI dataset by extracting 1,047 points of interest from OpenStreetMap (OSM) within the Shibuya station area of Tokyo, Japan. Each POI was enriched with computed spatial metadata including Haversine distance and compass direction (eight cardinal/intercardinal directions) from the Shibuya station reference point (35.658034°N, 139.701636°E). To validate generalization across different urban contexts, we additionally collected POIs from three other major Tokyo districts—Shinjuku, Ikebukuro, and Tokyo Station areas—totaling approximately 3,600 POIs. All POI embeddings were generated using the multilingual-e5-base model (768 dimensions) and stored in ChromaDB, an open-source vector database.

To address these research questions, we designed and compared six RAG architectures, all sharing the same underlying LLM (Qwen2.5-7B-Instruct, 4-bit quantized) and embedding model: (1) Vector RAG (Baseline), using cosine similarity retrieval with no spatial computation; (2) Structured RAG, introducing four deterministic spatial processing modules—proximity sorting, sensitivity analysis across varying search radii, directional comparison, and category aggregation—triggered by question-type classification and applied in combination rather than as mutually exclusive alternatives, with vector search always contributing as a complementary source; (3) GraphRAG, constructing a knowledge graph with 1,090 nodes and approximately 82,000 edges across seven relationship types for relational context retrieval; (4) Hybrid RAG, combining structured spatial processing with vector retrieval in a fixed pipeline; (5) Adaptive RAG, dynamically selecting between Structured RAG and GraphRAG based on query characteristics; and (6) Agentic RAG, employing a ReAct-style reasoning loop where the LLM autonomously decides which retrieval tools to invoke.

Evaluation relied on a hierarchical five-level test prompt framework designed to capture the diverse reasoning demands inherent in geospatial queries, with a total of 55 prompts (L1: 10, L2: 15, L3: 10, L4: 10, L5: 10) spanning 12 subcategories: L1 Basic Retrieval (factual POI lookup by name or category), L2 Spatial Reasoning (proximity ranking, density estimation, directional comparison), L3 Constraint Satisfaction (queries combining spatial and categorical filters), L4 Decision Support (location selection and business-oriented recommendations requiring multi-criteria reasoning), and L5 Advanced Reasoning (sensitivity analysis under varying search radii, comparative evaluation, and uncertainty quantification). All systems were evaluated using identical test prompts and a multi-dimensional scoring function comprising: keyword success rate, composite score (0–100), reasoning quality (0–5), evidence citation (0–5), constraint satisfaction (0–5), and uncertainty acknowledgment (0–5). All metrics were computed via rule-based automatic evaluation using regular expression and keyword matching—not LLM-as-judge—ensuring deterministic and reproducible scoring.

In the architecture comparison, Structured RAG achieved 89.1% overall accuracy, outperforming GraphRAG at 76.7% and Adaptive RAG at 86.1%. Notably, Structured RAG outperformed GraphRAG even on graph-specific test cases designed to favor relational retrieval (86.9% vs. 80.2%), demonstrating that coordinate-based deterministic computation renders explicit graph edge representations largely redundant for spatial relationship reasoning. Performance differences were most pronounced at L2 and L5, where deterministic spatial computation provided precise distance and directional data that graph traversal could not match.

In the multi-area generalization evaluation across four Tokyo districts, Hybrid RAG achieved the highest keyword success rate of 96.9%, followed by Adaptive RAG (96.9%), Graph RAG (100.0% keyword but lower composite), and Agentic RAG (89.2%). Multi-dimensional composite scoring revealed a persistent quality gap across all systems: Hybrid RAG scored 67.1/100 in composite quality despite near-perfect keyword accuracy, with the gap widening at L4 (Decision Support) and L5 (Advanced Reasoning), where keyword matching alone failed to capture reasoning quality.

We attribute the consistent superiority of Structured RAG over GraphRAG to the inherent computability of spatial relationships: distance and direction between POIs are directly calculable from coordinates, making explicit graph edges redundant for the majority of geographic queries. The quality gap observed in Hybrid RAG—near-perfect keyword success yet composite scores well below that level—indicates that retrieving relevant POIs is necessary but insufficient; generating well-reasoned answers at L4–L5 requires structured context that guides the LLM's reasoning process, not merely correct entity retrieval. This finding empirically demonstrates that single-metric evaluation based on keyword accuracy is insufficient for geospatial QA, underscoring the importance of hierarchical, multi-dimensional evaluation frameworks.

Our systematic comparison yields three principal findings for the geospatial open-source community: (1) deterministic structured processing using coordinate-based computation consistently outperforms graph-based retrieval for POI-centric spatial queries; (2) structured processing and vector search are complementary—not competing—retrieval strategies, and combining them yields the best results; (3) controlled multi-architecture comparison under identical evaluation conditions is essential for advancing geospatial RAG research. All experiments were conducted using exclusively open-source software (ChromaDB, NetworkX, Hugging Face Transformers) and open data (OpenStreetMap), ensuring full reproducibility. Future work will extend this evaluation framework to larger-scale multilingual POI datasets beyond the Tokyo metropolitan area and investigate adaptive weighting between structured and vector retrieval components.