Ida Dittrich e0c4193e36 extended PEK routes and included swisstopo connector

2025-12-15 07:16:21 +01:00

25 KiB

Raw Blame History

Swiss Topo STAC API Integration Guide

This document provides comprehensive conceptual documentation for integrating the Swiss Federal Office of Topography (Swisstopo) STAC (SpatioTemporal Asset Catalog) API into the Real Estate feature. This integration enables two primary user workflows: creating projects with enriched parcel data, and browsing maps to explore parcels and their project associations.

Overview
Architecture
User Story 1: Create Project and Connect Parcels
User Story 2: Browse Map and Explore Parcels
API Routes and Endpoints
Implementation Components
Data Flow Summary
Error Handling and Security
Testing and Configuration
Summary

Overview

Purpose

The STAC API integration enables the Real Estate feature to support two distinct user workflows:

User Story 1: Create a new project, select parcels on a map, automatically enrich them with geospatial data, and store everything in the database for project management.

User Story 2: Browse a map, click on parcels to fetch their data from STAC, and see whether each parcel is already connected to an existing project or available for new projects.

STAC API Overview

The Swiss Federal Office of Topography provides a STAC-compliant API that follows open standards for describing geospatial data. The API provides:

Collections: Groups of related geospatial data (e.g., cadastral parcels, zoning maps)
Items: Individual geospatial features (e.g., a specific parcel, a zoning polygon)
Assets: Downloadable files associated with items (e.g., GeoPackage files, PDFs)

Key STAC Collections for Real Estate

Collection ID	Description	Use Case
`ch.swisstopo.amtliche-vermessung`	Official cadastral survey	Identify parcel boundaries
`ch.swisstopo.swissboundaries3d-gemeinde-flaeche`	Municipality boundaries	Resolve Gemeinde from coordinates
`ch.swisstopo.swissboundaries3d-kanton-flaeche`	Canton boundaries	Resolve Kanton from coordinates
`ch.are.nutzungsplanung`	Land use planning (zoning)	Extract bauzone, baulinie
`ch.bafu.laermbelastung`	Noise pollution zones	Set laermschutzzone
`ch.bafu.hochwassergefahrenkarte`	Flood hazard map	Set hochwasserschutzzone
`ch.bafu.grundwasserschutz`	Groundwater protection zones	Set grundwasserschutzzone

Architecture

The integration follows the existing Gateway architecture pattern with clear separation of concerns:

Frontend Layer: User interaction components including map widgets, project forms, and parcel exploration views that communicate with the backend via HTTP/REST.

API Routes Layer: HTTP endpoint definitions that handle request validation, authentication, and response formatting. New domain-focused endpoints will replace generic table-based routes.

Feature Logic Layer: Business rules and orchestration logic that coordinates STAC queries, data transformation, and enrichment processes. This layer contains the core intelligence for processing location selections, fetching parcel data, and checking parcel-project associations.

Service Layer: Specialized services including AI-powered document extraction and text processing capabilities that can analyze zoning regulations from PDF documents.

Interface Layer: Normalized data access layer providing CRUD operations on Real Estate entities (Projekt, Parzelle, Gemeinde, etc.) with proper access control and mandate filtering.

Connector Layer: External API communication layer that handles STAC API interactions, including query construction, response parsing, error handling, and retry logic.

Data Storage: PostgreSQL database for persistent storage of project data, and the Swiss Topo STAC API as the external geospatial data source.

User Story 1: Create Project and Connect Parcels

User Goal: Create a new construction project, select parcels on a map, automatically enrich them with geospatial data, and store everything in the database for project management.

Workflow Steps

Create New Project: User provides project name and mandate identifier. Backend creates Projekt object with initial status "Analyse" and empty collections. ✅ Already implemented
Select Location on Map: User clicks on map at location of interest. Frontend sends coordinates (LV95/EPSG:2056) to backend. ❌ New endpoint needed
Identify Parcel from STAC: Backend queries STAC cadastral collection for parcels intersecting the clicked point. Returns parcel geometry, identifier, properties, and assets. ❌ New functionality needed
Create Parzelle and Connect: Backend converts STAC geometry to internal GeoPolylinie format, creates Parzelle object, and attaches it to Projekt. Parcel becomes part of project data model for enrichment, annotation, and offline access. ❌ New functionality needed
Enrich Administrative Context: Backend searches STAC for Gemeinde and Kanton boundaries, creates/updates entities, and establishes hierarchy: Parzelle → Gemeinde → Kanton → Land. ❌ New functionality needed
Extract Zoning Information: Backend searches zoning collection, calculates intersection areas, selects dominant zone by largest overlap, and extracts bauzone and baulinie. ❌ New functionality needed
Identify Protection Zones: Backend searches noise, flood, and groundwater collections, finds zones with largest overlap, and extracts zone designations (laermschutzzone, hochwasserschutzzone, grundwasserschutzzone). ❌ New functionality needed
Link Source Documents: Backend extracts assets from STAC items (GeoPackages, PDFs), creates Dokument objects with metadata, and links them to Parzelle for audit trail. ❌ New functionality needed
AI Extraction (Optional): Backend filters PDF documents, calls AI service to extract building parameters (AZ, BZ, vollgeschossZahl, gebaeudehoeheMax, regulations), and updates Parzelle with extracted values. ⚠️ Partial - AI service exists, extraction method needs implementation
Save to Database: All entities (Projekt, Parzelle, Gemeinde, Kanton, Land, Dokument) are saved to PostgreSQL. Frontend receives enriched Parzelle with enrichment status. ❌ New functionality needed
User Continues Planning: Users can edit project status, add more parcels, upload documents, add notes, override AI values, and update parcel properties. ⚠️ Partial - Update endpoints exist but may need enhancement

User Story 2: Browse Map and Explore Parcels

User Goal: Browse a map, click on parcels to fetch their data from STAC, and see whether each parcel is already connected to an existing project or available for new projects.

Workflow Steps

Open Map Browser: User navigates to map browsing interface displaying Switzerland map. ❌ New interface needed
Click on Map: User clicks at any location. Frontend sends GET request with coordinates and optional mandate ID. ❌ New endpoint needed
Query STAC for Parcel: Backend queries STAC cadastral collection for parcels at coordinates. Returns parcel data if found. ❌ New functionality needed
Check Project Associations: Backend searches database for Parzelle records matching STAC parcel identifier, retrieves associated Projekte, filters by mandate if provided, and determines status: Connected (list projects) or Available (not connected). ❌ New functionality needed
Fetch Basic Context (Optional): Backend performs lightweight enrichment: administrative boundaries (Gemeinde, Kanton), basic zoning, and protection zones. Uses caching for performance. ❌ New functionality needed
Display Parcel Information: Frontend shows parcel details, geometry on map, project association status with action buttons (View Project or Create New Project), protection zones, and STAC source indication. ❌ New interface needed
Explore Multiple Parcels: User can continue clicking to explore multiple parcels. Map shows multiple parcels with color-coded association status. ❌ New functionality needed
Connect Parcel to Project (Optional): If parcel is available, user can create new project or add to existing project. System validates permissions, fetches full parcel data, creates Parzelle, triggers full enrichment (User Story 1 Steps 5-9), and saves to database. ❌ New functionality needed

Key Differences from User Story 1

User Story 2 focuses on exploration and discovery:

No project creation required upfront
Lightweight data fetching (can be cached)
Read-only exploration mode
Project association checking
Optional connection to projects

User Story 1 focuses on project creation and data persistence:

Project must exist first
Full data enrichment and storage
All data saved to database
Complete administrative, zoning, and protection zone context
AI extraction of building rules

API Routes and Endpoints

Route Migration Overview

With STAC API integration, the route structure is refactored from generic table-based endpoints to domain-focused endpoints that better align with user workflows and business logic.

Routes to Deprecate

POST /api/realestate/query: Direct SQL queries are a security risk. Use /command endpoint for natural language queries or specific GET endpoints for structured data retrieval. ⚠️ Deprecate

POST /api/realestate/table/{table}: Generic table creation doesn't reflect domain model. Replace with dedicated project and parcel endpoints. ⚠️ Deprecate

Routes to Keep

POST /api/realestate/command: Keep for natural language commands. May need AI prompt updates for STAC workflows. ✅ Keep

GET /api/realestate/tables: Keep for frontend schema discovery. ✅ Keep

GET /api/realestate/table/{table}: Keep for admin/debugging. Consider adding domain-specific alternatives. ✅ Keep

New Domain-Focused Routes

Project Management

POST /api/realestate/projects: Create new project (replaces POST /api/realestate/table/Projekt)
GET /api/realestate/projects: List projects with filtering and pagination
GET /api/realestate/projects/{projekt_id}: Get single project with full details
PATCH /api/realestate/projects/{projekt_id}: Update project properties
DELETE /api/realestate/projects/{projekt_id}: Delete project with access control

Parcel Management (User Story 1)

POST /api/realestate/projects/{projekt_id}/locations: Select location and create Parzelle with full enrichment
GET /api/realestate/projects/{projekt_id}/parcels: List all parcels for a project
GET /api/realestate/projects/{projekt_id}/parcels/{parzelle_id}: Get single parcel with full details
PATCH /api/realestate/projects/{projekt_id}/parcels/{parzelle_id}: Update parcel properties
DELETE /api/realestate/projects/{projekt_id}/parcels/{parzelle_id}: Remove parcel from project
POST /api/realestate/projects/{projekt_id}/parcels/{parzelle_id}/refresh-context: Re-enrich parcel with updated STAC data

Map Browsing (User Story 2)

GET /api/realestate/parcels/explore: Browse map and fetch parcel data from STAC with project association check
POST /api/realestate/projects/{projekt_id}/parcels/connect: Connect explored parcel to project (triggers full enrichment)

Migration Strategy

Phase 1: Add new domain-focused endpoints alongside existing routes
Phase 2: Update frontend to use new endpoints
Phase 3: Mark old endpoints as deprecated with warnings
Phase 4: Remove deprecated endpoints after migration period

Implementation Components

STAC Connector

Purpose: Encapsulate all STAC API interactions following Gateway connector pattern.

Key Methods:

Search parcels by point (cadastral collection)
Search administrative boundaries (Gemeinde, Kanton)
Search zoning (land use planning)
Search protection zones (noise, flood, groundwater)
Get item assets (downloadable files)

Features: Retry logic with exponential backoff, coordinate system transformations, error handling, response transformation to internal data structures.

Caching Strategy: Cache administrative boundaries (rarely change), cache map browsing lookups (short TTL), don't cache project creation searches (user-specific), cache collection metadata.

Feature Logic Extensions

For User Story 1:

Process location selection and create Parzelle
Enrich parcel with STAC data (orchestrates full enrichment)
Extract zoning by calculating intersection areas
Extract protection zones by overlap calculation
Geometry conversion helpers (STAC GeoJSON ↔ internal GeoPolylinie)

For User Story 2:

Fetch parcel data from STAC (lightweight)
Check parcel-project associations (database queries)
Get basic parcel context (quick administrative/zoning lookup)
Connect parcel to project (transition to User Story 1)

AI Service Extension

Purpose: Extract building rules from zoning PDF documents.

Process: Download PDFs, extract text (OCR if needed), use LLM with structured schema to extract AZ, BZ, vollgeschossZahl, gebaeudehoeheMax, and regulation descriptions.

Status: ⚠️ Partial - AI service exists, but zoning rule extraction method needs implementation

Data Flow Summary

User Story 1: Location Selection Flow

sequenceDiagram
    participant User
    participant Frontend
    participant Route
    participant FeatureLogic
    participant STACConnector
    participant STACAPI
    participant Database
    participant AIService

    User->>Frontend: Click on map
    Frontend->>Route: POST /projects/{id}/locations<br/>(coordinates, project_id)
    Route->>Route: Validate CSRF token<br/>Validate coordinates<br/>Check project access
    
    Route->>FeatureLogic: processLocationSelection()
    FeatureLogic->>STACConnector: search_parcels_by_point(x, y)
    STACConnector->>STACAPI: Query cadastral collection<br/>(point geometry)
    STACAPI-->>STACConnector: Parcel items (geometry, properties, assets)
    STACConnector-->>FeatureLogic: Parcel data
    
    FeatureLogic->>FeatureLogic: Convert STAC geometry<br/>to GeoPolylinie
    FeatureLogic->>FeatureLogic: Create Parzelle object
    FeatureLogic->>Database: Attach Parzelle to Projekt
    
    FeatureLogic->>FeatureLogic: Trigger enrichment
    
    FeatureLogic->>STACConnector: search_administrative_boundaries()
    STACConnector->>STACAPI: Query Gemeinde/Kanton collections
    STACAPI-->>STACConnector: Boundary data
    STACConnector-->>FeatureLogic: Administrative boundaries
    FeatureLogic->>Database: Create/update Gemeinde, Kanton, Land
    
    FeatureLogic->>STACConnector: search_zoning()
    STACConnector->>STACAPI: Query zoning collection
    STACAPI-->>STACConnector: Zoning polygons
    STACConnector-->>FeatureLogic: Zoning data
    FeatureLogic->>FeatureLogic: Calculate intersection areas<br/>Select dominant zone
    FeatureLogic->>FeatureLogic: Extract bauzone, baulinie
    
    FeatureLogic->>STACConnector: search_protection_zones()
    STACConnector->>STACAPI: Query noise/flood/groundwater collections
    STACAPI-->>STACConnector: Protection zone data
    STACConnector-->>FeatureLogic: Protection zones
    FeatureLogic->>FeatureLogic: Extract zone designations
    
    FeatureLogic->>FeatureLogic: Extract assets from STAC items
    FeatureLogic->>Database: Create Dokument objects
    
    opt AI Extraction Enabled
        FeatureLogic->>FeatureLogic: Filter PDF documents
        FeatureLogic->>AIService: extractZoningRules(documents, bauzone)
        AIService->>AIService: Download PDFs<br/>Extract text (OCR if needed)
        AIService->>AIService: LLM analysis with structured schema
        AIService-->>FeatureLogic: Extracted values (AZ, BZ, etc.)
        FeatureLogic->>FeatureLogic: Update Parzelle with AI results
        FeatureLogic->>Database: Add Kontext entry
    end
    
    FeatureLogic->>Database: Save all entities<br/>(Projekt, Parzelle, Gemeinde, etc.)
    FeatureLogic-->>Route: Enriched Parzelle + status
    Route-->>Frontend: Response with enriched Parzelle
    Frontend->>User: Display enriched parcel data

User Story 2: Map Browse Flow

sequenceDiagram
    participant User
    participant Frontend
    participant Route
    participant FeatureLogic
    participant STACConnector
    participant STACAPI
    participant Database

    User->>Frontend: Click on map (browse mode)
    Frontend->>Route: GET /parcels/explore<br/>(coordinates, mandate_id?)
    Route->>Route: Validate coordinates<br/>Validate authentication
    
    Route->>FeatureLogic: fetchParcelData()
    FeatureLogic->>STACConnector: search_parcels_by_point(x, y)
    STACConnector->>STACAPI: Query cadastral collection<br/>(point geometry)
    STACAPI-->>STACConnector: Parcel items (geometry, properties)
    STACConnector-->>FeatureLogic: Parcel data
    
    FeatureLogic->>Database: Check parcel-project associations<br/>(search Parzelle by STAC identifier)
    Database-->>FeatureLogic: Matching Parzelle records<br/>with Projekt IDs
    
    alt Parcels Found in Database
        FeatureLogic->>Database: Get Projekt details
        Database-->>FeatureLogic: Projekt information
        FeatureLogic->>FeatureLogic: Status: Connected<br/>(list projects)
    else No Parcels Found
        FeatureLogic->>FeatureLogic: Status: Available
    end
    
    opt Basic Context Requested
        FeatureLogic->>STACConnector: search_administrative_boundaries()<br/>search_zoning()<br/>search_protection_zones()
        STACConnector->>STACAPI: Query collections (cached if possible)
        STACAPI-->>STACConnector: Context data
        STACConnector-->>FeatureLogic: Basic context
    end
    
    FeatureLogic-->>Route: Parcel data + association status + context
    Route-->>Frontend: Response with parcel info
    Frontend->>User: Display parcel details<br/>Show project association status<br/>Show action buttons
    
    opt User Chooses to Connect
        User->>Frontend: Click "Create Project" or "Add to Project"
        Frontend->>Route: POST /projects/{id}/parcels/connect<br/>(parcel_id, project_id)
        Route->>FeatureLogic: connectParcelToProject()
        Note over FeatureLogic: Triggers User Story 1<br/>enrichment process
        FeatureLogic->>Database: Create Parzelle + enrichment
        FeatureLogic-->>Route: Created Parzelle
        Route-->>Frontend: Success response
    end

AI Extraction Flow (User Story 1)

sequenceDiagram
    participant FeatureLogic
    participant Parzelle
    participant AIService
    participant PDFSource
    participant LLM
    participant Database

    Note over FeatureLogic: After STAC enrichment completes
    
    FeatureLogic->>Parzelle: Get dokumente collection
    Parzelle-->>FeatureLogic: List of Dokument objects
    
    FeatureLogic->>FeatureLogic: Filter documents:<br/>- mimeType == "application/pdf"<br/>- dokumentTyp in [BZO, Bauverordnung]
    
    FeatureLogic->>AIService: extractZoningRules(<br/>parzelleId,<br/>documents[],<br/>bauzone)
    
    loop For each PDF document
        AIService->>PDFSource: Download PDF from dokumentReferenz
        PDFSource-->>AIService: PDF file
        
        AIService->>AIService: Extract text content<br/>(OCR if scanned PDF)
        AIService->>AIService: Preprocess text
    end
    
    AIService->>LLM: Analyze text with structured schema<br/>(prompt: extract building parameters)
    LLM->>LLM: Parse zoning regulations
    LLM-->>AIService: Structured extraction:<br/>- az (float)<br/>- bz (float)<br/>- vollgeschossZahl (int)<br/>- gebaeudehoeheMax (float)<br/>- regelnGrenzabstand (array)<br/>- regelnMehrlaengenzuschlag (array)<br/>- regelnMehrhoehenzuschlag (array)
    
    AIService-->>FeatureLogic: Extraction results
    
    FeatureLogic->>FeatureLogic: Parse and validate results
    
    FeatureLogic->>Parzelle: Update fields:<br/>parzelle.az = result.az<br/>parzelle.bz = result.bz<br/>parzelle.vollgeschossZahl = result.vollgeschossZahl<br/>parzelle.gebaeudehoeheMax = result.gebaeudehoeheMax<br/>parzelle.regelnGrenzabstand = result.regelnGrenzabstand<br/>parzelle.regelnMehrlaengenzuschlag = result.regelnMehrlaengenzuschlag<br/>parzelle.regelnMehrhoehenzuschlag = result.regelnMehrhoehenzuschlag
    
    FeatureLogic->>Database: Create Kontext entry:<br/>thema: "Automatisch extrahierte Baukennzahlen"<br/>inhalt: "AZ/BZ etc. wurden aus BZO [Gemeinde] Stand [Date] extrahiert."
    
    FeatureLogic->>Database: Save updated Parzelle
    
    Database-->>FeatureLogic: Confirmation
    
    FeatureLogic-->>FeatureLogic: Return enrichment results<br/>including AI extraction status

Error Handling and Security

Error Handling

STAC API Errors: Retry with exponential backoff (3 attempts), validate coordinates (Switzerland bounds), handle no results found (404 for Story 1, informative response for Story 2), continue with available collections if some fail.

Database Errors: Transaction management with rollback, validate relationships before saving, handle parcel already connected scenarios.

AI Extraction Errors: Skip failed documents, return partial results, don't fail entire enrichment process.

Security

CSRF Protection: All POST/PATCH endpoints require CSRF tokens. GET endpoints validate authentication.

Input Validation: Validate coordinate ranges (Switzerland bounds), whitelist coordinate systems (EPSG:2056, EPSG:4326), validate UUID formats.

Rate Limiting: 60 requests/minute for location selection, 120 requests/minute for map browsing, 30 requests/minute for context refresh.

Data Privacy: STAC queries are read-only, parcel data stored per mandate with access control, mandate filtering for project associations.

Access Control: Users must have permission to create/modify projects, project association information filtered by mandate access, parcel connection requires project modification permissions.

Testing and Configuration

Testing Approach

Unit Tests: STAC connector methods with mock responses, feature logic with mocked connector, project association checking with various database states.

Integration Tests: End-to-end flows for both user stories, enrichment pipeline, project association scenarios (not connected, one project, multiple projects, different mandates).

Mock STAC Responses: Create realistic mock responses for development and testing without external API dependency.

Configuration Requirements

Environment Variables: STAC base URL, timeout settings, retry configuration, caching settings (different for User Story 1 vs User Story 2), map browse cache TTL.

Configuration Files: API endpoint URLs, timeout and retry parameters, caching behavior, collection identifiers.

Dependencies

Python Packages: STAC API client, geospatial operations library, GeoJSON support library.

System Requirements: Python 3.10+, internet connection for STAC API, PostgreSQL database, optional PostGIS extension.

External Services: Swiss Topo STAC API (public, no authentication), AI service infrastructure (existing).

Summary

This integration guide provides a comprehensive conceptual blueprint for integrating the Swiss Topo STAC API into the Real Estate feature. The implementation enables two distinct user workflows:

User Story 1: Create Project and Connect Parcels

Create new projects and select parcels on a map
Automatic enrichment with administrative, zoning, and protection zone data
Full data persistence in database
AI-powered extraction of building rules from documents
Complete project management workflow

User Story 2: Browse Map and Explore Parcels

Explore parcels on a map without project creation
Lightweight data fetching from STAC
Check if parcels are connected to existing projects
Optional connection to projects with full enrichment

The architecture follows the existing Gateway patterns:

Connectors for external API communication
Interfaces for normalized data access
Features for business logic
Routes for API endpoints

Both user stories share the STAC connector and enrichment logic, but serve different purposes: User Story 1 focuses on project creation and data persistence, while User Story 2 focuses on exploration and discovery before committing to project creation.

The route structure is refactored from generic table-based endpoints to domain-focused endpoints that better align with user workflows, with a phased migration strategy to ensure backward compatibility.

Document Version: 2.1
Last Updated: 2025-01-28
Author: Gateway Development Team

25 KiB Raw Blame History

Swiss Topo STAC API Integration Guide

Table of Contents

Overview

Purpose

STAC API Overview

Key STAC Collections for Real Estate

Architecture

User Story 1: Create Project and Connect Parcels

Workflow Steps

User Story 2: Browse Map and Explore Parcels

Workflow Steps

Key Differences from User Story 1

API Routes and Endpoints

Route Migration Overview

Routes to Deprecate

Routes to Keep

New Domain-Focused Routes

Project Management

Parcel Management (User Story 1)

Map Browsing (User Story 2)

Migration Strategy

Implementation Components

STAC Connector

Feature Logic Extensions

AI Service Extension

Data Flow Summary

User Story 1: Location Selection Flow

User Story 2: Map Browse Flow

AI Extraction Flow (User Story 1)

Error Handling and Security

Error Handling

Security

Testing and Configuration

Testing Approach

Configuration Requirements

Dependencies

Summary

25 KiB

Raw Blame History