# Datamodels and Interfaces Component ## Overview The Datamodels and Interfaces components form the core data layer of the Gateway application. They provide a clean separation between data structures (datamodels) and data access logic (interfaces), enabling type-safe, maintainable, and scalable data operations throughout the application. ## Component Architecture ```mermaid graph TB subgraph "Application Layer" App[Application
Routes, Services, Features] end subgraph "Interfaces Layer" IF_RealEstate[Real Estate Interface] IF_Chat[Chat Interface] IF_App[App Interface] IF_Component[Component Interface] IF_AI[AI Interface] IF_Ticket[Ticket Interface] IF_Voice[Voice Interface] end subgraph "Access Control Layer" AC_RealEstate[Real Estate Access] AC_Chat[Chat Access] AC_App[App Access] AC_Component[Component Access] end subgraph "Database Connectors" Connector_Postgre[PostgreSQL Connector] end subgraph "Databases" DB_RealEstate[(Real Estate
Database)] DB_Chat[(Chat
Database)] DB_App[(App
Database)] DB_Component[(Component
Database)] end subgraph "External Systems" External_AI[AI APIs
OpenAI, Anthropic, etc.] External_Tickets[Ticket Systems
Jira, ClickUp] External_Voice[Voice Services
Google Cloud] end App -.->|uses| IF_RealEstate App -.->|uses| IF_Chat App -.->|uses| IF_App App -.->|uses| IF_Component App -.->|uses| IF_AI App -.->|uses| IF_Ticket App -.->|uses| IF_Voice IF_RealEstate --> AC_RealEstate IF_Chat --> AC_Chat IF_App --> AC_App IF_Component --> AC_Component AC_RealEstate --> Connector_Postgre AC_Chat --> Connector_Postgre AC_App --> Connector_Postgre AC_Component --> Connector_Postgre Connector_Postgre --> DB_RealEstate Connector_Postgre --> DB_Chat Connector_Postgre --> DB_App Connector_Postgre --> DB_Component IF_AI --> External_AI IF_Ticket --> External_Tickets IF_Voice --> External_Voice IF_RealEstate -.->|uses| DM_RealEstate[Real Estate
Datamodels] IF_Chat -.->|uses| DM_Chat[Chat
Datamodels] IF_App -.->|uses| DM_UAM[User & Mandate
Datamodels] IF_Component -.->|uses| DM_Files[File
Datamodels] IF_AI -.->|uses| DM_AI[AI
Datamodels] ``` ## Data Flow ```mermaid sequenceDiagram participant Route as API Route participant Service as Service Layer participant Interface as Interface participant Access as Access Control participant Connector as Database Connector participant DB as Database Route->>Service: Request with User Context Service->>Interface: Initialize with User Interface->>Access: Check Permissions Access-->>Interface: Permission Granted Service->>Interface: CRUD Operation Interface->>Access: Validate Access Access-->>Interface: Access Validated Interface->>Connector: Execute Query Connector->>DB: SQL Query DB-->>Connector: Result Set Connector-->>Interface: Data Objects Interface->>Access: Apply Filtering Access-->>Interface: Filtered Data Interface-->>Service: Datamodel Instances Service-->>Route: Response Data ``` ## Component Structure ### Datamodels Structure ``` modules/datamodels/ ├── datamodelRealEstate.py # Real estate domain models ├── datamodelChat.py # Chat workflow models ├── datamodelAi.py # AI operation models ├── datamodelUam.py # User and mandate models ├── datamodelSecurity.py # Security and authentication models ├── datamodelFiles.py # File management models ├── datamodelDocument.py # Document structure models ├── datamodelExtraction.py # Content extraction models ├── datamodelPagination.py # Pagination models ├── datamodelVoice.py # Voice settings models ├── datamodelTickets.py # Ticket system models ├── datamodelNeutralizer.py # Data neutralization models ├── datamodelTools.py # Tool definitions ├── datamodelUtils.py # Utility models └── __init__.py # Package exports ``` ### Interfaces Structure ``` modules/interfaces/ ├── interfaceDbRealEstateObjects.py # Real estate data access ├── interfaceDbRealEstateAccess.py # Real estate access control ├── interfaceDbChatObjects.py # Chat data access ├── interfaceDbChatAccess.py # Chat access control ├── interfaceDbAppObjects.py # App/user management access ├── interfaceDbAppAccess.py # App access control ├── interfaceDbComponentObjects.py # Component management access ├── interfaceDbComponentAccess.py # Component access control ├── interfaceAiObjects.py # AI operations interface ├── interfaceTicketObjects.py # Ticket system interface └── interfaceVoiceObjects.py # Voice operations interface ``` --- ## Datamodels Component ### datamodelRealEstate.py ```mermaid erDiagram Projekt { string id PK string mandateId string label string statusProzess json perimeter json baulinie json parzellen json dokumente json kontextInformationen } Parzelle { string id PK string mandateId string label string kontextGemeinde FK json perimeter json baulinie string bauzone float az float bz json dokumente json kontextInformationen } Land { string id PK string mandateId string label string abk json dokumente json kontextInformationen } Kanton { string id PK string mandateId string label string id_land FK string abk json dokumente json kontextInformationen } Gemeinde { string id PK string mandateId string label string id_kanton FK string plz json dokumente json kontextInformationen } Dokument { string id PK string mandateId string label string versionsbezeichnung string dokumentTyp string dokumentReferenz string quelle string mimeType json kategorienTags } Kontext { string id PK string thema string inhalt } GeoPunkt { string koordinatensystem float x float y float z string referenz } GeoPolylinie { string id PK bool closed json punkte } Projekt ||--o{ Parzelle : contains Projekt ||--o{ Dokument : references Projekt ||--o{ Kontext : has Parzelle ||--o{ GeoPolylinie : contains Parzelle ||--o{ GeoPunkt : contains Parzelle }o--|| Gemeinde : located_in Land ||--o{ Kanton : contains Kanton ||--o{ Gemeinde : contains Land ||--o{ Dokument : has Land ||--o{ Kontext : has Kanton ||--o{ Dokument : has Kanton ||--o{ Kontext : has Gemeinde ||--o{ Dokument : has Gemeinde ||--o{ Kontext : has GeoPolylinie ||--o{ GeoPunkt : contains ``` ### datamodelChat.py ```mermaid erDiagram ChatWorkflow { string id PK string mandateId string status string name int currentRound int currentTask int currentAction int totalTasks int totalActions float lastActivity float startedAt string workflowMode int maxSteps json logs json messages json stats json tasks } ChatMessage { string id PK string workflowId FK string parentMessageId FK string message string summary string role string status int sequenceNr float publishedAt bool success string actionId json documents } ChatLog { string id PK string workflowId FK string message string type float timestamp string status float progress json performance } ChatStat { string id PK string workflowId FK float processingTime int bytesSent int bytesReceived int errorCount string process string engine float priceUsd } ChatDocument { string id PK string messageId FK string fileId FK string fileName int fileSize string mimeType int roundNumber int taskNumber int actionNumber string actionId } TaskPlan { string overview json tasks string userMessage } TaskItem { string id PK string workflowId FK string userInput string status string error float startedAt float finishedAt json actionList int retryCount int retryMax bool rollbackOnFailure json dependencies string feedback float processingTime json resultLabels } TaskStep { string id PK string objective json dependencies json successCriteria string estimatedComplexity string userMessage string dataType json expectedFormats json qualityRequirements } ActionItem { string id PK string execMethod string execAction json execParameters string execResultLabel json expectedDocumentFormats string userMessage string status string error int retryCount int retryMax float processingTime float timestamp string result } ActionResult { bool success string error json documents string resultLabel } AutomationDefinition { string id PK string mandateId string label string schedule string template json placeholders bool active string eventId string status json executionLogs } ChatWorkflow ||--o{ ChatMessage : contains ChatWorkflow ||--o{ ChatLog : contains ChatWorkflow ||--o{ ChatStat : contains ChatWorkflow ||--o{ TaskPlan : has ChatWorkflow ||--o{ AutomationDefinition : defines ChatMessage ||--o{ ChatDocument : references TaskPlan ||--o{ TaskStep : contains TaskItem ||--o{ ActionItem : contains ActionItem ||--o{ ActionResult : produces ``` ### datamodelAi.py ```mermaid erDiagram AiModel { string name PK string displayName string connectorType string apiUrl float temperature int maxTokens int contextLength float costPer1kTokensInput float costPer1kTokensOutput int speedRating int qualityRating string priority string processingMode json operationTypes int minContextLength bool isAvailable string version string lastUpdated } OperationTypeRating { string operationType int rating } AiCallOptions { string operationType string priority bool compressPrompt bool compressContext bool processDocumentsIndividually float maxCost int maxProcessingTime string processingMode string resultFormat float safetyMargin float temperature int maxParts } AiCallRequest { string prompt string context json options json contentParts } AiCallResponse { string content string modelName float priceUsd float processingTime int bytesSent int bytesReceived int errorCount } AiModelCall { json messages json model json options } AiModelResponse { string content bool success string error string modelId float processingTime json tokensUsed json metadata } AiModel ||--o{ OperationTypeRating : has AiCallRequest ||--|| AiCallOptions : uses AiCallRequest ||--|| AiCallResponse : produces AiCallRequest }o--|| AiModel : uses AiModelCall }o--|| AiModel : uses AiModelCall ||--|| AiCallOptions : uses AiModelCall ||--|| AiModelResponse : produces ``` ### datamodelUam.py ```mermaid erDiagram Mandate { string id PK string name string language bool enabled } User { string id PK string username string email string fullName string language bool enabled string privilege string authenticationAuthority string mandateId FK } UserConnection { string id PK string userId FK string authority string externalId string externalUsername string externalEmail string status float connectedAt float lastChecked float expiresAt string tokenStatus float tokenExpiresAt } UserInDB { string id PK string username string email string fullName string language bool enabled string privilege string authenticationAuthority string mandateId FK string hashedPassword } Mandate ||--o{ User : contains User ||--o{ UserConnection : has User ||--|| UserInDB : extends ``` ### datamodelSecurity.py ```mermaid erDiagram Token { string id PK string userId FK string authority string connectionId FK string tokenAccess string tokenType float expiresAt string tokenRefresh float createdAt string status float revokedAt string revokedBy string reason string sessionId string mandateId FK } AuthEvent { string id PK string userId FK string eventType float timestamp string ipAddress string userAgent bool success string details } Token ||--o{ AuthEvent : generates ``` ### datamodelFiles.py ```mermaid erDiagram FileItem { string id PK string mandateId FK string fileName string mimeType string fileHash int fileSize float creationDate } FilePreview { string content string mimeType string fileName bool isText string encoding int size } FileData { string id PK string data bool base64Encoded } FileItem ||--|| FilePreview : generates FileItem ||--|| FileData : contains ``` ### datamodelDocument.py ```mermaid erDiagram StructuredDocument { json metadata json sections string summary json tags } DocumentMetadata { string title string author datetime createdAt json sourceDocuments string extractionMethod string version } DocumentSection { string id PK string title string contentType json elements int order json metadata } Paragraph { string text json formatting json metadata } Heading { string text int level json metadata } CodeBlock { string code string language json metadata } Image { string data string altText string caption json metadata } BulletList { json items string listType json metadata } ListItem { string text json subitems json metadata } TableData { json headers json rows string caption json metadata } StructuredDocument ||--|| DocumentMetadata : has StructuredDocument ||--o{ DocumentSection : contains DocumentSection ||--o{ Paragraph : can_contain DocumentSection ||--o{ Heading : can_contain DocumentSection ||--o{ CodeBlock : can_contain DocumentSection ||--o{ Image : can_contain DocumentSection ||--o{ BulletList : can_contain DocumentSection ||--o{ TableData : can_contain BulletList ||--o{ ListItem : contains ListItem ||--o{ ListItem : contains ``` ### datamodelExtraction.py ```mermaid erDiagram ContentExtracted { string id PK json parts json summary } ContentPart { string id PK string parentId FK string label string typeGroup string mimeType string data json metadata } ExtractionOptions { string prompt string operationType bool processDocumentsIndividually int imageMaxPixels int imageQuality json mergeStrategy bool chunkAllowed int maxSize int textChunkSize int imageChunkSize bool enableParallelProcessing int maxConcurrentChunks } MergeStrategy { string groupBy string orderBy string mergeType int maxSize json textMerge json tableMerge json structureMerge json aiResultMerge bool preserveChunks string chunkSeparator bool preserveMetadata json metadataFields string onError bool validateContent bool useIntelligentMerging string prompt json capabilities } PartResult { json originalPart string aiResult int partIndex string documentId float processingTime json metadata } ChunkResult { json originalChunk string aiResult int chunkIndex string documentId float processingTime json metadata } ContentExtracted ||--o{ ContentPart : contains ContentPart ||--o{ ContentPart : parent_of ContentExtracted ||--|| ExtractionOptions : uses ExtractionOptions ||--|| MergeStrategy : uses ContentPart ||--|| PartResult : produces ContentPart ||--|| ChunkResult : produces ``` ### datamodelPagination.py ```mermaid erDiagram PaginationParams { int page int pageSize string sortBy string sortOrder } PaginationRequest { json params json sortFields } SortField { string field string order } PaginatedResult { json items json metadata json params } PaginationMetadata { int page int pageSize int totalItems int totalPages bool hasNext bool hasPrevious } PaginationRequest ||--|| PaginationParams : uses PaginationRequest ||--o{ SortField : contains PaginatedResult ||--|| PaginationMetadata : has PaginatedResult ||--o{ PaginationParams : uses ``` ### datamodelVoice.py ```mermaid erDiagram VoiceSettings { string id PK string userId FK string language string voice json settings } ``` ### datamodelTickets.py ```mermaid erDiagram TicketFieldAttribute { string fieldName PK string fieldType json fieldConfig } ``` ### datamodelNeutralizer.py ```mermaid erDiagram DataNeutraliserConfig { string id PK string mandateId FK string name bool enabled json attributes } DataNeutralizerAttributes { string fieldName PK string neutralizationType json options } DataNeutraliserConfig ||--o{ DataNeutralizerAttributes : contains ``` ### datamodelUtils.py ```mermaid erDiagram Prompt { string id PK string name string content json metadata } ``` ### datamodelTools.py ```mermaid erDiagram CountryCodes { string ISO2Code PK string tavilyName string perplexityName } ``` **Note**: `CountryCodes` is a utility class (not a Pydantic BaseModel) that provides static methods for country code mapping. It contains a mapping dictionary but is not persisted to the database. ### datamodelJson.py No database models - contains JSON template constants and supported section types. --- ## Interfaces Component ### Overview The Interfaces component provides a clean abstraction layer for data access operations. Interfaces handle CRUD operations, user context management, access control, and integration with database connectors and external systems. ### Objects vs Access Files Interfaces are split into two file types: #### Objects Files (`interface*Objects.py`) **Purpose**: Business logic and CRUD operations for data entities. **Responsibilities**: - CRUD operations (Create, Read, Update, Delete) - Data validation and transformation - Business rule enforcement - Database/external system communication - User context management - Pagination and filtering **Pattern**: Each Objects file contains methods for manipulating domain entities (e.g., `createProjekt()`, `getWorkflow()`, `updateUser()`). #### Access Files (`interface*Access.py`) **Purpose**: Permission checking and data filtering based on user privileges. **Responsibilities**: - User privilege validation - Mandate-based filtering - Record ownership checking - Access control attribute generation (`_hideView`, `_hideEdit`, `_hideDelete`) - Permission decision logic **Pattern**: Access files contain two main methods: - `uam()`: Unified Access Management - filters recordsets and adds access control flags - `canModify()`: Checks if user can create/update/delete records **Relationship**: ```mermaid graph TB Objects[Objects File] --> Access[Access File] Access --> UAM[uam Method] Access --> CanModify[canModify Method] Objects --> CRUD[CRUD Operations] CRUD --> AccessCheck{Check Access} AccessCheck -->|Read| UAM AccessCheck -->|Write| CanModify UAM --> Filter[Filter Records] UAM --> Flags[Add Access Flags] CanModify --> Permission{Permission?} Permission -->|Yes| Allow[Allow Operation] Permission -->|No| Deny[Deny Operation] ``` ### Interface Types Interfaces are categorized into two types based on their data source: #### Database Interfaces **Why Database Connectors?**: These interfaces manage persistent data stored in PostgreSQL databases. They use database connectors to: - Store structured data with relationships - Ensure data consistency and integrity - Provide ACID transactions - Support complex queries and filtering - Enable mandate-based data isolation **Characteristics**: - Use `DatabaseConnector` for PostgreSQL access - Implement Access classes for permission control - Support pagination and sorting - Apply mandate-based filtering automatically - Track record ownership (`_createdBy`) #### External System Interfaces **Why External Connectors?**: These interfaces integrate with external APIs and services. They use connectors to: - Communicate with third-party systems - Transform data between formats - Handle API authentication and rate limiting - Provide abstraction over external service complexity **Characteristics**: - Use specialized connectors (e.g., `ConnectorGoogleSpeech`, `ConnectorTicketJira`) - May not require user context (system-level operations) - Focus on data transformation and synchronization - Handle external API errors and retries ### Real Estate Interface (`interfaceDbRealEstateObjects.py`) **Type**: Database Interface **Database**: PostgreSQL (Real Estate database) **Access Control**: `interfaceDbRealEstateAccess.py` → `RealEstateAccess` **Purpose**: Manages real estate domain data including projects, parcels, administrative entities, and geographic information. **Why Database Connector**: Real estate data requires persistent storage with complex relationships (projects → parcels → administrative units), geographic data (polygons, points), and mandate-based isolation for multi-tenant scenarios. **CRUD Operations**: ```mermaid graph TB subgraph "Projekt Operations" PCreate[createProjekt] PGet[getProjekt] PGetAll[getProjekte] PUpdate[updateProjekt] PDelete[deleteProjekt] end subgraph "Parzelle Operations" ParCreate[createParzelle] ParGet[getParzelle] ParGetAll[getParzellen] ParUpdate[updateParzelle] ParDelete[deleteParzelle] end subgraph "Dokument Operations" DocCreate[createDokument] DocGet[getDokument] DocGetAll[getDokumente] DocUpdate[updateDokument] DocDelete[deleteDokument] end subgraph "Administrative Hierarchy" GemCreate[createGemeinde] GemGet[getGemeinde] GemGetAll[getGemeinden] GemUpdate[updateGemeinde] GemDelete[deleteGemeinde] KanCreate[createKanton] KanGet[getKanton] KanGetAll[getKantone] KanUpdate[updateKanton] KanDelete[deleteKanton] LanCreate[createLand] LanGet[getLand] LanGetAll[getLaender] LanUpdate[updateLand] LanDelete[deleteLand] end subgraph "Kontext Operations" KonCreate[createKontext] KonGet[getKontext] KonGetAll[getKontexte] KonUpdate[updateKontext] KonDelete[deleteKontext] end ``` **Complete CRUD List**: **Projekt**: - `createProjekt(projekt: Projekt) → Projekt` - `getProjekt(projektId: str) → Optional[Projekt]` - `getProjekte(recordFilter: Optional[Dict]) → List[Projekt]` - `updateProjekt(projektId: str, updateData: Dict) → Optional[Projekt]` - `deleteProjekt(projektId: str) → bool` **Parzelle**: - `createParzelle(parzelle: Parzelle) → Parzelle` - `getParzelle(parzelleId: str) → Optional[Parzelle]` - `getParzellen(recordFilter: Optional[Dict]) → List[Parzelle]` - `updateParzelle(parzelleId: str, updateData: Dict) → Optional[Parzelle]` - `deleteParzelle(parzelleId: str) → bool` **Dokument**: - `createDokument(dokument: Dokument) → Dokument` - `getDokument(dokumentId: str) → Optional[Dokument]` - `getDokumente(recordFilter: Optional[Dict]) → List[Dokument]` - `updateDokument(dokumentId: str, updateData: Dict) → Optional[Dokument]` - `deleteDokument(dokumentId: str) → bool` **Gemeinde**: - `createGemeinde(gemeinde: Gemeinde) → Gemeinde` - `getGemeinde(gemeindeId: str) → Optional[Gemeinde]` - `getGemeinden(recordFilter: Optional[Dict]) → List[Gemeinde]` - `updateGemeinde(gemeindeId: str, updateData: Dict) → Optional[Gemeinde]` - `deleteGemeinde(gemeindeId: str) → bool` **Kanton**: - `createKanton(kanton: Kanton) → Kanton` - `getKanton(kantonId: str) → Optional[Kanton]` - `getKantone(recordFilter: Optional[Dict]) → List[Kanton]` - `updateKanton(kantonId: str, updateData: Dict) → Optional[Kanton]` - `deleteKanton(kantonId: str) → bool` **Land**: - `createLand(land: Land) → Land` - `getLand(landId: str) → Optional[Land]` - `getLaender(recordFilter: Optional[Dict]) → List[Land]` - `updateLand(landId: str, updateData: Dict) → Optional[Land]` - `deleteLand(landId: str) → bool` **Kontext**: - `createKontext(kontext: Kontext) → Kontext` - `getKontext(kontextId: str) → Optional[Kontext]` - `getKontexte(recordFilter: Optional[Dict]) → List[Kontext]` - `updateKontext(kontextId: str, updateData: Dict) → Optional[Kontext]` - `deleteKontext(kontextId: str) → bool` **Access Control Flow**: ```mermaid graph TB Request[CRUD Request] --> Objects[RealEstateObjects] Objects --> Access[RealEstateAccess] Access --> CheckPriv[Check Privilege] CheckPriv -->|SYSADMIN| AllData[All Records] CheckPriv -->|ADMIN| MandateData[Mandate Records] CheckPriv -->|USER| OwnData[Own Records] AllData --> UAM[uam Method] MandateData --> UAM OwnData --> UAM UAM --> Filter[Filter by mandateId] UAM --> CheckOwn[Check _createdBy] UAM --> AddFlags[Add _hideView/_hideEdit/_hideDelete] Filter --> Return[Return Filtered Data] CheckOwn --> Return AddFlags --> Return ``` **Key Features**: - Geographic data support (GeoPolylinie, GeoPunkt) - Multi-level administrative hierarchy (Land → Kanton → Gemeinde) - Location name resolution (converts names to IDs for filtering) - Document versioning and management - Context information for projects and administrative units ### Chat Interface (`interfaceDbChatObjects.py`) **Type**: Database Interface **Database**: PostgreSQL (Chat database) **Access Control**: `interfaceDbChatAccess.py` → `ChatAccess` **Purpose**: Manages chat workflows, messages, logs, statistics, and automation definitions for AI-powered conversation workflows. **Why Database Connector**: Chat workflows require persistent storage for conversation history, workflow state, performance metrics, and automation configurations. Data must be queryable, filterable, and mandate-isolated. **CRUD Operations**: ```mermaid graph TB subgraph "ChatWorkflow Operations" WfGet[getWorkflows - with pagination] WfGetOne[getWorkflow - by ID] WfCreate[createWorkflow] WfUpdate[updateWorkflow] WfDelete[deleteWorkflow - cascade] end subgraph "ChatMessage Operations" MsgGet[getMessages - by workflowId, pagination] MsgCreate[createMessage] MsgUpdate[updateMessage] MsgDelete[deleteMessage] MsgDeleteFile[deleteFileFromMessage] end subgraph "ChatDocument Operations" DocGet[getDocuments - by messageId] DocCreate[createDocument] end subgraph "ChatLog Operations" LogGet[getLogs - by workflowId, pagination] LogCreate[createLog] end subgraph "ChatStat Operations" StatGet[getStats - by workflowId] StatCreate[createStat] end subgraph "AutomationDefinition Operations" AutoGet[getAllAutomationDefinitions - pagination] AutoGetOne[getAutomationDefinition - by ID] AutoCreate[createAutomationDefinition] AutoUpdate[updateAutomationDefinition] AutoDelete[deleteAutomationDefinition] end subgraph "Utility Operations" Unified[getUnifiedChatData - workflow snapshot] end ``` **Complete CRUD List**: **ChatWorkflow**: - `getWorkflows(pagination: Optional[PaginationParams]) → Union[List[Dict], PaginatedResult]` - `getWorkflow(workflowId: str) → Optional[ChatWorkflow]` - `createWorkflow(workflowData: Dict) → ChatWorkflow` - `updateWorkflow(workflowId: str, workflowData: Dict) → ChatWorkflow` - `deleteWorkflow(workflowId: str) → bool` (cascades to messages, logs, stats) **ChatMessage**: - `getMessages(workflowId: str, pagination: Optional[PaginationParams]) → Union[List[ChatMessage], PaginatedResult]` - `createMessage(messageData: Dict) → ChatMessage` - `updateMessage(messageId: str, messageData: Dict) → Dict` - `deleteMessage(workflowId: str, messageId: str) → bool` - `deleteFileFromMessage(workflowId: str, messageId: str, fileId: str) → bool` **ChatDocument**: - `getDocuments(messageId: str) → List[ChatDocument]` - `createDocument(documentData: Dict) → ChatDocument` **ChatLog**: - `getLogs(workflowId: str, pagination: Optional[PaginationParams]) → Union[List[ChatLog], PaginatedResult]` - `createLog(logData: Dict) → ChatLog` **ChatStat**: - `getStats(workflowId: str) → List[ChatStat]` - `createStat(statData: Dict) → ChatStat` **AutomationDefinition**: - `getAllAutomationDefinitions(pagination: Optional[PaginationParams]) → Union[List[Dict], PaginatedResult]` - `getAutomationDefinition(automationId: str) → Optional[Dict]` - `createAutomationDefinition(automationData: Dict) → Dict` - `updateAutomationDefinition(automationId: str, automationData: Dict) → Dict` - `deleteAutomationDefinition(automationId: str) → bool` **Utility Methods**: - `getUnifiedChatData(workflowId: str, afterTimestamp: Optional[float]) → Dict` (returns workflow snapshot with messages, logs, stats) **Access Control Flow**: ```mermaid graph TB Request[CRUD Request] --> Objects[ChatObjects] Objects --> Access[ChatAccess] Access --> CheckPriv[Check Privilege] CheckPriv -->|SYSADMIN| AllWorkflows[All Workflows] CheckPriv -->|ADMIN| MandateWorkflows[Mandate Workflows] CheckPriv -->|USER| OwnWorkflows[Own Workflows] AllWorkflows --> UAM[uam Method] MandateWorkflows --> UAM OwnWorkflows --> UAM UAM --> FilterWorkflow[Filter by workflowId mandate] UAM --> CheckOwn[Check _createdBy] UAM --> AddFlags[Add access flags] FilterWorkflow --> CheckChild[Check Child Access] CheckChild -->|Message| CheckWorkflow[Check workflow ownership] CheckChild -->|Log| CheckWorkflow CheckChild -->|Stat| CheckWorkflow CheckWorkflow --> Return[Return Filtered Data] CheckOwn --> Return AddFlags --> Return ``` **Key Features**: - Multi-round workflow support with state tracking - Normalized data model (workflows, messages, logs, stats in separate tables) - Cascade delete (deleting workflow removes all related data) - Pagination support for large datasets - Unified data retrieval for workflow snapshots - Automation workflow definitions - Document attachment management ### App Interface (`interfaceDbAppObjects.py`) **Type**: Database Interface **Database**: PostgreSQL (App database) **Access Control**: `interfaceDbAppAccess.py` → `AppAccess` **Purpose**: Manages users, mandates, authentication tokens, and application-level configuration. **Why Database Connector**: User accounts, mandates, and authentication data require secure, persistent storage with strict access control. This is the foundation for all other interfaces' user context. **CRUD Operations**: ```mermaid graph TB subgraph "User Operations" UserGet[getUsersByMandate - pagination] UserGetByUsername[getUserByUsername] UserGetOne[getUser - by ID] UserCreate[createUser - with password hash] UserUpdate[updateUser] UserDelete[deleteUser] end subgraph "UserConnection Operations" ConnGet[getUserConnections - by userId] ConnGetToken[getConnectionToken - by connectionId] end subgraph "Mandate Operations" ManGet[getAllMandates - pagination] ManGetOne[getMandate - by ID] ManCreate[createMandate] ManUpdate[updateMandate] ManDelete[deleteMandate] end subgraph "Neutralization Config" NeuGet[getNeutralizationConfig] NeuCreate[createOrUpdateNeutralizationConfig] NeuGetAttrs[getNeutralizationAttributes] NeuDeleteAttrs[deleteNeutralizationAttributes] end subgraph "Initialization" InitRoot[getRootInterface - system init] InitRecords[_initRootMandate, _initAdminUser, _initEventUser] end ``` **Complete CRUD List**: **User**: - `getUsersByMandate(mandateId: str, pagination: Optional[PaginationParams]) → Union[List[User], PaginatedResult]` - `getUserByUsername(username: str) → Optional[User]` - `getUser(userId: str) → Optional[User]` - `createUser(userData: Dict, password: Optional[str]) → User` (hashes password with Argon2) - `updateUser(userId: str, updateData: Union[Dict, User]) → User` - `deleteUser(userId: str) → bool` **UserConnection**: - `getUserConnections(userId: str) → List[UserConnection]` - `getConnectionToken(connectionId: str) → Optional[Token]` **Mandate**: - `getAllMandates(pagination: Optional[PaginationParams]) → Union[List[Mandate], PaginatedResult]` - `getMandate(mandateId: str) → Optional[Mandate]` - `createMandate(name: str, language: str) → Mandate` - `updateMandate(mandateId: str, updateData: Dict) → Mandate` - `deleteMandate(mandateId: str) → bool` **DataNeutraliserConfig**: - `getNeutralizationConfig() → Optional[DataNeutraliserConfig]` - `createOrUpdateNeutralizationConfig(configData: Dict) → DataNeutraliserConfig` - `getNeutralizationAttributes(file_id: str) → List[Dict]` - `deleteNeutralizationAttributes(file_id: str) → bool` **Special Methods**: - `getRootInterface() → AppObjects` (creates interface with system admin privileges for initialization) - `getInitialId(model_class: type) → Optional[str]` (gets first record ID for a model) **Access Control Flow**: ```mermaid graph TB Request[CRUD Request] --> Objects[AppObjects] Objects --> Access[AppAccess] Access --> CheckPriv[Check Privilege] CheckPriv -->|SYSADMIN| FullAccess[Full System Access] CheckPriv -->|ADMIN| MandateAccess[Mandate Access Only] CheckPriv -->|USER| SelfAccess[Own User Record Only] FullAccess --> UserOps[User Operations] MandateAccess --> UserOps SelfAccess --> UserOps UserOps --> CheckOwn[Check Ownership] CheckOwn -->|User CRUD| SYSADMINOnly{SYSADMIN?} CheckOwn -->|Mandate CRUD| SYSADMINOnly CheckOwn -->|Own Profile| Allow[Allow] SYSADMINOnly -->|Yes| Allow SYSADMINOnly -->|No| Deny[Deny] ``` **Key Features**: - Password hashing with Argon2 - Multi-provider authentication support (local, external) - System initialization (Root mandate, Admin user, Event user) - Mandate-based user isolation - Token management for external connections - Data neutralization configuration ### Component Interface (`interfaceDbComponentObjects.py`) **Type**: Database Interface **Database**: PostgreSQL (Component/Management database) **Access Control**: `interfaceDbComponentAccess.py` → `ComponentAccess` **Purpose**: Manages component-level data including files, prompts, and voice settings used across the application. **Why Database Connector**: Files, prompts, and voice settings require persistent storage with metadata, ownership tracking, and mandate isolation. Files need binary storage with preview generation. **CRUD Operations**: ```mermaid graph TB subgraph "File Operations" FileGetAll[getAllFiles - pagination] FileGet[getFile - by ID] FileCreate[createFile - name, mimeType, content] FileUpdate[updateFile] FileDelete[deleteFile] FileGetData[getFileData - binary content] FileGetContent[getFileContent - preview] FileCreateData[createFileData - store binary] end subgraph "Prompt Operations" PromptGetAll[getAllPrompts - pagination] PromptGet[getPrompt - by ID] PromptCreate[createPrompt] PromptUpdate[updatePrompt] PromptDelete[deletePrompt] end subgraph "VoiceSettings Operations" VoiceGet[getVoiceSettings - by userId] VoiceCreate[createVoiceSettings] VoiceUpdate[updateVoiceSettings] VoiceDelete[deleteVoiceSettings] VoiceGetOrCreate[getOrCreateVoiceSettings] end subgraph "Utilities" MimeType[getMimeType - from fileName] end ``` **Complete CRUD List**: **FileItem**: - `getAllFiles(pagination: Optional[PaginationParams]) → Union[List[FileItem], PaginatedResult]` - `getFile(fileId: str) → Optional[FileItem]` - `createFile(name: str, mimeType: str, content: bytes) → FileItem` (creates FileItem and FileData) - `updateFile(fileId: str, updateData: Dict) → Dict` - `deleteFile(fileId: str) → bool` (deletes FileItem and FileData) - `getFileData(fileId: str) → Optional[bytes]` (raw binary content) - `getFileContent(fileId: str) → Optional[FilePreview]` (generates preview) - `createFileData(fileId: str, data: bytes) → bool` (stores binary data) **Prompt**: - `getAllPrompts(pagination: Optional[PaginationParams]) → Union[List[Prompt], PaginatedResult]` - `getPrompt(promptId: str) → Optional[Prompt]` - `createPrompt(promptData: Dict) → Dict` - `updatePrompt(promptId: str, updateData: Dict) → Dict` - `deletePrompt(promptId: str) → bool` **VoiceSettings**: - `getVoiceSettings(userId: Optional[str]) → Optional[VoiceSettings]` - `createVoiceSettings(settingsData: Dict) → Dict` - `updateVoiceSettings(userId: str, updateData: Dict) → Dict` - `deleteVoiceSettings(userId: str) → bool` - `getOrCreateVoiceSettings(userId: Optional[str]) → VoiceSettings` **Utility Methods**: - `getMimeType(fileName: str) → str` (detects MIME type from extension) **Access Control Flow**: ```mermaid graph TB Request[CRUD Request] --> Objects[ComponentObjects] Objects --> Access[ComponentAccess] Access --> CheckPriv[Check Privilege] CheckPriv -->|SYSADMIN| AllFiles[All Files] CheckPriv -->|ADMIN| MandateFiles[Mandate Files] CheckPriv -->|USER| OwnFiles[Own Files] AllFiles --> UAM[uam Method] MandateFiles --> UAM OwnFiles --> UAM UAM --> FilterMandate[Filter by mandateId] UAM --> CheckOwn[Check _createdBy] UAM --> AddFlags[Add access flags] FilterMandate --> Return[Return Filtered Data] CheckOwn --> Return AddFlags --> Return ``` **Key Features**: - Binary file storage with metadata - Automatic preview generation (text, images, etc.) - MIME type detection - Prompt template management with initialization - Voice settings per user - File hash calculation for deduplication ### AI Interface (`interfaceAiObjects.py`) **Type**: External System Interface **Connectors**: Dynamic discovery via `modelRegistry` **Access Control**: None (system-level operations) **Purpose**: Provides centralized AI operations with dynamic model discovery, automatic model selection, and failover handling. **Why External Connector**: AI operations require communication with external APIs (OpenAI, Anthropic, Perplexity, Tavily) and internal AI services. The interface abstracts model selection, handles API calls, manages failover, and tracks costs. **Operations**: ```mermaid graph TB subgraph "AI Call Operations" Call[call - main entry point] CallText[call with text/context] CallParts[call with content parts] end subgraph "Model Information" ModelInfo[getModelInfo - by displayName] ModelsByTag[getModelsByTag - filter by tag] end subgraph "Internal Processing" SelectModel[_selectModel - dynamic selection] ProcessPart[_processContentPartWithFallback] MergeResults[_mergePartResults] CallWithModel[_callWithModel - execute API call] end ``` **Complete Operations List**: **AI Calls**: - `call(request: AiCallRequest, progressCallback=None) → AiCallResponse` (main entry point, handles text/context or content parts) - `getModelInfo(displayName: str) → Dict[str, Any]` (get model metadata) - `getModelsByTag(tag: str) → List[str]` (filter models by tag) **Internal Methods** (used by call): - `_selectModel(prompt: str, context: str, options: AiCallOptions) → str` (selects best model) - `_callWithTextContext(request: AiCallRequest) → AiCallResponse` (handles traditional text/context calls) - `_callWithContentParts(request: AiCallRequest, progressCallback) → AiCallResponse` (handles content parts with chunking) - `_processContentPartWithFallback(...) → AiCallResponse` (processes single part with failover) - `_callWithModel(model, prompt, context, options) → AiCallResponse` (executes actual API call) **Model Selection and Failover Flow**: ```mermaid graph TB Request[AI Call Request] --> CheckType{Request Type?} CheckType -->|Text/Context| TextPath[Text/Context Path] CheckType -->|Content Parts| PartsPath[Content Parts Path] TextPath --> GetFailover[Get Failover Model List] PartsPath --> GetFailover GetFailover --> SelectModel[Select Best Model] SelectModel --> FilterOp[Filter by Operation Type] FilterOp --> RateCap[Rate by Capabilities] RateCap --> ApplyPriority[Apply Priority Rules] ApplyPriority --> TryModel[Try Model Call] TryModel --> Success{Success?} Success -->|Yes| Return[Return Response] Success -->|No| NextModel{More Models?} NextModel -->|Yes| TryModel NextModel -->|No| Error[Return Error Response] PartsPath --> ProcessEach[Process Each Part] ProcessEach --> Chunk[Model-Aware Chunking] Chunk --> TryModel ProcessEach --> Merge[Merge Results] Merge --> Return ``` **Supported External Systems**: - **OpenAI**: GPT models via `aicorePluginOpenai` - **Anthropic**: Claude models via `aicorePluginAnthropic` - **Perplexity**: Search-enabled models via `aicorePluginPerplexity` - **Tavily**: Web search API via `aicorePluginTavily` - **Internal**: Custom models via `aicorePluginInternal` **Key Features**: - Dynamic model discovery (auto-registers available connectors) - Operation type-based model selection (e.g., IMAGE_ANALYSE, TEXT_GENERATION) - Automatic failover (tries multiple models on failure) - Model-aware chunking (respects model context limits) - Content part processing (handles images, text, tables) - Cost tracking (calculates USD cost per call) - Progress callbacks for long-running operations ### Ticket Interface (`interfaceTicketObjects.py`) **Type**: External System Interface **Connectors**: `ConnectorTicketJira`, `ConnectorTicketClickup` **Access Control**: Connector-level (API credentials) **Purpose**: Synchronizes data with external ticket systems (Jira, ClickUp) by transforming tickets to/from list format for Excel-like operations. **Why External Connector**: Ticket systems are external services with their own APIs. The interface provides bidirectional synchronization, field mapping, and data transformation between the application's list format and ticket system formats. **Operations**: ```mermaid graph TB subgraph "Factory Method" Factory[createTicketInterfaceByType - connectorType, params] end subgraph "Export Operations" Export[exportTicketsAsList - read from external] end subgraph "Import Operations" Import[importListToTickets - write to external] end subgraph "Internal Transformation" Transform[_transformTicketRecords - field mapping] Extract[_extractFieldValue - path-based extraction] FormatDate[_formatDateForExcel - date formatting] FilterEmpty[_filterEmptyRecords - validation] end ``` **Complete Operations List**: **Factory**: - `createTicketInterfaceByType(taskSyncDefinition: dict, connectorType: str, connectorParams: dict) → TicketInterface` (creates interface with appropriate connector) **Export**: - `exportTicketsAsList() → list[dict]` (reads tickets from external system, transforms to list format) **Import**: - `importListToTickets(records: list[dict]) → None` (transforms list format, writes to external system) **Internal Methods**: - `_transformTicketRecords(tasks: list[dict], includePut: bool) → list[dict]` (transforms according to task_sync_definition) - `_extractFieldValue(issue_data: dict, field_path: list[str], field_name: str) → Any` (extracts value using path) - `_formatDateForExcel(date_value: Any) → Optional[str]` (formats dates for Excel compatibility) - `_isDateField(field_name: str) → bool` (detects date fields) - `_filterEmptyRecords(records: list[dict]) → list[dict]` (removes invalid records) **Synchronization Flow**: ```mermaid graph TB subgraph "Export Flow" ExportStart[exportTicketsAsList] --> ReadConn[Read from Connector] ReadConn --> GetTasks[Get Tasks from API] GetTasks --> Transform[Transform Fields] Transform --> MapFields[Map via task_sync_definition] MapFields --> FormatDates[Format Dates] FormatDates --> Filter[Filter Empty Records] Filter --> ReturnList[Return List Format] end subgraph "Import Flow" ImportStart[importListToTickets] --> ReceiveList[Receive List Format] ReceiveList --> ExtractFields[Extract Fields] ExtractFields --> MapToTicket[Map to Ticket Format] MapToTicket --> BuildUpdate[Build Update Payload] BuildUpdate --> WriteConn[Write via Connector] WriteConn --> UpdateAPI[Update External API] end subgraph "Field Mapping" SyncDef[task_sync_definition] --> Direction[Direction: get/put] SyncDef --> Path[Field Path: nested access] Direction --> Transform Path --> MapFields Path --> MapToTicket end ``` **Supported External Systems**: - **Jira**: Via `ConnectorTicketJira` (uses Jira REST API) - **ClickUp**: Via `ConnectorTicketClickup` (uses ClickUp API) **Key Features**: - Dynamic connector selection (Jira or ClickUp) - Field mapping configuration (task_sync_definition maps fields bidirectionally) - Path-based field extraction (supports nested JSON structures) - Date format handling (converts various formats to Excel-compatible) - Bidirectional sync (export to list, import from list) - Empty record filtering (validates records have IDs) ### Voice Interface (`interfaceVoiceObjects.py`) **Type**: External System Interface **Connector**: `ConnectorGoogleSpeech` **Access Control**: User context for settings (stored in Component database) **Purpose**: Provides speech-to-text, text-to-speech, and translation services using Google Cloud APIs. **Why External Connector**: Voice operations require Google Cloud Speech-to-Text, Text-to-Speech, and Translation APIs. The interface abstracts API complexity, handles audio format conversion, and manages user voice settings (stored in database via Component interface). **Operations**: ```mermaid graph TB subgraph "Speech-to-Text Operations" STT[speechToText - audio to text] STTTrans[speechToTranslatedText - audio to translated text] end subgraph "Text-to-Speech Operations" TTS[textToSpeech - text to audio] TTSTrans[textToTranslatedSpeech - text to translated audio] end subgraph "Translation Operations" Trans[translateText - text translation] end subgraph "Voice Settings" GetSettings[getVoiceSettings - from Component DB] CreateSettings[createVoiceSettings - to Component DB] UpdateSettings[updateVoiceSettings - in Component DB] GetOrCreate[getOrCreateVoiceSettings] end subgraph "Metadata Operations" GetLangs[getAvailableLanguages - from Google API] GetVoices[getAvailableVoices - from Google API] end ``` **Complete Operations List**: **Speech-to-Text**: - `speechToText(audioContent: bytes, language: str, sampleRate: Optional[int], channels: Optional[int]) → Dict[str, Any]` (converts audio to text) - `speechToTranslatedText(audioContent: bytes, fromLanguage: str, toLanguage: str, sampleRate: Optional[int], channels: Optional[int]) → Dict[str, Any]` (converts audio to translated text) **Text-to-Speech**: - `textToSpeech(text: str, language: Optional[str], voice: Optional[str]) → Dict[str, Any]` (converts text to audio) - `textToTranslatedSpeech(text: str, fromLanguage: str, toLanguage: str, voice: Optional[str]) → Dict[str, Any]` (converts text to translated audio) **Translation**: - `translateText(text: str, sourceLanguage: str, targetLanguage: str) → Dict[str, Any]` (translates text) **Voice Settings** (delegates to Component interface): - `getVoiceSettings(userId: str) → Optional[VoiceSettings]` - `createVoiceSettings(settingsData: Dict) → Optional[VoiceSettings]` - `updateVoiceSettings(userId: str, settingsData: Dict) → Optional[VoiceSettings]` - `getOrCreateVoiceSettings(userId: str) → Optional[VoiceSettings]` **Metadata**: - `getAvailableLanguages() → Dict[str, Any]` (lists supported languages from Google API) - `getAvailableVoices(languageCode: Optional[str]) → Dict[str, Any]` (lists available voices, optionally filtered by language) **Operation Flow**: ```mermaid graph TB Request[Voice Operation Request] --> Voice[VoiceObjects] Voice --> CheckSettings{Need Settings?} CheckSettings -->|Yes| GetSettings[Get from Component DB] CheckSettings -->|No| Connector[Get Connector] GetSettings --> Connector Connector --> GoogleAPI[Google Cloud API] GoogleAPI --> STT{Operation Type?} STT -->|Speech-to-Text| STTAPI[Speech-to-Text API] STT -->|Text-to-Speech| TTSAPI[Text-to-Speech API] STT -->|Translation| TransAPI[Translation API] STTAPI --> Process[Process Response] TTSAPI --> Process TransAPI --> Process Process --> Format[Format Response] Format --> Return[Return Result] ``` **Supported External Systems**: - **Google Cloud Speech-to-Text**: Audio transcription - **Google Cloud Text-to-Speech**: Audio synthesis - **Google Cloud Translation**: Text translation **Key Features**: - Multi-language support (detects and supports many languages) - Audio format handling (auto-detects sample rate, channels) - Combined operations (speech-to-translated-text, text-to-translated-speech) - User voice preferences (stored in Component database) - Language and voice discovery (queries Google API for available options) - Error handling with detailed error messages ### Access Control Summary **Privilege Levels**: 1. **SYSADMIN**: Full system access, all mandates 2. **ADMIN**: Full access within mandate 3. **USER**: Access to own records only **Access Methods**: - `uam()`: Filters recordsets by privilege, adds `_hideView`, `_hideEdit`, `_hideDelete` flags - `canModify()`: Checks if user can create/update/delete records based on ownership and privilege **Singleton Pattern**: Interfaces use factory functions that cache instances per user context for efficient memory usage. **User Context**: All database interfaces require user context (User object with mandateId, userId, privilege) for access control and data filtering. ---