# Datamodels and Interfaces Component
## Overview
The Datamodels and Interfaces components form the core data layer of the Gateway application. They provide a clean separation between data structures (datamodels) and data access logic (interfaces), enabling type-safe, maintainable, and scalable data operations throughout the application.
## Component Architecture
```mermaid
graph TB
subgraph "Application Layer"
App[Application
Routes, Services, Features]
end
subgraph "Interfaces Layer"
IF_RealEstate[Real Estate Interface]
IF_Chat[Chat Interface]
IF_App[App Interface]
IF_Component[Component Interface]
IF_AI[AI Interface]
IF_Ticket[Ticket Interface]
IF_Voice[Voice Interface]
end
subgraph "Access Control Layer"
AC_RealEstate[Real Estate Access]
AC_Chat[Chat Access]
AC_App[App Access]
AC_Component[Component Access]
end
subgraph "Database Connectors"
Connector_Postgre[PostgreSQL Connector]
end
subgraph "Databases"
DB_RealEstate[(Real Estate
Database)]
DB_Chat[(Chat
Database)]
DB_App[(App
Database)]
DB_Component[(Component
Database)]
end
subgraph "External Systems"
External_AI[AI APIs
OpenAI, Anthropic, etc.]
External_Tickets[Ticket Systems
Jira, ClickUp]
External_Voice[Voice Services
Google Cloud]
end
App -.->|uses| IF_RealEstate
App -.->|uses| IF_Chat
App -.->|uses| IF_App
App -.->|uses| IF_Component
App -.->|uses| IF_AI
App -.->|uses| IF_Ticket
App -.->|uses| IF_Voice
IF_RealEstate --> AC_RealEstate
IF_Chat --> AC_Chat
IF_App --> AC_App
IF_Component --> AC_Component
AC_RealEstate --> Connector_Postgre
AC_Chat --> Connector_Postgre
AC_App --> Connector_Postgre
AC_Component --> Connector_Postgre
Connector_Postgre --> DB_RealEstate
Connector_Postgre --> DB_Chat
Connector_Postgre --> DB_App
Connector_Postgre --> DB_Component
IF_AI --> External_AI
IF_Ticket --> External_Tickets
IF_Voice --> External_Voice
IF_RealEstate -.->|uses| DM_RealEstate[Real Estate
Datamodels]
IF_Chat -.->|uses| DM_Chat[Chat
Datamodels]
IF_App -.->|uses| DM_UAM[User & Mandate
Datamodels]
IF_Component -.->|uses| DM_Files[File
Datamodels]
IF_AI -.->|uses| DM_AI[AI
Datamodels]
```
## Data Flow
```mermaid
sequenceDiagram
participant Route as API Route
participant Service as Service Layer
participant Interface as Interface
participant Access as Access Control
participant Connector as Database Connector
participant DB as Database
Route->>Service: Request with User Context
Service->>Interface: Initialize with User
Interface->>Access: Check Permissions
Access-->>Interface: Permission Granted
Service->>Interface: CRUD Operation
Interface->>Access: Validate Access
Access-->>Interface: Access Validated
Interface->>Connector: Execute Query
Connector->>DB: SQL Query
DB-->>Connector: Result Set
Connector-->>Interface: Data Objects
Interface->>Access: Apply Filtering
Access-->>Interface: Filtered Data
Interface-->>Service: Datamodel Instances
Service-->>Route: Response Data
```
## Component Structure
### Datamodels Structure
```
modules/datamodels/
├── datamodelRealEstate.py # Real estate domain models
├── datamodelChat.py # Chat workflow models
├── datamodelAi.py # AI operation models
├── datamodelUam.py # User and mandate models
├── datamodelSecurity.py # Security and authentication models
├── datamodelFiles.py # File management models
├── datamodelDocument.py # Document structure models
├── datamodelExtraction.py # Content extraction models
├── datamodelPagination.py # Pagination models
├── datamodelVoice.py # Voice settings models
├── datamodelTickets.py # Ticket system models
├── datamodelNeutralizer.py # Data neutralization models
├── datamodelTools.py # Tool definitions
├── datamodelUtils.py # Utility models
└── __init__.py # Package exports
```
### Interfaces Structure
```
modules/interfaces/
├── interfaceDbRealEstateObjects.py # Real estate data access
├── interfaceDbRealEstateAccess.py # Real estate access control
├── interfaceDbChatObjects.py # Chat data access
├── interfaceDbChatAccess.py # Chat access control
├── interfaceDbAppObjects.py # App/user management access
├── interfaceDbAppAccess.py # App access control
├── interfaceDbComponentObjects.py # Component management access
├── interfaceDbComponentAccess.py # Component access control
├── interfaceAiObjects.py # AI operations interface
├── interfaceTicketObjects.py # Ticket system interface
└── interfaceVoiceObjects.py # Voice operations interface
```
---
## Datamodels Component
### datamodelRealEstate.py
```mermaid
erDiagram
Projekt {
string id PK
string mandateId
string label
string statusProzess
json perimeter
json baulinie
json parzellen
json dokumente
json kontextInformationen
}
Parzelle {
string id PK
string mandateId
string label
string kontextGemeinde FK
json perimeter
json baulinie
string bauzone
float az
float bz
json dokumente
json kontextInformationen
}
Land {
string id PK
string mandateId
string label
string abk
json dokumente
json kontextInformationen
}
Kanton {
string id PK
string mandateId
string label
string id_land FK
string abk
json dokumente
json kontextInformationen
}
Gemeinde {
string id PK
string mandateId
string label
string id_kanton FK
string plz
json dokumente
json kontextInformationen
}
Dokument {
string id PK
string mandateId
string label
string versionsbezeichnung
string dokumentTyp
string dokumentReferenz
string quelle
string mimeType
json kategorienTags
}
Kontext {
string id PK
string thema
string inhalt
}
GeoPunkt {
string koordinatensystem
float x
float y
float z
string referenz
}
GeoPolylinie {
string id PK
bool closed
json punkte
}
Projekt ||--o{ Parzelle : contains
Projekt ||--o{ Dokument : references
Projekt ||--o{ Kontext : has
Parzelle ||--o{ GeoPolylinie : contains
Parzelle ||--o{ GeoPunkt : contains
Parzelle }o--|| Gemeinde : located_in
Land ||--o{ Kanton : contains
Kanton ||--o{ Gemeinde : contains
Land ||--o{ Dokument : has
Land ||--o{ Kontext : has
Kanton ||--o{ Dokument : has
Kanton ||--o{ Kontext : has
Gemeinde ||--o{ Dokument : has
Gemeinde ||--o{ Kontext : has
GeoPolylinie ||--o{ GeoPunkt : contains
```
### datamodelChat.py
```mermaid
erDiagram
ChatWorkflow {
string id PK
string mandateId
string status
string name
int currentRound
int currentTask
int currentAction
int totalTasks
int totalActions
float lastActivity
float startedAt
string workflowMode
int maxSteps
json logs
json messages
json stats
json tasks
}
ChatMessage {
string id PK
string workflowId FK
string parentMessageId FK
string message
string summary
string role
string status
int sequenceNr
float publishedAt
bool success
string actionId
json documents
}
ChatLog {
string id PK
string workflowId FK
string message
string type
float timestamp
string status
float progress
json performance
}
ChatStat {
string id PK
string workflowId FK
float processingTime
int bytesSent
int bytesReceived
int errorCount
string process
string engine
float priceUsd
}
ChatDocument {
string id PK
string messageId FK
string fileId FK
string fileName
int fileSize
string mimeType
int roundNumber
int taskNumber
int actionNumber
string actionId
}
TaskPlan {
string overview
json tasks
string userMessage
}
TaskItem {
string id PK
string workflowId FK
string userInput
string status
string error
float startedAt
float finishedAt
json actionList
int retryCount
int retryMax
bool rollbackOnFailure
json dependencies
string feedback
float processingTime
json resultLabels
}
TaskStep {
string id PK
string objective
json dependencies
json successCriteria
string estimatedComplexity
string userMessage
string dataType
json expectedFormats
json qualityRequirements
}
ActionItem {
string id PK
string execMethod
string execAction
json execParameters
string execResultLabel
json expectedDocumentFormats
string userMessage
string status
string error
int retryCount
int retryMax
float processingTime
float timestamp
string result
}
ActionResult {
bool success
string error
json documents
string resultLabel
}
AutomationDefinition {
string id PK
string mandateId
string label
string schedule
string template
json placeholders
bool active
string eventId
string status
json executionLogs
}
ChatWorkflow ||--o{ ChatMessage : contains
ChatWorkflow ||--o{ ChatLog : contains
ChatWorkflow ||--o{ ChatStat : contains
ChatWorkflow ||--o{ TaskPlan : has
ChatWorkflow ||--o{ AutomationDefinition : defines
ChatMessage ||--o{ ChatDocument : references
TaskPlan ||--o{ TaskStep : contains
TaskItem ||--o{ ActionItem : contains
ActionItem ||--o{ ActionResult : produces
```
### datamodelAi.py
```mermaid
erDiagram
AiModel {
string name PK
string displayName
string connectorType
string apiUrl
float temperature
int maxTokens
int contextLength
float costPer1kTokensInput
float costPer1kTokensOutput
int speedRating
int qualityRating
string priority
string processingMode
json operationTypes
int minContextLength
bool isAvailable
string version
string lastUpdated
}
OperationTypeRating {
string operationType
int rating
}
AiCallOptions {
string operationType
string priority
bool compressPrompt
bool compressContext
bool processDocumentsIndividually
float maxCost
int maxProcessingTime
string processingMode
string resultFormat
float safetyMargin
float temperature
int maxParts
}
AiCallRequest {
string prompt
string context
json options
json contentParts
}
AiCallResponse {
string content
string modelName
float priceUsd
float processingTime
int bytesSent
int bytesReceived
int errorCount
}
AiModelCall {
json messages
json model
json options
}
AiModelResponse {
string content
bool success
string error
string modelId
float processingTime
json tokensUsed
json metadata
}
AiModel ||--o{ OperationTypeRating : has
AiCallRequest ||--|| AiCallOptions : uses
AiCallRequest ||--|| AiCallResponse : produces
AiCallRequest }o--|| AiModel : uses
AiModelCall }o--|| AiModel : uses
AiModelCall ||--|| AiCallOptions : uses
AiModelCall ||--|| AiModelResponse : produces
```
### datamodelUam.py
```mermaid
erDiagram
Mandate {
string id PK
string name
string language
bool enabled
}
User {
string id PK
string username
string email
string fullName
string language
bool enabled
string privilege
string authenticationAuthority
string mandateId FK
}
UserConnection {
string id PK
string userId FK
string authority
string externalId
string externalUsername
string externalEmail
string status
float connectedAt
float lastChecked
float expiresAt
string tokenStatus
float tokenExpiresAt
}
UserInDB {
string id PK
string username
string email
string fullName
string language
bool enabled
string privilege
string authenticationAuthority
string mandateId FK
string hashedPassword
}
Mandate ||--o{ User : contains
User ||--o{ UserConnection : has
User ||--|| UserInDB : extends
```
### datamodelSecurity.py
```mermaid
erDiagram
Token {
string id PK
string userId FK
string authority
string connectionId FK
string tokenAccess
string tokenType
float expiresAt
string tokenRefresh
float createdAt
string status
float revokedAt
string revokedBy
string reason
string sessionId
string mandateId FK
}
AuthEvent {
string id PK
string userId FK
string eventType
float timestamp
string ipAddress
string userAgent
bool success
string details
}
Token ||--o{ AuthEvent : generates
```
### datamodelFiles.py
```mermaid
erDiagram
FileItem {
string id PK
string mandateId FK
string fileName
string mimeType
string fileHash
int fileSize
float creationDate
}
FilePreview {
string content
string mimeType
string fileName
bool isText
string encoding
int size
}
FileData {
string id PK
string data
bool base64Encoded
}
FileItem ||--|| FilePreview : generates
FileItem ||--|| FileData : contains
```
### datamodelDocument.py
```mermaid
erDiagram
StructuredDocument {
json metadata
json sections
string summary
json tags
}
DocumentMetadata {
string title
string author
datetime createdAt
json sourceDocuments
string extractionMethod
string version
}
DocumentSection {
string id PK
string title
string contentType
json elements
int order
json metadata
}
Paragraph {
string text
json formatting
json metadata
}
Heading {
string text
int level
json metadata
}
CodeBlock {
string code
string language
json metadata
}
Image {
string data
string altText
string caption
json metadata
}
BulletList {
json items
string listType
json metadata
}
ListItem {
string text
json subitems
json metadata
}
TableData {
json headers
json rows
string caption
json metadata
}
StructuredDocument ||--|| DocumentMetadata : has
StructuredDocument ||--o{ DocumentSection : contains
DocumentSection ||--o{ Paragraph : can_contain
DocumentSection ||--o{ Heading : can_contain
DocumentSection ||--o{ CodeBlock : can_contain
DocumentSection ||--o{ Image : can_contain
DocumentSection ||--o{ BulletList : can_contain
DocumentSection ||--o{ TableData : can_contain
BulletList ||--o{ ListItem : contains
ListItem ||--o{ ListItem : contains
```
### datamodelExtraction.py
```mermaid
erDiagram
ContentExtracted {
string id PK
json parts
json summary
}
ContentPart {
string id PK
string parentId FK
string label
string typeGroup
string mimeType
string data
json metadata
}
ExtractionOptions {
string prompt
string operationType
bool processDocumentsIndividually
int imageMaxPixels
int imageQuality
json mergeStrategy
bool chunkAllowed
int maxSize
int textChunkSize
int imageChunkSize
bool enableParallelProcessing
int maxConcurrentChunks
}
MergeStrategy {
string groupBy
string orderBy
string mergeType
int maxSize
json textMerge
json tableMerge
json structureMerge
json aiResultMerge
bool preserveChunks
string chunkSeparator
bool preserveMetadata
json metadataFields
string onError
bool validateContent
bool useIntelligentMerging
string prompt
json capabilities
}
PartResult {
json originalPart
string aiResult
int partIndex
string documentId
float processingTime
json metadata
}
ChunkResult {
json originalChunk
string aiResult
int chunkIndex
string documentId
float processingTime
json metadata
}
ContentExtracted ||--o{ ContentPart : contains
ContentPart ||--o{ ContentPart : parent_of
ContentExtracted ||--|| ExtractionOptions : uses
ExtractionOptions ||--|| MergeStrategy : uses
ContentPart ||--|| PartResult : produces
ContentPart ||--|| ChunkResult : produces
```
### datamodelPagination.py
```mermaid
erDiagram
PaginationParams {
int page
int pageSize
string sortBy
string sortOrder
}
PaginationRequest {
json params
json sortFields
}
SortField {
string field
string order
}
PaginatedResult {
json items
json metadata
json params
}
PaginationMetadata {
int page
int pageSize
int totalItems
int totalPages
bool hasNext
bool hasPrevious
}
PaginationRequest ||--|| PaginationParams : uses
PaginationRequest ||--o{ SortField : contains
PaginatedResult ||--|| PaginationMetadata : has
PaginatedResult ||--o{ PaginationParams : uses
```
### datamodelVoice.py
```mermaid
erDiagram
VoiceSettings {
string id PK
string userId FK
string language
string voice
json settings
}
```
### datamodelTickets.py
```mermaid
erDiagram
TicketFieldAttribute {
string fieldName PK
string fieldType
json fieldConfig
}
```
### datamodelNeutralizer.py
```mermaid
erDiagram
DataNeutraliserConfig {
string id PK
string mandateId FK
string name
bool enabled
json attributes
}
DataNeutralizerAttributes {
string fieldName PK
string neutralizationType
json options
}
DataNeutraliserConfig ||--o{ DataNeutralizerAttributes : contains
```
### datamodelUtils.py
```mermaid
erDiagram
Prompt {
string id PK
string name
string content
json metadata
}
```
### datamodelTools.py
```mermaid
erDiagram
CountryCodes {
string ISO2Code PK
string tavilyName
string perplexityName
}
```
**Note**: `CountryCodes` is a utility class (not a Pydantic BaseModel) that provides static methods for country code mapping. It contains a mapping dictionary but is not persisted to the database.
### datamodelJson.py
No database models - contains JSON template constants and supported section types.
---
## Interfaces Component
### Overview
The Interfaces component provides a clean abstraction layer for data access operations. Interfaces handle CRUD operations, user context management, access control, and integration with database connectors and external systems.
### Objects vs Access Files
Interfaces are split into two file types:
#### Objects Files (`interface*Objects.py`)
**Purpose**: Business logic and CRUD operations for data entities.
**Responsibilities**:
- CRUD operations (Create, Read, Update, Delete)
- Data validation and transformation
- Business rule enforcement
- Database/external system communication
- User context management
- Pagination and filtering
**Pattern**: Each Objects file contains methods for manipulating domain entities (e.g., `createProjekt()`, `getWorkflow()`, `updateUser()`).
#### Access Files (`interface*Access.py`)
**Purpose**: Permission checking and data filtering based on user privileges.
**Responsibilities**:
- User privilege validation
- Mandate-based filtering
- Record ownership checking
- Access control attribute generation (`_hideView`, `_hideEdit`, `_hideDelete`)
- Permission decision logic
**Pattern**: Access files contain two main methods:
- `uam()`: Unified Access Management - filters recordsets and adds access control flags
- `canModify()`: Checks if user can create/update/delete records
**Relationship**:
```mermaid
graph TB
Objects[Objects File] --> Access[Access File]
Access --> UAM[uam Method]
Access --> CanModify[canModify Method]
Objects --> CRUD[CRUD Operations]
CRUD --> AccessCheck{Check Access}
AccessCheck -->|Read| UAM
AccessCheck -->|Write| CanModify
UAM --> Filter[Filter Records]
UAM --> Flags[Add Access Flags]
CanModify --> Permission{Permission?}
Permission -->|Yes| Allow[Allow Operation]
Permission -->|No| Deny[Deny Operation]
```
### Interface Types
Interfaces are categorized into two types based on their data source:
#### Database Interfaces
**Why Database Connectors?**: These interfaces manage persistent data stored in PostgreSQL databases. They use database connectors to:
- Store structured data with relationships
- Ensure data consistency and integrity
- Provide ACID transactions
- Support complex queries and filtering
- Enable mandate-based data isolation
**Characteristics**:
- Use `DatabaseConnector` for PostgreSQL access
- Implement Access classes for permission control
- Support pagination and sorting
- Apply mandate-based filtering automatically
- Track record ownership (`_createdBy`)
#### External System Interfaces
**Why External Connectors?**: These interfaces integrate with external APIs and services. They use connectors to:
- Communicate with third-party systems
- Transform data between formats
- Handle API authentication and rate limiting
- Provide abstraction over external service complexity
**Characteristics**:
- Use specialized connectors (e.g., `ConnectorGoogleSpeech`, `ConnectorTicketJira`)
- May not require user context (system-level operations)
- Focus on data transformation and synchronization
- Handle external API errors and retries
### Real Estate Interface (`interfaceDbRealEstateObjects.py`)
**Type**: Database Interface
**Database**: PostgreSQL (Real Estate database)
**Access Control**: `interfaceDbRealEstateAccess.py` → `RealEstateAccess`
**Purpose**: Manages real estate domain data including projects, parcels, administrative entities, and geographic information.
**Why Database Connector**: Real estate data requires persistent storage with complex relationships (projects → parcels → administrative units), geographic data (polygons, points), and mandate-based isolation for multi-tenant scenarios.
**CRUD Operations**:
```mermaid
graph TB
subgraph "Projekt Operations"
PCreate[createProjekt]
PGet[getProjekt]
PGetAll[getProjekte]
PUpdate[updateProjekt]
PDelete[deleteProjekt]
end
subgraph "Parzelle Operations"
ParCreate[createParzelle]
ParGet[getParzelle]
ParGetAll[getParzellen]
ParUpdate[updateParzelle]
ParDelete[deleteParzelle]
end
subgraph "Dokument Operations"
DocCreate[createDokument]
DocGet[getDokument]
DocGetAll[getDokumente]
DocUpdate[updateDokument]
DocDelete[deleteDokument]
end
subgraph "Administrative Hierarchy"
GemCreate[createGemeinde]
GemGet[getGemeinde]
GemGetAll[getGemeinden]
GemUpdate[updateGemeinde]
GemDelete[deleteGemeinde]
KanCreate[createKanton]
KanGet[getKanton]
KanGetAll[getKantone]
KanUpdate[updateKanton]
KanDelete[deleteKanton]
LanCreate[createLand]
LanGet[getLand]
LanGetAll[getLaender]
LanUpdate[updateLand]
LanDelete[deleteLand]
end
subgraph "Kontext Operations"
KonCreate[createKontext]
KonGet[getKontext]
KonGetAll[getKontexte]
KonUpdate[updateKontext]
KonDelete[deleteKontext]
end
```
**Complete CRUD List**:
**Projekt**:
- `createProjekt(projekt: Projekt) → Projekt`
- `getProjekt(projektId: str) → Optional[Projekt]`
- `getProjekte(recordFilter: Optional[Dict]) → List[Projekt]`
- `updateProjekt(projektId: str, updateData: Dict) → Optional[Projekt]`
- `deleteProjekt(projektId: str) → bool`
**Parzelle**:
- `createParzelle(parzelle: Parzelle) → Parzelle`
- `getParzelle(parzelleId: str) → Optional[Parzelle]`
- `getParzellen(recordFilter: Optional[Dict]) → List[Parzelle]`
- `updateParzelle(parzelleId: str, updateData: Dict) → Optional[Parzelle]`
- `deleteParzelle(parzelleId: str) → bool`
**Dokument**:
- `createDokument(dokument: Dokument) → Dokument`
- `getDokument(dokumentId: str) → Optional[Dokument]`
- `getDokumente(recordFilter: Optional[Dict]) → List[Dokument]`
- `updateDokument(dokumentId: str, updateData: Dict) → Optional[Dokument]`
- `deleteDokument(dokumentId: str) → bool`
**Gemeinde**:
- `createGemeinde(gemeinde: Gemeinde) → Gemeinde`
- `getGemeinde(gemeindeId: str) → Optional[Gemeinde]`
- `getGemeinden(recordFilter: Optional[Dict]) → List[Gemeinde]`
- `updateGemeinde(gemeindeId: str, updateData: Dict) → Optional[Gemeinde]`
- `deleteGemeinde(gemeindeId: str) → bool`
**Kanton**:
- `createKanton(kanton: Kanton) → Kanton`
- `getKanton(kantonId: str) → Optional[Kanton]`
- `getKantone(recordFilter: Optional[Dict]) → List[Kanton]`
- `updateKanton(kantonId: str, updateData: Dict) → Optional[Kanton]`
- `deleteKanton(kantonId: str) → bool`
**Land**:
- `createLand(land: Land) → Land`
- `getLand(landId: str) → Optional[Land]`
- `getLaender(recordFilter: Optional[Dict]) → List[Land]`
- `updateLand(landId: str, updateData: Dict) → Optional[Land]`
- `deleteLand(landId: str) → bool`
**Kontext**:
- `createKontext(kontext: Kontext) → Kontext`
- `getKontext(kontextId: str) → Optional[Kontext]`
- `getKontexte(recordFilter: Optional[Dict]) → List[Kontext]`
- `updateKontext(kontextId: str, updateData: Dict) → Optional[Kontext]`
- `deleteKontext(kontextId: str) → bool`
**Access Control Flow**:
```mermaid
graph TB
Request[CRUD Request] --> Objects[RealEstateObjects]
Objects --> Access[RealEstateAccess]
Access --> CheckPriv[Check Privilege]
CheckPriv -->|SYSADMIN| AllData[All Records]
CheckPriv -->|ADMIN| MandateData[Mandate Records]
CheckPriv -->|USER| OwnData[Own Records]
AllData --> UAM[uam Method]
MandateData --> UAM
OwnData --> UAM
UAM --> Filter[Filter by mandateId]
UAM --> CheckOwn[Check _createdBy]
UAM --> AddFlags[Add _hideView/_hideEdit/_hideDelete]
Filter --> Return[Return Filtered Data]
CheckOwn --> Return
AddFlags --> Return
```
**Key Features**:
- Geographic data support (GeoPolylinie, GeoPunkt)
- Multi-level administrative hierarchy (Land → Kanton → Gemeinde)
- Location name resolution (converts names to IDs for filtering)
- Document versioning and management
- Context information for projects and administrative units
### Chat Interface (`interfaceDbChatObjects.py`)
**Type**: Database Interface
**Database**: PostgreSQL (Chat database)
**Access Control**: `interfaceDbChatAccess.py` → `ChatAccess`
**Purpose**: Manages chat workflows, messages, logs, statistics, and automation definitions for AI-powered conversation workflows.
**Why Database Connector**: Chat workflows require persistent storage for conversation history, workflow state, performance metrics, and automation configurations. Data must be queryable, filterable, and mandate-isolated.
**CRUD Operations**:
```mermaid
graph TB
subgraph "ChatWorkflow Operations"
WfGet[getWorkflows - with pagination]
WfGetOne[getWorkflow - by ID]
WfCreate[createWorkflow]
WfUpdate[updateWorkflow]
WfDelete[deleteWorkflow - cascade]
end
subgraph "ChatMessage Operations"
MsgGet[getMessages - by workflowId, pagination]
MsgCreate[createMessage]
MsgUpdate[updateMessage]
MsgDelete[deleteMessage]
MsgDeleteFile[deleteFileFromMessage]
end
subgraph "ChatDocument Operations"
DocGet[getDocuments - by messageId]
DocCreate[createDocument]
end
subgraph "ChatLog Operations"
LogGet[getLogs - by workflowId, pagination]
LogCreate[createLog]
end
subgraph "ChatStat Operations"
StatGet[getStats - by workflowId]
StatCreate[createStat]
end
subgraph "AutomationDefinition Operations"
AutoGet[getAllAutomationDefinitions - pagination]
AutoGetOne[getAutomationDefinition - by ID]
AutoCreate[createAutomationDefinition]
AutoUpdate[updateAutomationDefinition]
AutoDelete[deleteAutomationDefinition]
end
subgraph "Utility Operations"
Unified[getUnifiedChatData - workflow snapshot]
end
```
**Complete CRUD List**:
**ChatWorkflow**:
- `getWorkflows(pagination: Optional[PaginationParams]) → Union[List[Dict], PaginatedResult]`
- `getWorkflow(workflowId: str) → Optional[ChatWorkflow]`
- `createWorkflow(workflowData: Dict) → ChatWorkflow`
- `updateWorkflow(workflowId: str, workflowData: Dict) → ChatWorkflow`
- `deleteWorkflow(workflowId: str) → bool` (cascades to messages, logs, stats)
**ChatMessage**:
- `getMessages(workflowId: str, pagination: Optional[PaginationParams]) → Union[List[ChatMessage], PaginatedResult]`
- `createMessage(messageData: Dict) → ChatMessage`
- `updateMessage(messageId: str, messageData: Dict) → Dict`
- `deleteMessage(workflowId: str, messageId: str) → bool`
- `deleteFileFromMessage(workflowId: str, messageId: str, fileId: str) → bool`
**ChatDocument**:
- `getDocuments(messageId: str) → List[ChatDocument]`
- `createDocument(documentData: Dict) → ChatDocument`
**ChatLog**:
- `getLogs(workflowId: str, pagination: Optional[PaginationParams]) → Union[List[ChatLog], PaginatedResult]`
- `createLog(logData: Dict) → ChatLog`
**ChatStat**:
- `getStats(workflowId: str) → List[ChatStat]`
- `createStat(statData: Dict) → ChatStat`
**AutomationDefinition**:
- `getAllAutomationDefinitions(pagination: Optional[PaginationParams]) → Union[List[Dict], PaginatedResult]`
- `getAutomationDefinition(automationId: str) → Optional[Dict]`
- `createAutomationDefinition(automationData: Dict) → Dict`
- `updateAutomationDefinition(automationId: str, automationData: Dict) → Dict`
- `deleteAutomationDefinition(automationId: str) → bool`
**Utility Methods**:
- `getUnifiedChatData(workflowId: str, afterTimestamp: Optional[float]) → Dict` (returns workflow snapshot with messages, logs, stats)
**Access Control Flow**:
```mermaid
graph TB
Request[CRUD Request] --> Objects[ChatObjects]
Objects --> Access[ChatAccess]
Access --> CheckPriv[Check Privilege]
CheckPriv -->|SYSADMIN| AllWorkflows[All Workflows]
CheckPriv -->|ADMIN| MandateWorkflows[Mandate Workflows]
CheckPriv -->|USER| OwnWorkflows[Own Workflows]
AllWorkflows --> UAM[uam Method]
MandateWorkflows --> UAM
OwnWorkflows --> UAM
UAM --> FilterWorkflow[Filter by workflowId mandate]
UAM --> CheckOwn[Check _createdBy]
UAM --> AddFlags[Add access flags]
FilterWorkflow --> CheckChild[Check Child Access]
CheckChild -->|Message| CheckWorkflow[Check workflow ownership]
CheckChild -->|Log| CheckWorkflow
CheckChild -->|Stat| CheckWorkflow
CheckWorkflow --> Return[Return Filtered Data]
CheckOwn --> Return
AddFlags --> Return
```
**Key Features**:
- Multi-round workflow support with state tracking
- Normalized data model (workflows, messages, logs, stats in separate tables)
- Cascade delete (deleting workflow removes all related data)
- Pagination support for large datasets
- Unified data retrieval for workflow snapshots
- Automation workflow definitions
- Document attachment management
### App Interface (`interfaceDbAppObjects.py`)
**Type**: Database Interface
**Database**: PostgreSQL (App database)
**Access Control**: `interfaceDbAppAccess.py` → `AppAccess`
**Purpose**: Manages users, mandates, authentication tokens, and application-level configuration.
**Why Database Connector**: User accounts, mandates, and authentication data require secure, persistent storage with strict access control. This is the foundation for all other interfaces' user context.
**CRUD Operations**:
```mermaid
graph TB
subgraph "User Operations"
UserGet[getUsersByMandate - pagination]
UserGetByUsername[getUserByUsername]
UserGetOne[getUser - by ID]
UserCreate[createUser - with password hash]
UserUpdate[updateUser]
UserDelete[deleteUser]
end
subgraph "UserConnection Operations"
ConnGet[getUserConnections - by userId]
ConnGetToken[getConnectionToken - by connectionId]
end
subgraph "Mandate Operations"
ManGet[getAllMandates - pagination]
ManGetOne[getMandate - by ID]
ManCreate[createMandate]
ManUpdate[updateMandate]
ManDelete[deleteMandate]
end
subgraph "Neutralization Config"
NeuGet[getNeutralizationConfig]
NeuCreate[createOrUpdateNeutralizationConfig]
NeuGetAttrs[getNeutralizationAttributes]
NeuDeleteAttrs[deleteNeutralizationAttributes]
end
subgraph "Initialization"
InitRoot[getRootInterface - system init]
InitRecords[_initRootMandate, _initAdminUser, _initEventUser]
end
```
**Complete CRUD List**:
**User**:
- `getUsersByMandate(mandateId: str, pagination: Optional[PaginationParams]) → Union[List[User], PaginatedResult]`
- `getUserByUsername(username: str) → Optional[User]`
- `getUser(userId: str) → Optional[User]`
- `createUser(userData: Dict, password: Optional[str]) → User` (hashes password with Argon2)
- `updateUser(userId: str, updateData: Union[Dict, User]) → User`
- `deleteUser(userId: str) → bool`
**UserConnection**:
- `getUserConnections(userId: str) → List[UserConnection]`
- `getConnectionToken(connectionId: str) → Optional[Token]`
**Mandate**:
- `getAllMandates(pagination: Optional[PaginationParams]) → Union[List[Mandate], PaginatedResult]`
- `getMandate(mandateId: str) → Optional[Mandate]`
- `createMandate(name: str, language: str) → Mandate`
- `updateMandate(mandateId: str, updateData: Dict) → Mandate`
- `deleteMandate(mandateId: str) → bool`
**DataNeutraliserConfig**:
- `getNeutralizationConfig() → Optional[DataNeutraliserConfig]`
- `createOrUpdateNeutralizationConfig(configData: Dict) → DataNeutraliserConfig`
- `getNeutralizationAttributes(file_id: str) → List[Dict]`
- `deleteNeutralizationAttributes(file_id: str) → bool`
**Special Methods**:
- `getRootInterface() → AppObjects` (creates interface with system admin privileges for initialization)
- `getInitialId(model_class: type) → Optional[str]` (gets first record ID for a model)
**Access Control Flow**:
```mermaid
graph TB
Request[CRUD Request] --> Objects[AppObjects]
Objects --> Access[AppAccess]
Access --> CheckPriv[Check Privilege]
CheckPriv -->|SYSADMIN| FullAccess[Full System Access]
CheckPriv -->|ADMIN| MandateAccess[Mandate Access Only]
CheckPriv -->|USER| SelfAccess[Own User Record Only]
FullAccess --> UserOps[User Operations]
MandateAccess --> UserOps
SelfAccess --> UserOps
UserOps --> CheckOwn[Check Ownership]
CheckOwn -->|User CRUD| SYSADMINOnly{SYSADMIN?}
CheckOwn -->|Mandate CRUD| SYSADMINOnly
CheckOwn -->|Own Profile| Allow[Allow]
SYSADMINOnly -->|Yes| Allow
SYSADMINOnly -->|No| Deny[Deny]
```
**Key Features**:
- Password hashing with Argon2
- Multi-provider authentication support (local, external)
- System initialization (Root mandate, Admin user, Event user)
- Mandate-based user isolation
- Token management for external connections
- Data neutralization configuration
### Component Interface (`interfaceDbComponentObjects.py`)
**Type**: Database Interface
**Database**: PostgreSQL (Component/Management database)
**Access Control**: `interfaceDbComponentAccess.py` → `ComponentAccess`
**Purpose**: Manages component-level data including files, prompts, and voice settings used across the application.
**Why Database Connector**: Files, prompts, and voice settings require persistent storage with metadata, ownership tracking, and mandate isolation. Files need binary storage with preview generation.
**CRUD Operations**:
```mermaid
graph TB
subgraph "File Operations"
FileGetAll[getAllFiles - pagination]
FileGet[getFile - by ID]
FileCreate[createFile - name, mimeType, content]
FileUpdate[updateFile]
FileDelete[deleteFile]
FileGetData[getFileData - binary content]
FileGetContent[getFileContent - preview]
FileCreateData[createFileData - store binary]
end
subgraph "Prompt Operations"
PromptGetAll[getAllPrompts - pagination]
PromptGet[getPrompt - by ID]
PromptCreate[createPrompt]
PromptUpdate[updatePrompt]
PromptDelete[deletePrompt]
end
subgraph "VoiceSettings Operations"
VoiceGet[getVoiceSettings - by userId]
VoiceCreate[createVoiceSettings]
VoiceUpdate[updateVoiceSettings]
VoiceDelete[deleteVoiceSettings]
VoiceGetOrCreate[getOrCreateVoiceSettings]
end
subgraph "Utilities"
MimeType[getMimeType - from fileName]
end
```
**Complete CRUD List**:
**FileItem**:
- `getAllFiles(pagination: Optional[PaginationParams]) → Union[List[FileItem], PaginatedResult]`
- `getFile(fileId: str) → Optional[FileItem]`
- `createFile(name: str, mimeType: str, content: bytes) → FileItem` (creates FileItem and FileData)
- `updateFile(fileId: str, updateData: Dict) → Dict`
- `deleteFile(fileId: str) → bool` (deletes FileItem and FileData)
- `getFileData(fileId: str) → Optional[bytes]` (raw binary content)
- `getFileContent(fileId: str) → Optional[FilePreview]` (generates preview)
- `createFileData(fileId: str, data: bytes) → bool` (stores binary data)
**Prompt**:
- `getAllPrompts(pagination: Optional[PaginationParams]) → Union[List[Prompt], PaginatedResult]`
- `getPrompt(promptId: str) → Optional[Prompt]`
- `createPrompt(promptData: Dict) → Dict`
- `updatePrompt(promptId: str, updateData: Dict) → Dict`
- `deletePrompt(promptId: str) → bool`
**VoiceSettings**:
- `getVoiceSettings(userId: Optional[str]) → Optional[VoiceSettings]`
- `createVoiceSettings(settingsData: Dict) → Dict`
- `updateVoiceSettings(userId: str, updateData: Dict) → Dict`
- `deleteVoiceSettings(userId: str) → bool`
- `getOrCreateVoiceSettings(userId: Optional[str]) → VoiceSettings`
**Utility Methods**:
- `getMimeType(fileName: str) → str` (detects MIME type from extension)
**Access Control Flow**:
```mermaid
graph TB
Request[CRUD Request] --> Objects[ComponentObjects]
Objects --> Access[ComponentAccess]
Access --> CheckPriv[Check Privilege]
CheckPriv -->|SYSADMIN| AllFiles[All Files]
CheckPriv -->|ADMIN| MandateFiles[Mandate Files]
CheckPriv -->|USER| OwnFiles[Own Files]
AllFiles --> UAM[uam Method]
MandateFiles --> UAM
OwnFiles --> UAM
UAM --> FilterMandate[Filter by mandateId]
UAM --> CheckOwn[Check _createdBy]
UAM --> AddFlags[Add access flags]
FilterMandate --> Return[Return Filtered Data]
CheckOwn --> Return
AddFlags --> Return
```
**Key Features**:
- Binary file storage with metadata
- Automatic preview generation (text, images, etc.)
- MIME type detection
- Prompt template management with initialization
- Voice settings per user
- File hash calculation for deduplication
### AI Interface (`interfaceAiObjects.py`)
**Type**: External System Interface
**Connectors**: Dynamic discovery via `modelRegistry`
**Access Control**: None (system-level operations)
**Purpose**: Provides centralized AI operations with dynamic model discovery, automatic model selection, and failover handling.
**Why External Connector**: AI operations require communication with external APIs (OpenAI, Anthropic, Perplexity, Tavily) and internal AI services. The interface abstracts model selection, handles API calls, manages failover, and tracks costs.
**Operations**:
```mermaid
graph TB
subgraph "AI Call Operations"
Call[call - main entry point]
CallText[call with text/context]
CallParts[call with content parts]
end
subgraph "Model Information"
ModelInfo[getModelInfo - by displayName]
ModelsByTag[getModelsByTag - filter by tag]
end
subgraph "Internal Processing"
SelectModel[_selectModel - dynamic selection]
ProcessPart[_processContentPartWithFallback]
MergeResults[_mergePartResults]
CallWithModel[_callWithModel - execute API call]
end
```
**Complete Operations List**:
**AI Calls**:
- `call(request: AiCallRequest, progressCallback=None) → AiCallResponse` (main entry point, handles text/context or content parts)
- `getModelInfo(displayName: str) → Dict[str, Any]` (get model metadata)
- `getModelsByTag(tag: str) → List[str]` (filter models by tag)
**Internal Methods** (used by call):
- `_selectModel(prompt: str, context: str, options: AiCallOptions) → str` (selects best model)
- `_callWithTextContext(request: AiCallRequest) → AiCallResponse` (handles traditional text/context calls)
- `_callWithContentParts(request: AiCallRequest, progressCallback) → AiCallResponse` (handles content parts with chunking)
- `_processContentPartWithFallback(...) → AiCallResponse` (processes single part with failover)
- `_callWithModel(model, prompt, context, options) → AiCallResponse` (executes actual API call)
**Model Selection and Failover Flow**:
```mermaid
graph TB
Request[AI Call Request] --> CheckType{Request Type?}
CheckType -->|Text/Context| TextPath[Text/Context Path]
CheckType -->|Content Parts| PartsPath[Content Parts Path]
TextPath --> GetFailover[Get Failover Model List]
PartsPath --> GetFailover
GetFailover --> SelectModel[Select Best Model]
SelectModel --> FilterOp[Filter by Operation Type]
FilterOp --> RateCap[Rate by Capabilities]
RateCap --> ApplyPriority[Apply Priority Rules]
ApplyPriority --> TryModel[Try Model Call]
TryModel --> Success{Success?}
Success -->|Yes| Return[Return Response]
Success -->|No| NextModel{More Models?}
NextModel -->|Yes| TryModel
NextModel -->|No| Error[Return Error Response]
PartsPath --> ProcessEach[Process Each Part]
ProcessEach --> Chunk[Model-Aware Chunking]
Chunk --> TryModel
ProcessEach --> Merge[Merge Results]
Merge --> Return
```
**Supported External Systems**:
- **OpenAI**: GPT models via `aicorePluginOpenai`
- **Anthropic**: Claude models via `aicorePluginAnthropic`
- **Perplexity**: Search-enabled models via `aicorePluginPerplexity`
- **Tavily**: Web search API via `aicorePluginTavily`
- **Internal**: Custom models via `aicorePluginInternal`
**Key Features**:
- Dynamic model discovery (auto-registers available connectors)
- Operation type-based model selection (e.g., IMAGE_ANALYSE, TEXT_GENERATION)
- Automatic failover (tries multiple models on failure)
- Model-aware chunking (respects model context limits)
- Content part processing (handles images, text, tables)
- Cost tracking (calculates USD cost per call)
- Progress callbacks for long-running operations
### Ticket Interface (`interfaceTicketObjects.py`)
**Type**: External System Interface
**Connectors**: `ConnectorTicketJira`, `ConnectorTicketClickup`
**Access Control**: Connector-level (API credentials)
**Purpose**: Synchronizes data with external ticket systems (Jira, ClickUp) by transforming tickets to/from list format for Excel-like operations.
**Why External Connector**: Ticket systems are external services with their own APIs. The interface provides bidirectional synchronization, field mapping, and data transformation between the application's list format and ticket system formats.
**Operations**:
```mermaid
graph TB
subgraph "Factory Method"
Factory[createTicketInterfaceByType - connectorType, params]
end
subgraph "Export Operations"
Export[exportTicketsAsList - read from external]
end
subgraph "Import Operations"
Import[importListToTickets - write to external]
end
subgraph "Internal Transformation"
Transform[_transformTicketRecords - field mapping]
Extract[_extractFieldValue - path-based extraction]
FormatDate[_formatDateForExcel - date formatting]
FilterEmpty[_filterEmptyRecords - validation]
end
```
**Complete Operations List**:
**Factory**:
- `createTicketInterfaceByType(taskSyncDefinition: dict, connectorType: str, connectorParams: dict) → TicketInterface` (creates interface with appropriate connector)
**Export**:
- `exportTicketsAsList() → list[dict]` (reads tickets from external system, transforms to list format)
**Import**:
- `importListToTickets(records: list[dict]) → None` (transforms list format, writes to external system)
**Internal Methods**:
- `_transformTicketRecords(tasks: list[dict], includePut: bool) → list[dict]` (transforms according to task_sync_definition)
- `_extractFieldValue(issue_data: dict, field_path: list[str], field_name: str) → Any` (extracts value using path)
- `_formatDateForExcel(date_value: Any) → Optional[str]` (formats dates for Excel compatibility)
- `_isDateField(field_name: str) → bool` (detects date fields)
- `_filterEmptyRecords(records: list[dict]) → list[dict]` (removes invalid records)
**Synchronization Flow**:
```mermaid
graph TB
subgraph "Export Flow"
ExportStart[exportTicketsAsList] --> ReadConn[Read from Connector]
ReadConn --> GetTasks[Get Tasks from API]
GetTasks --> Transform[Transform Fields]
Transform --> MapFields[Map via task_sync_definition]
MapFields --> FormatDates[Format Dates]
FormatDates --> Filter[Filter Empty Records]
Filter --> ReturnList[Return List Format]
end
subgraph "Import Flow"
ImportStart[importListToTickets] --> ReceiveList[Receive List Format]
ReceiveList --> ExtractFields[Extract Fields]
ExtractFields --> MapToTicket[Map to Ticket Format]
MapToTicket --> BuildUpdate[Build Update Payload]
BuildUpdate --> WriteConn[Write via Connector]
WriteConn --> UpdateAPI[Update External API]
end
subgraph "Field Mapping"
SyncDef[task_sync_definition] --> Direction[Direction: get/put]
SyncDef --> Path[Field Path: nested access]
Direction --> Transform
Path --> MapFields
Path --> MapToTicket
end
```
**Supported External Systems**:
- **Jira**: Via `ConnectorTicketJira` (uses Jira REST API)
- **ClickUp**: Via `ConnectorTicketClickup` (uses ClickUp API)
**Key Features**:
- Dynamic connector selection (Jira or ClickUp)
- Field mapping configuration (task_sync_definition maps fields bidirectionally)
- Path-based field extraction (supports nested JSON structures)
- Date format handling (converts various formats to Excel-compatible)
- Bidirectional sync (export to list, import from list)
- Empty record filtering (validates records have IDs)
### Voice Interface (`interfaceVoiceObjects.py`)
**Type**: External System Interface
**Connector**: `ConnectorGoogleSpeech`
**Access Control**: User context for settings (stored in Component database)
**Purpose**: Provides speech-to-text, text-to-speech, and translation services using Google Cloud APIs.
**Why External Connector**: Voice operations require Google Cloud Speech-to-Text, Text-to-Speech, and Translation APIs. The interface abstracts API complexity, handles audio format conversion, and manages user voice settings (stored in database via Component interface).
**Operations**:
```mermaid
graph TB
subgraph "Speech-to-Text Operations"
STT[speechToText - audio to text]
STTTrans[speechToTranslatedText - audio to translated text]
end
subgraph "Text-to-Speech Operations"
TTS[textToSpeech - text to audio]
TTSTrans[textToTranslatedSpeech - text to translated audio]
end
subgraph "Translation Operations"
Trans[translateText - text translation]
end
subgraph "Voice Settings"
GetSettings[getVoiceSettings - from Component DB]
CreateSettings[createVoiceSettings - to Component DB]
UpdateSettings[updateVoiceSettings - in Component DB]
GetOrCreate[getOrCreateVoiceSettings]
end
subgraph "Metadata Operations"
GetLangs[getAvailableLanguages - from Google API]
GetVoices[getAvailableVoices - from Google API]
end
```
**Complete Operations List**:
**Speech-to-Text**:
- `speechToText(audioContent: bytes, language: str, sampleRate: Optional[int], channels: Optional[int]) → Dict[str, Any]` (converts audio to text)
- `speechToTranslatedText(audioContent: bytes, fromLanguage: str, toLanguage: str, sampleRate: Optional[int], channels: Optional[int]) → Dict[str, Any]` (converts audio to translated text)
**Text-to-Speech**:
- `textToSpeech(text: str, language: Optional[str], voice: Optional[str]) → Dict[str, Any]` (converts text to audio)
- `textToTranslatedSpeech(text: str, fromLanguage: str, toLanguage: str, voice: Optional[str]) → Dict[str, Any]` (converts text to translated audio)
**Translation**:
- `translateText(text: str, sourceLanguage: str, targetLanguage: str) → Dict[str, Any]` (translates text)
**Voice Settings** (delegates to Component interface):
- `getVoiceSettings(userId: str) → Optional[VoiceSettings]`
- `createVoiceSettings(settingsData: Dict) → Optional[VoiceSettings]`
- `updateVoiceSettings(userId: str, settingsData: Dict) → Optional[VoiceSettings]`
- `getOrCreateVoiceSettings(userId: str) → Optional[VoiceSettings]`
**Metadata**:
- `getAvailableLanguages() → Dict[str, Any]` (lists supported languages from Google API)
- `getAvailableVoices(languageCode: Optional[str]) → Dict[str, Any]` (lists available voices, optionally filtered by language)
**Operation Flow**:
```mermaid
graph TB
Request[Voice Operation Request] --> Voice[VoiceObjects]
Voice --> CheckSettings{Need Settings?}
CheckSettings -->|Yes| GetSettings[Get from Component DB]
CheckSettings -->|No| Connector[Get Connector]
GetSettings --> Connector
Connector --> GoogleAPI[Google Cloud API]
GoogleAPI --> STT{Operation Type?}
STT -->|Speech-to-Text| STTAPI[Speech-to-Text API]
STT -->|Text-to-Speech| TTSAPI[Text-to-Speech API]
STT -->|Translation| TransAPI[Translation API]
STTAPI --> Process[Process Response]
TTSAPI --> Process
TransAPI --> Process
Process --> Format[Format Response]
Format --> Return[Return Result]
```
**Supported External Systems**:
- **Google Cloud Speech-to-Text**: Audio transcription
- **Google Cloud Text-to-Speech**: Audio synthesis
- **Google Cloud Translation**: Text translation
**Key Features**:
- Multi-language support (detects and supports many languages)
- Audio format handling (auto-detects sample rate, channels)
- Combined operations (speech-to-translated-text, text-to-translated-speech)
- User voice preferences (stored in Component database)
- Language and voice discovery (queries Google API for available options)
- Error handling with detailed error messages
### Access Control Summary
**Privilege Levels**:
1. **SYSADMIN**: Full system access, all mandates
2. **ADMIN**: Full access within mandate
3. **USER**: Access to own records only
**Access Methods**:
- `uam()`: Filters recordsets by privilege, adds `_hideView`, `_hideEdit`, `_hideDelete` flags
- `canModify()`: Checks if user can create/update/delete records based on ownership and privilege
**Singleton Pattern**: Interfaces use factory functions that cache instances per user context for efficient memory usage.
**User Context**: All database interfaces require user context (User object with mandateId, userId, privilege) for access control and data filtering.
---