53 KiB
Datamodels and Interfaces Component
Overview
The Datamodels and Interfaces components form the core data layer of the Gateway application. They provide a clean separation between data structures (datamodels) and data access logic (interfaces), enabling type-safe, maintainable, and scalable data operations throughout the application.
Component Architecture
graph TB
subgraph "Application Layer"
App[Application<br/>Routes, Services, Features]
end
subgraph "Interfaces Layer"
IF_RealEstate[Real Estate Interface]
IF_Chat[Chat Interface]
IF_App[App Interface]
IF_Component[Component Interface]
IF_AI[AI Interface]
IF_Ticket[Ticket Interface]
IF_Voice[Voice Interface]
end
subgraph "Access Control Layer"
AC_RealEstate[Real Estate Access]
AC_Chat[Chat Access]
AC_App[App Access]
AC_Component[Component Access]
end
subgraph "Database Connectors"
Connector_Postgre[PostgreSQL Connector]
end
subgraph "Databases"
DB_RealEstate[(Real Estate<br/>Database)]
DB_Chat[(Chat<br/>Database)]
DB_App[(App<br/>Database)]
DB_Component[(Component<br/>Database)]
end
subgraph "External Systems"
External_AI[AI APIs<br/>OpenAI, Anthropic, etc.]
External_Tickets[Ticket Systems<br/>Jira, ClickUp]
External_Voice[Voice Services<br/>Google Cloud]
end
App -.->|uses| IF_RealEstate
App -.->|uses| IF_Chat
App -.->|uses| IF_App
App -.->|uses| IF_Component
App -.->|uses| IF_AI
App -.->|uses| IF_Ticket
App -.->|uses| IF_Voice
IF_RealEstate --> AC_RealEstate
IF_Chat --> AC_Chat
IF_App --> AC_App
IF_Component --> AC_Component
AC_RealEstate --> Connector_Postgre
AC_Chat --> Connector_Postgre
AC_App --> Connector_Postgre
AC_Component --> Connector_Postgre
Connector_Postgre --> DB_RealEstate
Connector_Postgre --> DB_Chat
Connector_Postgre --> DB_App
Connector_Postgre --> DB_Component
IF_AI --> External_AI
IF_Ticket --> External_Tickets
IF_Voice --> External_Voice
IF_RealEstate -.->|uses| DM_RealEstate[Real Estate<br/>Datamodels]
IF_Chat -.->|uses| DM_Chat[Chat<br/>Datamodels]
IF_App -.->|uses| DM_UAM[User & Mandate<br/>Datamodels]
IF_Component -.->|uses| DM_Files[File<br/>Datamodels]
IF_AI -.->|uses| DM_AI[AI<br/>Datamodels]
Data Flow
sequenceDiagram
participant Route as API Route
participant Service as Service Layer
participant Interface as Interface
participant Access as Access Control
participant Connector as Database Connector
participant DB as Database
Route->>Service: Request with User Context
Service->>Interface: Initialize with User
Interface->>Access: Check Permissions
Access-->>Interface: Permission Granted
Service->>Interface: CRUD Operation
Interface->>Access: Validate Access
Access-->>Interface: Access Validated
Interface->>Connector: Execute Query
Connector->>DB: SQL Query
DB-->>Connector: Result Set
Connector-->>Interface: Data Objects
Interface->>Access: Apply Filtering
Access-->>Interface: Filtered Data
Interface-->>Service: Datamodel Instances
Service-->>Route: Response Data
Component Structure
Datamodels Structure
modules/datamodels/
├── datamodelRealEstate.py # Real estate domain models
├── datamodelChat.py # Chat workflow models
├── datamodelAi.py # AI operation models
├── datamodelUam.py # User and mandate models
├── datamodelSecurity.py # Security and authentication models
├── datamodelFiles.py # File management models
├── datamodelDocument.py # Document structure models
├── datamodelExtraction.py # Content extraction models
├── datamodelPagination.py # Pagination models
├── datamodelVoice.py # Voice settings models
├── datamodelTickets.py # Ticket system models
├── datamodelNeutralizer.py # Data neutralization models
├── datamodelTools.py # Tool definitions
├── datamodelUtils.py # Utility models
└── __init__.py # Package exports
Interfaces Structure
modules/interfaces/
├── interfaceDbRealEstateObjects.py # Real estate data access
├── interfaceDbRealEstateAccess.py # Real estate access control
├── interfaceDbChatObjects.py # Chat data access
├── interfaceDbChatAccess.py # Chat access control
├── interfaceDbAppObjects.py # App/user management access
├── interfaceDbAppAccess.py # App access control
├── interfaceDbComponentObjects.py # Component management access
├── interfaceDbComponentAccess.py # Component access control
├── interfaceAiObjects.py # AI operations interface
├── interfaceTicketObjects.py # Ticket system interface
└── interfaceVoiceObjects.py # Voice operations interface
Datamodels Component
datamodelRealEstate.py
erDiagram
Projekt {
string id PK
string mandateId
string label
string statusProzess
json perimeter
json baulinie
json parzellen
json dokumente
json kontextInformationen
}
Parzelle {
string id PK
string mandateId
string label
string kontextGemeinde FK
json perimeter
json baulinie
string bauzone
float az
float bz
json dokumente
json kontextInformationen
}
Land {
string id PK
string mandateId
string label
string abk
json dokumente
json kontextInformationen
}
Kanton {
string id PK
string mandateId
string label
string id_land FK
string abk
json dokumente
json kontextInformationen
}
Gemeinde {
string id PK
string mandateId
string label
string id_kanton FK
string plz
json dokumente
json kontextInformationen
}
Dokument {
string id PK
string mandateId
string label
string versionsbezeichnung
string dokumentTyp
string dokumentReferenz
string quelle
string mimeType
json kategorienTags
}
Kontext {
string id PK
string thema
string inhalt
}
GeoPunkt {
string koordinatensystem
float x
float y
float z
string referenz
}
GeoPolylinie {
string id PK
bool closed
json punkte
}
Projekt ||--o{ Parzelle : contains
Projekt ||--o{ Dokument : references
Projekt ||--o{ Kontext : has
Parzelle ||--o{ GeoPolylinie : contains
Parzelle ||--o{ GeoPunkt : contains
Parzelle }o--|| Gemeinde : located_in
Land ||--o{ Kanton : contains
Kanton ||--o{ Gemeinde : contains
Land ||--o{ Dokument : has
Land ||--o{ Kontext : has
Kanton ||--o{ Dokument : has
Kanton ||--o{ Kontext : has
Gemeinde ||--o{ Dokument : has
Gemeinde ||--o{ Kontext : has
GeoPolylinie ||--o{ GeoPunkt : contains
datamodelChat.py
erDiagram
ChatWorkflow {
string id PK
string mandateId
string status
string name
int currentRound
int currentTask
int currentAction
int totalTasks
int totalActions
float lastActivity
float startedAt
string workflowMode
int maxSteps
json logs
json messages
json stats
json tasks
}
ChatMessage {
string id PK
string workflowId FK
string parentMessageId FK
string message
string summary
string role
string status
int sequenceNr
float publishedAt
bool success
string actionId
json documents
}
ChatLog {
string id PK
string workflowId FK
string message
string type
float timestamp
string status
float progress
json performance
}
ChatStat {
string id PK
string workflowId FK
float processingTime
int bytesSent
int bytesReceived
int errorCount
string process
string engine
float priceUsd
}
ChatDocument {
string id PK
string messageId FK
string fileId FK
string fileName
int fileSize
string mimeType
int roundNumber
int taskNumber
int actionNumber
string actionId
}
TaskPlan {
string overview
json tasks
string userMessage
}
TaskItem {
string id PK
string workflowId FK
string userInput
string status
string error
float startedAt
float finishedAt
json actionList
int retryCount
int retryMax
bool rollbackOnFailure
json dependencies
string feedback
float processingTime
json resultLabels
}
TaskStep {
string id PK
string objective
json dependencies
json successCriteria
string estimatedComplexity
string userMessage
string dataType
json expectedFormats
json qualityRequirements
}
ActionItem {
string id PK
string execMethod
string execAction
json execParameters
string execResultLabel
json expectedDocumentFormats
string userMessage
string status
string error
int retryCount
int retryMax
float processingTime
float timestamp
string result
}
ActionResult {
bool success
string error
json documents
string resultLabel
}
AutomationDefinition {
string id PK
string mandateId
string label
string schedule
string template
json placeholders
bool active
string eventId
string status
json executionLogs
}
ChatWorkflow ||--o{ ChatMessage : contains
ChatWorkflow ||--o{ ChatLog : contains
ChatWorkflow ||--o{ ChatStat : contains
ChatWorkflow ||--o{ TaskPlan : has
ChatWorkflow ||--o{ AutomationDefinition : defines
ChatMessage ||--o{ ChatDocument : references
TaskPlan ||--o{ TaskStep : contains
TaskItem ||--o{ ActionItem : contains
ActionItem ||--o{ ActionResult : produces
datamodelAi.py
erDiagram
AiModel {
string name PK
string displayName
string connectorType
string apiUrl
float temperature
int maxTokens
int contextLength
float costPer1kTokensInput
float costPer1kTokensOutput
int speedRating
int qualityRating
string priority
string processingMode
json operationTypes
int minContextLength
bool isAvailable
string version
string lastUpdated
}
OperationTypeRating {
string operationType
int rating
}
AiCallOptions {
string operationType
string priority
bool compressPrompt
bool compressContext
bool processDocumentsIndividually
float maxCost
int maxProcessingTime
string processingMode
string resultFormat
float safetyMargin
float temperature
int maxParts
}
AiCallRequest {
string prompt
string context
json options
json contentParts
}
AiCallResponse {
string content
string modelName
float priceUsd
float processingTime
int bytesSent
int bytesReceived
int errorCount
}
AiModelCall {
json messages
json model
json options
}
AiModelResponse {
string content
bool success
string error
string modelId
float processingTime
json tokensUsed
json metadata
}
AiModel ||--o{ OperationTypeRating : has
AiCallRequest ||--|| AiCallOptions : uses
AiCallRequest ||--|| AiCallResponse : produces
AiCallRequest }o--|| AiModel : uses
AiModelCall }o--|| AiModel : uses
AiModelCall ||--|| AiCallOptions : uses
AiModelCall ||--|| AiModelResponse : produces
datamodelUam.py
erDiagram
Mandate {
string id PK
string name
string language
bool enabled
}
User {
string id PK
string username
string email
string fullName
string language
bool enabled
string privilege
string authenticationAuthority
string mandateId FK
}
UserConnection {
string id PK
string userId FK
string authority
string externalId
string externalUsername
string externalEmail
string status
float connectedAt
float lastChecked
float expiresAt
string tokenStatus
float tokenExpiresAt
}
UserInDB {
string id PK
string username
string email
string fullName
string language
bool enabled
string privilege
string authenticationAuthority
string mandateId FK
string hashedPassword
}
Mandate ||--o{ User : contains
User ||--o{ UserConnection : has
User ||--|| UserInDB : extends
datamodelSecurity.py
erDiagram
Token {
string id PK
string userId FK
string authority
string connectionId FK
string tokenAccess
string tokenType
float expiresAt
string tokenRefresh
float createdAt
string status
float revokedAt
string revokedBy
string reason
string sessionId
string mandateId FK
}
AuthEvent {
string id PK
string userId FK
string eventType
float timestamp
string ipAddress
string userAgent
bool success
string details
}
Token ||--o{ AuthEvent : generates
datamodelFiles.py
erDiagram
FileItem {
string id PK
string mandateId FK
string fileName
string mimeType
string fileHash
int fileSize
float creationDate
}
FilePreview {
string content
string mimeType
string fileName
bool isText
string encoding
int size
}
FileData {
string id PK
string data
bool base64Encoded
}
FileItem ||--|| FilePreview : generates
FileItem ||--|| FileData : contains
datamodelDocument.py
erDiagram
StructuredDocument {
json metadata
json sections
string summary
json tags
}
DocumentMetadata {
string title
string author
datetime createdAt
json sourceDocuments
string extractionMethod
string version
}
DocumentSection {
string id PK
string title
string contentType
json elements
int order
json metadata
}
Paragraph {
string text
json formatting
json metadata
}
Heading {
string text
int level
json metadata
}
CodeBlock {
string code
string language
json metadata
}
Image {
string data
string altText
string caption
json metadata
}
BulletList {
json items
string listType
json metadata
}
ListItem {
string text
json subitems
json metadata
}
TableData {
json headers
json rows
string caption
json metadata
}
StructuredDocument ||--|| DocumentMetadata : has
StructuredDocument ||--o{ DocumentSection : contains
DocumentSection ||--o{ Paragraph : can_contain
DocumentSection ||--o{ Heading : can_contain
DocumentSection ||--o{ CodeBlock : can_contain
DocumentSection ||--o{ Image : can_contain
DocumentSection ||--o{ BulletList : can_contain
DocumentSection ||--o{ TableData : can_contain
BulletList ||--o{ ListItem : contains
ListItem ||--o{ ListItem : contains
datamodelExtraction.py
erDiagram
ContentExtracted {
string id PK
json parts
json summary
}
ContentPart {
string id PK
string parentId FK
string label
string typeGroup
string mimeType
string data
json metadata
}
ExtractionOptions {
string prompt
string operationType
bool processDocumentsIndividually
int imageMaxPixels
int imageQuality
json mergeStrategy
bool chunkAllowed
int maxSize
int textChunkSize
int imageChunkSize
bool enableParallelProcessing
int maxConcurrentChunks
}
MergeStrategy {
string groupBy
string orderBy
string mergeType
int maxSize
json textMerge
json tableMerge
json structureMerge
json aiResultMerge
bool preserveChunks
string chunkSeparator
bool preserveMetadata
json metadataFields
string onError
bool validateContent
bool useIntelligentMerging
string prompt
json capabilities
}
PartResult {
json originalPart
string aiResult
int partIndex
string documentId
float processingTime
json metadata
}
ChunkResult {
json originalChunk
string aiResult
int chunkIndex
string documentId
float processingTime
json metadata
}
ContentExtracted ||--o{ ContentPart : contains
ContentPart ||--o{ ContentPart : parent_of
ContentExtracted ||--|| ExtractionOptions : uses
ExtractionOptions ||--|| MergeStrategy : uses
ContentPart ||--|| PartResult : produces
ContentPart ||--|| ChunkResult : produces
datamodelPagination.py
erDiagram
PaginationParams {
int page
int pageSize
string sortBy
string sortOrder
}
PaginationRequest {
json params
json sortFields
}
SortField {
string field
string order
}
PaginatedResult {
json items
json metadata
json params
}
PaginationMetadata {
int page
int pageSize
int totalItems
int totalPages
bool hasNext
bool hasPrevious
}
PaginationRequest ||--|| PaginationParams : uses
PaginationRequest ||--o{ SortField : contains
PaginatedResult ||--|| PaginationMetadata : has
PaginatedResult ||--o{ PaginationParams : uses
datamodelVoice.py
erDiagram
VoiceSettings {
string id PK
string userId FK
string language
string voice
json settings
}
datamodelTickets.py
erDiagram
TicketFieldAttribute {
string fieldName PK
string fieldType
json fieldConfig
}
datamodelNeutralizer.py
erDiagram
DataNeutraliserConfig {
string id PK
string mandateId FK
string name
bool enabled
json attributes
}
DataNeutralizerAttributes {
string fieldName PK
string neutralizationType
json options
}
DataNeutraliserConfig ||--o{ DataNeutralizerAttributes : contains
datamodelUtils.py
erDiagram
Prompt {
string id PK
string name
string content
json metadata
}
datamodelTools.py
erDiagram
CountryCodes {
string ISO2Code PK
string tavilyName
string perplexityName
}
Note: CountryCodes is a utility class (not a Pydantic BaseModel) that provides static methods for country code mapping. It contains a mapping dictionary but is not persisted to the database.
datamodelJson.py
No database models - contains JSON template constants and supported section types.
Interfaces Component
Overview
The Interfaces component provides a clean abstraction layer for data access operations. Interfaces handle CRUD operations, user context management, access control, and integration with database connectors and external systems.
Objects vs Access Files
Interfaces are split into two file types:
Objects Files (interface*Objects.py)
Purpose: Business logic and CRUD operations for data entities.
Responsibilities:
- CRUD operations (Create, Read, Update, Delete)
- Data validation and transformation
- Business rule enforcement
- Database/external system communication
- User context management
- Pagination and filtering
Pattern: Each Objects file contains methods for manipulating domain entities (e.g., createProjekt(), getWorkflow(), updateUser()).
Access Files (interface*Access.py)
Purpose: Permission checking and data filtering based on user privileges.
Responsibilities:
- User privilege validation
- Mandate-based filtering
- Record ownership checking
- Access control attribute generation (
_hideView,_hideEdit,_hideDelete) - Permission decision logic
Pattern: Access files contain two main methods:
uam(): Unified Access Management - filters recordsets and adds access control flagscanModify(): Checks if user can create/update/delete records
Relationship:
graph TB
Objects[Objects File] --> Access[Access File]
Access --> UAM[uam Method]
Access --> CanModify[canModify Method]
Objects --> CRUD[CRUD Operations]
CRUD --> AccessCheck{Check Access}
AccessCheck -->|Read| UAM
AccessCheck -->|Write| CanModify
UAM --> Filter[Filter Records]
UAM --> Flags[Add Access Flags]
CanModify --> Permission{Permission?}
Permission -->|Yes| Allow[Allow Operation]
Permission -->|No| Deny[Deny Operation]
Interface Types
Interfaces are categorized into two types based on their data source:
Database Interfaces
Why Database Connectors?: These interfaces manage persistent data stored in PostgreSQL databases. They use database connectors to:
- Store structured data with relationships
- Ensure data consistency and integrity
- Provide ACID transactions
- Support complex queries and filtering
- Enable mandate-based data isolation
Characteristics:
- Use
DatabaseConnectorfor PostgreSQL access - Implement Access classes for permission control
- Support pagination and sorting
- Apply mandate-based filtering automatically
- Track record ownership (
_createdBy)
External System Interfaces
Why External Connectors?: These interfaces integrate with external APIs and services. They use connectors to:
- Communicate with third-party systems
- Transform data between formats
- Handle API authentication and rate limiting
- Provide abstraction over external service complexity
Characteristics:
- Use specialized connectors (e.g.,
ConnectorGoogleSpeech,ConnectorTicketJira) - May not require user context (system-level operations)
- Focus on data transformation and synchronization
- Handle external API errors and retries
Real Estate Interface (interfaceDbRealEstateObjects.py)
Type: Database Interface
Database: PostgreSQL (Real Estate database)
Access Control: interfaceDbRealEstateAccess.py → RealEstateAccess
Purpose: Manages real estate domain data including projects, parcels, administrative entities, and geographic information.
Why Database Connector: Real estate data requires persistent storage with complex relationships (projects → parcels → administrative units), geographic data (polygons, points), and mandate-based isolation for multi-tenant scenarios.
CRUD Operations:
graph TB
subgraph "Projekt Operations"
PCreate[createProjekt]
PGet[getProjekt]
PGetAll[getProjekte]
PUpdate[updateProjekt]
PDelete[deleteProjekt]
end
subgraph "Parzelle Operations"
ParCreate[createParzelle]
ParGet[getParzelle]
ParGetAll[getParzellen]
ParUpdate[updateParzelle]
ParDelete[deleteParzelle]
end
subgraph "Dokument Operations"
DocCreate[createDokument]
DocGet[getDokument]
DocGetAll[getDokumente]
DocUpdate[updateDokument]
DocDelete[deleteDokument]
end
subgraph "Administrative Hierarchy"
GemCreate[createGemeinde]
GemGet[getGemeinde]
GemGetAll[getGemeinden]
GemUpdate[updateGemeinde]
GemDelete[deleteGemeinde]
KanCreate[createKanton]
KanGet[getKanton]
KanGetAll[getKantone]
KanUpdate[updateKanton]
KanDelete[deleteKanton]
LanCreate[createLand]
LanGet[getLand]
LanGetAll[getLaender]
LanUpdate[updateLand]
LanDelete[deleteLand]
end
subgraph "Kontext Operations"
KonCreate[createKontext]
KonGet[getKontext]
KonGetAll[getKontexte]
KonUpdate[updateKontext]
KonDelete[deleteKontext]
end
Complete CRUD List:
Projekt:
createProjekt(projekt: Projekt) → ProjektgetProjekt(projektId: str) → Optional[Projekt]getProjekte(recordFilter: Optional[Dict]) → List[Projekt]updateProjekt(projektId: str, updateData: Dict) → Optional[Projekt]deleteProjekt(projektId: str) → bool
Parzelle:
createParzelle(parzelle: Parzelle) → ParzellegetParzelle(parzelleId: str) → Optional[Parzelle]getParzellen(recordFilter: Optional[Dict]) → List[Parzelle]updateParzelle(parzelleId: str, updateData: Dict) → Optional[Parzelle]deleteParzelle(parzelleId: str) → bool
Dokument:
createDokument(dokument: Dokument) → DokumentgetDokument(dokumentId: str) → Optional[Dokument]getDokumente(recordFilter: Optional[Dict]) → List[Dokument]updateDokument(dokumentId: str, updateData: Dict) → Optional[Dokument]deleteDokument(dokumentId: str) → bool
Gemeinde:
createGemeinde(gemeinde: Gemeinde) → GemeindegetGemeinde(gemeindeId: str) → Optional[Gemeinde]getGemeinden(recordFilter: Optional[Dict]) → List[Gemeinde]updateGemeinde(gemeindeId: str, updateData: Dict) → Optional[Gemeinde]deleteGemeinde(gemeindeId: str) → bool
Kanton:
createKanton(kanton: Kanton) → KantongetKanton(kantonId: str) → Optional[Kanton]getKantone(recordFilter: Optional[Dict]) → List[Kanton]updateKanton(kantonId: str, updateData: Dict) → Optional[Kanton]deleteKanton(kantonId: str) → bool
Land:
createLand(land: Land) → LandgetLand(landId: str) → Optional[Land]getLaender(recordFilter: Optional[Dict]) → List[Land]updateLand(landId: str, updateData: Dict) → Optional[Land]deleteLand(landId: str) → bool
Kontext:
createKontext(kontext: Kontext) → KontextgetKontext(kontextId: str) → Optional[Kontext]getKontexte(recordFilter: Optional[Dict]) → List[Kontext]updateKontext(kontextId: str, updateData: Dict) → Optional[Kontext]deleteKontext(kontextId: str) → bool
Access Control Flow:
graph TB
Request[CRUD Request] --> Objects[RealEstateObjects]
Objects --> Access[RealEstateAccess]
Access --> CheckPriv[Check Privilege]
CheckPriv -->|SYSADMIN| AllData[All Records]
CheckPriv -->|ADMIN| MandateData[Mandate Records]
CheckPriv -->|USER| OwnData[Own Records]
AllData --> UAM[uam Method]
MandateData --> UAM
OwnData --> UAM
UAM --> Filter[Filter by mandateId]
UAM --> CheckOwn[Check _createdBy]
UAM --> AddFlags[Add _hideView/_hideEdit/_hideDelete]
Filter --> Return[Return Filtered Data]
CheckOwn --> Return
AddFlags --> Return
Key Features:
- Geographic data support (GeoPolylinie, GeoPunkt)
- Multi-level administrative hierarchy (Land → Kanton → Gemeinde)
- Location name resolution (converts names to IDs for filtering)
- Document versioning and management
- Context information for projects and administrative units
Chat Interface (interfaceDbChatObjects.py)
Type: Database Interface
Database: PostgreSQL (Chat database)
Access Control: interfaceDbChatAccess.py → ChatAccess
Purpose: Manages chat workflows, messages, logs, statistics, and automation definitions for AI-powered conversation workflows.
Why Database Connector: Chat workflows require persistent storage for conversation history, workflow state, performance metrics, and automation configurations. Data must be queryable, filterable, and mandate-isolated.
CRUD Operations:
graph TB
subgraph "ChatWorkflow Operations"
WfGet[getWorkflows - with pagination]
WfGetOne[getWorkflow - by ID]
WfCreate[createWorkflow]
WfUpdate[updateWorkflow]
WfDelete[deleteWorkflow - cascade]
end
subgraph "ChatMessage Operations"
MsgGet[getMessages - by workflowId, pagination]
MsgCreate[createMessage]
MsgUpdate[updateMessage]
MsgDelete[deleteMessage]
MsgDeleteFile[deleteFileFromMessage]
end
subgraph "ChatDocument Operations"
DocGet[getDocuments - by messageId]
DocCreate[createDocument]
end
subgraph "ChatLog Operations"
LogGet[getLogs - by workflowId, pagination]
LogCreate[createLog]
end
subgraph "ChatStat Operations"
StatGet[getStats - by workflowId]
StatCreate[createStat]
end
subgraph "AutomationDefinition Operations"
AutoGet[getAllAutomationDefinitions - pagination]
AutoGetOne[getAutomationDefinition - by ID]
AutoCreate[createAutomationDefinition]
AutoUpdate[updateAutomationDefinition]
AutoDelete[deleteAutomationDefinition]
end
subgraph "Utility Operations"
Unified[getUnifiedChatData - workflow snapshot]
end
Complete CRUD List:
ChatWorkflow:
getWorkflows(pagination: Optional[PaginationParams]) → Union[List[Dict], PaginatedResult]getWorkflow(workflowId: str) → Optional[ChatWorkflow]createWorkflow(workflowData: Dict) → ChatWorkflowupdateWorkflow(workflowId: str, workflowData: Dict) → ChatWorkflowdeleteWorkflow(workflowId: str) → bool(cascades to messages, logs, stats)
ChatMessage:
getMessages(workflowId: str, pagination: Optional[PaginationParams]) → Union[List[ChatMessage], PaginatedResult]createMessage(messageData: Dict) → ChatMessageupdateMessage(messageId: str, messageData: Dict) → DictdeleteMessage(workflowId: str, messageId: str) → booldeleteFileFromMessage(workflowId: str, messageId: str, fileId: str) → bool
ChatDocument:
getDocuments(messageId: str) → List[ChatDocument]createDocument(documentData: Dict) → ChatDocument
ChatLog:
getLogs(workflowId: str, pagination: Optional[PaginationParams]) → Union[List[ChatLog], PaginatedResult]createLog(logData: Dict) → ChatLog
ChatStat:
getStats(workflowId: str) → List[ChatStat]createStat(statData: Dict) → ChatStat
AutomationDefinition:
getAllAutomationDefinitions(pagination: Optional[PaginationParams]) → Union[List[Dict], PaginatedResult]getAutomationDefinition(automationId: str) → Optional[Dict]createAutomationDefinition(automationData: Dict) → DictupdateAutomationDefinition(automationId: str, automationData: Dict) → DictdeleteAutomationDefinition(automationId: str) → bool
Utility Methods:
getUnifiedChatData(workflowId: str, afterTimestamp: Optional[float]) → Dict(returns workflow snapshot with messages, logs, stats)
Access Control Flow:
graph TB
Request[CRUD Request] --> Objects[ChatObjects]
Objects --> Access[ChatAccess]
Access --> CheckPriv[Check Privilege]
CheckPriv -->|SYSADMIN| AllWorkflows[All Workflows]
CheckPriv -->|ADMIN| MandateWorkflows[Mandate Workflows]
CheckPriv -->|USER| OwnWorkflows[Own Workflows]
AllWorkflows --> UAM[uam Method]
MandateWorkflows --> UAM
OwnWorkflows --> UAM
UAM --> FilterWorkflow[Filter by workflowId mandate]
UAM --> CheckOwn[Check _createdBy]
UAM --> AddFlags[Add access flags]
FilterWorkflow --> CheckChild[Check Child Access]
CheckChild -->|Message| CheckWorkflow[Check workflow ownership]
CheckChild -->|Log| CheckWorkflow
CheckChild -->|Stat| CheckWorkflow
CheckWorkflow --> Return[Return Filtered Data]
CheckOwn --> Return
AddFlags --> Return
Key Features:
- Multi-round workflow support with state tracking
- Normalized data model (workflows, messages, logs, stats in separate tables)
- Cascade delete (deleting workflow removes all related data)
- Pagination support for large datasets
- Unified data retrieval for workflow snapshots
- Automation workflow definitions
- Document attachment management
App Interface (interfaceDbAppObjects.py)
Type: Database Interface
Database: PostgreSQL (App database)
Access Control: interfaceDbAppAccess.py → AppAccess
Purpose: Manages users, mandates, authentication tokens, and application-level configuration.
Why Database Connector: User accounts, mandates, and authentication data require secure, persistent storage with strict access control. This is the foundation for all other interfaces' user context.
CRUD Operations:
graph TB
subgraph "User Operations"
UserGet[getUsersByMandate - pagination]
UserGetByUsername[getUserByUsername]
UserGetOne[getUser - by ID]
UserCreate[createUser - with password hash]
UserUpdate[updateUser]
UserDelete[deleteUser]
end
subgraph "UserConnection Operations"
ConnGet[getUserConnections - by userId]
ConnGetToken[getConnectionToken - by connectionId]
end
subgraph "Mandate Operations"
ManGet[getAllMandates - pagination]
ManGetOne[getMandate - by ID]
ManCreate[createMandate]
ManUpdate[updateMandate]
ManDelete[deleteMandate]
end
subgraph "Neutralization Config"
NeuGet[getNeutralizationConfig]
NeuCreate[createOrUpdateNeutralizationConfig]
NeuGetAttrs[getNeutralizationAttributes]
NeuDeleteAttrs[deleteNeutralizationAttributes]
end
subgraph "Initialization"
InitRoot[getRootInterface - system init]
InitRecords[_initRootMandate, _initAdminUser, _initEventUser]
end
Complete CRUD List:
User:
getUsersByMandate(mandateId: str, pagination: Optional[PaginationParams]) → Union[List[User], PaginatedResult]getUserByUsername(username: str) → Optional[User]getUser(userId: str) → Optional[User]createUser(userData: Dict, password: Optional[str]) → User(hashes password with Argon2)updateUser(userId: str, updateData: Union[Dict, User]) → UserdeleteUser(userId: str) → bool
UserConnection:
getUserConnections(userId: str) → List[UserConnection]getConnectionToken(connectionId: str) → Optional[Token]
Mandate:
getAllMandates(pagination: Optional[PaginationParams]) → Union[List[Mandate], PaginatedResult]getMandate(mandateId: str) → Optional[Mandate]createMandate(name: str, language: str) → MandateupdateMandate(mandateId: str, updateData: Dict) → MandatedeleteMandate(mandateId: str) → bool
DataNeutraliserConfig:
getNeutralizationConfig() → Optional[DataNeutraliserConfig]createOrUpdateNeutralizationConfig(configData: Dict) → DataNeutraliserConfiggetNeutralizationAttributes(file_id: str) → List[Dict]deleteNeutralizationAttributes(file_id: str) → bool
Special Methods:
getRootInterface() → AppObjects(creates interface with system admin privileges for initialization)getInitialId(model_class: type) → Optional[str](gets first record ID for a model)
Access Control Flow:
graph TB
Request[CRUD Request] --> Objects[AppObjects]
Objects --> Access[AppAccess]
Access --> CheckPriv[Check Privilege]
CheckPriv -->|SYSADMIN| FullAccess[Full System Access]
CheckPriv -->|ADMIN| MandateAccess[Mandate Access Only]
CheckPriv -->|USER| SelfAccess[Own User Record Only]
FullAccess --> UserOps[User Operations]
MandateAccess --> UserOps
SelfAccess --> UserOps
UserOps --> CheckOwn[Check Ownership]
CheckOwn -->|User CRUD| SYSADMINOnly{SYSADMIN?}
CheckOwn -->|Mandate CRUD| SYSADMINOnly
CheckOwn -->|Own Profile| Allow[Allow]
SYSADMINOnly -->|Yes| Allow
SYSADMINOnly -->|No| Deny[Deny]
Key Features:
- Password hashing with Argon2
- Multi-provider authentication support (local, external)
- System initialization (Root mandate, Admin user, Event user)
- Mandate-based user isolation
- Token management for external connections
- Data neutralization configuration
Component Interface (interfaceDbComponentObjects.py)
Type: Database Interface
Database: PostgreSQL (Component/Management database)
Access Control: interfaceDbComponentAccess.py → ComponentAccess
Purpose: Manages component-level data including files, prompts, and voice settings used across the application.
Why Database Connector: Files, prompts, and voice settings require persistent storage with metadata, ownership tracking, and mandate isolation. Files need binary storage with preview generation.
CRUD Operations:
graph TB
subgraph "File Operations"
FileGetAll[getAllFiles - pagination]
FileGet[getFile - by ID]
FileCreate[createFile - name, mimeType, content]
FileUpdate[updateFile]
FileDelete[deleteFile]
FileGetData[getFileData - binary content]
FileGetContent[getFileContent - preview]
FileCreateData[createFileData - store binary]
end
subgraph "Prompt Operations"
PromptGetAll[getAllPrompts - pagination]
PromptGet[getPrompt - by ID]
PromptCreate[createPrompt]
PromptUpdate[updatePrompt]
PromptDelete[deletePrompt]
end
subgraph "VoiceSettings Operations"
VoiceGet[getVoiceSettings - by userId]
VoiceCreate[createVoiceSettings]
VoiceUpdate[updateVoiceSettings]
VoiceDelete[deleteVoiceSettings]
VoiceGetOrCreate[getOrCreateVoiceSettings]
end
subgraph "Utilities"
MimeType[getMimeType - from fileName]
end
Complete CRUD List:
FileItem:
getAllFiles(pagination: Optional[PaginationParams]) → Union[List[FileItem], PaginatedResult]getFile(fileId: str) → Optional[FileItem]createFile(name: str, mimeType: str, content: bytes) → FileItem(creates FileItem and FileData)updateFile(fileId: str, updateData: Dict) → DictdeleteFile(fileId: str) → bool(deletes FileItem and FileData)getFileData(fileId: str) → Optional[bytes](raw binary content)getFileContent(fileId: str) → Optional[FilePreview](generates preview)createFileData(fileId: str, data: bytes) → bool(stores binary data)
Prompt:
getAllPrompts(pagination: Optional[PaginationParams]) → Union[List[Prompt], PaginatedResult]getPrompt(promptId: str) → Optional[Prompt]createPrompt(promptData: Dict) → DictupdatePrompt(promptId: str, updateData: Dict) → DictdeletePrompt(promptId: str) → bool
VoiceSettings:
getVoiceSettings(userId: Optional[str]) → Optional[VoiceSettings]createVoiceSettings(settingsData: Dict) → DictupdateVoiceSettings(userId: str, updateData: Dict) → DictdeleteVoiceSettings(userId: str) → boolgetOrCreateVoiceSettings(userId: Optional[str]) → VoiceSettings
Utility Methods:
getMimeType(fileName: str) → str(detects MIME type from extension)
Access Control Flow:
graph TB
Request[CRUD Request] --> Objects[ComponentObjects]
Objects --> Access[ComponentAccess]
Access --> CheckPriv[Check Privilege]
CheckPriv -->|SYSADMIN| AllFiles[All Files]
CheckPriv -->|ADMIN| MandateFiles[Mandate Files]
CheckPriv -->|USER| OwnFiles[Own Files]
AllFiles --> UAM[uam Method]
MandateFiles --> UAM
OwnFiles --> UAM
UAM --> FilterMandate[Filter by mandateId]
UAM --> CheckOwn[Check _createdBy]
UAM --> AddFlags[Add access flags]
FilterMandate --> Return[Return Filtered Data]
CheckOwn --> Return
AddFlags --> Return
Key Features:
- Binary file storage with metadata
- Automatic preview generation (text, images, etc.)
- MIME type detection
- Prompt template management with initialization
- Voice settings per user
- File hash calculation for deduplication
AI Interface (interfaceAiObjects.py)
Type: External System Interface
Connectors: Dynamic discovery via modelRegistry
Access Control: None (system-level operations)
Purpose: Provides centralized AI operations with dynamic model discovery, automatic model selection, and failover handling.
Why External Connector: AI operations require communication with external APIs (OpenAI, Anthropic, Perplexity, Tavily) and internal AI services. The interface abstracts model selection, handles API calls, manages failover, and tracks costs.
Operations:
graph TB
subgraph "AI Call Operations"
Call[call - main entry point]
CallText[call with text/context]
CallParts[call with content parts]
end
subgraph "Model Information"
ModelInfo[getModelInfo - by displayName]
ModelsByTag[getModelsByTag - filter by tag]
end
subgraph "Internal Processing"
SelectModel[_selectModel - dynamic selection]
ProcessPart[_processContentPartWithFallback]
MergeResults[_mergePartResults]
CallWithModel[_callWithModel - execute API call]
end
Complete Operations List:
AI Calls:
call(request: AiCallRequest, progressCallback=None) → AiCallResponse(main entry point, handles text/context or content parts)getModelInfo(displayName: str) → Dict[str, Any](get model metadata)getModelsByTag(tag: str) → List[str](filter models by tag)
Internal Methods (used by call):
_selectModel(prompt: str, context: str, options: AiCallOptions) → str(selects best model)_callWithTextContext(request: AiCallRequest) → AiCallResponse(handles traditional text/context calls)_callWithContentParts(request: AiCallRequest, progressCallback) → AiCallResponse(handles content parts with chunking)_processContentPartWithFallback(...) → AiCallResponse(processes single part with failover)_callWithModel(model, prompt, context, options) → AiCallResponse(executes actual API call)
Model Selection and Failover Flow:
graph TB
Request[AI Call Request] --> CheckType{Request Type?}
CheckType -->|Text/Context| TextPath[Text/Context Path]
CheckType -->|Content Parts| PartsPath[Content Parts Path]
TextPath --> GetFailover[Get Failover Model List]
PartsPath --> GetFailover
GetFailover --> SelectModel[Select Best Model]
SelectModel --> FilterOp[Filter by Operation Type]
FilterOp --> RateCap[Rate by Capabilities]
RateCap --> ApplyPriority[Apply Priority Rules]
ApplyPriority --> TryModel[Try Model Call]
TryModel --> Success{Success?}
Success -->|Yes| Return[Return Response]
Success -->|No| NextModel{More Models?}
NextModel -->|Yes| TryModel
NextModel -->|No| Error[Return Error Response]
PartsPath --> ProcessEach[Process Each Part]
ProcessEach --> Chunk[Model-Aware Chunking]
Chunk --> TryModel
ProcessEach --> Merge[Merge Results]
Merge --> Return
Supported External Systems:
- OpenAI: GPT models via
aicorePluginOpenai - Anthropic: Claude models via
aicorePluginAnthropic - Perplexity: Search-enabled models via
aicorePluginPerplexity - Tavily: Web search API via
aicorePluginTavily - Internal: Custom models via
aicorePluginInternal
Key Features:
- Dynamic model discovery (auto-registers available connectors)
- Operation type-based model selection (e.g., IMAGE_ANALYSE, TEXT_GENERATION)
- Automatic failover (tries multiple models on failure)
- Model-aware chunking (respects model context limits)
- Content part processing (handles images, text, tables)
- Cost tracking (calculates USD cost per call)
- Progress callbacks for long-running operations
Ticket Interface (interfaceTicketObjects.py)
Type: External System Interface
Connectors: ConnectorTicketJira, ConnectorTicketClickup
Access Control: Connector-level (API credentials)
Purpose: Synchronizes data with external ticket systems (Jira, ClickUp) by transforming tickets to/from list format for Excel-like operations.
Why External Connector: Ticket systems are external services with their own APIs. The interface provides bidirectional synchronization, field mapping, and data transformation between the application's list format and ticket system formats.
Operations:
graph TB
subgraph "Factory Method"
Factory[createTicketInterfaceByType - connectorType, params]
end
subgraph "Export Operations"
Export[exportTicketsAsList - read from external]
end
subgraph "Import Operations"
Import[importListToTickets - write to external]
end
subgraph "Internal Transformation"
Transform[_transformTicketRecords - field mapping]
Extract[_extractFieldValue - path-based extraction]
FormatDate[_formatDateForExcel - date formatting]
FilterEmpty[_filterEmptyRecords - validation]
end
Complete Operations List:
Factory:
createTicketInterfaceByType(taskSyncDefinition: dict, connectorType: str, connectorParams: dict) → TicketInterface(creates interface with appropriate connector)
Export:
exportTicketsAsList() → list[dict](reads tickets from external system, transforms to list format)
Import:
importListToTickets(records: list[dict]) → None(transforms list format, writes to external system)
Internal Methods:
_transformTicketRecords(tasks: list[dict], includePut: bool) → list[dict](transforms according to task_sync_definition)_extractFieldValue(issue_data: dict, field_path: list[str], field_name: str) → Any(extracts value using path)_formatDateForExcel(date_value: Any) → Optional[str](formats dates for Excel compatibility)_isDateField(field_name: str) → bool(detects date fields)_filterEmptyRecords(records: list[dict]) → list[dict](removes invalid records)
Synchronization Flow:
graph TB
subgraph "Export Flow"
ExportStart[exportTicketsAsList] --> ReadConn[Read from Connector]
ReadConn --> GetTasks[Get Tasks from API]
GetTasks --> Transform[Transform Fields]
Transform --> MapFields[Map via task_sync_definition]
MapFields --> FormatDates[Format Dates]
FormatDates --> Filter[Filter Empty Records]
Filter --> ReturnList[Return List Format]
end
subgraph "Import Flow"
ImportStart[importListToTickets] --> ReceiveList[Receive List Format]
ReceiveList --> ExtractFields[Extract Fields]
ExtractFields --> MapToTicket[Map to Ticket Format]
MapToTicket --> BuildUpdate[Build Update Payload]
BuildUpdate --> WriteConn[Write via Connector]
WriteConn --> UpdateAPI[Update External API]
end
subgraph "Field Mapping"
SyncDef[task_sync_definition] --> Direction[Direction: get/put]
SyncDef --> Path[Field Path: nested access]
Direction --> Transform
Path --> MapFields
Path --> MapToTicket
end
Supported External Systems:
- Jira: Via
ConnectorTicketJira(uses Jira REST API) - ClickUp: Via
ConnectorTicketClickup(uses ClickUp API)
Key Features:
- Dynamic connector selection (Jira or ClickUp)
- Field mapping configuration (task_sync_definition maps fields bidirectionally)
- Path-based field extraction (supports nested JSON structures)
- Date format handling (converts various formats to Excel-compatible)
- Bidirectional sync (export to list, import from list)
- Empty record filtering (validates records have IDs)
Voice Interface (interfaceVoiceObjects.py)
Type: External System Interface
Connector: ConnectorGoogleSpeech
Access Control: User context for settings (stored in Component database)
Purpose: Provides speech-to-text, text-to-speech, and translation services using Google Cloud APIs.
Why External Connector: Voice operations require Google Cloud Speech-to-Text, Text-to-Speech, and Translation APIs. The interface abstracts API complexity, handles audio format conversion, and manages user voice settings (stored in database via Component interface).
Operations:
graph TB
subgraph "Speech-to-Text Operations"
STT[speechToText - audio to text]
STTTrans[speechToTranslatedText - audio to translated text]
end
subgraph "Text-to-Speech Operations"
TTS[textToSpeech - text to audio]
TTSTrans[textToTranslatedSpeech - text to translated audio]
end
subgraph "Translation Operations"
Trans[translateText - text translation]
end
subgraph "Voice Settings"
GetSettings[getVoiceSettings - from Component DB]
CreateSettings[createVoiceSettings - to Component DB]
UpdateSettings[updateVoiceSettings - in Component DB]
GetOrCreate[getOrCreateVoiceSettings]
end
subgraph "Metadata Operations"
GetLangs[getAvailableLanguages - from Google API]
GetVoices[getAvailableVoices - from Google API]
end
Complete Operations List:
Speech-to-Text:
speechToText(audioContent: bytes, language: str, sampleRate: Optional[int], channels: Optional[int]) → Dict[str, Any](converts audio to text)speechToTranslatedText(audioContent: bytes, fromLanguage: str, toLanguage: str, sampleRate: Optional[int], channels: Optional[int]) → Dict[str, Any](converts audio to translated text)
Text-to-Speech:
textToSpeech(text: str, language: Optional[str], voice: Optional[str]) → Dict[str, Any](converts text to audio)textToTranslatedSpeech(text: str, fromLanguage: str, toLanguage: str, voice: Optional[str]) → Dict[str, Any](converts text to translated audio)
Translation:
translateText(text: str, sourceLanguage: str, targetLanguage: str) → Dict[str, Any](translates text)
Voice Settings (delegates to Component interface):
getVoiceSettings(userId: str) → Optional[VoiceSettings]createVoiceSettings(settingsData: Dict) → Optional[VoiceSettings]updateVoiceSettings(userId: str, settingsData: Dict) → Optional[VoiceSettings]getOrCreateVoiceSettings(userId: str) → Optional[VoiceSettings]
Metadata:
getAvailableLanguages() → Dict[str, Any](lists supported languages from Google API)getAvailableVoices(languageCode: Optional[str]) → Dict[str, Any](lists available voices, optionally filtered by language)
Operation Flow:
graph TB
Request[Voice Operation Request] --> Voice[VoiceObjects]
Voice --> CheckSettings{Need Settings?}
CheckSettings -->|Yes| GetSettings[Get from Component DB]
CheckSettings -->|No| Connector[Get Connector]
GetSettings --> Connector
Connector --> GoogleAPI[Google Cloud API]
GoogleAPI --> STT{Operation Type?}
STT -->|Speech-to-Text| STTAPI[Speech-to-Text API]
STT -->|Text-to-Speech| TTSAPI[Text-to-Speech API]
STT -->|Translation| TransAPI[Translation API]
STTAPI --> Process[Process Response]
TTSAPI --> Process
TransAPI --> Process
Process --> Format[Format Response]
Format --> Return[Return Result]
Supported External Systems:
- Google Cloud Speech-to-Text: Audio transcription
- Google Cloud Text-to-Speech: Audio synthesis
- Google Cloud Translation: Text translation
Key Features:
- Multi-language support (detects and supports many languages)
- Audio format handling (auto-detects sample rate, channels)
- Combined operations (speech-to-translated-text, text-to-translated-speech)
- User voice preferences (stored in Component database)
- Language and voice discovery (queries Google API for available options)
- Error handling with detailed error messages
Access Control Summary
Privilege Levels:
- SYSADMIN: Full system access, all mandates
- ADMIN: Full access within mandate
- USER: Access to own records only
Access Methods:
uam(): Filters recordsets by privilege, adds_hideView,_hideEdit,_hideDeleteflagscanModify(): Checks if user can create/update/delete records based on ownership and privilege
Singleton Pattern: Interfaces use factory functions that cache instances per user context for efficient memory usage.
User Context: All database interfaces require user context (User object with mandateId, userId, privilege) for access control and data filtering.