PowerOn/gateway
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
gateway/docs/code-documentation/connectors-component.md

31 KiB
Raw Blame History

Connectors Component Documentation

Table of Contents

  1. Overview
  2. Architecture
  3. Database Connectors
  4. Voice Connector
  5. Ticket Connectors
  6. Integration Patterns
  7. Configuration
  8. Design Principles

Overview

The Connectors component provides abstraction layers for external systems and data storage mechanisms. It acts as the bridge between the application's business logic and external services, databases, and third-party APIs. This component implements the Adapter Pattern to provide consistent interfaces regardless of the underlying technology.

Purpose and Scope

The connectors component serves three primary functions:

  1. Data Persistence - Abstracts database operations for both JSON file-based and PostgreSQL storage
  2. Voice Processing - Integrates Google Cloud Speech services for voice recognition, translation, and synthesis
  3. Ticket Management - Connects to external ticketing systems (JIRA, ClickUp) for synchronization

Component Structure

modules/connectors/
├── connectorDbJson.py          # JSON file-based database
├── connectorDbPostgre.py       # PostgreSQL database
├── connectorVoiceGoogle.py     # Google Cloud Speech services
├── connectorTicketsJira.py     # JIRA integration
└── connectorTicketsClickup.py  # ClickUp integration

Architecture

Connector Hierarchy

graph TD
    A[Application Layer<br/>routes/, workflows/, features/] --> B[Interface Layer<br/>modules/interfaces/]
    B --> C[Connector Layer<br/>modules/connectors/]
    C --> D[External Systems]
    
    B --> B1[interfaceDbAppObjects.py<br/>AppObjects]
    B --> B2[interfaceDbChatObjects.py<br/>ChatObjects]
    B --> B3[interfaceVoiceObjects.py<br/>VoiceObjects]
    B --> B4[interfaceTicketObjects.py<br/>TicketInterface]
    
    C --> C1[connectorDbJson.py<br/>DatabaseConnector]
    C --> C2[connectorDbPostgre.py<br/>DatabaseConnector]
    C --> C3[connectorVoiceGoogle.py<br/>ConnectorGoogleSpeech]
    C --> C4[connectorTicketsJira.py<br/>ConnectorTicketJira]
    C --> C5[connectorTicketsClickup.py<br/>ConnectorTicketClickup]
    
    B1 --> C1
    B1 --> C2
    B2 --> C1
    B2 --> C2
    B3 --> C3
    B4 --> C4
    B4 --> C5
    
    C1 --> D1[JSON Files]
    C2 --> D2[PostgreSQL]
    C3 --> D3[Google Cloud APIs]
    C4 --> D4[JIRA API]
    C5 --> D5[ClickUp API]
    
    style A fill:#e1f5ff
    style B fill:#fff9e6
    style C fill:#e8f5e9
    style D fill:#fce4ec

Layered Architecture Pattern

The connectors follow a three-tier architecture:

  1. Application Layer: Business logic, workflows, services
  2. Interface Layer: Domain-specific abstractions (AppObjects, ChatObjects, etc.)
  3. Connector Layer: Technology-specific implementations
  4. External Systems: Databases, APIs, cloud services

This separation ensures:

  • Loose Coupling: Application code doesn't depend on specific technologies
  • Testability: Connectors can be mocked or swapped
  • Flexibility: Easy migration between storage backends or service providers
  • Maintainability: Changes to external systems are isolated to connector layer

Database Connectors

Overview

The application supports two database connector implementations that provide identical public APIs but different storage mechanisms. This allows deployment flexibility without code changes.

DatabaseConnector Interface

Both database connectors implement a common interface using duck typing (no formal interface class). They provide:

  • CRUD Operations: Create, read, update, delete records
  • Schema Management: Dynamic table creation from Pydantic models
  • Context Management: User-aware operations for audit trails
  • Concurrency Control: Thread-safe operations with locking mechanisms
  • Initial Record Tracking: System table for bootstrap data

JSON Database Connector

Purpose and Use Cases

The JSON connector is ideal for:

  • Development Environments: Fast setup without database infrastructure
  • Small Deployments: Low-volume applications
  • Portable Data: Easy backup and version control
  • Testing: Simplified test data management

Storage Structure

graph LR
    A[Database Host Directory] --> B[Database Name Directory]
    B --> C[Table1 Directory]
    B --> D[Table2 Directory]
    B --> E[_system.json]
    
    C --> C1[record1.json]
    C --> C2[record2.json]
    C --> C3[_metadata.json]
    
    D --> D1[record3.json]
    D --> D2[record4.json]
    
    style E fill:#ffeb3b
    style C3 fill:#ffeb3b

File System Layout:

  • Each database is a directory
  • Each table is a subdirectory
  • Each record is a separate JSON file
  • Metadata files track record IDs and indexes
  • System table stores initial record references

Key Features

Atomic Operations:

  • Temporary file creation with validation
  • Atomic move operations to prevent corruption
  • Lock management for concurrent access

Caching Strategy:

  • In-memory table cache for performance
  • Metadata cache for quick record lookups
  • Intelligent cache invalidation

Concurrency Control:

  • File-level locks with timeout protection
  • Table-level locks for metadata operations
  • Deadlock prevention through lock ordering

PostgreSQL Database Connector

Purpose and Use Cases

The PostgreSQL connector is designed for:

  • Production Environments: High-performance, reliable storage
  • Multi-User Systems: Concurrent access with ACID guarantees
  • Large Datasets: Efficient querying and indexing
  • Scalability: Horizontal and vertical scaling capabilities

Schema Architecture

erDiagram
    _system ||--o{ Tables : tracks_initial_records
    Tables ||--o{ Records : contains
    
    _system {
        varchar table_name PK
        varchar initial_id
        double _createdAt
        double _modifiedAt
    }
    
    Tables {
        varchar id PK
        text field1
        jsonb field2
        double _createdAt
        double _modifiedAt
        varchar _createdBy
        varchar _modifiedBy
    }

Dynamic Schema Generation

The connector automatically:

  • Creates tables from Pydantic models
  • Maps Python types to SQL types
  • Adds metadata columns automatically
  • Creates indexes for foreign key fields
  • Performs additive migrations (adds missing columns)

Type Mapping:

  • str → TEXT
  • int → INTEGER
  • float → DOUBLE PRECISION
  • bool → BOOLEAN
  • dict/list → JSONB (enables flexible document storage)

JSONB Support

The connector uses PostgreSQL's JSONB type for complex fields:

  • Efficient binary JSON storage
  • Indexable JSON content
  • Native JSON operators
  • Flexible schema evolution

Database Connector Selection

The system selects connectors through import statements in interface files:

graph TD
    A[Interface Initialization] --> B{Check DB_HOST Config}
    B -->|File Path| C[Import connectorDbJson]
    B -->|Host:Port| D[Import connectorDbPostgre]
    C --> E[Create DatabaseConnector]
    D --> E
    E --> F[Initialize System]
    
    style B fill:#fff3e0

Selection Criteria:

  • Configuration-driven through config.ini
  • Import statement determines implementation
  • Transparent to application layer
  • No runtime switching (decided at startup)

Common Operations Flow

sequenceDiagram
    participant App as Application
    participant Iface as Interface Layer
    participant DB as DatabaseConnector
    participant Storage as Storage Backend
    
    App->>Iface: getRecordset(Model, filters)
    Iface->>DB: getRecordset(model_class, recordFilter)
    
    alt PostgreSQL
        DB->>Storage: SELECT * FROM table WHERE...
        Storage-->>DB: Rows
    else JSON
        DB->>Storage: Read files from directory
        Storage-->>DB: JSON objects
    end
    
    DB->>DB: Apply filters
    DB->>DB: Handle JSONB parsing
    DB-->>Iface: List of records
    Iface->>Iface: Apply UAM filters
    Iface-->>App: Filtered records
    
    Note over DB,Storage: Both connectors provide<br/>identical interface

Transaction Handling

PostgreSQL:

  • Uses database transactions
  • ACID compliance
  • Automatic rollback on errors
  • Connection pooling and retry logic

JSON:

  • File-level atomicity
  • Lock-based isolation
  • Manual rollback through file operations
  • Lock timeout protection

Performance Considerations

Aspect JSON Connector PostgreSQL Connector
Read Speed Fast for small datasets, degrades with size Consistent, optimized with indexes
Write Speed Fast for single records Fast with connection pooling
Concurrent Access Limited by file locking Excellent with MVCC
Query Capability In-memory filtering only Full SQL with JSONB operators
Scalability Limited to single server Horizontal and vertical scaling
Memory Usage High (full table caching) Low (database managed)

Voice Connector

Overview

The ConnectorGoogleSpeech provides integration with Google Cloud AI services for voice processing, offering a complete pipeline for speech recognition, translation, and text-to-speech synthesis.

Architecture

graph TB
    A[Voice Interface] --> B[ConnectorGoogleSpeech]
    
    B --> C[Speech-to-Text Client]
    B --> D[Translation Client]
    B --> E[Text-to-Speech Client]
    
    C --> F[Google Cloud Speech-to-Text API]
    D --> G[Google Cloud Translation API]
    E --> H[Google Cloud Text-to-Speech API]
    
    F --> I[Audio Processing]
    G --> J[Language Translation]
    H --> K[Voice Synthesis]
    
    style B fill:#e8f5e9
    style F fill:#e3f2fd
    style G fill:#e3f2fd
    style H fill:#e3f2fd

Core Capabilities

1. Speech-to-Text Processing

Audio Format Support:

  • WEBM OPUS (primary web recording format)
  • WAV (Linear PCM)
  • MP3
  • FLAC
  • OGG

Processing Pipeline:

sequenceDiagram
    participant Client
    participant Connector
    participant Validator
    participant API as Google Speech API
    
    Client->>Connector: speechToText(audioContent)
    Connector->>Validator: validateAudioFormat()
    
    Validator->>Validator: Detect format
    Validator->>Validator: Extract sample rate
    Validator->>Validator: Determine channels
    Validator-->>Connector: Format metadata
    
    Connector->>API: recognize(config, audio)
    
    alt Success
        API-->>Connector: Transcription + confidence
        Connector-->>Client: Success response
    else API Error
        API-->>Connector: Error
        Connector->>Connector: Try fallback configs
        Connector->>API: recognize(fallback_config)
        API-->>Connector: Result
        Connector-->>Client: Response
    end

Audio Format Detection:

  • Magic byte pattern recognition
  • Header parsing for metadata extraction
  • Automatic format-specific configuration
  • Deep scanning for ambiguous formats

Fallback Strategy: Multiple configurations tried automatically:

  1. Detected format with detected parameters
  2. Alternative encodings (LINEAR16, WEBM_OPUS)
  3. Standard sample rates (8kHz, 16kHz, 44.1kHz, 48kHz)
  4. Different recognition models (latest_long, phone_call, latest_short)

2. Translation Services

Features:

  • Automatic language detection
  • HTML entity decoding
  • Bidirectional translation
  • Preserves text formatting

Translation Flow:

graph LR
    A[Input Text] --> B[Google Translation API]
    B --> C[Detect Source Language]
    C --> D[Translate to Target]
    D --> E[Decode HTML Entities]
    E --> F[Return Result]
    
    style B fill:#e3f2fd

3. Text-to-Speech Synthesis

Voice Selection:

  • Language-specific voices
  • Gender-based voice selection
  • Neural voice quality
  • Multiple voice variants per language

Synthesis Process:

sequenceDiagram
    participant Client
    participant Connector
    participant API as Google TTS API
    
    Client->>Connector: textToSpeech(text, language, voice)
    
    alt Voice Specified
        Connector->>API: synthesize_speech(voice)
    else No Voice
        Connector->>Client: Error: no default voice
    end
    
    API->>API: Generate audio
    API-->>Connector: MP3 audio data
    Connector-->>Client: Audio content + metadata

Complete Pipeline: Speech-to-Translated-Text

The connector provides an integrated pipeline:

graph TD
    A[Audio Input] --> B[Speech-to-Text]
    B --> C[Original Text]
    C --> D[Translation]
    D --> E[Translated Text]
    
    B -.->|Confidence Score| F[Metadata]
    D -.->|Source Language| F
    F --> G[Complete Response]
    
    style A fill:#ffebee
    style C fill:#fff3e0
    style E fill:#e8f5e9
    style G fill:#e1f5fe

Use Cases:

  • Real-time voice translation
  • Multilingual voice assistants
  • International call centers
  • Language learning applications

Authentication and Configuration

Credential Management:

  • Service account JSON key stored in configuration
  • Parsed and loaded at initialization
  • No file system dependency
  • Credentials object creation from JSON

Configuration Parameters:

  • Connector_GoogleSpeech_API_KEY_SECRET: Service account JSON (encrypted)

Error Handling and Resilience

Retry Mechanisms:

  • Multiple encoding attempts
  • Sample rate fallbacks
  • Model fallbacks
  • Graceful degradation

Validation:

  • Audio length verification
  • Format compatibility checks
  • Content quality analysis
  • Silence detection

Integration Points

graph TB
    A[Routes Layer] --> B[VoiceObjects Interface]
    B --> C[ConnectorGoogleSpeech]
    
    A1[/voice-google/speech-to-text] --> B
    A2[/voice-google/translate] --> B
    A3[/voice-google/text-to-speech] --> B
    A4[/voice-google/languages] --> B
    A5[/voice-google/voices] --> B
    A6[WebSocket /ws/realtime-interpreter] --> B
    
    style A1 fill:#e8f5e9
    style A2 fill:#e8f5e9
    style A3 fill:#e8f5e9
    style A4 fill:#fff3e0
    style A5 fill:#fff3e0
    style A6 fill:#ffebee

Ticket Connectors

Overview

Ticket connectors provide unified access to external project management and ticketing systems. They enable bidirectional synchronization of tasks and tickets with external platforms.

Common Interface Pattern

Both ticket connectors implement a common base pattern:

Core Operations:

  • readAttributes(): Fetch field metadata from the system
  • readTasks(): Read tickets/tasks with pagination
  • writeTasks(): Update tickets/tasks in bulk
classDiagram
    class TicketBase {
        <<abstract>>
        +readAttributes() list~TicketFieldAttribute~
        +readTasks(limit) list~dict~
        +writeTasks(tasklist) None
    }
    
    class ConnectorTicketJira {
        -apiUsername: str
        -apiToken: str
        -apiUrl: str
        -projectCode: str
        -ticketType: str
        +readAttributes()
        +readTasks()
        +writeTasks()
    }
    
    class ConnectorTicketClickup {
        -apiToken: str
        -teamId: str
        -listId: str
        -apiUrl: str
        +readAttributes()
        +readTasks()
        +writeTasks()
    }
    
    TicketBase <|-- ConnectorTicketJira
    TicketBase <|-- ConnectorTicketClickup

JIRA Connector

Authentication and Configuration

Required Parameters:

  • apiUsername: JIRA account username
  • apiToken: API authentication token
  • apiUrl: JIRA instance URL
  • projectCode: Project identifier
  • ticketType: Issue type filter

Field Discovery

sequenceDiagram
    participant App
    participant Connector
    participant JIRA as JIRA API
    
    App->>Connector: readAttributes()
    Connector->>JIRA: POST /search/jql
    JIRA-->>Connector: Issue with all fields
    
    alt Fields Available
        Connector->>Connector: Extract field mappings
        Connector-->>App: List of attributes
    else No Fields
        Connector->>JIRA: GET /field
        JIRA-->>Connector: All field definitions
        Connector-->>App: Field list
    end

Field Mapping:

  • Maps JIRA field IDs to human-readable names
  • Supports custom fields
  • Handles complex field types (ADF, arrays, objects)

Pagination Strategy

The connector uses JIRA's cursor-based pagination:

graph TD
    A[Start] --> B[Initial Request]
    B --> C{Issues Returned?}
    C -->|No| D[End]
    C -->|Yes| E[Process Issues]
    E --> F{Has Next Page Token?}
    F -->|No| D
    F -->|Yes| G[Request Next Page]
    G --> H{Safety Cap Reached?}
    H -->|Yes| D
    H -->|No| C
    
    style D fill:#e8f5e9
    style H fill:#ffebee

Pagination Features:

  • Cursor-based continuation
  • Duplicate detection
  • Safety cap (1000 pages max)
  • Configurable page size
  • Loop prevention

Task Updates

Update Flow:

sequenceDiagram
    participant App
    participant Connector
    participant JIRA
    
    App->>Connector: writeTasks([task1, task2])
    
    loop For each task
        Connector->>Connector: Extract task ID
        Connector->>Connector: Map fields
        Connector->>Connector: Convert to ADF format
        Connector->>JIRA: PUT /issue/{id}
        
        alt Success
            JIRA-->>Connector: 204 No Content
        else Error
            JIRA-->>Connector: Error response
            Connector->>Connector: Log error
        end
    end
    
    Connector-->>App: Complete

Field Processing:

  • Automatic ADF (Atlassian Document Format) conversion for rich text
  • Custom field handling
  • Empty field validation
  • Selective field updates

ClickUp Connector

Authentication and Configuration

Required Parameters:

  • apiToken: ClickUp API token
  • teamId: Workspace/team identifier
  • listId: Optional list filter
  • apiUrl: API endpoint (default: https://api.clickup.com/api/v2)

Hierarchical Data Access

graph TD
    A[Team Level] --> B[Space Level]
    B --> C[Folder Level]
    C --> D[List Level]
    D --> E[Task Level]
    E --> F[Subtask Level]
    
    G[Connector] -.->|listId specified| D
    G -.->|listId not specified| A
    
    style G fill:#fff3e0

Access Patterns:

  • List-specific access when listId provided
  • Team-wide search when no listId
  • Automatic subtask inclusion

Field Discovery

ClickUp provides both:

  1. Custom Fields: From list-specific field API
  2. Core Fields: Standard task properties
graph LR
    A[readAttributes] --> B{listId Present?}
    B -->|Yes| C[GET /list/id/field]
    B -->|No| D[Return Core Fields Only]
    
    C --> E[Merge Custom + Core Fields]
    D --> F[Return Fields]
    E --> F
    
    style C fill:#e3f2fd

Core Fields:

  • ID
  • Name
  • Status
  • Assignees
  • Date Created
  • Due Date

Task Retrieval

Pagination:

  • Page-based pagination
  • Configurable page size (100 default)
  • Automatic page iteration
sequenceDiagram
    participant Connector
    participant API as ClickUp API
    
    loop Until no more tasks
        Connector->>API: GET /list/id/task?page={n}
        API-->>Connector: Tasks array
        
        alt Tasks returned < page size
            Connector->>Connector: Stop pagination
        else More tasks possible
            Connector->>Connector: Increment page
        end
    end

Task Updates

Update Strategy:

graph TD
    A[Task Update Request] --> B[Extract Task ID]
    B --> C[Extract Fields]
    C --> D{Field Type?}
    
    D -->|name/summary| E[Update name]
    D -->|status| F[Update status]
    D -->|custom field| G[Add to custom_fields array]
    D -->|other| H[Add to description]
    
    E --> I[Build Payload]
    F --> I
    G --> I
    H --> I
    
    I --> J[PUT /task/id]
    
    style J fill:#e3f2fd

Field Mapping:

  • Heuristic field name matching
  • Custom field special handling
  • Best-effort unknown field mapping

Ticket Interface Integration

The ticket connectors are wrapped by the TicketInterface for field mapping:

graph TB
    A[Workflow/Feature] --> B[TicketService]
    B --> C[TicketInterface]
    C --> D[Connector Factory]
    
    D -->|connectorType='Jira'| E[ConnectorTicketJira]
    D -->|connectorType='ClickUp'| F[ConnectorTicketClickup]
    
    C --> G[Task Sync Definition]
    G --> H[Field Mapping]
    
    E --> I[External System]
    F --> I
    
    style C fill:#fff3e0
    style G fill:#e8f5e9

Task Sync Definition:

  • Maps internal field names to external field paths
  • Specifies read/write directions
  • Handles nested field access
  • Enables field transformations

Example Flow:

sequenceDiagram
    participant Workflow
    participant Interface as TicketInterface
    participant Connector
    participant External as External System
    
    Workflow->>Interface: exportTicketsAsList()
    Interface->>Connector: readTasks()
    Connector->>External: API Request
    External-->>Connector: Raw tickets
    Connector-->>Interface: Tickets list
    Interface->>Interface: _transformTicketRecords()
    Interface-->>Workflow: Transformed data
    
    Note over Interface: Applies field mapping<br/>from sync definition

Integration Patterns

Connector Initialization Patterns

1. Singleton Pattern (Database Connectors)

Database connectors use singleton-like patterns through interface factories:

graph TD
    A[Request 1] --> B[getAppInterface]
    C[Request 2] --> B
    D[Request 3] --> B
    
    B --> E{Instance Exists?}
    E -->|No| F[Create AppObjects]
    E -->|Yes| G[Return Cached Instance]
    
    F --> H[Initialize DatabaseConnector]
    H --> I[Store in _gatewayInterfaces]
    
    G --> J[Return Interface]
    I --> J
    
    style B fill:#fff3e0
    style I fill:#e8f5e9

Benefits:

  • Reuses database connections
  • Maintains context consistency
  • Reduces initialization overhead
  • Per-user instances for security

2. Factory Pattern (Ticket Connectors)

Ticket connectors use factory pattern for runtime selection:

graph TD
    A[Service Request] --> B[createTicketInterfaceByType]
    B --> C{Connector Type?}
    
    C -->|'jira'| D[Import JIRA Connector]
    C -->|'clickup'| E[Import ClickUp Connector]
    C -->|unknown| F[Raise ValueError]
    
    D --> G[Create Connector Instance]
    E --> G
    
    G --> H[Wrap in TicketInterface]
    H --> I[Return Interface]
    
    style B fill:#fff3e0

Advantages:

  • Runtime connector selection
  • Easy addition of new connectors
  • Consistent interface wrapping
  • Configuration-driven behavior

3. Dependency Injection (Voice Connector)

Voice connector uses lazy initialization with dependency injection:

sequenceDiagram
    participant Route
    participant Interface as VoiceObjects
    participant Connector as ConnectorGoogleSpeech
    
    Route->>Interface: getVoiceInterface(user)
    Interface->>Interface: _getGoogleSpeechConnector()
    
    alt First Call
        Interface->>Connector: __init__()
        Connector->>Connector: Load credentials
        Connector->>Connector: Initialize clients
        Connector-->>Interface: Connector instance
        Interface->>Interface: Cache connector
    else Subsequent Calls
        Interface-->>Route: Cached connector
    end
    
    Interface-->>Route: Voice interface

Context Management Pattern

All connectors support context updates for audit trails:

graph LR
    A[User Login] --> B[Create Interface]
    B --> C[Initialize Connector]
    C --> D[Set userId Context]
    
    E[User Switch] --> F[updateContext]
    F --> G[Update userId]
    F --> H[Clear Caches]
    
    I[All Operations] --> J[Include userId in metadata]
    J --> K[_createdBy]
    J --> L[_modifiedBy]
    
    style D fill:#e8f5e9
    style G fill:#fff3e0

Context Metadata:

  • _createdBy: User who created record
  • _modifiedBy: User who last modified record
  • _createdAt: Creation timestamp
  • _modifiedAt: Modification timestamp

Error Handling Pattern

Connectors implement consistent error handling:

graph TD
    A[Operation Start] --> B{Try Operation}
    B -->|Success| C[Return Result]
    B -->|Error| D[Log Error]
    
    D --> E{Retry Possible?}
    E -->|Yes| F[Execute Fallback]
    E -->|No| G[Return Error Response]
    
    F --> H{Success?}
    H -->|Yes| C
    H -->|No| G
    
    G --> I[Structured Error Response]
    
    style C fill:#e8f5e9
    style G fill:#ffebee

Error Response Structure:

  • Consistent dictionary format
  • success: Boolean indicator
  • error: Descriptive error message
  • Additional context fields
  • No exceptions propagated to application layer

Configuration

Database Configuration

Each database interface reads specific configuration keys:

App Database (User/Mandate Management):

  • DB_APP_HOST: Database host or file path
  • DB_APP_DATABASE: Database name
  • DB_APP_USER: Database username
  • DB_APP_PASSWORD_SECRET: Encrypted password
  • DB_APP_PORT: Database port (default: 5432)

Chat Database:

  • DB_CHAT_HOST
  • DB_CHAT_DATABASE
  • DB_CHAT_USER
  • DB_CHAT_PASSWORD_SECRET
  • DB_CHAT_PORT

Management Database:

  • DB_MANAGEMENT_HOST
  • DB_MANAGEMENT_DATABASE
  • DB_MANAGEMENT_USER
  • DB_MANAGEMENT_PASSWORD_SECRET
  • DB_MANAGEMENT_PORT

Real Estate Database:

  • DB_REAL_ESTATE_HOST
  • DB_REAL_ESTATE_DATABASE
  • DB_REAL_ESTATE_USER
  • DB_REAL_ESTATE_PASSWORD_SECRET
  • DB_REAL_ESTATE_PORT

Voice Connector Configuration

Google Cloud Credentials:

  • Connector_GoogleSpeech_API_KEY_SECRET: Complete service account JSON key (encrypted)

Ticket Connector Configuration

Ticket connectors receive configuration at runtime through connectorParams:

JIRA Configuration:

  • apiUsername: JIRA username
  • apiToken: API token
  • apiUrl: JIRA instance URL
  • projectCode: Project key
  • ticketType: Issue type filter

ClickUp Configuration:

  • apiToken: ClickUp API token
  • teamId: Workspace ID
  • listId: Optional list ID
  • apiUrl: API base URL

Configuration Flow

sequenceDiagram
    participant Config as config.ini
    participant Security as Security Module
    participant Interface
    participant Connector
    
    Config->>Security: Read encrypted values
    Security->>Security: Decrypt secrets
    Security-->>Interface: Configuration values
    
    Interface->>Connector: Initialize with config
    Connector->>Connector: Validate configuration
    
    alt Valid Config
        Connector->>Connector: Establish connections
        Connector-->>Interface: Ready
    else Invalid Config
        Connector-->>Interface: Raise Exception
    end

Design Principles

1. Abstraction and Encapsulation

Principle: Hide implementation details behind consistent interfaces.

graph LR
    A[Application Code] --> B[Interface Layer]
    B -.->|Never directly accesses| C[Connector Details]
    B --> D[Public API]
    D --> C
    
    style B fill:#e8f5e9
    style C fill:#ffebee

Benefits:

  • Technology independence
  • Easy testing with mocks
  • Simplified application code
  • Future-proof architecture

2. Duck Typing over Formal Interfaces

Rationale: Python's duck typing provides flexibility without interface boilerplate.

Both database connectors provide identical methods without inheriting from a common base class. This allows:

  • Natural Python idioms
  • Easy addition of connector-specific features
  • No multiple inheritance complexity
  • Freedom in implementation

3. Configuration over Code

Principle: Behavior should be configurable without code changes.

graph TD
    A[Deployment Requirements] --> B[Configuration Files]
    B --> C[Runtime Behavior]
    
    B1[Development] --> B
    B2[Staging] --> B
    B3[Production] --> B
    
    C --> C1[Connector Selection]
    C --> C2[Connection Parameters]
    C --> C3[Feature Flags]
    
    style B fill:#fff3e0

Implementation:

  • External configuration files
  • Environment-specific settings
  • Encrypted secrets support
  • No hardcoded credentials

4. Fail-Safe Defaults

Principle: System should work out-of-the-box with sensible defaults.

Examples:

  • JSON connector for development (no DB setup)
  • Default sample rates for audio processing
  • Automatic format detection
  • Graceful degradation

5. Explicit Error Handling

Principle: Errors should be caught, logged, and returned as data structures.

graph TD
    A[Operation] --> B{Success?}
    B -->|Yes| C[Return Success Response]
    B -->|No| D[Catch Exception]
    
    D --> E[Log Error with Context]
    E --> F[Create Error Response]
    F --> G[Return Error Structure]
    
    style C fill:#e8f5e9
    style G fill:#ffebee

Benefits:

  • No unexpected exceptions
  • Consistent error format
  • Rich error context for debugging
  • Application can handle errors gracefully

6. Single Responsibility

Principle: Each connector has one clear purpose.

  • Database Connectors: Only handle data persistence
  • Voice Connector: Only handle voice processing
  • Ticket Connectors: Only handle external ticket systems

Business logic, validation, and transformations belong in higher layers.

7. Dependency Inversion

Principle: High-level modules don't depend on low-level modules.

graph TD
    A[Workflow Layer] --> B[Service Layer]
    B --> C[Interface Layer]
    C --> D[Connector Layer]
    
    A -.->|Does not depend on| D
    B -.->|Does not depend on| D
    
    style A fill:#e3f2fd
    style D fill:#e8f5e9

The application depends on interfaces (duck-typed contracts), not concrete implementations.

8. Idempotency Where Possible

Principle: Operations should be safe to retry.

Implementation:

  • Record updates are idempotent (same result if repeated)
  • Duplicate detection in pagination
  • Transaction rollback on errors
  • Atomic file operations

9. Progressive Enhancement

Principle: Core functionality works simply; advanced features add complexity only when needed.

Examples:

  • Basic audio format → Automatic fallbacks
  • Simple field mapping → Complex transformations
  • Single database → Multiple database support
  • Direct API calls → Retry logic

10. Audit Trail by Design

Principle: All data modifications tracked automatically.

graph LR
    A[Create/Modify Operation] --> B[Add Metadata]
    B --> C[_createdAt]
    B --> D[_createdBy]
    B --> E[_modifiedAt]
    B --> F[_modifiedBy]
    
    G[User Context] --> B
    H[Current Timestamp] --> B
    
    style B fill:#fff3e0

Benefits:

  • Automatic compliance
  • Debugging support
  • Security auditing
  • User accountability

Summary

The Connectors component provides a robust, flexible abstraction layer for external system integration. Key strengths include:

  • Technology Independence: Application code unaware of specific storage or service implementations
  • Flexibility: Easy swapping between implementations without code changes
  • Reliability: Comprehensive error handling and retry mechanisms
  • Performance: Optimized for each technology (caching for JSON, connection pooling for PostgreSQL)
  • Maintainability: Clear separation of concerns and consistent patterns
  • Extensibility: New connectors can be added with minimal impact

The component enables the application to work seamlessly across different deployment scenarios while maintaining clean architecture and separation of concerns.