gateway/docs/code-documentation/connectors-component.md

# Connectors Component Documentation

## Table of Contents

1. [Overview](#overview)
2. [Architecture](#architecture)
3. [Database Connectors](#database-connectors)
4. [Voice Connector](#voice-connector)
5. [Ticket Connectors](#ticket-connectors)
6. [Integration Patterns](#integration-patterns)
7. [Configuration](#configuration)
8. [Design Principles](#design-principles)

---

## Overview

The Connectors component provides abstraction layers for external systems and data storage mechanisms. It acts as the bridge between the application's business logic and external services, databases, and third-party APIs. This component implements the **Adapter Pattern** to provide consistent interfaces regardless of the underlying technology.

### Purpose and Scope

The connectors component serves three primary functions:

1. **Data Persistence** - Abstracts database operations for both JSON file-based and PostgreSQL storage
2. **Voice Processing** - Integrates Google Cloud Speech services for voice recognition, translation, and synthesis
3. **Ticket Management** - Connects to external ticketing systems (JIRA, ClickUp) for synchronization

### Component Structure

```
modules/connectors/
├── connectorDbJson.py          # JSON file-based database
├── connectorDbPostgre.py       # PostgreSQL database
├── connectorVoiceGoogle.py     # Google Cloud Speech services
├── connectorTicketsJira.py     # JIRA integration
└── connectorTicketsClickup.py  # ClickUp integration
```

---

## Architecture

### Connector Hierarchy

```mermaid
graph TD
    A[Application Layer<br/>routes/, workflows/, features/] --> B[Interface Layer<br/>modules/interfaces/]
    B --> C[Connector Layer<br/>modules/connectors/]
    C --> D[External Systems]
    
    B --> B1[interfaceDbAppObjects.py<br/>AppObjects]
    B --> B2[interfaceDbChatObjects.py<br/>ChatObjects]
    B --> B3[interfaceVoiceObjects.py<br/>VoiceObjects]
    B --> B4[interfaceTicketObjects.py<br/>TicketInterface]
    
    C --> C1[connectorDbJson.py<br/>DatabaseConnector]
    C --> C2[connectorDbPostgre.py<br/>DatabaseConnector]
    C --> C3[connectorVoiceGoogle.py<br/>ConnectorGoogleSpeech]
    C --> C4[connectorTicketsJira.py<br/>ConnectorTicketJira]
    C --> C5[connectorTicketsClickup.py<br/>ConnectorTicketClickup]
    
    B1 --> C1
    B1 --> C2
    B2 --> C1
    B2 --> C2
    B3 --> C3
    B4 --> C4
    B4 --> C5
    
    C1 --> D1[JSON Files]
    C2 --> D2[PostgreSQL]
    C3 --> D3[Google Cloud APIs]
    C4 --> D4[JIRA API]
    C5 --> D5[ClickUp API]
    
    style A fill:#e1f5ff
    style B fill:#fff9e6
    style C fill:#e8f5e9
    style D fill:#fce4ec
```

### Layered Architecture Pattern

The connectors follow a three-tier architecture:

1. **Application Layer**: Business logic, workflows, services
2. **Interface Layer**: Domain-specific abstractions (AppObjects, ChatObjects, etc.)
3. **Connector Layer**: Technology-specific implementations
4. **External Systems**: Databases, APIs, cloud services

This separation ensures:
- **Loose Coupling**: Application code doesn't depend on specific technologies
- **Testability**: Connectors can be mocked or swapped
- **Flexibility**: Easy migration between storage backends or service providers
- **Maintainability**: Changes to external systems are isolated to connector layer

---

## Database Connectors

### Overview

The application supports two database connector implementations that provide identical public APIs but different storage mechanisms. This allows deployment flexibility without code changes.

### DatabaseConnector Interface

Both database connectors implement a common interface using duck typing (no formal interface class). They provide:

- **CRUD Operations**: Create, read, update, delete records
- **Schema Management**: Dynamic table creation from Pydantic models
- **Context Management**: User-aware operations for audit trails
- **Concurrency Control**: Thread-safe operations with locking mechanisms
- **Initial Record Tracking**: System table for bootstrap data

### JSON Database Connector

#### Purpose and Use Cases

The JSON connector is ideal for:
- **Development Environments**: Fast setup without database infrastructure
- **Small Deployments**: Low-volume applications
- **Portable Data**: Easy backup and version control
- **Testing**: Simplified test data management

#### Storage Structure

```mermaid
graph LR
    A[Database Host Directory] --> B[Database Name Directory]
    B --> C[Table1 Directory]
    B --> D[Table2 Directory]
    B --> E[_system.json]
    
    C --> C1[record1.json]
    C --> C2[record2.json]
    C --> C3[_metadata.json]
    
    D --> D1[record3.json]
    D --> D2[record4.json]
    
    style E fill:#ffeb3b
    style C3 fill:#ffeb3b
```

**File System Layout:**
- Each database is a directory
- Each table is a subdirectory
- Each record is a separate JSON file
- Metadata files track record IDs and indexes
- System table stores initial record references

#### Key Features

**Atomic Operations:**
- Temporary file creation with validation
- Atomic move operations to prevent corruption
- Lock management for concurrent access

**Caching Strategy:**
- In-memory table cache for performance
- Metadata cache for quick record lookups
- Intelligent cache invalidation

**Concurrency Control:**
- File-level locks with timeout protection
- Table-level locks for metadata operations
- Deadlock prevention through lock ordering

### PostgreSQL Database Connector

#### Purpose and Use Cases

The PostgreSQL connector is designed for:
- **Production Environments**: High-performance, reliable storage
- **Multi-User Systems**: Concurrent access with ACID guarantees
- **Large Datasets**: Efficient querying and indexing
- **Scalability**: Horizontal and vertical scaling capabilities

#### Schema Architecture

```mermaid
erDiagram
    _system ||--o{ Tables : tracks_initial_records
    Tables ||--o{ Records : contains
    
    _system {
        varchar table_name PK
        varchar initial_id
        double _createdAt
        double _modifiedAt
    }
    
    Tables {
        varchar id PK
        text field1
        jsonb field2
        double _createdAt
        double _modifiedAt
        varchar _createdBy
        varchar _modifiedBy
    }
```

#### Dynamic Schema Generation

The connector automatically:
- Creates tables from Pydantic models
- Maps Python types to SQL types
- Adds metadata columns automatically
- Creates indexes for foreign key fields
- Performs additive migrations (adds missing columns)

**Type Mapping:**
- `str` → `TEXT`
- `int` → `INTEGER`
- `float` → `DOUBLE PRECISION`
- `bool` → `BOOLEAN`
- `dict/list` → `JSONB` (enables flexible document storage)

#### JSONB Support

The connector uses PostgreSQL's JSONB type for complex fields:
- Efficient binary JSON storage
- Indexable JSON content
- Native JSON operators
- Flexible schema evolution

### Database Connector Selection

The system selects connectors through import statements in interface files:

```mermaid
graph TD
    A[Interface Initialization] --> B{Check DB_HOST Config}
    B -->|File Path| C[Import connectorDbJson]
    B -->|Host:Port| D[Import connectorDbPostgre]
    C --> E[Create DatabaseConnector]
    D --> E
    E --> F[Initialize System]
    
    style B fill:#fff3e0
```

**Selection Criteria:**
- Configuration-driven through `config.ini`
- Import statement determines implementation
- Transparent to application layer
- No runtime switching (decided at startup)

### Common Operations Flow

```mermaid
sequenceDiagram
    participant App as Application
    participant Iface as Interface Layer
    participant DB as DatabaseConnector
    participant Storage as Storage Backend
    
    App->>Iface: getRecordset(Model, filters)
    Iface->>DB: getRecordset(model_class, recordFilter)
    
    alt PostgreSQL
        DB->>Storage: SELECT * FROM table WHERE...
        Storage-->>DB: Rows
    else JSON
        DB->>Storage: Read files from directory
        Storage-->>DB: JSON objects
    end
    
    DB->>DB: Apply filters
    DB->>DB: Handle JSONB parsing
    DB-->>Iface: List of records
    Iface->>Iface: Apply UAM filters
    Iface-->>App: Filtered records
    
    Note over DB,Storage: Both connectors provide<br/>identical interface
```

### Transaction Handling

**PostgreSQL:**
- Uses database transactions
- ACID compliance
- Automatic rollback on errors
- Connection pooling and retry logic

**JSON:**
- File-level atomicity
- Lock-based isolation
- Manual rollback through file operations
- Lock timeout protection

### Performance Considerations

| Aspect | JSON Connector | PostgreSQL Connector |
|--------|---------------|---------------------|
| **Read Speed** | Fast for small datasets, degrades with size | Consistent, optimized with indexes |
| **Write Speed** | Fast for single records | Fast with connection pooling |
| **Concurrent Access** | Limited by file locking | Excellent with MVCC |
| **Query Capability** | In-memory filtering only | Full SQL with JSONB operators |
| **Scalability** | Limited to single server | Horizontal and vertical scaling |
| **Memory Usage** | High (full table caching) | Low (database managed) |

---

## Voice Connector

### Overview

The `ConnectorGoogleSpeech` provides integration with Google Cloud AI services for voice processing, offering a complete pipeline for speech recognition, translation, and text-to-speech synthesis.

### Architecture

```mermaid
graph TB
    A[Voice Interface] --> B[ConnectorGoogleSpeech]
    
    B --> C[Speech-to-Text Client]
    B --> D[Translation Client]
    B --> E[Text-to-Speech Client]
    
    C --> F[Google Cloud Speech-to-Text API]
    D --> G[Google Cloud Translation API]
    E --> H[Google Cloud Text-to-Speech API]
    
    F --> I[Audio Processing]
    G --> J[Language Translation]
    H --> K[Voice Synthesis]
    
    style B fill:#e8f5e9
    style F fill:#e3f2fd
    style G fill:#e3f2fd
    style H fill:#e3f2fd
```

### Core Capabilities

#### 1. Speech-to-Text Processing

**Audio Format Support:**
- WEBM OPUS (primary web recording format)
- WAV (Linear PCM)
- MP3
- FLAC
- OGG

**Processing Pipeline:**

```mermaid
sequenceDiagram
    participant Client
    participant Connector
    participant Validator
    participant API as Google Speech API
    
    Client->>Connector: speechToText(audioContent)
    Connector->>Validator: validateAudioFormat()
    
    Validator->>Validator: Detect format
    Validator->>Validator: Extract sample rate
    Validator->>Validator: Determine channels
    Validator-->>Connector: Format metadata
    
    Connector->>API: recognize(config, audio)
    
    alt Success
        API-->>Connector: Transcription + confidence
        Connector-->>Client: Success response
    else API Error
        API-->>Connector: Error
        Connector->>Connector: Try fallback configs
        Connector->>API: recognize(fallback_config)
        API-->>Connector: Result
        Connector-->>Client: Response
    end
```

**Audio Format Detection:**
- Magic byte pattern recognition
- Header parsing for metadata extraction
- Automatic format-specific configuration
- Deep scanning for ambiguous formats

**Fallback Strategy:**
Multiple configurations tried automatically:
1. Detected format with detected parameters
2. Alternative encodings (LINEAR16, WEBM_OPUS)
3. Standard sample rates (8kHz, 16kHz, 44.1kHz, 48kHz)
4. Different recognition models (latest_long, phone_call, latest_short)

#### 2. Translation Services

**Features:**
- Automatic language detection
- HTML entity decoding
- Bidirectional translation
- Preserves text formatting

**Translation Flow:**

```mermaid
graph LR
    A[Input Text] --> B[Google Translation API]
    B --> C[Detect Source Language]
    C --> D[Translate to Target]
    D --> E[Decode HTML Entities]
    E --> F[Return Result]
    
    style B fill:#e3f2fd
```

#### 3. Text-to-Speech Synthesis

**Voice Selection:**
- Language-specific voices
- Gender-based voice selection
- Neural voice quality
- Multiple voice variants per language

**Synthesis Process:**

```mermaid
sequenceDiagram
    participant Client
    participant Connector
    participant API as Google TTS API
    
    Client->>Connector: textToSpeech(text, language, voice)
    
    alt Voice Specified
        Connector->>API: synthesize_speech(voice)
    else No Voice
        Connector->>Client: Error: no default voice
    end
    
    API->>API: Generate audio
    API-->>Connector: MP3 audio data
    Connector-->>Client: Audio content + metadata
```

### Complete Pipeline: Speech-to-Translated-Text

The connector provides an integrated pipeline:

```mermaid
graph TD
    A[Audio Input] --> B[Speech-to-Text]
    B --> C[Original Text]
    C --> D[Translation]
    D --> E[Translated Text]
    
    B -.->|Confidence Score| F[Metadata]
    D -.->|Source Language| F
    F --> G[Complete Response]
    
    style A fill:#ffebee
    style C fill:#fff3e0
    style E fill:#e8f5e9
    style G fill:#e1f5fe
```

**Use Cases:**
- Real-time voice translation
- Multilingual voice assistants
- International call centers
- Language learning applications

### Authentication and Configuration

**Credential Management:**
- Service account JSON key stored in configuration
- Parsed and loaded at initialization
- No file system dependency
- Credentials object creation from JSON

**Configuration Parameters:**
- `Connector_GoogleSpeech_API_KEY_SECRET`: Service account JSON (encrypted)

### Error Handling and Resilience

**Retry Mechanisms:**
- Multiple encoding attempts
- Sample rate fallbacks
- Model fallbacks
- Graceful degradation

**Validation:**
- Audio length verification
- Format compatibility checks
- Content quality analysis
- Silence detection

### Integration Points

```mermaid
graph TB
    A[Routes Layer] --> B[VoiceObjects Interface]
    B --> C[ConnectorGoogleSpeech]
    
    A1[/voice-google/speech-to-text] --> B
    A2[/voice-google/translate] --> B
    A3[/voice-google/text-to-speech] --> B
    A4[/voice-google/languages] --> B
    A5[/voice-google/voices] --> B
    A6[WebSocket /ws/realtime-interpreter] --> B
    
    style A1 fill:#e8f5e9
    style A2 fill:#e8f5e9
    style A3 fill:#e8f5e9
    style A4 fill:#fff3e0
    style A5 fill:#fff3e0
    style A6 fill:#ffebee
```

---

## Ticket Connectors

### Overview

Ticket connectors provide unified access to external project management and ticketing systems. They enable bidirectional synchronization of tasks and tickets with external platforms.

### Common Interface Pattern

Both ticket connectors implement a common base pattern:

**Core Operations:**
- `readAttributes()`: Fetch field metadata from the system
- `readTasks()`: Read tickets/tasks with pagination
- `writeTasks()`: Update tickets/tasks in bulk

```mermaid
classDiagram
    class TicketBase {
        <<abstract>>
        +readAttributes() list~TicketFieldAttribute~
        +readTasks(limit) list~dict~
        +writeTasks(tasklist) None
    }
    
    class ConnectorTicketJira {
        -apiUsername: str
        -apiToken: str
        -apiUrl: str
        -projectCode: str
        -ticketType: str
        +readAttributes()
        +readTasks()
        +writeTasks()
    }
    
    class ConnectorTicketClickup {
        -apiToken: str
        -teamId: str
        -listId: str
        -apiUrl: str
        +readAttributes()
        +readTasks()
        +writeTasks()
    }
    
    TicketBase <|-- ConnectorTicketJira
    TicketBase <|-- ConnectorTicketClickup
```

### JIRA Connector

#### Authentication and Configuration

**Required Parameters:**
- `apiUsername`: JIRA account username
- `apiToken`: API authentication token
- `apiUrl`: JIRA instance URL
- `projectCode`: Project identifier
- `ticketType`: Issue type filter

#### Field Discovery

```mermaid
sequenceDiagram
    participant App
    participant Connector
    participant JIRA as JIRA API
    
    App->>Connector: readAttributes()
    Connector->>JIRA: POST /search/jql
    JIRA-->>Connector: Issue with all fields
    
    alt Fields Available
        Connector->>Connector: Extract field mappings
        Connector-->>App: List of attributes
    else No Fields
        Connector->>JIRA: GET /field
        JIRA-->>Connector: All field definitions
        Connector-->>App: Field list
    end
```

**Field Mapping:**
- Maps JIRA field IDs to human-readable names
- Supports custom fields
- Handles complex field types (ADF, arrays, objects)

#### Pagination Strategy

The connector uses JIRA's cursor-based pagination:

```mermaid
graph TD
    A[Start] --> B[Initial Request]
    B --> C{Issues Returned?}
    C -->|No| D[End]
    C -->|Yes| E[Process Issues]
    E --> F{Has Next Page Token?}
    F -->|No| D
    F -->|Yes| G[Request Next Page]
    G --> H{Safety Cap Reached?}
    H -->|Yes| D
    H -->|No| C
    
    style D fill:#e8f5e9
    style H fill:#ffebee
```

**Pagination Features:**
- Cursor-based continuation
- Duplicate detection
- Safety cap (1000 pages max)
- Configurable page size
- Loop prevention

#### Task Updates

**Update Flow:**

```mermaid
sequenceDiagram
    participant App
    participant Connector
    participant JIRA
    
    App->>Connector: writeTasks([task1, task2])
    
    loop For each task
        Connector->>Connector: Extract task ID
        Connector->>Connector: Map fields
        Connector->>Connector: Convert to ADF format
        Connector->>JIRA: PUT /issue/{id}
        
        alt Success
            JIRA-->>Connector: 204 No Content
        else Error
            JIRA-->>Connector: Error response
            Connector->>Connector: Log error
        end
    end
    
    Connector-->>App: Complete
```

**Field Processing:**
- Automatic ADF (Atlassian Document Format) conversion for rich text
- Custom field handling
- Empty field validation
- Selective field updates

### ClickUp Connector

#### Authentication and Configuration

**Required Parameters:**
- `apiToken`: ClickUp API token
- `teamId`: Workspace/team identifier
- `listId`: Optional list filter
- `apiUrl`: API endpoint (default: https://api.clickup.com/api/v2)

#### Hierarchical Data Access

```mermaid
graph TD
    A[Team Level] --> B[Space Level]
    B --> C[Folder Level]
    C --> D[List Level]
    D --> E[Task Level]
    E --> F[Subtask Level]
    
    G[Connector] -.->|listId specified| D
    G -.->|listId not specified| A
    
    style G fill:#fff3e0
```

**Access Patterns:**
- List-specific access when `listId` provided
- Team-wide search when no `listId`
- Automatic subtask inclusion

#### Field Discovery

ClickUp provides both:
1. **Custom Fields**: From list-specific field API
2. **Core Fields**: Standard task properties

```mermaid
graph LR
    A[readAttributes] --> B{listId Present?}
    B -->|Yes| C[GET /list/id/field]
    B -->|No| D[Return Core Fields Only]
    
    C --> E[Merge Custom + Core Fields]
    D --> F[Return Fields]
    E --> F
    
    style C fill:#e3f2fd
```

**Core Fields:**
- ID
- Name
- Status
- Assignees
- Date Created
- Due Date

#### Task Retrieval

**Pagination:**
- Page-based pagination
- Configurable page size (100 default)
- Automatic page iteration

```mermaid
sequenceDiagram
    participant Connector
    participant API as ClickUp API
    
    loop Until no more tasks
        Connector->>API: GET /list/id/task?page={n}
        API-->>Connector: Tasks array
        
        alt Tasks returned < page size
            Connector->>Connector: Stop pagination
        else More tasks possible
            Connector->>Connector: Increment page
        end
    end
```

#### Task Updates

**Update Strategy:**

```mermaid
graph TD
    A[Task Update Request] --> B[Extract Task ID]
    B --> C[Extract Fields]
    C --> D{Field Type?}
    
    D -->|name/summary| E[Update name]
    D -->|status| F[Update status]
    D -->|custom field| G[Add to custom_fields array]
    D -->|other| H[Add to description]
    
    E --> I[Build Payload]
    F --> I
    G --> I
    H --> I
    
    I --> J[PUT /task/id]
    
    style J fill:#e3f2fd
```

**Field Mapping:**
- Heuristic field name matching
- Custom field special handling
- Best-effort unknown field mapping

### Ticket Interface Integration

The ticket connectors are wrapped by the `TicketInterface` for field mapping:

```mermaid
graph TB
    A[Workflow/Feature] --> B[TicketService]
    B --> C[TicketInterface]
    C --> D[Connector Factory]
    
    D -->|connectorType='Jira'| E[ConnectorTicketJira]
    D -->|connectorType='ClickUp'| F[ConnectorTicketClickup]
    
    C --> G[Task Sync Definition]
    G --> H[Field Mapping]
    
    E --> I[External System]
    F --> I
    
    style C fill:#fff3e0
    style G fill:#e8f5e9
```

**Task Sync Definition:**
- Maps internal field names to external field paths
- Specifies read/write directions
- Handles nested field access
- Enables field transformations

**Example Flow:**

```mermaid
sequenceDiagram
    participant Workflow
    participant Interface as TicketInterface
    participant Connector
    participant External as External System
    
    Workflow->>Interface: exportTicketsAsList()
    Interface->>Connector: readTasks()
    Connector->>External: API Request
    External-->>Connector: Raw tickets
    Connector-->>Interface: Tickets list
    Interface->>Interface: _transformTicketRecords()
    Interface-->>Workflow: Transformed data
    
    Note over Interface: Applies field mapping<br/>from sync definition
```

---

## Integration Patterns

### Connector Initialization Patterns

#### 1. Singleton Pattern (Database Connectors)

Database connectors use singleton-like patterns through interface factories:

```mermaid
graph TD
    A[Request 1] --> B[getAppInterface]
    C[Request 2] --> B
    D[Request 3] --> B
    
    B --> E{Instance Exists?}
    E -->|No| F[Create AppObjects]
    E -->|Yes| G[Return Cached Instance]
    
    F --> H[Initialize DatabaseConnector]
    H --> I[Store in _gatewayInterfaces]
    
    G --> J[Return Interface]
    I --> J
    
    style B fill:#fff3e0
    style I fill:#e8f5e9
```

**Benefits:**
- Reuses database connections
- Maintains context consistency
- Reduces initialization overhead
- Per-user instances for security

#### 2. Factory Pattern (Ticket Connectors)

Ticket connectors use factory pattern for runtime selection:

```mermaid
graph TD
    A[Service Request] --> B[createTicketInterfaceByType]
    B --> C{Connector Type?}
    
    C -->|'jira'| D[Import JIRA Connector]
    C -->|'clickup'| E[Import ClickUp Connector]
    C -->|unknown| F[Raise ValueError]
    
    D --> G[Create Connector Instance]
    E --> G
    
    G --> H[Wrap in TicketInterface]
    H --> I[Return Interface]
    
    style B fill:#fff3e0
```

**Advantages:**
- Runtime connector selection
- Easy addition of new connectors
- Consistent interface wrapping
- Configuration-driven behavior

#### 3. Dependency Injection (Voice Connector)

Voice connector uses lazy initialization with dependency injection:

```mermaid
sequenceDiagram
    participant Route
    participant Interface as VoiceObjects
    participant Connector as ConnectorGoogleSpeech
    
    Route->>Interface: getVoiceInterface(user)
    Interface->>Interface: _getGoogleSpeechConnector()
    
    alt First Call
        Interface->>Connector: __init__()
        Connector->>Connector: Load credentials
        Connector->>Connector: Initialize clients
        Connector-->>Interface: Connector instance
        Interface->>Interface: Cache connector
    else Subsequent Calls
        Interface-->>Route: Cached connector
    end
    
    Interface-->>Route: Voice interface
```

### Context Management Pattern

All connectors support context updates for audit trails:

```mermaid
graph LR
    A[User Login] --> B[Create Interface]
    B --> C[Initialize Connector]
    C --> D[Set userId Context]
    
    E[User Switch] --> F[updateContext]
    F --> G[Update userId]
    F --> H[Clear Caches]
    
    I[All Operations] --> J[Include userId in metadata]
    J --> K[_createdBy]
    J --> L[_modifiedBy]
    
    style D fill:#e8f5e9
    style G fill:#fff3e0
```

**Context Metadata:**
- `_createdBy`: User who created record
- `_modifiedBy`: User who last modified record
- `_createdAt`: Creation timestamp
- `_modifiedAt`: Modification timestamp

### Error Handling Pattern

Connectors implement consistent error handling:

```mermaid
graph TD
    A[Operation Start] --> B{Try Operation}
    B -->|Success| C[Return Result]
    B -->|Error| D[Log Error]
    
    D --> E{Retry Possible?}
    E -->|Yes| F[Execute Fallback]
    E -->|No| G[Return Error Response]
    
    F --> H{Success?}
    H -->|Yes| C
    H -->|No| G
    
    G --> I[Structured Error Response]
    
    style C fill:#e8f5e9
    style G fill:#ffebee
```

**Error Response Structure:**
- Consistent dictionary format
- `success`: Boolean indicator
- `error`: Descriptive error message
- Additional context fields
- No exceptions propagated to application layer

---

## Configuration

### Database Configuration

Each database interface reads specific configuration keys:

**App Database (User/Mandate Management):**
- `DB_APP_HOST`: Database host or file path
- `DB_APP_DATABASE`: Database name
- `DB_APP_USER`: Database username
- `DB_APP_PASSWORD_SECRET`: Encrypted password
- `DB_APP_PORT`: Database port (default: 5432)

**Chat Database:**
- `DB_CHAT_HOST`
- `DB_CHAT_DATABASE`
- `DB_CHAT_USER`
- `DB_CHAT_PASSWORD_SECRET`
- `DB_CHAT_PORT`

**Management Database:**
- `DB_MANAGEMENT_HOST`
- `DB_MANAGEMENT_DATABASE`
- `DB_MANAGEMENT_USER`
- `DB_MANAGEMENT_PASSWORD_SECRET`
- `DB_MANAGEMENT_PORT`

**Real Estate Database:**
- `DB_REAL_ESTATE_HOST`
- `DB_REAL_ESTATE_DATABASE`
- `DB_REAL_ESTATE_USER`
- `DB_REAL_ESTATE_PASSWORD_SECRET`
- `DB_REAL_ESTATE_PORT`

### Voice Connector Configuration

**Google Cloud Credentials:**
- `Connector_GoogleSpeech_API_KEY_SECRET`: Complete service account JSON key (encrypted)

### Ticket Connector Configuration

Ticket connectors receive configuration at runtime through `connectorParams`:

**JIRA Configuration:**
- `apiUsername`: JIRA username
- `apiToken`: API token
- `apiUrl`: JIRA instance URL
- `projectCode`: Project key
- `ticketType`: Issue type filter

**ClickUp Configuration:**
- `apiToken`: ClickUp API token
- `teamId`: Workspace ID
- `listId`: Optional list ID
- `apiUrl`: API base URL

### Configuration Flow

```mermaid
sequenceDiagram
    participant Config as config.ini
    participant Security as Security Module
    participant Interface
    participant Connector
    
    Config->>Security: Read encrypted values
    Security->>Security: Decrypt secrets
    Security-->>Interface: Configuration values
    
    Interface->>Connector: Initialize with config
    Connector->>Connector: Validate configuration
    
    alt Valid Config
        Connector->>Connector: Establish connections
        Connector-->>Interface: Ready
    else Invalid Config
        Connector-->>Interface: Raise Exception
    end
```

---

## Design Principles

### 1. Abstraction and Encapsulation

**Principle:** Hide implementation details behind consistent interfaces.

```mermaid
graph LR
    A[Application Code] --> B[Interface Layer]
    B -.->|Never directly accesses| C[Connector Details]
    B --> D[Public API]
    D --> C
    
    style B fill:#e8f5e9
    style C fill:#ffebee
```

**Benefits:**
- Technology independence
- Easy testing with mocks
- Simplified application code
- Future-proof architecture

### 2. Duck Typing over Formal Interfaces

**Rationale:** Python's duck typing provides flexibility without interface boilerplate.

Both database connectors provide identical methods without inheriting from a common base class. This allows:
- Natural Python idioms
- Easy addition of connector-specific features
- No multiple inheritance complexity
- Freedom in implementation

### 3. Configuration over Code

**Principle:** Behavior should be configurable without code changes.

```mermaid
graph TD
    A[Deployment Requirements] --> B[Configuration Files]
    B --> C[Runtime Behavior]
    
    B1[Development] --> B
    B2[Staging] --> B
    B3[Production] --> B
    
    C --> C1[Connector Selection]
    C --> C2[Connection Parameters]
    C --> C3[Feature Flags]
    
    style B fill:#fff3e0
```

**Implementation:**
- External configuration files
- Environment-specific settings
- Encrypted secrets support
- No hardcoded credentials

### 4. Fail-Safe Defaults

**Principle:** System should work out-of-the-box with sensible defaults.

**Examples:**
- JSON connector for development (no DB setup)
- Default sample rates for audio processing
- Automatic format detection
- Graceful degradation

### 5. Explicit Error Handling

**Principle:** Errors should be caught, logged, and returned as data structures.

```mermaid
graph TD
    A[Operation] --> B{Success?}
    B -->|Yes| C[Return Success Response]
    B -->|No| D[Catch Exception]
    
    D --> E[Log Error with Context]
    E --> F[Create Error Response]
    F --> G[Return Error Structure]
    
    style C fill:#e8f5e9
    style G fill:#ffebee
```

**Benefits:**
- No unexpected exceptions
- Consistent error format
- Rich error context for debugging
- Application can handle errors gracefully

### 6. Single Responsibility

**Principle:** Each connector has one clear purpose.

- **Database Connectors**: Only handle data persistence
- **Voice Connector**: Only handle voice processing
- **Ticket Connectors**: Only handle external ticket systems

Business logic, validation, and transformations belong in higher layers.

### 7. Dependency Inversion

**Principle:** High-level modules don't depend on low-level modules.

```mermaid
graph TD
    A[Workflow Layer] --> B[Service Layer]
    B --> C[Interface Layer]
    C --> D[Connector Layer]
    
    A -.->|Does not depend on| D
    B -.->|Does not depend on| D
    
    style A fill:#e3f2fd
    style D fill:#e8f5e9
```

The application depends on interfaces (duck-typed contracts), not concrete implementations.

### 8. Idempotency Where Possible

**Principle:** Operations should be safe to retry.

**Implementation:**
- Record updates are idempotent (same result if repeated)
- Duplicate detection in pagination
- Transaction rollback on errors
- Atomic file operations

### 9. Progressive Enhancement

**Principle:** Core functionality works simply; advanced features add complexity only when needed.

**Examples:**
- Basic audio format → Automatic fallbacks
- Simple field mapping → Complex transformations
- Single database → Multiple database support
- Direct API calls → Retry logic

### 10. Audit Trail by Design

**Principle:** All data modifications tracked automatically.

```mermaid
graph LR
    A[Create/Modify Operation] --> B[Add Metadata]
    B --> C[_createdAt]
    B --> D[_createdBy]
    B --> E[_modifiedAt]
    B --> F[_modifiedBy]
    
    G[User Context] --> B
    H[Current Timestamp] --> B
    
    style B fill:#fff3e0
```

**Benefits:**
- Automatic compliance
- Debugging support
- Security auditing
- User accountability

---

## Summary

The Connectors component provides a robust, flexible abstraction layer for external system integration. Key strengths include:

- **Technology Independence**: Application code unaware of specific storage or service implementations
- **Flexibility**: Easy swapping between implementations without code changes
- **Reliability**: Comprehensive error handling and retry mechanisms
- **Performance**: Optimized for each technology (caching for JSON, connection pooling for PostgreSQL)
- **Maintainability**: Clear separation of concerns and consistent patterns
- **Extensibility**: New connectors can be added with minimal impact

The component enables the application to work seamlessly across different deployment scenarios while maintaining clean architecture and separation of concerns.