wiki/reviews/connector_generic_refactoring_analysis.md

Connector Generic Refactoring Analysis

Problem

The connectorDbPostgre.py has hardcoded field names that make it non-generic:

• Lines 61-70: Hardcoded field names in _get_model_fields() to force JSONB type
• Lines 689-700: Hardcoded field names in _loadTable() for None value defaults
• Lines 891-902: Hardcoded field names in getRecordset() for None value defaults

Hardcoded Fields in Connector

Location 1: _get_model_fields() (Lines 61-70)

or field_name in [
    "execParameters",
    "expectedDocumentFormats",
    "resultDocuments",
    "logs",
    "messages",
    "stats",
    "tasks",
]

Location 2: _loadTable() (Lines 689-700)

if field_name in [
    "logs",
    "messages",
    "tasks",
    "expectedDocumentFormats",
    "resultDocuments",
]:
    record[field_name] = []
elif field_name in ["execParameters", "stats"]:
    record[field_name] = {}

Location 3: getRecordset() (Lines 891-902)

if field_name in [
    "logs",
    "messages",
    "tasks",
    "expectedDocumentFormats",
    "resultDocuments",
]:
    record[field_name] = []
elif field_name in ["execParameters", "stats"]:
    record[field_name] = {}

---

Field Usage Analysis

Fields Used in Models:

1. logs, messages, stats, tasks → ChatWorkflow model (datamodelChat.py)

   • Used by: interfaceDbChatObjects.py
   • Type: List[...] (should be JSONB)

2. execParameters, expectedDocumentFormats → ActionItem model (datamodelChat.py)

   • Used by: interfaceDbChatObjects.py (via AutomationDefinition)
   • Type: Dict[str, Any] and Optional[List[Dict[str, str]]] (should be JSONB)

3. resultDocuments → Not found in current models (may be legacy or unused)

---

Current Interface Handling

✅ interfaceDbChatObjects.py - HAS Field Separation Logic

• Has: _separateObjectFields() method (lines 210-254)
• Handles: Separates documents, stats to objectFields (lines 226-228)
• Still has hardcoded fields: Lines 234: execParameters, expectedDocumentFormats, resultDocuments
• Status: ✅ Partially handles field separation, but still relies on connector for JSONB detection

❌ interfaceDbComponentObjects.py - NO Field Separation Logic

• Has: Direct calls to db.recordCreate(), db.recordModify()
• Handles: None - relies entirely on connector's hardcoded logic
• Status: ❌ Needs field separation logic

❌ interfaceDbAppObjects.py - NO Field Separation Logic

• Has: Direct calls to db.recordCreate(), db.recordModify()
• Handles: None - relies entirely on connector's hardcoded logic
• Status: ❌ Needs field separation logic

⚠️ interfaceRbac.py - HAS Hardcoded Field Names

• Has: Helper function getRecordsetWithRBAC() (lines 113-120)
• Handles: Hardcoded field names for None defaults (same as connector)
• Status: ⚠️ Also needs to be made generic

---

Required Changes

1. connectorDbPostgre.py - REMOVE Hardcoded Logic

Change 1: _get_model_fields() (Lines 43-87)

REMOVE:

or field_name in [
    "execParameters",
    "expectedDocumentFormats",
    "resultDocuments",
    "logs",
    "messages",
    "stats",
    "tasks",
]

REPLACE WITH: Pure type-based detection only

# Check for JSONB fields (Dict, List, or complex types)
if (
    field_type == dict
    or field_type == list
    or (
        hasattr(field_type, "__origin__")
        and field_type.__origin__ in (dict, list)
    )
):
    fields[field_name] = "JSONB"

Change 2: _loadTable() (Lines 689-700)

REMOVE: Hardcoded field name checks for None defaults REPLACE WITH: Generic type-based handling or remove entirely (let interfaces handle)

Change 3: getRecordset() (Lines 891-902)

REMOVE: Hardcoded field name checks for None defaults REPLACE WITH: Generic type-based handling or remove entirely (let interfaces handle)

---

2. interfaceDbChatObjects.py - ENHANCE Field Separation

Change: _separateObjectFields() (Lines 210-254)

CURRENT: Lines 226-228 handle documents, stats as object fields CURRENT: Lines 234 still has hardcoded execParameters, expectedDocumentFormats, resultDocuments

UPDATE TO:

• Remove hardcoded field names from line 234
• Rely purely on type detection (Dict, List)
• Handle relational fields (documents, stats, logs, messages) as object fields (not stored in main table)
• Handle JSONB fields (execParameters, expectedDocumentFormats) as simple fields (stored as JSONB)

Logic should be:

# Relational fields (stored in separate normalized tables)
if fieldName in ['documents', 'stats', 'logs', 'messages']:
    objectFields[fieldName] = value
    continue

# JSONB fields (stored as JSONB in main table)
if (fieldType == dict or fieldType == list or 
    (hasattr(fieldType, '__origin__') and fieldType.__origin__ in (dict, list))):
    simpleFields[fieldName] = value  # Store as JSONB

---

3. interfaceDbComponentObjects.py - ADD Field Separation

Add: _separateObjectFields() method

LOCATION: After _initializeDatabase() method (around line 140)

IMPLEMENTATION: Similar to interfaceDbChatObjects.py but adapted for ComponentObjects models

• Check which models used by this interface have relational/JSONB fields
• Implement type-based field separation
• Use before calling db.recordCreate() and db.recordModify()

Models used: Prompt, FileItem, FileData, VoiceSettings, MessagingSubscription, etc.

• Check if any have relational fields that need separate handling

---

4. interfaceDbAppObjects.py - ADD Field Separation

Add: _separateObjectFields() method

LOCATION: After _initializeDatabase() method (around line 135)

IMPLEMENTATION: Similar to interfaceDbChatObjects.py but adapted for AppObjects models

• Check which models used by this interface have relational/JSONB fields
• Implement type-based field separation
• Use before calling db.recordCreate() and db.recordModify()

Models used: User, Mandate, UserInDB, Token, AuthEvent, UserConnection, etc.

• Check if any have relational fields that need separate handling

---

Impact Analysis

Models Affected:

1. ChatWorkflow (datamodelChat.py)

   • Fields: logs, messages, stats, tasks (List types)
   • Currently: Hardcoded in connector
   • After: Handled by interfaceDbChatObjects._separateObjectFields()

2. ActionItem (datamodelChat.py)

   • Fields: execParameters (Dict), expectedDocumentFormats (List)
   • Currently: Hardcoded in connector
   • After: Handled by interfaceDbChatObjects._separateObjectFields()

3. AutomationDefinition (datamodelChat.py)

   • Contains ActionItem objects
   • Currently: Hardcoded in connector
   • After: Handled by interfaceDbChatObjects._separateObjectFields()

---

Migration Steps

Step 1: Update interfaceDbChatObjects.py

1. Enhance _separateObjectFields() to handle all JSONB fields based on type only
2. Remove hardcoded field names
3. Test with ChatWorkflow and AutomationDefinition models

Step 2: Add Field Separation to Other Interfaces

1. Add _separateObjectFields() to interfaceDbComponentObjects.py
2. Add _separateObjectFields() to interfaceDbAppObjects.py
3. Update all recordCreate() and recordModify() calls to use field separation
4. Update interfaceRbac.py to remove hardcoded field names from getRecordsetWithRBAC()

Step 3: Clean Up Connector

1. Remove hardcoded field names from _get_model_fields()
2. Remove hardcoded None defaults from _loadTable() and getRecordset()
3. Make connector purely type-based

Step 4: Testing

1. Test all CRUD operations for ChatWorkflow
2. Test all CRUD operations for AutomationDefinition
3. Test all CRUD operations for ComponentObjects models
4. Test all CRUD operations for AppObjects models
5. Verify JSONB fields are stored/loaded correctly

---

Summary of Changes Needed

File	Change Type	Description
connectorDbPostgre.py	REMOVE	Remove hardcoded field names from _get_model_fields() (lines 61-70)
connectorDbPostgre.py	REMOVE	Remove hardcoded None defaults from _loadTable() (lines 689-700)
connectorDbPostgre.py	REMOVE	Remove hardcoded None defaults from getRecordset() (lines 891-902)
interfaceDbChatObjects.py	UPDATE	Remove hardcoded fields from _separateObjectFields() line 234
interfaceDbChatObjects.py	UPDATE	Add logs, messages to relational fields list (line 226)
interfaceDbComponentObjects.py	ADD	Add _separateObjectFields() method
interfaceDbComponentObjects.py	UPDATE	Use field separation before all recordCreate()/recordModify() calls
interfaceDbAppObjects.py	ADD	Add _separateObjectFields() method
interfaceDbAppObjects.py	UPDATE	Use field separation before all recordCreate()/recordModify() calls
interfaceRbac.py	UPDATE	Remove hardcoded field names from getRecordsetWithRBAC() (lines 115-118)
interfaceRbac.py	UPDATE	Use type-based None defaults instead of hardcoded field names

---

Notes

• The connector should be completely generic and only use type information from Pydantic models
• Interfaces should handle domain-specific logic (which fields are relational vs JSONB)
• The _separateObjectFields() pattern in interfaceDbChatObjects.py is the correct approach
• Other interfaces should follow the same pattern