265 lines
9.5 KiB
Markdown
265 lines
9.5 KiB
Markdown
# Connector Generic Refactoring Analysis
|
|
|
|
## Problem
|
|
The `connectorDbPostgre.py` has **hardcoded field names** that make it non-generic:
|
|
- Lines 61-70: Hardcoded field names in `_get_model_fields()` to force JSONB type
|
|
- Lines 689-700: Hardcoded field names in `_loadTable()` for None value defaults
|
|
- Lines 891-902: Hardcoded field names in `getRecordset()` for None value defaults
|
|
|
|
## Hardcoded Fields in Connector
|
|
|
|
### Location 1: `_get_model_fields()` (Lines 61-70)
|
|
```python
|
|
or field_name in [
|
|
"execParameters",
|
|
"expectedDocumentFormats",
|
|
"resultDocuments",
|
|
"logs",
|
|
"messages",
|
|
"stats",
|
|
"tasks",
|
|
]
|
|
```
|
|
|
|
### Location 2: `_loadTable()` (Lines 689-700)
|
|
```python
|
|
if field_name in [
|
|
"logs",
|
|
"messages",
|
|
"tasks",
|
|
"expectedDocumentFormats",
|
|
"resultDocuments",
|
|
]:
|
|
record[field_name] = []
|
|
elif field_name in ["execParameters", "stats"]:
|
|
record[field_name] = {}
|
|
```
|
|
|
|
### Location 3: `getRecordset()` (Lines 891-902)
|
|
```python
|
|
if field_name in [
|
|
"logs",
|
|
"messages",
|
|
"tasks",
|
|
"expectedDocumentFormats",
|
|
"resultDocuments",
|
|
]:
|
|
record[field_name] = []
|
|
elif field_name in ["execParameters", "stats"]:
|
|
record[field_name] = {}
|
|
```
|
|
|
|
---
|
|
|
|
## Field Usage Analysis
|
|
|
|
### Fields Used in Models:
|
|
|
|
1. **`logs`, `messages`, `stats`, `tasks`** → `ChatWorkflow` model (`datamodelChat.py`)
|
|
- Used by: `interfaceDbChatObjects.py`
|
|
- Type: `List[...]` (should be JSONB)
|
|
|
|
2. **`execParameters`, `expectedDocumentFormats`** → `ActionItem` model (`datamodelChat.py`)
|
|
- Used by: `interfaceDbChatObjects.py` (via `AutomationDefinition`)
|
|
- Type: `Dict[str, Any]` and `Optional[List[Dict[str, str]]]` (should be JSONB)
|
|
|
|
3. **`resultDocuments`** → Not found in current models (may be legacy or unused)
|
|
|
|
---
|
|
|
|
## Current Interface Handling
|
|
|
|
### ✅ `interfaceDbChatObjects.py` - HAS Field Separation Logic
|
|
- **Has:** `_separateObjectFields()` method (lines 210-254)
|
|
- **Handles:** Separates `documents`, `stats` to `objectFields` (lines 226-228)
|
|
- **Still has hardcoded fields:** Lines 234: `execParameters`, `expectedDocumentFormats`, `resultDocuments`
|
|
- **Status:** ✅ Partially handles field separation, but still relies on connector for JSONB detection
|
|
|
|
### ❌ `interfaceDbComponentObjects.py` - NO Field Separation Logic
|
|
- **Has:** Direct calls to `db.recordCreate()`, `db.recordModify()`
|
|
- **Handles:** None - relies entirely on connector's hardcoded logic
|
|
- **Status:** ❌ Needs field separation logic
|
|
|
|
### ❌ `interfaceDbAppObjects.py` - NO Field Separation Logic
|
|
- **Has:** Direct calls to `db.recordCreate()`, `db.recordModify()`
|
|
- **Handles:** None - relies entirely on connector's hardcoded logic
|
|
- **Status:** ❌ Needs field separation logic
|
|
|
|
### ⚠️ `interfaceRbac.py` - HAS Hardcoded Field Names
|
|
- **Has:** Helper function `getRecordsetWithRBAC()` (lines 113-120)
|
|
- **Handles:** Hardcoded field names for None defaults (same as connector)
|
|
- **Status:** ⚠️ Also needs to be made generic
|
|
|
|
---
|
|
|
|
## Required Changes
|
|
|
|
### 1. **connectorDbPostgre.py** - REMOVE Hardcoded Logic
|
|
|
|
#### Change 1: `_get_model_fields()` (Lines 43-87)
|
|
**REMOVE:**
|
|
```python
|
|
or field_name in [
|
|
"execParameters",
|
|
"expectedDocumentFormats",
|
|
"resultDocuments",
|
|
"logs",
|
|
"messages",
|
|
"stats",
|
|
"tasks",
|
|
]
|
|
```
|
|
|
|
**REPLACE WITH:** Pure type-based detection only
|
|
```python
|
|
# Check for JSONB fields (Dict, List, or complex types)
|
|
if (
|
|
field_type == dict
|
|
or field_type == list
|
|
or (
|
|
hasattr(field_type, "__origin__")
|
|
and field_type.__origin__ in (dict, list)
|
|
)
|
|
):
|
|
fields[field_name] = "JSONB"
|
|
```
|
|
|
|
#### Change 2: `_loadTable()` (Lines 689-700)
|
|
**REMOVE:** Hardcoded field name checks for None defaults
|
|
**REPLACE WITH:** Generic type-based handling or remove entirely (let interfaces handle)
|
|
|
|
#### Change 3: `getRecordset()` (Lines 891-902)
|
|
**REMOVE:** Hardcoded field name checks for None defaults
|
|
**REPLACE WITH:** Generic type-based handling or remove entirely (let interfaces handle)
|
|
|
|
---
|
|
|
|
### 2. **interfaceDbChatObjects.py** - ENHANCE Field Separation
|
|
|
|
#### Change: `_separateObjectFields()` (Lines 210-254)
|
|
**CURRENT:** Lines 226-228 handle `documents`, `stats` as object fields
|
|
**CURRENT:** Lines 234 still has hardcoded `execParameters`, `expectedDocumentFormats`, `resultDocuments`
|
|
|
|
**UPDATE TO:**
|
|
- Remove hardcoded field names from line 234
|
|
- Rely purely on type detection (Dict, List)
|
|
- Handle relational fields (`documents`, `stats`, `logs`, `messages`) as object fields (not stored in main table)
|
|
- Handle JSONB fields (`execParameters`, `expectedDocumentFormats`) as simple fields (stored as JSONB)
|
|
|
|
**Logic should be:**
|
|
```python
|
|
# Relational fields (stored in separate normalized tables)
|
|
if fieldName in ['documents', 'stats', 'logs', 'messages']:
|
|
objectFields[fieldName] = value
|
|
continue
|
|
|
|
# JSONB fields (stored as JSONB in main table)
|
|
if (fieldType == dict or fieldType == list or
|
|
(hasattr(fieldType, '__origin__') and fieldType.__origin__ in (dict, list))):
|
|
simpleFields[fieldName] = value # Store as JSONB
|
|
```
|
|
|
|
---
|
|
|
|
### 3. **interfaceDbComponentObjects.py** - ADD Field Separation
|
|
|
|
#### Add: `_separateObjectFields()` method
|
|
**LOCATION:** After `_initializeDatabase()` method (around line 140)
|
|
|
|
**IMPLEMENTATION:** Similar to `interfaceDbChatObjects.py` but adapted for ComponentObjects models
|
|
- Check which models used by this interface have relational/JSONB fields
|
|
- Implement type-based field separation
|
|
- Use before calling `db.recordCreate()` and `db.recordModify()`
|
|
|
|
**Models used:** `Prompt`, `FileItem`, `FileData`, `VoiceSettings`, `MessagingSubscription`, etc.
|
|
- Check if any have relational fields that need separate handling
|
|
|
|
---
|
|
|
|
### 4. **interfaceDbAppObjects.py** - ADD Field Separation
|
|
|
|
#### Add: `_separateObjectFields()` method
|
|
**LOCATION:** After `_initializeDatabase()` method (around line 135)
|
|
|
|
**IMPLEMENTATION:** Similar to `interfaceDbChatObjects.py` but adapted for AppObjects models
|
|
- Check which models used by this interface have relational/JSONB fields
|
|
- Implement type-based field separation
|
|
- Use before calling `db.recordCreate()` and `db.recordModify()`
|
|
|
|
**Models used:** `User`, `Mandate`, `UserInDB`, `Token`, `AuthEvent`, `UserConnection`, etc.
|
|
- Check if any have relational fields that need separate handling
|
|
|
|
---
|
|
|
|
## Impact Analysis
|
|
|
|
### Models Affected:
|
|
|
|
1. **ChatWorkflow** (`datamodelChat.py`)
|
|
- Fields: `logs`, `messages`, `stats`, `tasks` (List types)
|
|
- Currently: Hardcoded in connector
|
|
- After: Handled by `interfaceDbChatObjects._separateObjectFields()`
|
|
|
|
2. **ActionItem** (`datamodelChat.py`)
|
|
- Fields: `execParameters` (Dict), `expectedDocumentFormats` (List)
|
|
- Currently: Hardcoded in connector
|
|
- After: Handled by `interfaceDbChatObjects._separateObjectFields()`
|
|
|
|
3. **AutomationDefinition** (`datamodelChat.py`)
|
|
- Contains `ActionItem` objects
|
|
- Currently: Hardcoded in connector
|
|
- After: Handled by `interfaceDbChatObjects._separateObjectFields()`
|
|
|
|
---
|
|
|
|
## Migration Steps
|
|
|
|
### Step 1: Update `interfaceDbChatObjects.py`
|
|
1. Enhance `_separateObjectFields()` to handle all JSONB fields based on type only
|
|
2. Remove hardcoded field names
|
|
3. Test with ChatWorkflow and AutomationDefinition models
|
|
|
|
### Step 2: Add Field Separation to Other Interfaces
|
|
1. Add `_separateObjectFields()` to `interfaceDbComponentObjects.py`
|
|
2. Add `_separateObjectFields()` to `interfaceDbAppObjects.py`
|
|
3. Update all `recordCreate()` and `recordModify()` calls to use field separation
|
|
4. Update `interfaceRbac.py` to remove hardcoded field names from `getRecordsetWithRBAC()`
|
|
|
|
### Step 3: Clean Up Connector
|
|
1. Remove hardcoded field names from `_get_model_fields()`
|
|
2. Remove hardcoded None defaults from `_loadTable()` and `getRecordset()`
|
|
3. Make connector purely type-based
|
|
|
|
### Step 4: Testing
|
|
1. Test all CRUD operations for ChatWorkflow
|
|
2. Test all CRUD operations for AutomationDefinition
|
|
3. Test all CRUD operations for ComponentObjects models
|
|
4. Test all CRUD operations for AppObjects models
|
|
5. Verify JSONB fields are stored/loaded correctly
|
|
|
|
---
|
|
|
|
## Summary of Changes Needed
|
|
|
|
| File | Change Type | Description |
|
|
|------|-------------|-------------|
|
|
| `connectorDbPostgre.py` | **REMOVE** | Remove hardcoded field names from `_get_model_fields()` (lines 61-70) |
|
|
| `connectorDbPostgre.py` | **REMOVE** | Remove hardcoded None defaults from `_loadTable()` (lines 689-700) |
|
|
| `connectorDbPostgre.py` | **REMOVE** | Remove hardcoded None defaults from `getRecordset()` (lines 891-902) |
|
|
| `interfaceDbChatObjects.py` | **UPDATE** | Remove hardcoded fields from `_separateObjectFields()` line 234 |
|
|
| `interfaceDbChatObjects.py` | **UPDATE** | Add `logs`, `messages` to relational fields list (line 226) |
|
|
| `interfaceDbComponentObjects.py` | **ADD** | Add `_separateObjectFields()` method |
|
|
| `interfaceDbComponentObjects.py` | **UPDATE** | Use field separation before all `recordCreate()`/`recordModify()` calls |
|
|
| `interfaceDbAppObjects.py` | **ADD** | Add `_separateObjectFields()` method |
|
|
| `interfaceDbAppObjects.py` | **UPDATE** | Use field separation before all `recordCreate()`/`recordModify()` calls |
|
|
| `interfaceRbac.py` | **UPDATE** | Remove hardcoded field names from `getRecordsetWithRBAC()` (lines 115-118) |
|
|
| `interfaceRbac.py` | **UPDATE** | Use type-based None defaults instead of hardcoded field names |
|
|
|
|
---
|
|
|
|
## Notes
|
|
|
|
- The connector should be **completely generic** and only use type information from Pydantic models
|
|
- Interfaces should handle domain-specific logic (which fields are relational vs JSONB)
|
|
- The `_separateObjectFields()` pattern in `interfaceDbChatObjects.py` is the correct approach
|
|
- Other interfaces should follow the same pattern
|