239 lines
10 KiB
Markdown
239 lines
10 KiB
Markdown
# Stats Refactory Concept and Implementation Plan
|
|
|
|
Date: 2025-10-16
|
|
Owner: Workflow/Data Layer
|
|
|
|
## Goals
|
|
- Clean, consistent statistics (ChatStat) model and lifecycle.
|
|
- Workflow is the authoritative container for stats (similar to logs).
|
|
- Simple, predictable DB APIs for reading full stats history per workflow.
|
|
- Lightweight instrumentation points at key producers (AI calls, extraction, generation) to emit stats.
|
|
|
|
---
|
|
|
|
## Datamodel Changes (modules/datamodels/datamodelChat.py)
|
|
|
|
- ChatStat (all fields optional)
|
|
- Keep: id, workflowId, processingTime, bytesSent, bytesReceived, errorCount
|
|
- Remove: successRate, tokenCount, tokenPriceUnit, tokenPriceAmount
|
|
- Add: process: str
|
|
- Example values: "action.outlook.readMails", "ai.process.document.name"
|
|
- Add: engine: str
|
|
- Example values: "ai.anthropic.35", "ai.tavily.basic", "renderer.docx"
|
|
- Add: priceCHF: float (calculated price in USD for the operation)
|
|
|
|
- ChatWorkflow
|
|
- stats: change from Optional[ChatStat] to List[ChatStat] (default [])
|
|
|
|
- ChatMessage
|
|
- Remove stats field entirely to avoid coupling to message lifecycles.
|
|
- Remove messageId from ChatStat (no longer needed)
|
|
|
|
Rationale:
|
|
- Stats are workflow-scope artifacts for billing/analytics. They should not depend on transient message presence.
|
|
- Attaching stats to workflow aligns with logs; UI polling returns consolidated workflow data including stats list.
|
|
|
|
---
|
|
|
|
## Interface/API Changes (modules/interfaces/interfaceDbChatObjects.py)
|
|
|
|
- Replace getWorkflowStats(workflowId) returning single latest with:
|
|
- getWorkflowStatsList(workflowId) -> List[ChatStat]
|
|
- Add createWorkflowStat(statData: Dict[str, Any]) -> ChatStat (write-only per record)
|
|
- Keep updateWorkflowStats(...) only if absolutely needed; prefer append-only stats records for auditability
|
|
- Update unified polling (getUnifiedChatData) to include full current stats list for workflow for each poll
|
|
|
|
DB logic:
|
|
- Store each stat emission as one row (append-only). No modify unless explicitly required.
|
|
- Deletion remains cascading on workflow delete.
|
|
|
|
---
|
|
|
|
## Service Layer Changes (modules/services/serviceWorkflow/mainServiceWorkflow.py)
|
|
|
|
- Replace storeWorkflowStat(workflow, statData) to:
|
|
- Coerce statData to ChatStat fields (process, engine, priceCHF, bytesSent, bytesReceived, processingTime, errorCount) --> to handover pydantic model (with id empty), not the single values
|
|
- Set id and workflowId
|
|
- Persist via interfaceDbChat.createWorkflowStat
|
|
- Append to workflow.stats in memory
|
|
- Return ChatStat
|
|
|
|
- Remove storeMessageStat (no message-level stats in new model)
|
|
|
|
---
|
|
|
|
## Instrumentation Points (where to emit stats)
|
|
|
|
1) AI Calls (modules/interfaces/interfaceAiObjects.py)
|
|
- Each AI call measures processing time, calculates priceCHF, and tracks bytes sent/received
|
|
- Returns standardized AiCallResponse with priceCHF, processingTime, bytesSent, bytesReceived, errorCount included (to adapt pydantic model)
|
|
- All AI functions (call, callImage, generateImage, webQuery, etc.) return this standardized response
|
|
- Model-specific pricing functions calculate priceCHF based on (processingTime, bytesSent, bytesReceived)
|
|
|
|
2) Service Layer (modules/services/serviceWorkflow/mainServiceWorkflow.py)
|
|
- Receives AiCallResponse from AI interface
|
|
- Creates ChatStat objects with data from AiCallResponse
|
|
- Emits stats to database via interfaceDbChat.createWorkflowStat
|
|
|
|
3) Extraction/Generation Services
|
|
- services/serviceExtraction/mainServiceExtraction.py and related extractors
|
|
- services/serviceGeneration/* (document rendering/generation)
|
|
- Emit ChatStat per completed operation with process, engine, processingTime, bytes, priceCHF, etc.
|
|
|
|
4) Workflow-level summaries (optional)
|
|
- At workflow completion, compute a summary stat (aggregated bytes/costs) if desired
|
|
|
|
---
|
|
|
|
## AiCallResponse Changes (modules/datamodels/datamodelAi.py)
|
|
|
|
Update AiCallResponse to include standardized stats fields:
|
|
|
|
- Remove: usedTokens, costEstimate
|
|
- Add: priceCHF: float (calculated price in USD)
|
|
- Add: processingTime: float (duration in seconds)
|
|
- Add: bytesSent: int (input data size in bytes)
|
|
- Add: bytesReceived: int (output data size in bytes)
|
|
- Add: errorCount: int (0 for success, 1+ for errors)
|
|
|
|
## Model-Specific Pricing Functions
|
|
|
|
Each AI model should implement a pricing calculation function that takes the operation parameters and returns the cost in USD:
|
|
|
|
- Function signature: `_calculatepriceCHF(processingTime: float, bytesSent: int, bytesReceived: int) -> float`
|
|
- Parameters:
|
|
- `processingTime`: Duration of the operation in seconds
|
|
- `bytesSent`: Size of input data in bytes
|
|
- `bytesReceived`: Size of output data in bytes
|
|
- Returns: Price in USD as float
|
|
|
|
Implementation approach:
|
|
- Each model class (e.g., OpenAI, Anthropic, etc.) implements its own pricing logic
|
|
- Complex pricing models can handle different rates for input/output, time-based costs, etc.
|
|
- Models can access their specific pricing configuration (rates, tiers, etc.)
|
|
- No fallback! each model MUST have its pricing calculation
|
|
|
|
Example implementations:
|
|
- OpenAI: Token-based pricing with different rates for input/output tokens
|
|
- Anthropic: Similar token-based pricing with model-specific rates
|
|
- Tavily: Simple per-request pricing
|
|
- Custom models: Any complex pricing logic (tiered, time-based, etc.)
|
|
|
|
---
|
|
|
|
## Routing/Polling (UI)
|
|
|
|
- Update route (unified polling) to include stats list for workflow:
|
|
- interfaceDbChatObjects.getUnifiedChatData(workflowId): include
|
|
- { type: "stat", createdAt: stat_timestamp, item: ChatStat(dict) } for all stats
|
|
|
|
- UI will receive a complete and growing list of stats with each poll; no special latest logic required.
|
|
|
|
---
|
|
|
|
## Step-by-Step Implementation Plan
|
|
|
|
### Phase 1: Core Data Model Changes
|
|
1. **Update ChatStat model** (modules/datamodels/datamodelChat.py)
|
|
- Remove: successRate, tokenCount, tokenPriceUnit, tokenPriceAmount, messageId
|
|
- Add: process: str, engine: str, priceCHF: float
|
|
- Keep: id, workflowId, processingTime, bytesSent, bytesReceived, errorCount
|
|
|
|
2. **Update ChatWorkflow model** (modules/datamodels/datamodelChat.py)
|
|
- Change stats from Optional[ChatStat] to List[ChatStat] (default [])
|
|
|
|
3. **Update ChatMessage model** (modules/datamodels/datamodelChat.py)
|
|
- Remove stats field entirely
|
|
- Remove messageId from ChatStat (no longer needed)
|
|
|
|
4. **Update AiCallResponse model** (modules/datamodels/datamodelAi.py)
|
|
- Remove: usedTokens, costEstimate
|
|
- Add: priceCHF: float, processingTime: float, bytesSent: int, bytesReceived: int, errorCount: int
|
|
|
|
### Phase 2: Database Interface Changes
|
|
5. **Update interfaceDbChatObjects.py**
|
|
- Implement createWorkflowStat(workflow: ChatWorkflow, statData: Dict[str, Any]) -> ChatStat (the function creates a new ChatStat object, persists it to the database, and includes it into the workflows stats list)
|
|
- Implement getWorkflowStatsList(workflowId) -> List[ChatStat] (the function returns the list of stats for a given workflow)
|
|
- Update getUnifiedChatData to include full stats list
|
|
- Remove getWorkflowStats (single latest) - replace with getWorkflowStatsList
|
|
|
|
### Phase 3: AI Interface Layer Changes
|
|
6. **Add model-specific pricing functions** (modules/interfaces/interfaceAiObjects.py)
|
|
- Add _calculatepriceCHF() method to each model in aiModels registry
|
|
- Implement pricing logic for each connector type (OpenAI, Anthropic, Perplexity, Tavily)
|
|
|
|
7. **Update all AI call methods** (modules/interfaces/interfaceAiObjects.py)
|
|
- Add timing measurement (start/end timestamps)
|
|
- Calculate bytesSent (input data size)
|
|
- Calculate bytesReceived (output data size)
|
|
- Call model-specific _calculatepriceCHF()
|
|
- Return standardized AiCallResponse with all stats fields
|
|
|
|
8. **Update specific AI methods:**
|
|
- call() - text generation
|
|
- callImage() - image analysis
|
|
- generateImage() - image generation
|
|
- webQuery() - web research
|
|
- All web methods (search_websites, crawl_websites, etc.)
|
|
|
|
### Phase 4: Service Layer Changes
|
|
9. **Update mainServiceWorkflow.py**
|
|
- Implement storeWorkflowStat(workflow: ChatWorkflow, aiResponse: AiCallResponse, process: str) -> ChatStat
|
|
- Remove storeMessageStat (no message-level stats)
|
|
- Update storeWorkflowStat to use new ChatStat fields
|
|
|
|
10. **Update workflow service methods**
|
|
- Replace old stats handling with new storeWorkflowStat calls
|
|
- Ensure all AI calls go through storeWorkflowStat
|
|
|
|
### Phase 5: Extraction/Generation Services
|
|
11. **Update serviceExtraction services**
|
|
- Add stats emission for extraction operations
|
|
- Calculate processingTime, bytesSent, bytesReceived
|
|
- Emit ChatStat via storeWorkflowStat
|
|
|
|
12. **Update serviceGeneration services**
|
|
- Add stats emission for generation operations
|
|
- Calculate processingTime, bytesSent, bytesReceived
|
|
- Emit ChatStat via storeWorkflowStat
|
|
|
|
### Phase 6: Integration and Testing
|
|
13. **Update all AI call sites**
|
|
- Replace old stats handling with new AiCallResponse usage
|
|
- Ensure all AI calls emit stats via storeWorkflowStat
|
|
|
|
14. **Update routing/polling**
|
|
- Ensure getUnifiedChatData includes full stats list
|
|
- Update UI polling to handle stats array
|
|
|
|
15. **Add comprehensive logging**
|
|
- Debug logs for each stat emission (process, engine, bytes, priceCHF)
|
|
- Error handling for stats emission failures
|
|
|
|
### Phase 7: Cleanup and Validation
|
|
16. **Remove deprecated code**
|
|
- Remove old getWorkflowStats calls
|
|
- Remove old updateWorkflowStats calls
|
|
- Remove token-based pricing logic
|
|
|
|
17. **Add validation and tests**
|
|
- Test stats emission for all AI operations
|
|
- Validate pricing calculations
|
|
- Test database operations
|
|
|
|
18. **Performance optimization**
|
|
- Ensure stats emission doesn't impact AI call performance
|
|
- Optimize database queries for stats retrieval
|
|
|
|
---
|
|
|
|
## Smart Implementation Notes
|
|
|
|
- **Centralized measurement**: AI interface layer handles all timing and byte counting, ensuring consistency.
|
|
- **Standardized responses**: All AI calls return AiCallResponse with complete stats data.
|
|
- **Model-specific pricing**: Each model implements `_calculatepriceCHF()` with its own pricing strategy.
|
|
- **Service layer simplicity**: Service layer just creates ChatStat objects from AiCallResponse data.
|
|
- **Append-only stats**: Simplifies auditing and invoicing; no implicit overwrites.
|
|
- **Workflow-scoped stats**: All stats attached to workflow, not messages; UI gets complete history.
|
|
- **Performance consideration**: Stats emission should not impact AI call performance.
|
|
- **Error handling**: Graceful degradation if stats emission fails; AI calls should still succeed.
|