PowerOn/gateway
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
gateway/docs/code-documentation/services-component.md

1530 lines
46 KiB
Markdown
Raw Blame History

# Services Component Documentation
Comprehensive documentation of the Services layer in the Gateway application, explaining the architecture, patterns, and implementation details of all service modules.
> 📚 **Looking for detailed API reference?** See the [Services API Reference](./services-api-reference.md) for complete method signatures, parameters, return types, and usage examples.
## Table of Contents
1. [Overview](#overview)
2. [What is a Service?](#what-is-a-service)
3. [Service Architecture](#service-architecture)
4. [Service Initialization](#service-initialization)
5. [Core Services](#core-services)
6. [Service Usage Patterns](#service-usage-patterns)
7. [Service Lifecycle](#service-lifecycle)
8. [Best Practices](#best-practices)
---
## Overview
The **Services Layer** is a critical component of the Gateway application's business logic layer. Services provide reusable, composable functionality that can be used by features, routes, and other services. They encapsulate complex operations and act as orchestrators between the API layer and the data access layer.
```mermaid
graph TB
subgraph "Application Layers"
Routes[Routes Layer<br/>API Endpoints]
Features[Features Layer<br/>Business Logic]
Services[Services Layer<br/>Reusable Components]
Interfaces[Interfaces Layer<br/>Data Access]
Connectors[Connectors Layer<br/>External Systems]
end
Routes --> Features
Routes --> Services
Features --> Services
Services --> Interfaces
Services --> Services
Interfaces --> Connectors
style Services fill:#e8f5e9,stroke:#1b5e20,stroke-width:3px
```
### Key Characteristics
- **Reusable**: Services provide functionality that can be used across multiple features and routes
- **Composable**: Services can call other services to build complex workflows
- **Stateless**: Services operate without maintaining session state
- **User-Context Aware**: All services receive user context through the Services container
- **Interface-Dependent**: Services use interfaces to access data and external systems
---
## What is a Service?
A **Service** is a Python class that encapsulates a specific domain of functionality within the business logic layer. Services are designed to:
1. **Abstract Complexity**: Hide implementation details behind clean, well-defined APIs
2. **Promote Reusability**: Provide operations that can be used in multiple contexts
3. **Orchestrate Operations**: Coordinate between multiple interfaces and other services
4. **Process Data**: Transform, validate, and enrich data between layers
5. **Enforce Business Rules**: Implement domain-specific logic and constraints
### Service vs Feature vs Interface
```mermaid
graph LR
subgraph "Comparison"
A[Route] -->|Delegates to| B[Feature]
A -->|Can call directly| C[Service]
B -->|Uses| C
C -->|Calls| D[Interface]
C -->|Calls| E[Other Services]
D -->|Uses| F[Connector]
end
style A fill:#f3e5f5,stroke:#4a148c
style B fill:#e8f5e9,stroke:#1b5e20
style C fill:#e8f5e9,stroke:#1b5e20
style D fill:#fff3e0,stroke:#e65100
style F fill:#fce4ec,stroke:#880e4f
```
| Aspect | Feature | Service | Interface |
|--------|---------|---------|-----------|
| **Purpose** | Domain-specific business logic | Cross-cutting, reusable functionality | Data access abstraction |
| **Scope** | Single use case or domain | Multiple use cases | Database/API operations |
| **Reusability** | Low (domain-specific) | High (cross-domain) | High (data operations) |
| **Dependencies** | Can use services | Can use other services and interfaces | Can use connectors |
| **Examples** | Real Estate workflow | AI processing, Document extraction | Database queries, API calls |
---
## Service Architecture
The Services layer follows a **Service-Oriented Architecture (SOA)** pattern with a centralized service container.
### Service Container Pattern
All services are initialized and accessed through a central `Services` container that provides:
- **User Context**: Every service has access to the current user
- **Workflow Context**: Access to the current workflow state
- **Interface Access**: Direct access to all data interfaces
- **Service Composition**: Services can call other services via `self.services`
```mermaid
graph TB
subgraph "Services Container"
SC[Services Instance]
SC -->|Holds| User[User Context]
SC -->|Holds| Workflow[Workflow Context]
SC -->|Provides| IntChat[Chat Interface]
SC -->|Provides| IntApp[App Interface]
SC -->|Provides| IntComp[Component Interface]
SC -->|Initializes| SvcAI[AI Service]
SC -->|Initializes| SvcChat[Chat Service]
SC -->|Initializes| SvcExtract[Extraction Service]
SC -->|Initializes| SvcGen[Generation Service]
SC -->|Initializes| SvcNeut[Neutralization Service]
SC -->|Initializes| SvcUtils[Utils Service]
SC -->|Initializes| SvcTicket[Ticket Service]
SC -->|Initializes| SvcSP[SharePoint Service]
SvcAI -.->|Uses| SC
SvcChat -.->|Uses| SC
SvcExtract -.->|Uses| SC
end
style SC fill:#e1f5ff,stroke:#01579b,stroke-width:3px
style User fill:#fff3e0,stroke:#e65100
style Workflow fill:#fff3e0,stroke:#e65100
```
### Service Initialization Flow
```mermaid
sequenceDiagram
participant Route
participant Services Container
participant Service
participant Interface
participant Connector
Route->>Services Container: Create(user, workflow)
Services Container->>Interface: Initialize Interfaces
Interface-->>Services Container: Ready
Services Container->>Service: Initialize Service(serviceCenter)
Service->>Service: Store serviceCenter reference
Service-->>Services Container: Ready
Services Container-->>Route: Services Ready
Route->>Services Container: services.ai.someMethod()
Services Container->>Service: someMethod()
Service->>Services Container: self.services.chat.getContext()
Services Container->>Service: Context Data
Service->>Interface: Query Data
Interface->>Connector: Execute Query
Connector-->>Interface: Results
Interface-->>Service: Domain Objects
Service-->>Services Container: Result
Services Container-->>Route: Result
```
---
## Service Initialization
Services are initialized through the `Services` class in `modules/services/__init__.py`.
### Services Class Structure
```python
class Services:
def __init__(self, user: User, workflow: ChatWorkflow = None):
# Core Context
self.user: User = user
self.workflow: ChatWorkflow = workflow
self.currentUserPrompt: str = ""
self.rawUserPrompt: str = ""
# Initialize Interfaces
self.interfaceDbChat = getChatInterface(user)
self.interfaceDbApp = getAppInterface(user)
self.interfaceDbComponent = getComponentInterface(user)
# Initialize Services
self.extraction = PublicService(ExtractionService(self))
self.generation = PublicService(GenerationService(self))
self.neutralization = PublicService(NeutralizationService(self))
self.sharepoint = PublicService(SharepointService(self))
self.ai = PublicService(AiService(self), functionsOnly=False)
self.ticket = PublicService(TicketService(self))
self.chat = PublicService(ChatService(self))
self.utils = PublicService(UtilsService(self))
```
### PublicService Wrapper
The `PublicService` wrapper provides:
- **Method Access Control**: Exposes only public methods (non-underscore prefixed)
- **Attribute Protection**: Prevents direct attribute access from outside
- **Clean API Surface**: Provides a well-defined public interface
```python
class PublicService:
"""Wrapper that exposes only public methods of a service"""
def __init__(self, service_instance, functionsOnly=True):
self._service = service_instance
self._functionsOnly = functionsOnly
def __getattr__(self, name):
# Only expose public methods/attributes
if name.startswith('_'):
raise AttributeError(f"Access to private attributes is not allowed")
return getattr(self._service, name)
```
---
## Core Services
The Gateway application includes 8 core services, each handling a specific domain of functionality.
### Service Overview Diagram
```mermaid
graph TB
subgraph "Core Services"
AI[AI Service<br/>AI model operations]
Chat[Chat Service<br/>Workflow & documents]
Extract[Extraction Service<br/>Content extraction]
Gen[Generation Service<br/>Document generation]
Neut[Neutralization Service<br/>Data anonymization]
SP[SharePoint Service<br/>SharePoint integration]
Ticket[Ticket Service<br/>Ticket system integration]
Utils[Utils Service<br/>Common utilities]
end
subgraph "Service Dependencies"
AI -.->|Uses| Extract
Gen -.->|Uses| AI
Chat -.->|Uses| Utils
end
style AI fill:#bbdefb,stroke:#1976d2,stroke-width:2px
style Chat fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
style Extract fill:#fff9c4,stroke:#fbc02d,stroke-width:2px
style Gen fill:#f8bbd0,stroke:#c2185b,stroke-width:2px
style Neut fill:#d1c4e9,stroke:#512da8,stroke-width:2px
style SP fill:#ffccbc,stroke:#e64a19,stroke-width:2px
style Ticket fill:#b2dfdb,stroke:#00796b,stroke-width:2px
style Utils fill:#e0e0e0,stroke:#616161,stroke-width:2px
```
---
### 1. AI Service
**Location**: `modules/services/serviceAi/mainServiceAi.py`
**Purpose**: Orchestrates all AI model operations including text generation, document processing, image generation, web search, and web crawling.
#### Key Responsibilities
- AI model selection and routing
- Prompt building and placeholder replacement
- Iterative generation with continuation support
- Document processing with chunking
- Model-aware content handling
- Progress tracking and statistics
#### AI Service Architecture
```mermaid
graph TB
subgraph "AI Service Components"
Main[AiService Main]
Main --> Planning[Planning Call]
Main --> Documents[Documents Call]
Main --> Text[Text Call]
Documents --> ImageGen[Image Generation]
Documents --> WebOps[Web Operations]
Documents --> DocGen[Document Generation]
Documents --> TextProc[Text Processing]
Main --> Helpers[Helper Methods]
Helpers --> Placeholders[Placeholder Building]
Helpers --> Analysis[Prompt Analysis]
Helpers --> Looping[Iterative Generation]
Helpers --> Parsing[Response Parsing]
end
Main -.->|Uses| ExtractSvc[Extraction Service]
Main -.->|Uses| AiObjects[AI Objects Interface]
style Main fill:#bbdefb,stroke:#1976d2,stroke-width:3px
```
#### Core Capabilities
**Planning Operations**
- Task and action planning with optimized parameters
- Intent analysis and workflow design
- Returns structured JSON plans
**Document Operations**
- Multi-format document processing
- Document generation (PDF, DOCX, XLSX, HTML, etc.)
- Image generation
- Web search and crawling
**Text Processing**
- Content extraction and summarization
- Data analysis and insights
- Model-aware chunking for large documents
#### Special Features
1. **Iterative Generation**: Supports multi-iteration generation with continuation context
2. **Repair-Based Looping**: Automatically repairs broken JSON and continues generation
3. **Model-Aware Processing**: Automatically chunks content based on model limits
4. **Progress Tracking**: Provides granular progress updates for long operations
5. **Statistics Collection**: Tracks costs, processing time, and token usage
> 📖 **See [Services API Reference](./services-api-reference.md#ai-service-api)** for detailed method signatures, parameters, and examples.
---
### 2. Chat Service
**Location**: `modules/services/serviceChat/mainServiceChat.py`
**Purpose**: Manages workflow operations, message handling, document resolution, and chat context management.
#### Key Responsibilities
- Workflow CRUD operations
- Message and document management
- Document reference resolution
- Connection management
- Progress logging
- Workflow statistics
#### Chat Service Data Flow
```mermaid
sequenceDiagram
participant Feature
participant ChatService
participant Interface
participant Database
Feature->>ChatService: createWorkflow(data)
ChatService->>Interface: createWorkflow(data)
Interface->>Database: INSERT workflow
Database-->>Interface: Workflow object
Interface-->>ChatService: Workflow
ChatService-->>Feature: Workflow
Feature->>ChatService: storeMessageWithDocuments(workflow, message, docs)
ChatService->>Interface: createMessage(messageData)
Interface->>Database: INSERT message
Database-->>Interface: Message object
Interface->>Database: INSERT documents
Database-->>Interface: Document objects
ChatService->>ChatService: Sync in-memory workflow.messages
ChatService-->>Feature: ChatMessage with documents
Feature->>ChatService: getChatDocumentsFromDocumentList(refs)
ChatService->>ChatService: Parse document references
ChatService->>ChatService: Resolve from workflow.messages
ChatService-->>Feature: List[ChatDocument]
```
#### Document Reference System
The Chat Service implements a sophisticated document reference system with three formats:
1. **docItem**: Single document by ID - `docItem:<id>:<filename>`
2. **docList with message ID**: Documents from specific message - `docList:<messageId>:<label>`
3. **docList by label**: Documents by label (newest) - `docList:<label>`
#### Core Capabilities
**Workflow Management**
- Create, update, retrieve workflows
- Context tracking (round, task, action numbers)
- Workflow statistics and metadata
**Document Operations**
- Resolve document references to ChatDocument objects
- Store messages with attached documents
- Track document lineage through workflow
**Progress Tracking**
- Start, update, finish operation tracking
- Real-time progress reporting
- Success/failure status
**Connection Management**
- User connection references
- Token refresh and management
- Connection status tracking
> 📖 **See [Services API Reference](./services-api-reference.md#chat-service-api)** for detailed method signatures, parameters, and examples.
---
### 3. Extraction Service
**Location**: `modules/services/serviceExtraction/mainServiceExtraction.py`
**Purpose**: Extracts and processes content from various document formats (PDF, DOCX, XLSX, images, etc.) with intelligent chunking and merging strategies.
#### Key Responsibilities
- Multi-format document extraction
- Content chunking for large documents
- Intelligent content merging
- Type-specific processing (text, tables, images, structures)
- Model-aware processing with parallel execution
#### Extraction Service Architecture
```mermaid
graph TB
subgraph "Extraction Service"
Main[ExtractionService Main]
Main --> Extract[extractContent]
Main --> Merge[mergeAiResults]
Main --> Process[processDocumentsPerChunk]
Extract --> Registry[Extractor Registry]
Registry --> PDF[PDF Extractor]
Registry --> DOCX[DOCX Extractor]
Registry --> XLSX[XLSX Extractor]
Registry --> IMG[Image Extractor]
Registry --> TXT[Text Extractor]
Registry --> CSV[CSV Extractor]
Registry --> JSON[JSON Extractor]
Registry --> XML[XML Extractor]
Extract --> Chunkers[Chunker Registry]
Chunkers --> TextChunk[Text Chunker]
Chunkers --> TableChunk[Table Chunker]
Chunkers --> ImgChunk[Image Chunker]
Chunkers --> StructChunk[Structure Chunker]
end
Main -.->|Uses| AiObjects[AI Objects]
style Main fill:#fff9c4,stroke:#fbc02d,stroke-width:3px
```
#### Extraction Pipeline
```mermaid
sequenceDiagram
participant Client
participant ExtractSvc as Extraction Service
participant Extractor
participant Chunker
participant Merger
Client->>ExtractSvc: extractContent(documents, options)
loop For each document
ExtractSvc->>Extractor: Extract by MIME type
Extractor-->>ExtractSvc: ContentParts
ExtractSvc->>Chunker: Chunk if needed
Chunker-->>ExtractSvc: Chunked ContentParts
end
ExtractSvc-->>Client: List[ContentExtracted]
Client->>ExtractSvc: processDocumentsPerChunk(docs, prompt)
ExtractSvc->>ExtractSvc: Extract content
loop For each content part
ExtractSvc->>AiObjects: Process part with AI
AiObjects-->>ExtractSvc: AI result
end
ExtractSvc->>Merger: Merge part results
Merger-->>ExtractSvc: Merged content
ExtractSvc-->>Client: Final merged text
```
#### Supported Formats
| Format | Extractor | Chunking | Special Handling |
|--------|-----------|----------|------------------|
| PDF | extractorPdf.py | ✅ | Page-aware extraction |
| DOCX | extractorDocx.py | ✅ | Paragraph structure |
| XLSX | extractorXlsx.py | ✅ | Sheet-based processing |
| PPTX | extractorPptx.py | ✅ | Slide-based extraction |
| CSV | extractorCsv.py | ✅ | Row/column chunking |
| JSON | extractorJson.py | ✅ | Tree-based chunking |
| XML | extractorXml.py | ✅ | Element-based chunking |
| HTML | extractorHtml.py | ✅ | DOM-based extraction |
| Images | extractorImage.py | ✅ | OCR + metadata |
| Text | extractorText.py | ✅ | Paragraph chunking |
> 📖 **See [Services API Reference](./services-api-reference.md#extraction-service-api)** for detailed method signatures, parameters, and examples.
---
### 4. Generation Service
**Location**: `modules/services/serviceGeneration/mainServiceGeneration.py`
**Purpose**: Generates documents in various formats from structured content (HTML, PDF, DOCX, XLSX, etc.).
#### Key Responsibilities
- Document rendering in multiple formats
- Action result document processing
- File creation and storage
- Workflow context attachment
- MIME type detection and normalization
#### Generation Service Architecture
```mermaid
graph TB
subgraph "Generation Service"
Main[GenerationService Main]
Main --> Process[Process Action Results]
Main --> Render[renderReport]
Main --> Create[Create Documents]
Render --> RendererRegistry[Renderer Registry]
RendererRegistry --> HTML[HTML Renderer]
RendererRegistry --> PDF[PDF Renderer]
RendererRegistry --> DOCX[DOCX Renderer]
RendererRegistry --> XLSX[XLSX Renderer]
RendererRegistry --> CSV[CSV Renderer]
RendererRegistry --> MD[Markdown Renderer]
RendererRegistry --> JSON[JSON Renderer]
RendererRegistry --> TXT[Text Renderer]
RendererRegistry --> IMG[Image Renderer]
end
Main -.->|Uses| ComponentInterface[Component Interface]
Main -.->|Uses| ChatInterface[Chat Interface]
style Main fill:#f8bbd0,stroke:#c2185b,stroke-width:3px
```
#### Document Generation Flow
```mermaid
sequenceDiagram
participant Action
participant GenSvc as Generation Service
participant Renderer
participant CompInterface as Component Interface
participant Storage
Action->>GenSvc: processActionResultDocuments(result, action)
GenSvc->>GenSvc: Extract document data
GenSvc->>GenSvc: Detect MIME type
GenSvc-->>Action: Processed documents list
Action->>GenSvc: createDocumentsFromActionResult(result, action, workflow)
loop For each document
GenSvc->>GenSvc: Normalize filename & extension
GenSvc->>GenSvc: Convert to string content
GenSvc->>CompInterface: createFile(name, mimeType, content)
CompInterface->>Storage: Store file
Storage-->>CompInterface: File object
GenSvc->>GenSvc: Create ChatDocument
GenSvc->>GenSvc: Set workflow context
end
GenSvc-->>Action: List[ChatDocument]
Action->>GenSvc: renderReport(content, format, title)
GenSvc->>Renderer: Get renderer for format
Renderer->>Renderer: render(content, title)
Renderer-->>GenSvc: (rendered_content, mime_type)
GenSvc-->>Action: Rendered document
```
#### Supported Output Formats
| Format | Renderer | Use Case | Special Features |
|--------|----------|----------|------------------|
| HTML | rendererHtml.py | Web display | CSS styling, responsive |
| PDF | rendererPdf.py | Print-ready docs | Page layout, headers/footers |
| DOCX | rendererDocx.py | Editable documents | Styles, tables, images |
| XLSX | rendererXlsx.py | Spreadsheets | Multiple sheets, formulas |
| PPTX | rendererPptx.py | Presentations | Slides, layouts, transitions |
| CSV | rendererCsv.py | Data export | Tabular data |
| JSON | rendererJson.py | Data exchange | Structured data |
| Markdown | rendererMarkdown.py | Documentation | GitHub-flavored markdown |
| Text | rendererText.py | Plain text | Simple formatting |
| Image | rendererImage.py | Charts, diagrams | PNG, JPEG |
> 📖 **See [Services API Reference](./services-api-reference.md#generation-service-api)** for detailed method signatures, parameters, and examples.
---
### 5. Neutralization Service
**Location**: `modules/services/serviceNeutralization/mainServiceNeutralization.py`
**Purpose**: Anonymizes sensitive data in documents for GDPR compliance, replacing personal information with placeholders.
#### Key Responsibilities
- PII detection and replacement
- Multi-language support (DE, EN, FR, IT)
- Format-specific processing (text, JSON, CSV, XML)
- Attribute tracking and resolution
- Configuration management
#### Neutralization Service Architecture
```mermaid
graph TB
subgraph "Neutralization Service"
Main[NeutralizationService Main]
Main --> Config[Configuration Mgmt]
Main --> Process[Process Text/File]
Main --> Resolve[Resolve Placeholders]
Main --> Attributes[Attribute Management]
Process --> Detection[Pattern Detection]
Detection --> HeaderPat[Header Patterns]
Detection --> DataPat[Data Patterns]
Detection --> TablePat[Table Patterns]
Process --> Processors[Content Processors]
Processors --> TextProc[Text Processor]
Processors --> ListProc[List Processor]
Processors --> BinaryProc[Binary Processor]
TextProc --> PlainText[Plain Text Handler]
ListProc --> CSV[CSV Handler]
ListProc --> JSON[JSON Handler]
ListProc --> XML[XML Handler]
ListProc --> TableData[Table Data Handler]
end
Main -.->|Uses| AppInterface[App Interface]
Main -.->|Uses| ComponentInterface[Component Interface]
style Main fill:#d1c4e9,stroke:#512da8,stroke-width:3px
```
#### Neutralization Flow
```mermaid
sequenceDiagram
participant Client
participant NeutSvc as Neutralization Service
participant Processor
participant Patterns
participant AppInterface as App Interface
participant Database
Client->>NeutSvc: processText(text)
NeutSvc->>NeutSvc: _reloadNamesFromConfig()
NeutSvc->>AppInterface: getNeutralizationConfig()
AppInterface->>Database: Query config
Database-->>AppInterface: Config with names
AppInterface-->>NeutSvc: Config
NeutSvc->>NeutSvc: Auto-detect content type
alt Text Content
NeutSvc->>Processor: processTextContent(text)
Processor->>Patterns: Match PII patterns
Patterns-->>Processor: Matches
Processor->>Processor: Replace with placeholders
else JSON/CSV/XML
NeutSvc->>Processor: processJsonContent(text)
Processor->>Processor: Parse structure
Processor->>Patterns: Match in fields
Processor->>Processor: Replace in-place
end
Processor-->>NeutSvc: (data, mapping, replacements, info)
NeutSvc->>AppInterface: createNeutralizationAttributes(mapping)
AppInterface->>Database: Store attributes
NeutSvc-->>Client: NeutralizationResult
```
#### PII Detection Patterns
The service detects and neutralizes:
- **Names**: First names, last names, full names
- **Email Addresses**: Standard email formats
- **Phone Numbers**: International formats
- **Addresses**: Street addresses, postal codes
- **IDs**: SSN, passport numbers, employee IDs
- **Financial**: Credit cards, bank accounts, IBAN
- **Dates**: Birth dates, dates of service
- **Custom**: User-configured patterns
> 📖 **See [Services API Reference](./services-api-reference.md#neutralization-service-api)** for detailed method signatures, parameters, and examples.
---
### 6. SharePoint Service
**Location**: `modules/services/serviceSharepoint/mainServiceSharepoint.py`
**Purpose**: Integrates with Microsoft SharePoint for document management and synchronization.
#### Key Responsibilities
- SharePoint authentication
- Document upload/download
- Folder management
- Metadata handling
- Permission management
#### Core Capabilities
- Connect to SharePoint sites with OAuth
- Upload and download documents
- List and create folders
- Manage document metadata
- Handle permissions
> 📖 **See [Services API Reference](./services-api-reference.md#sharepoint-service-api)** for detailed method signatures, parameters, and examples.
---
### 7. Ticket Service
**Location**: `modules/services/serviceTicket/mainServiceTicket.py`
**Purpose**: Integrates with ticket management systems (Jira, ClickUp, ServiceNow, etc.).
#### Key Responsibilities
- Ticket system authentication
- Ticket CRUD operations
- Field mapping and synchronization
- Status tracking
- Comment management
#### Ticket Service Architecture
```mermaid
graph TB
subgraph "Ticket Service"
Main[TicketService Main]
Main --> Connect[connectTicket]
Connect --> TicketInterface[Ticket Interface Factory]
TicketInterface --> Jira[Jira Interface]
TicketInterface --> ClickUp[ClickUp Interface]
TicketInterface --> ServiceNow[ServiceNow Interface]
end
Main -.->|Uses| Connectors[Ticket Connectors]
style Main fill:#b2dfdb,stroke:#00796b,stroke-width:3px
```
#### Core Capabilities
- Connect to multiple ticket systems
- Create and update tickets
- Map fields between systems
- Track ticket status
- Manage comments and attachments
> 📖 **See [Services API Reference](./services-api-reference.md#ticket-service-api)** for detailed method signatures, parameters, and examples.
---
### 8. Utils Service
**Location**: `modules/services/serviceUtils/mainServiceUtils.py`
**Purpose**: Provides common utility functions used across the application.
#### Key Responsibilities
- Configuration access
- Event scheduling (cron, interval jobs)
- Timestamp utilities
- Debug logging
- Prompt sanitization
- JSON utilities
- Enum mapping
#### Core Capabilities
**Event Management**
- Schedule cron jobs (daily, weekly, custom)
- Schedule interval jobs (every N seconds/minutes/hours)
- Remove scheduled jobs
**Configuration**
- Get/set configuration values
- User-specific configurations
- Default value support
**Time Utilities**
- UTC timestamp generation
- Timezone conversion
**Debug Tools**
- Write debug files
- Log to debug file
- Store debug artifacts
**Content Sanitization**
- Sanitize user input for prompts
- Escape special characters
- Prevent injection attacks
**JSON Utilities**
- Extract JSON from text
- Parse with error recovery
- Normalize JSON text
> 📖 **See [Services API Reference](./services-api-reference.md#utils-service-api)** for detailed method signatures, parameters, and examples.
---
## Service Usage Patterns
### Pattern 1: Direct Service Call from Route
```mermaid
sequenceDiagram
participant Client
participant Route
participant Services
participant Service
participant Interface
Client->>Route: POST /api/neutralize
Route->>Services: Create(user, workflow)
Services-->>Route: services instance
Route->>Service: services.neutralization.processText(text)
Service->>Interface: Query/Update data
Interface-->>Service: Result
Service-->>Route: Neutralized text
Route-->>Client: JSON Response
```
Example:
```python
@router.post("/neutralize-text")
async def neutralizeText(
request: NeutralizeRequest,
currentUser: User = Depends(getCurrentUser)
):
# Create services container
services = Services(user=currentUser)
# Call service directly
result = services.neutralization.processText(request.text)
return {
"success": True,
"data": result
}
```
### Pattern 2: Feature Using Multiple Services
```mermaid
sequenceDiagram
participant Route
participant Feature
participant Services
participant AIService
participant ChatService
participant GenService
Route->>Feature: execute(services)
Feature->>Services: services.ai.callAiDocuments()
Services->>AIService: Process documents
AIService->>ChatService: Get document context
ChatService-->>AIService: Context
AIService-->>Feature: AI result
Feature->>Services: services.generation.renderReport()
Services->>GenService: Render to PDF
GenService-->>Feature: PDF document
Feature->>Services: services.chat.storeMessageWithDocuments()
Services->>ChatService: Store result
ChatService-->>Feature: Message stored
Feature-->>Route: Feature result
```
Example:
```python
class RealEstateFeature:
async def execute(self, services: Services):
# Use AI service
analysis = await services.ai.callAiDocuments(
prompt=self.prompt,
documents=self.documents,
options=self.options
)
# Use generation service
report, mime = await services.generation.renderReport(
extractedContent=analysis,
outputFormat="pdf",
title="Real Estate Analysis"
)
# Use chat service
message = services.chat.storeMessageWithDocuments(
workflow=services.workflow,
messageData=message_data,
documents=[report_document]
)
return message
```
### Pattern 3: Service Composition
```mermaid
sequenceDiagram
participant Feature
participant AIService
participant ExtractService
participant ChatService
participant Interface
Feature->>AIService: callAiText(documents, prompt)
AIService->>ExtractService: processDocumentsPerChunk(docs, prompt)
ExtractService->>ExtractService: extractContent(documents)
ExtractService->>Interface: Get file data
Interface-->>ExtractService: Raw bytes
ExtractService->>ExtractService: Extract & chunk
loop For each chunk
ExtractService->>AIService: call(request)
AIService-->>ExtractService: AI result
end
ExtractService->>ExtractService: Merge results
ExtractService-->>AIService: Merged text
AIService->>ChatService: storeWorkflowStat()
ChatService-->>AIService: Stat stored
AIService-->>Feature: Final result
```
Example:
```python
class AiService:
async def callAiText(self, prompt, documents, options):
# AI service uses extraction service internally
result = await self.extractionService.processDocumentsPerChunk(
documents=documents,
prompt=prompt,
aiObjects=self.aiObjects,
options=options
)
# Also uses chat service for statistics
self.services.chat.storeWorkflowStat(
self.services.workflow,
response,
"ai.text_call"
)
return result
```
### Pattern 4: Progress Tracking
```mermaid
sequenceDiagram
participant Feature
participant AIService
participant ChatService
participant ProgressLogger
participant Database
Feature->>AIService: callAiDocuments(prompt, docs)
AIService->>ChatService: progressLogStart(opId, "AI Call", "Processing")
ChatService->>ProgressLogger: startOperation(opId, details)
ProgressLogger->>Database: Store progress log
AIService->>AIService: Process step 1
AIService->>ChatService: progressLogUpdate(opId, 0.3, "Extracting")
ChatService->>ProgressLogger: updateOperation(opId, 0.3)
ProgressLogger->>Database: Update progress
AIService->>AIService: Process step 2
AIService->>ChatService: progressLogUpdate(opId, 0.6, "Generating")
ChatService->>ProgressLogger: updateOperation(opId, 0.6)
AIService->>AIService: Complete
AIService->>ChatService: progressLogFinish(opId, True)
ChatService->>ProgressLogger: finishOperation(opId, True)
ProgressLogger->>Database: Mark complete
AIService-->>Feature: Result
```
Example:
```python
# In AI Service
async def callAiDocuments(self, prompt, documents, ...):
# Create operation ID
operationId = f"ai_docs_{workflow_id}_{timestamp}"
# Start progress tracking
self.services.chat.progressLogStart(
operationId,
"AI Service",
"Document Processing",
f"Processing {len(documents)} documents"
)
try:
# Step 1
self.services.chat.progressLogUpdate(
operationId, 0.2, "Extracting content"
)
extracted = await self.extract(documents)
# Step 2
self.services.chat.progressLogUpdate(
operationId, 0.5, "Calling AI model"
)
result = await self.callAI(extracted)
# Step 3
self.services.chat.progressLogUpdate(
operationId, 0.9, "Finalizing result"
)
final = self.finalize(result)
# Complete
self.services.chat.progressLogFinish(operationId, True)
return final
except Exception as e:
self.services.chat.progressLogFinish(operationId, False)
raise
```
---
## Service Lifecycle
### Lifecycle Stages
```mermaid
stateDiagram-v2
[*] --> Created: Services(user, workflow)
Created --> Initialized: Interface setup
Initialized --> Ready: Services ready
Ready --> InUse: Method calls
InUse --> Ready: Method completes
InUse --> Error: Exception
Error --> Ready: Error handled
Ready --> Disposed: Request complete
Disposed --> [*]
note right of Created
Services container created
User context set
Workflow context set
end note
note right of Initialized
All interfaces connected
All services instantiated
Dependencies resolved
end note
note right of InUse
Service methods executing
Can call other services
Progress tracked
end note
```
### Request Lifecycle Example
```mermaid
sequenceDiagram
participant Client
participant FastAPI
participant Route
participant Services
participant Service
Client->>FastAPI: HTTP Request
FastAPI->>FastAPI: Authentication
FastAPI->>Route: route_handler(user=currentUser)
Note over Route: Request Start
Route->>Services: Create Services(user)
Services->>Services: Initialize interfaces
Services->>Services: Initialize services
Services-->>Route: services instance
Note over Route,Service: Service Usage
Route->>Service: services.ai.callAiDocuments()
Service->>Service: Process
Service->>Service: self.services.chat.progressLog()
Service-->>Route: Result
Note over Route: Request End
Route->>Route: Format response
Route-->>FastAPI: JSON response
FastAPI->>FastAPI: services destroyed
FastAPI-->>Client: HTTP Response
```
### Service Initialization Order
1. **User Context**: User object retrieved from authentication
2. **Workflow Context** (optional): Workflow loaded if ID provided
3. **Interfaces**: Database interfaces initialized with user context
4. **Services**: All services instantiated with Services container reference
5. **Dependencies**: Services can now call other services
### Service Destruction
Services are automatically destroyed at the end of each request:
- No explicit cleanup required
- Database connections returned to pool
- Memory released
- No persistent state maintained
---
## Best Practices
### 1. Service Design Principles
#### Keep Services Stateless
```python
# ❌ BAD: Storing state in service
class BadService:
def __init__(self, services):
self.services = services
self.cached_results = {} # State!
def process(self, data):
if data.id in self.cached_results:
return self.cached_results[data.id]
# ...
# ✅ GOOD: Stateless service
class GoodService:
def __init__(self, services):
self.services = services
def process(self, data):
# Use database or pass state explicitly
cached = self.services.interfaceDbApp.getCache(data.id)
if cached:
return cached
# ...
```
#### Single Responsibility
```python
# ❌ BAD: Service doing too much
class BadService:
def processDocuments(self, docs):
# Extract content
# Call AI
# Generate report
# Send email
# Update database
pass
# ✅ GOOD: Focused services
class ExtractionService:
def extractContent(self, docs):
# Only extraction logic
pass
class GenerationService:
def renderReport(self, content, format):
# Only generation logic
pass
class EmailService:
def sendReport(self, report, recipient):
# Only email logic
pass
```
#### Clear Interface Boundaries
```python
# ✅ GOOD: Well-defined public API
class GoodService:
# Public API
def processDocument(self, doc: ChatDocument) -> str:
"""Process a document and return result."""
# Validate input
self._validateDocument(doc)
# Process
content = self._extractContent(doc)
result = self._transformContent(content)
return result
# Private helpers
def _validateDocument(self, doc):
# Internal validation
pass
def _extractContent(self, doc):
# Internal extraction
pass
def _transformContent(self, content):
# Internal transformation
pass
```
### 2. Error Handling
```python
import logging
logger = logging.getLogger(__name__)
class MyService:
def processData(self, data):
try:
# Process data
result = self._process(data)
return result
except ValidationError as e:
# Expected errors - log and re-raise
logger.warning(f"Validation error: {str(e)}")
raise
except Exception as e:
# Unexpected errors - log with details and re-raise
logger.error(f"Unexpected error in processData: {str(e)}",
exc_info=True)
raise ServiceException("Failed to process data") from e
```
### 3. Service Composition
```python
class CompositeService:
def __init__(self, services):
self.services = services
async def complexOperation(self, input_data):
# Use multiple services in sequence
# Step 1: Extract
extracted = await self.services.extraction.extractContent(
input_data.documents
)
# Step 2: Process with AI
processed = await self.services.ai.callAiText(
prompt=input_data.prompt,
documents=extracted
)
# Step 3: Generate output
result = await self.services.generation.renderReport(
content=processed,
format=input_data.format
)
# Step 4: Store result
message = self.services.chat.storeMessageWithDocuments(
workflow=self.services.workflow,
messageData={"content": "Result"},
documents=[result]
)
return message
```
### 4. Progress Tracking Best Practices
```python
class ProgressAwareService:
async def longRunningOperation(self, data, operation_id=None):
# Create operation ID if not provided
if not operation_id:
operation_id = f"operation_{uuid.uuid4()}"
# Always use try-finally for progress tracking
try:
# Start
self.services.chat.progressLogStart(
operation_id,
"My Service",
"Long Operation",
f"Processing {len(data)} items"
)
# Process with granular updates
total = len(data)
for i, item in enumerate(data):
# Process item
self._processItem(item)
# Update progress (0.0 to 1.0)
progress = (i + 1) / total
self.services.chat.progressLogUpdate(
operation_id,
progress,
f"Processed {i + 1}/{total} items"
)
# Success
self.services.chat.progressLogFinish(operation_id, True)
return results
except Exception as e:
# Failure
self.services.chat.progressLogFinish(operation_id, False)
raise
```
### 5. Testing Services
```python
import pytest
from unittest.mock import Mock, MagicMock
from modules.services import Services
from modules.datamodels.datamodelUam import User
@pytest.fixture
def mock_services():
"""Create a mock services container for testing"""
user = User(id="test-user", email="test@example.com")
services = Services(user=user)
# Mock interfaces
services.interfaceDbChat = MagicMock()
services.interfaceDbApp = MagicMock()
services.interfaceDbComponent = MagicMock()
return services
@pytest.mark.asyncio
async def test_service_method(mock_services):
"""Test a service method"""
# Setup
mock_services.interfaceDbChat.getWorkflow.return_value = Mock(
id="workflow-123",
messages=[]
)
# Execute
result = await mock_services.ai.callAiPlanning(
prompt="Test prompt"
)
# Assert
assert result is not None
mock_services.interfaceDbChat.getWorkflow.assert_called_once()
```
### 6. Documentation Standards
Every service method should include:
```python
def serviceMethod(
self,
required_param: str,
optional_param: Optional[int] = None
) -> Dict[str, Any]:
"""
Short one-line description of what the method does.
Longer description providing more details about the method's
purpose, behavior, and any important considerations.
Args:
required_param: Description of required parameter
optional_param: Description of optional parameter (default: None)
Returns:
Dict containing:
- field1: Description of field1
- field2: Description of field2
Raises:
ValueError: When required_param is invalid
ServiceException: When operation fails
Example:
```python
result = services.myservice.serviceMethod(
required_param="value",
optional_param=42
)
```
"""
# Implementation
pass
```
---
## Service Interaction Patterns
### Complete Request Flow with Multiple Services
```mermaid
sequenceDiagram
participant Client
participant Route
participant Services
participant AIService
participant ExtractionService
participant ChatService
participant GenerationService
participant Interface
participant Database
Client->>Route: POST /api/analyze-documents
Route->>Services: Create Services(user, workflow)
Services-->>Route: services instance
Route->>AIService: services.ai.callAiDocuments(prompt, docs, format="pdf")
Note over AIService,ExtractionService: Document Processing Phase
AIService->>ChatService: progressLogStart(opId, "AI Service", "Processing")
AIService->>ExtractionService: processDocumentsPerChunk(docs, prompt)
ExtractionService->>Interface: getFileData(fileId)
Interface->>Database: SELECT file_data
Database-->>Interface: Raw bytes
Interface-->>ExtractionService: File data
ExtractionService->>ExtractionService: Extract & chunk content
ExtractionService->>ChatService: storeWorkflowStat(extraction stats)
loop For each chunk
ExtractionService->>AIService: call(chunk, prompt)
AIService-->>ExtractionService: AI result
AIService->>ChatService: storeWorkflowStat(ai stats)
end
ExtractionService->>ExtractionService: Merge results
ExtractionService-->>AIService: Merged content
Note over AIService,GenerationService: Generation Phase
AIService->>ChatService: progressLogUpdate(opId, 0.7, "Generating")
AIService->>GenerationService: renderReport(content, "pdf", title)
GenerationService->>GenerationService: Render to PDF
GenerationService->>Interface: createFile(name, mimeType, content)
Interface->>Database: INSERT file
Database-->>Interface: File object
Interface-->>GenerationService: File stored
GenerationService-->>AIService: (pdf_content, mime_type)
Note over AIService,ChatService: Storage Phase
AIService->>ChatService: progressLogUpdate(opId, 0.9, "Storing")
AIService->>ChatService: storeMessageWithDocuments(workflow, msg, [doc])
ChatService->>Interface: createMessage(messageData)
Interface->>Database: INSERT message
Database-->>Interface: Message object
ChatService->>ChatService: Sync workflow.messages
ChatService-->>AIService: Message stored
AIService->>ChatService: progressLogFinish(opId, True)
AIService-->>Route: Final result with PDF
Route->>Route: Format response
Route-->>Client: JSON with PDF document
```
---
## Conclusion
The Services layer is the backbone of the Gateway application's business logic, providing:
- **Modularity**: Each service handles a specific domain
- **Reusability**: Services can be used across features and routes
- **Composability**: Services build on each other for complex workflows
- **Maintainability**: Clear boundaries and responsibilities
- **Testability**: Well-defined interfaces and dependencies
By following the patterns and best practices outlined in this documentation, developers can effectively use and extend the services layer to build robust, scalable features.
---
## Additional Resources
- [Services API Reference](./services-api-reference.md) - Complete API documentation with method signatures and examples
- [Architecture Overview](./architecture-overview.md) - High-level system architecture
- [Security Component](./security-component.md) - Security and authentication
- [Real Estate Feature Guide](../real-estate-feature-integration-guide/README.md) - Example feature implementation
- [Data Models Documentation](../../modules/datamodels/README.md) - Data model reference
---
**Document Version**: 1.0
**Last Updated**: 2025-01-25
**Status**: Complete
Reference in a new issue View git blame Copy permalink