Ida Dittrich f12ac49d32 finished pagination feature with filter and search options

2025-11-27 13:18:07 +01:00

46 KiB

Raw Blame History

Services Component Documentation

Comprehensive documentation of the Services layer in the Gateway application, explaining the architecture, patterns, and implementation details of all service modules.

📚 Looking for detailed API reference? See the Services API Reference for complete method signatures, parameters, return types, and usage examples.

Overview
What is a Service?
Service Architecture
Service Initialization
Core Services
Service Usage Patterns
Service Lifecycle
Best Practices

Overview

The Services Layer is a critical component of the Gateway application's business logic layer. Services provide reusable, composable functionality that can be used by features, routes, and other services. They encapsulate complex operations and act as orchestrators between the API layer and the data access layer.

graph TB
    subgraph "Application Layers"
        Routes[Routes Layer<br/>API Endpoints]
        Features[Features Layer<br/>Business Logic]
        Services[Services Layer<br/>Reusable Components]
        Interfaces[Interfaces Layer<br/>Data Access]
        Connectors[Connectors Layer<br/>External Systems]
    end
    
    Routes --> Features
    Routes --> Services
    Features --> Services
    Services --> Interfaces
    Services --> Services
    Interfaces --> Connectors
    
    style Services fill:#e8f5e9,stroke:#1b5e20,stroke-width:3px

Key Characteristics

Reusable: Services provide functionality that can be used across multiple features and routes
Composable: Services can call other services to build complex workflows
Stateless: Services operate without maintaining session state
User-Context Aware: All services receive user context through the Services container
Interface-Dependent: Services use interfaces to access data and external systems

What is a Service?

A Service is a Python class that encapsulates a specific domain of functionality within the business logic layer. Services are designed to:

Abstract Complexity: Hide implementation details behind clean, well-defined APIs
Promote Reusability: Provide operations that can be used in multiple contexts
Orchestrate Operations: Coordinate between multiple interfaces and other services
Process Data: Transform, validate, and enrich data between layers
Enforce Business Rules: Implement domain-specific logic and constraints

Service vs Feature vs Interface

graph LR
    subgraph "Comparison"
        A[Route] -->|Delegates to| B[Feature]
        A -->|Can call directly| C[Service]
        B -->|Uses| C
        C -->|Calls| D[Interface]
        C -->|Calls| E[Other Services]
        D -->|Uses| F[Connector]
    end
    
    style A fill:#f3e5f5,stroke:#4a148c
    style B fill:#e8f5e9,stroke:#1b5e20
    style C fill:#e8f5e9,stroke:#1b5e20
    style D fill:#fff3e0,stroke:#e65100
    style F fill:#fce4ec,stroke:#880e4f

Aspect	Feature	Service	Interface
Purpose	Domain-specific business logic	Cross-cutting, reusable functionality	Data access abstraction
Scope	Single use case or domain	Multiple use cases	Database/API operations
Reusability	Low (domain-specific)	High (cross-domain)	High (data operations)
Dependencies	Can use services	Can use other services and interfaces	Can use connectors
Examples	Real Estate workflow	AI processing, Document extraction	Database queries, API calls

Service Architecture

The Services layer follows a Service-Oriented Architecture (SOA) pattern with a centralized service container.

Service Container Pattern

All services are initialized and accessed through a central Services container that provides:

User Context: Every service has access to the current user
Workflow Context: Access to the current workflow state
Interface Access: Direct access to all data interfaces
Service Composition: Services can call other services via self.services

graph TB
    subgraph "Services Container"
        SC[Services Instance]
        
        SC -->|Holds| User[User Context]
        SC -->|Holds| Workflow[Workflow Context]
        SC -->|Provides| IntChat[Chat Interface]
        SC -->|Provides| IntApp[App Interface]
        SC -->|Provides| IntComp[Component Interface]
        
        SC -->|Initializes| SvcAI[AI Service]
        SC -->|Initializes| SvcChat[Chat Service]
        SC -->|Initializes| SvcExtract[Extraction Service]
        SC -->|Initializes| SvcGen[Generation Service]
        SC -->|Initializes| SvcNeut[Neutralization Service]
        SC -->|Initializes| SvcUtils[Utils Service]
        SC -->|Initializes| SvcTicket[Ticket Service]
        SC -->|Initializes| SvcSP[SharePoint Service]
        
        SvcAI -.->|Uses| SC
        SvcChat -.->|Uses| SC
        SvcExtract -.->|Uses| SC
    end
    
    style SC fill:#e1f5ff,stroke:#01579b,stroke-width:3px
    style User fill:#fff3e0,stroke:#e65100
    style Workflow fill:#fff3e0,stroke:#e65100

Service Initialization Flow

sequenceDiagram
    participant Route
    participant Services Container
    participant Service
    participant Interface
    participant Connector
    
    Route->>Services Container: Create(user, workflow)
    Services Container->>Interface: Initialize Interfaces
    Interface-->>Services Container: Ready
    Services Container->>Service: Initialize Service(serviceCenter)
    Service->>Service: Store serviceCenter reference
    Service-->>Services Container: Ready
    Services Container-->>Route: Services Ready
    
    Route->>Services Container: services.ai.someMethod()
    Services Container->>Service: someMethod()
    Service->>Services Container: self.services.chat.getContext()
    Services Container->>Service: Context Data
    Service->>Interface: Query Data
    Interface->>Connector: Execute Query
    Connector-->>Interface: Results
    Interface-->>Service: Domain Objects
    Service-->>Services Container: Result
    Services Container-->>Route: Result

Service Initialization

Services are initialized through the Services class in modules/services/__init__.py.

Services Class Structure

class Services:
    def __init__(self, user: User, workflow: ChatWorkflow = None):
        # Core Context
        self.user: User = user
        self.workflow: ChatWorkflow = workflow
        self.currentUserPrompt: str = ""
        self.rawUserPrompt: str = ""
        
        # Initialize Interfaces
        self.interfaceDbChat = getChatInterface(user)
        self.interfaceDbApp = getAppInterface(user)
        self.interfaceDbComponent = getComponentInterface(user)
        
        # Initialize Services
        self.extraction = PublicService(ExtractionService(self))
        self.generation = PublicService(GenerationService(self))
        self.neutralization = PublicService(NeutralizationService(self))
        self.sharepoint = PublicService(SharepointService(self))
        self.ai = PublicService(AiService(self), functionsOnly=False)
        self.ticket = PublicService(TicketService(self))
        self.chat = PublicService(ChatService(self))
        self.utils = PublicService(UtilsService(self))

PublicService Wrapper

The PublicService wrapper provides:

Method Access Control: Exposes only public methods (non-underscore prefixed)
Attribute Protection: Prevents direct attribute access from outside
Clean API Surface: Provides a well-defined public interface

class PublicService:
    """Wrapper that exposes only public methods of a service"""
    
    def __init__(self, service_instance, functionsOnly=True):
        self._service = service_instance
        self._functionsOnly = functionsOnly
    
    def __getattr__(self, name):
        # Only expose public methods/attributes
        if name.startswith('_'):
            raise AttributeError(f"Access to private attributes is not allowed")
        return getattr(self._service, name)

Core Services

The Gateway application includes 8 core services, each handling a specific domain of functionality.

Service Overview Diagram

graph TB
    subgraph "Core Services"
        AI[AI Service<br/>AI model operations]
        Chat[Chat Service<br/>Workflow & documents]
        Extract[Extraction Service<br/>Content extraction]
        Gen[Generation Service<br/>Document generation]
        Neut[Neutralization Service<br/>Data anonymization]
        SP[SharePoint Service<br/>SharePoint integration]
        Ticket[Ticket Service<br/>Ticket system integration]
        Utils[Utils Service<br/>Common utilities]
    end
    
    subgraph "Service Dependencies"
        AI -.->|Uses| Extract
        Gen -.->|Uses| AI
        Chat -.->|Uses| Utils
    end
    
    style AI fill:#bbdefb,stroke:#1976d2,stroke-width:2px
    style Chat fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
    style Extract fill:#fff9c4,stroke:#fbc02d,stroke-width:2px
    style Gen fill:#f8bbd0,stroke:#c2185b,stroke-width:2px
    style Neut fill:#d1c4e9,stroke:#512da8,stroke-width:2px
    style SP fill:#ffccbc,stroke:#e64a19,stroke-width:2px
    style Ticket fill:#b2dfdb,stroke:#00796b,stroke-width:2px
    style Utils fill:#e0e0e0,stroke:#616161,stroke-width:2px

1. AI Service

Location: modules/services/serviceAi/mainServiceAi.py

Purpose: Orchestrates all AI model operations including text generation, document processing, image generation, web search, and web crawling.

Key Responsibilities

AI model selection and routing
Prompt building and placeholder replacement
Iterative generation with continuation support
Document processing with chunking
Model-aware content handling
Progress tracking and statistics

AI Service Architecture

graph TB
    subgraph "AI Service Components"
        Main[AiService Main]
        Main --> Planning[Planning Call]
        Main --> Documents[Documents Call]
        Main --> Text[Text Call]
        
        Documents --> ImageGen[Image Generation]
        Documents --> WebOps[Web Operations]
        Documents --> DocGen[Document Generation]
        Documents --> TextProc[Text Processing]
        
        Main --> Helpers[Helper Methods]
        Helpers --> Placeholders[Placeholder Building]
        Helpers --> Analysis[Prompt Analysis]
        Helpers --> Looping[Iterative Generation]
        Helpers --> Parsing[Response Parsing]
    end
    
    Main -.->|Uses| ExtractSvc[Extraction Service]
    Main -.->|Uses| AiObjects[AI Objects Interface]
    
    style Main fill:#bbdefb,stroke:#1976d2,stroke-width:3px

Core Capabilities

Planning Operations

Task and action planning with optimized parameters
Intent analysis and workflow design
Returns structured JSON plans

Document Operations

Multi-format document processing
Document generation (PDF, DOCX, XLSX, HTML, etc.)
Image generation
Web search and crawling

Text Processing

Content extraction and summarization
Data analysis and insights
Model-aware chunking for large documents

Special Features

Iterative Generation: Supports multi-iteration generation with continuation context
Repair-Based Looping: Automatically repairs broken JSON and continues generation
Model-Aware Processing: Automatically chunks content based on model limits
Progress Tracking: Provides granular progress updates for long operations
Statistics Collection: Tracks costs, processing time, and token usage

📖 See Services API Reference for detailed method signatures, parameters, and examples.

2. Chat Service

Location: modules/services/serviceChat/mainServiceChat.py

Purpose: Manages workflow operations, message handling, document resolution, and chat context management.

Key Responsibilities

Workflow CRUD operations
Message and document management
Document reference resolution
Connection management
Progress logging
Workflow statistics

Chat Service Data Flow

sequenceDiagram
    participant Feature
    participant ChatService
    participant Interface
    participant Database
    
    Feature->>ChatService: createWorkflow(data)
    ChatService->>Interface: createWorkflow(data)
    Interface->>Database: INSERT workflow
    Database-->>Interface: Workflow object
    Interface-->>ChatService: Workflow
    ChatService-->>Feature: Workflow
    
    Feature->>ChatService: storeMessageWithDocuments(workflow, message, docs)
    ChatService->>Interface: createMessage(messageData)
    Interface->>Database: INSERT message
    Database-->>Interface: Message object
    Interface->>Database: INSERT documents
    Database-->>Interface: Document objects
    ChatService->>ChatService: Sync in-memory workflow.messages
    ChatService-->>Feature: ChatMessage with documents
    
    Feature->>ChatService: getChatDocumentsFromDocumentList(refs)
    ChatService->>ChatService: Parse document references
    ChatService->>ChatService: Resolve from workflow.messages
    ChatService-->>Feature: List[ChatDocument]

Document Reference System

The Chat Service implements a sophisticated document reference system with three formats:

docItem: Single document by ID - docItem:<id>:<filename>
docList with message ID: Documents from specific message - docList:<messageId>:<label>
docList by label: Documents by label (newest) - docList:<label>

Core Capabilities

Workflow Management

Create, update, retrieve workflows
Context tracking (round, task, action numbers)
Workflow statistics and metadata

Document Operations

Resolve document references to ChatDocument objects
Store messages with attached documents
Track document lineage through workflow

Progress Tracking

Start, update, finish operation tracking
Real-time progress reporting
Success/failure status

Connection Management

User connection references
Token refresh and management
Connection status tracking

📖 See Services API Reference for detailed method signatures, parameters, and examples.

3. Extraction Service

Location: modules/services/serviceExtraction/mainServiceExtraction.py

Purpose: Extracts and processes content from various document formats (PDF, DOCX, XLSX, images, etc.) with intelligent chunking and merging strategies.

Key Responsibilities

Multi-format document extraction
Content chunking for large documents
Intelligent content merging
Type-specific processing (text, tables, images, structures)
Model-aware processing with parallel execution

Extraction Service Architecture

graph TB
    subgraph "Extraction Service"
        Main[ExtractionService Main]
        Main --> Extract[extractContent]
        Main --> Merge[mergeAiResults]
        Main --> Process[processDocumentsPerChunk]
        
        Extract --> Registry[Extractor Registry]
        Registry --> PDF[PDF Extractor]
        Registry --> DOCX[DOCX Extractor]
        Registry --> XLSX[XLSX Extractor]
        Registry --> IMG[Image Extractor]
        Registry --> TXT[Text Extractor]
        Registry --> CSV[CSV Extractor]
        Registry --> JSON[JSON Extractor]
        Registry --> XML[XML Extractor]
        
        Extract --> Chunkers[Chunker Registry]
        Chunkers --> TextChunk[Text Chunker]
        Chunkers --> TableChunk[Table Chunker]
        Chunkers --> ImgChunk[Image Chunker]
        Chunkers --> StructChunk[Structure Chunker]
    end
    
    Main -.->|Uses| AiObjects[AI Objects]
    
    style Main fill:#fff9c4,stroke:#fbc02d,stroke-width:3px

Extraction Pipeline

sequenceDiagram
    participant Client
    participant ExtractSvc as Extraction Service
    participant Extractor
    participant Chunker
    participant Merger
    
    Client->>ExtractSvc: extractContent(documents, options)
    
    loop For each document
        ExtractSvc->>Extractor: Extract by MIME type
        Extractor-->>ExtractSvc: ContentParts
        
        ExtractSvc->>Chunker: Chunk if needed
        Chunker-->>ExtractSvc: Chunked ContentParts
    end
    
    ExtractSvc-->>Client: List[ContentExtracted]
    
    Client->>ExtractSvc: processDocumentsPerChunk(docs, prompt)
    ExtractSvc->>ExtractSvc: Extract content
    
    loop For each content part
        ExtractSvc->>AiObjects: Process part with AI
        AiObjects-->>ExtractSvc: AI result
    end
    
    ExtractSvc->>Merger: Merge part results
    Merger-->>ExtractSvc: Merged content
    ExtractSvc-->>Client: Final merged text

Supported Formats

Format	Extractor	Chunking	Special Handling
PDF	extractorPdf.py	✅	Page-aware extraction
DOCX	extractorDocx.py	✅	Paragraph structure
XLSX	extractorXlsx.py	✅	Sheet-based processing
PPTX	extractorPptx.py	✅	Slide-based extraction
CSV	extractorCsv.py	✅	Row/column chunking
JSON	extractorJson.py	✅	Tree-based chunking
XML	extractorXml.py	✅	Element-based chunking
HTML	extractorHtml.py	✅	DOM-based extraction
Images	extractorImage.py	✅	OCR + metadata
Text	extractorText.py	✅	Paragraph chunking

📖 See Services API Reference for detailed method signatures, parameters, and examples.

4. Generation Service

Location: modules/services/serviceGeneration/mainServiceGeneration.py

Purpose: Generates documents in various formats from structured content (HTML, PDF, DOCX, XLSX, etc.).

Key Responsibilities

Document rendering in multiple formats
Action result document processing
File creation and storage
Workflow context attachment
MIME type detection and normalization

Generation Service Architecture

graph TB
    subgraph "Generation Service"
        Main[GenerationService Main]
        Main --> Process[Process Action Results]
        Main --> Render[renderReport]
        Main --> Create[Create Documents]
        
        Render --> RendererRegistry[Renderer Registry]
        RendererRegistry --> HTML[HTML Renderer]
        RendererRegistry --> PDF[PDF Renderer]
        RendererRegistry --> DOCX[DOCX Renderer]
        RendererRegistry --> XLSX[XLSX Renderer]
        RendererRegistry --> CSV[CSV Renderer]
        RendererRegistry --> MD[Markdown Renderer]
        RendererRegistry --> JSON[JSON Renderer]
        RendererRegistry --> TXT[Text Renderer]
        RendererRegistry --> IMG[Image Renderer]
    end
    
    Main -.->|Uses| ComponentInterface[Component Interface]
    Main -.->|Uses| ChatInterface[Chat Interface]
    
    style Main fill:#f8bbd0,stroke:#c2185b,stroke-width:3px

Document Generation Flow

sequenceDiagram
    participant Action
    participant GenSvc as Generation Service
    participant Renderer
    participant CompInterface as Component Interface
    participant Storage
    
    Action->>GenSvc: processActionResultDocuments(result, action)
    GenSvc->>GenSvc: Extract document data
    GenSvc->>GenSvc: Detect MIME type
    GenSvc-->>Action: Processed documents list
    
    Action->>GenSvc: createDocumentsFromActionResult(result, action, workflow)
    
    loop For each document
        GenSvc->>GenSvc: Normalize filename & extension
        GenSvc->>GenSvc: Convert to string content
        GenSvc->>CompInterface: createFile(name, mimeType, content)
        CompInterface->>Storage: Store file
        Storage-->>CompInterface: File object
        GenSvc->>GenSvc: Create ChatDocument
        GenSvc->>GenSvc: Set workflow context
    end
    
    GenSvc-->>Action: List[ChatDocument]
    
    Action->>GenSvc: renderReport(content, format, title)
    GenSvc->>Renderer: Get renderer for format
    Renderer->>Renderer: render(content, title)
    Renderer-->>GenSvc: (rendered_content, mime_type)
    GenSvc-->>Action: Rendered document

Supported Output Formats

Format	Renderer	Use Case	Special Features
HTML	rendererHtml.py	Web display	CSS styling, responsive
PDF	rendererPdf.py	Print-ready docs	Page layout, headers/footers
DOCX	rendererDocx.py	Editable documents	Styles, tables, images
XLSX	rendererXlsx.py	Spreadsheets	Multiple sheets, formulas
PPTX	rendererPptx.py	Presentations	Slides, layouts, transitions
CSV	rendererCsv.py	Data export	Tabular data
JSON	rendererJson.py	Data exchange	Structured data
Markdown	rendererMarkdown.py	Documentation	GitHub-flavored markdown
Text	rendererText.py	Plain text	Simple formatting
Image	rendererImage.py	Charts, diagrams	PNG, JPEG

📖 See Services API Reference for detailed method signatures, parameters, and examples.

5. Neutralization Service

Location: modules/services/serviceNeutralization/mainServiceNeutralization.py

Purpose: Anonymizes sensitive data in documents for GDPR compliance, replacing personal information with placeholders.

Key Responsibilities

PII detection and replacement
Multi-language support (DE, EN, FR, IT)
Format-specific processing (text, JSON, CSV, XML)
Attribute tracking and resolution
Configuration management

Neutralization Service Architecture

graph TB
    subgraph "Neutralization Service"
        Main[NeutralizationService Main]
        Main --> Config[Configuration Mgmt]
        Main --> Process[Process Text/File]
        Main --> Resolve[Resolve Placeholders]
        Main --> Attributes[Attribute Management]
        
        Process --> Detection[Pattern Detection]
        Detection --> HeaderPat[Header Patterns]
        Detection --> DataPat[Data Patterns]
        Detection --> TablePat[Table Patterns]
        
        Process --> Processors[Content Processors]
        Processors --> TextProc[Text Processor]
        Processors --> ListProc[List Processor]
        Processors --> BinaryProc[Binary Processor]
        
        TextProc --> PlainText[Plain Text Handler]
        ListProc --> CSV[CSV Handler]
        ListProc --> JSON[JSON Handler]
        ListProc --> XML[XML Handler]
        ListProc --> TableData[Table Data Handler]
    end
    
    Main -.->|Uses| AppInterface[App Interface]
    Main -.->|Uses| ComponentInterface[Component Interface]
    
    style Main fill:#d1c4e9,stroke:#512da8,stroke-width:3px

Neutralization Flow

sequenceDiagram
    participant Client
    participant NeutSvc as Neutralization Service
    participant Processor
    participant Patterns
    participant AppInterface as App Interface
    participant Database
    
    Client->>NeutSvc: processText(text)
    NeutSvc->>NeutSvc: _reloadNamesFromConfig()
    NeutSvc->>AppInterface: getNeutralizationConfig()
    AppInterface->>Database: Query config
    Database-->>AppInterface: Config with names
    AppInterface-->>NeutSvc: Config
    
    NeutSvc->>NeutSvc: Auto-detect content type
    
    alt Text Content
        NeutSvc->>Processor: processTextContent(text)
        Processor->>Patterns: Match PII patterns
        Patterns-->>Processor: Matches
        Processor->>Processor: Replace with placeholders
    else JSON/CSV/XML
        NeutSvc->>Processor: processJsonContent(text)
        Processor->>Processor: Parse structure
        Processor->>Patterns: Match in fields
        Processor->>Processor: Replace in-place
    end
    
    Processor-->>NeutSvc: (data, mapping, replacements, info)
    
    NeutSvc->>AppInterface: createNeutralizationAttributes(mapping)
    AppInterface->>Database: Store attributes
    
    NeutSvc-->>Client: NeutralizationResult

PII Detection Patterns

The service detects and neutralizes:

Names: First names, last names, full names
Email Addresses: Standard email formats
Phone Numbers: International formats
Addresses: Street addresses, postal codes
IDs: SSN, passport numbers, employee IDs
Financial: Credit cards, bank accounts, IBAN
Dates: Birth dates, dates of service
Custom: User-configured patterns

📖 See Services API Reference for detailed method signatures, parameters, and examples.

6. SharePoint Service

Location: modules/services/serviceSharepoint/mainServiceSharepoint.py

Purpose: Integrates with Microsoft SharePoint for document management and synchronization.

Key Responsibilities

SharePoint authentication
Document upload/download
Folder management
Metadata handling
Permission management

Core Capabilities

Connect to SharePoint sites with OAuth
Upload and download documents
List and create folders
Manage document metadata
Handle permissions

📖 See Services API Reference for detailed method signatures, parameters, and examples.

7. Ticket Service

Location: modules/services/serviceTicket/mainServiceTicket.py

Purpose: Integrates with ticket management systems (Jira, ClickUp, ServiceNow, etc.).

Key Responsibilities

Ticket system authentication
Ticket CRUD operations
Field mapping and synchronization
Status tracking
Comment management

Ticket Service Architecture

graph TB
    subgraph "Ticket Service"
        Main[TicketService Main]
        Main --> Connect[connectTicket]
        
        Connect --> TicketInterface[Ticket Interface Factory]
        TicketInterface --> Jira[Jira Interface]
        TicketInterface --> ClickUp[ClickUp Interface]
        TicketInterface --> ServiceNow[ServiceNow Interface]
    end
    
    Main -.->|Uses| Connectors[Ticket Connectors]
    
    style Main fill:#b2dfdb,stroke:#00796b,stroke-width:3px

Core Capabilities

Connect to multiple ticket systems
Create and update tickets
Map fields between systems
Track ticket status
Manage comments and attachments

📖 See Services API Reference for detailed method signatures, parameters, and examples.

8. Utils Service

Location: modules/services/serviceUtils/mainServiceUtils.py

Purpose: Provides common utility functions used across the application.

Key Responsibilities

Configuration access
Event scheduling (cron, interval jobs)
Timestamp utilities
Debug logging
Prompt sanitization
JSON utilities
Enum mapping

Core Capabilities

Event Management

Schedule cron jobs (daily, weekly, custom)
Schedule interval jobs (every N seconds/minutes/hours)
Remove scheduled jobs

Configuration

Get/set configuration values
User-specific configurations
Default value support

Time Utilities

UTC timestamp generation
Timezone conversion

Debug Tools

Write debug files
Log to debug file
Store debug artifacts

Content Sanitization

Sanitize user input for prompts
Escape special characters
Prevent injection attacks

JSON Utilities

Extract JSON from text
Parse with error recovery
Normalize JSON text

📖 See Services API Reference for detailed method signatures, parameters, and examples.

Service Usage Patterns

Pattern 1: Direct Service Call from Route

sequenceDiagram
    participant Client
    participant Route
    participant Services
    participant Service
    participant Interface
    
    Client->>Route: POST /api/neutralize
    Route->>Services: Create(user, workflow)
    Services-->>Route: services instance
    Route->>Service: services.neutralization.processText(text)
    Service->>Interface: Query/Update data
    Interface-->>Service: Result
    Service-->>Route: Neutralized text
    Route-->>Client: JSON Response

Example:

@router.post("/neutralize-text")
async def neutralizeText(
    request: NeutralizeRequest,
    currentUser: User = Depends(getCurrentUser)
):
    # Create services container
    services = Services(user=currentUser)
    
    # Call service directly
    result = services.neutralization.processText(request.text)
    
    return {
        "success": True,
        "data": result
    }

Pattern 2: Feature Using Multiple Services

sequenceDiagram
    participant Route
    participant Feature
    participant Services
    participant AIService
    participant ChatService
    participant GenService
    
    Route->>Feature: execute(services)
    Feature->>Services: services.ai.callAiDocuments()
    Services->>AIService: Process documents
    AIService->>ChatService: Get document context
    ChatService-->>AIService: Context
    AIService-->>Feature: AI result
    
    Feature->>Services: services.generation.renderReport()
    Services->>GenService: Render to PDF
    GenService-->>Feature: PDF document
    
    Feature->>Services: services.chat.storeMessageWithDocuments()
    Services->>ChatService: Store result
    ChatService-->>Feature: Message stored
    
    Feature-->>Route: Feature result

Example:

class RealEstateFeature:
    async def execute(self, services: Services):
        # Use AI service
        analysis = await services.ai.callAiDocuments(
            prompt=self.prompt,
            documents=self.documents,
            options=self.options
        )
        
        # Use generation service
        report, mime = await services.generation.renderReport(
            extractedContent=analysis,
            outputFormat="pdf",
            title="Real Estate Analysis"
        )
        
        # Use chat service
        message = services.chat.storeMessageWithDocuments(
            workflow=services.workflow,
            messageData=message_data,
            documents=[report_document]
        )
        
        return message

Pattern 3: Service Composition

sequenceDiagram
    participant Feature
    participant AIService
    participant ExtractService
    participant ChatService
    participant Interface
    
    Feature->>AIService: callAiText(documents, prompt)
    AIService->>ExtractService: processDocumentsPerChunk(docs, prompt)
    ExtractService->>ExtractService: extractContent(documents)
    ExtractService->>Interface: Get file data
    Interface-->>ExtractService: Raw bytes
    ExtractService->>ExtractService: Extract & chunk
    
    loop For each chunk
        ExtractService->>AIService: call(request)
        AIService-->>ExtractService: AI result
    end
    
    ExtractService->>ExtractService: Merge results
    ExtractService-->>AIService: Merged text
    
    AIService->>ChatService: storeWorkflowStat()
    ChatService-->>AIService: Stat stored
    
    AIService-->>Feature: Final result

Example:

class AiService:
    async def callAiText(self, prompt, documents, options):
        # AI service uses extraction service internally
        result = await self.extractionService.processDocumentsPerChunk(
            documents=documents,
            prompt=prompt,
            aiObjects=self.aiObjects,
            options=options
        )
        
        # Also uses chat service for statistics
        self.services.chat.storeWorkflowStat(
            self.services.workflow,
            response,
            "ai.text_call"
        )
        
        return result

Pattern 4: Progress Tracking

sequenceDiagram
    participant Feature
    participant AIService
    participant ChatService
    participant ProgressLogger
    participant Database
    
    Feature->>AIService: callAiDocuments(prompt, docs)
    AIService->>ChatService: progressLogStart(opId, "AI Call", "Processing")
    ChatService->>ProgressLogger: startOperation(opId, details)
    ProgressLogger->>Database: Store progress log
    
    AIService->>AIService: Process step 1
    AIService->>ChatService: progressLogUpdate(opId, 0.3, "Extracting")
    ChatService->>ProgressLogger: updateOperation(opId, 0.3)
    ProgressLogger->>Database: Update progress
    
    AIService->>AIService: Process step 2
    AIService->>ChatService: progressLogUpdate(opId, 0.6, "Generating")
    ChatService->>ProgressLogger: updateOperation(opId, 0.6)
    
    AIService->>AIService: Complete
    AIService->>ChatService: progressLogFinish(opId, True)
    ChatService->>ProgressLogger: finishOperation(opId, True)
    ProgressLogger->>Database: Mark complete
    
    AIService-->>Feature: Result

Example:

# In AI Service
async def callAiDocuments(self, prompt, documents, ...):
    # Create operation ID
    operationId = f"ai_docs_{workflow_id}_{timestamp}"
    
    # Start progress tracking
    self.services.chat.progressLogStart(
        operationId,
        "AI Service",
        "Document Processing",
        f"Processing {len(documents)} documents"
    )
    
    try:
        # Step 1
        self.services.chat.progressLogUpdate(
            operationId, 0.2, "Extracting content"
        )
        extracted = await self.extract(documents)
        
        # Step 2
        self.services.chat.progressLogUpdate(
            operationId, 0.5, "Calling AI model"
        )
        result = await self.callAI(extracted)
        
        # Step 3
        self.services.chat.progressLogUpdate(
            operationId, 0.9, "Finalizing result"
        )
        final = self.finalize(result)
        
        # Complete
        self.services.chat.progressLogFinish(operationId, True)
        return final
        
    except Exception as e:
        self.services.chat.progressLogFinish(operationId, False)
        raise

Service Lifecycle

Lifecycle Stages

stateDiagram-v2
    [*] --> Created: Services(user, workflow)
    Created --> Initialized: Interface setup
    Initialized --> Ready: Services ready
    Ready --> InUse: Method calls
    InUse --> Ready: Method completes
    InUse --> Error: Exception
    Error --> Ready: Error handled
    Ready --> Disposed: Request complete
    Disposed --> [*]
    
    note right of Created
        Services container created
        User context set
        Workflow context set
    end note
    
    note right of Initialized
        All interfaces connected
        All services instantiated
        Dependencies resolved
    end note
    
    note right of InUse
        Service methods executing
        Can call other services
        Progress tracked
    end note

Request Lifecycle Example

sequenceDiagram
    participant Client
    participant FastAPI
    participant Route
    participant Services
    participant Service
    
    Client->>FastAPI: HTTP Request
    FastAPI->>FastAPI: Authentication
    FastAPI->>Route: route_handler(user=currentUser)
    
    Note over Route: Request Start
    Route->>Services: Create Services(user)
    Services->>Services: Initialize interfaces
    Services->>Services: Initialize services
    Services-->>Route: services instance
    
    Note over Route,Service: Service Usage
    Route->>Service: services.ai.callAiDocuments()
    Service->>Service: Process
    Service->>Service: self.services.chat.progressLog()
    Service-->>Route: Result
    
    Note over Route: Request End
    Route->>Route: Format response
    Route-->>FastAPI: JSON response
    FastAPI->>FastAPI: services destroyed
    FastAPI-->>Client: HTTP Response

Service Initialization Order

User Context: User object retrieved from authentication
Workflow Context (optional): Workflow loaded if ID provided
Interfaces: Database interfaces initialized with user context
Services: All services instantiated with Services container reference
Dependencies: Services can now call other services

Service Destruction

Services are automatically destroyed at the end of each request:

No explicit cleanup required
Database connections returned to pool
Memory released
No persistent state maintained

Best Practices

1. Service Design Principles

Keep Services Stateless

# ❌ BAD: Storing state in service
class BadService:
    def __init__(self, services):
        self.services = services
        self.cached_results = {}  # State!
        
    def process(self, data):
        if data.id in self.cached_results:
            return self.cached_results[data.id]
        # ...

# ✅ GOOD: Stateless service
class GoodService:
    def __init__(self, services):
        self.services = services
        
    def process(self, data):
        # Use database or pass state explicitly
        cached = self.services.interfaceDbApp.getCache(data.id)
        if cached:
            return cached
        # ...

Single Responsibility

# ❌ BAD: Service doing too much
class BadService:
    def processDocuments(self, docs):
        # Extract content
        # Call AI
        # Generate report
        # Send email
        # Update database
        pass

# ✅ GOOD: Focused services
class ExtractionService:
    def extractContent(self, docs):
        # Only extraction logic
        pass

class GenerationService:
    def renderReport(self, content, format):
        # Only generation logic
        pass

class EmailService:
    def sendReport(self, report, recipient):
        # Only email logic
        pass

Clear Interface Boundaries

# ✅ GOOD: Well-defined public API
class GoodService:
    # Public API
    def processDocument(self, doc: ChatDocument) -> str:
        """Process a document and return result."""
        # Validate input
        self._validateDocument(doc)
        # Process
        content = self._extractContent(doc)
        result = self._transformContent(content)
        return result
    
    # Private helpers
    def _validateDocument(self, doc):
        # Internal validation
        pass
    
    def _extractContent(self, doc):
        # Internal extraction
        pass
    
    def _transformContent(self, content):
        # Internal transformation
        pass

2. Error Handling

import logging

logger = logging.getLogger(__name__)

class MyService:
    def processData(self, data):
        try:
            # Process data
            result = self._process(data)
            return result
            
        except ValidationError as e:
            # Expected errors - log and re-raise
            logger.warning(f"Validation error: {str(e)}")
            raise
            
        except Exception as e:
            # Unexpected errors - log with details and re-raise
            logger.error(f"Unexpected error in processData: {str(e)}", 
                        exc_info=True)
            raise ServiceException("Failed to process data") from e

3. Service Composition

class CompositeService:
    def __init__(self, services):
        self.services = services
    
    async def complexOperation(self, input_data):
        # Use multiple services in sequence
        
        # Step 1: Extract
        extracted = await self.services.extraction.extractContent(
            input_data.documents
        )
        
        # Step 2: Process with AI
        processed = await self.services.ai.callAiText(
            prompt=input_data.prompt,
            documents=extracted
        )
        
        # Step 3: Generate output
        result = await self.services.generation.renderReport(
            content=processed,
            format=input_data.format
        )
        
        # Step 4: Store result
        message = self.services.chat.storeMessageWithDocuments(
            workflow=self.services.workflow,
            messageData={"content": "Result"},
            documents=[result]
        )
        
        return message

4. Progress Tracking Best Practices

class ProgressAwareService:
    async def longRunningOperation(self, data, operation_id=None):
        # Create operation ID if not provided
        if not operation_id:
            operation_id = f"operation_{uuid.uuid4()}"
        
        # Always use try-finally for progress tracking
        try:
            # Start
            self.services.chat.progressLogStart(
                operation_id,
                "My Service",
                "Long Operation",
                f"Processing {len(data)} items"
            )
            
            # Process with granular updates
            total = len(data)
            for i, item in enumerate(data):
                # Process item
                self._processItem(item)
                
                # Update progress (0.0 to 1.0)
                progress = (i + 1) / total
                self.services.chat.progressLogUpdate(
                    operation_id,
                    progress,
                    f"Processed {i + 1}/{total} items"
                )
            
            # Success
            self.services.chat.progressLogFinish(operation_id, True)
            return results
            
        except Exception as e:
            # Failure
            self.services.chat.progressLogFinish(operation_id, False)
            raise

5. Testing Services

import pytest
from unittest.mock import Mock, MagicMock
from modules.services import Services
from modules.datamodels.datamodelUam import User

@pytest.fixture
def mock_services():
    """Create a mock services container for testing"""
    user = User(id="test-user", email="test@example.com")
    services = Services(user=user)
    
    # Mock interfaces
    services.interfaceDbChat = MagicMock()
    services.interfaceDbApp = MagicMock()
    services.interfaceDbComponent = MagicMock()
    
    return services

@pytest.mark.asyncio
async def test_service_method(mock_services):
    """Test a service method"""
    # Setup
    mock_services.interfaceDbChat.getWorkflow.return_value = Mock(
        id="workflow-123",
        messages=[]
    )
    
    # Execute
    result = await mock_services.ai.callAiPlanning(
        prompt="Test prompt"
    )
    
    # Assert
    assert result is not None
    mock_services.interfaceDbChat.getWorkflow.assert_called_once()

6. Documentation Standards

Every service method should include:

def serviceMethod(
    self,
    required_param: str,
    optional_param: Optional[int] = None
) -> Dict[str, Any]:
    """
    Short one-line description of what the method does.
    
    Longer description providing more details about the method's
    purpose, behavior, and any important considerations.
    
    Args:
        required_param: Description of required parameter
        optional_param: Description of optional parameter (default: None)
        
    Returns:
        Dict containing:
            - field1: Description of field1
            - field2: Description of field2
            
    Raises:
        ValueError: When required_param is invalid
        ServiceException: When operation fails
        
    Example:
        ```python
        result = services.myservice.serviceMethod(
            required_param="value",
            optional_param=42
        )
        ```
    """
    # Implementation
    pass

Service Interaction Patterns

Complete Request Flow with Multiple Services

sequenceDiagram
    participant Client
    participant Route
    participant Services
    participant AIService
    participant ExtractionService
    participant ChatService
    participant GenerationService
    participant Interface
    participant Database
    
    Client->>Route: POST /api/analyze-documents
    Route->>Services: Create Services(user, workflow)
    Services-->>Route: services instance
    
    Route->>AIService: services.ai.callAiDocuments(prompt, docs, format="pdf")
    
    Note over AIService,ExtractionService: Document Processing Phase
    AIService->>ChatService: progressLogStart(opId, "AI Service", "Processing")
    AIService->>ExtractionService: processDocumentsPerChunk(docs, prompt)
    ExtractionService->>Interface: getFileData(fileId)
    Interface->>Database: SELECT file_data
    Database-->>Interface: Raw bytes
    Interface-->>ExtractionService: File data
    
    ExtractionService->>ExtractionService: Extract & chunk content
    ExtractionService->>ChatService: storeWorkflowStat(extraction stats)
    
    loop For each chunk
        ExtractionService->>AIService: call(chunk, prompt)
        AIService-->>ExtractionService: AI result
        AIService->>ChatService: storeWorkflowStat(ai stats)
    end
    
    ExtractionService->>ExtractionService: Merge results
    ExtractionService-->>AIService: Merged content
    
    Note over AIService,GenerationService: Generation Phase
    AIService->>ChatService: progressLogUpdate(opId, 0.7, "Generating")
    AIService->>GenerationService: renderReport(content, "pdf", title)
    GenerationService->>GenerationService: Render to PDF
    GenerationService->>Interface: createFile(name, mimeType, content)
    Interface->>Database: INSERT file
    Database-->>Interface: File object
    Interface-->>GenerationService: File stored
    GenerationService-->>AIService: (pdf_content, mime_type)
    
    Note over AIService,ChatService: Storage Phase
    AIService->>ChatService: progressLogUpdate(opId, 0.9, "Storing")
    AIService->>ChatService: storeMessageWithDocuments(workflow, msg, [doc])
    ChatService->>Interface: createMessage(messageData)
    Interface->>Database: INSERT message
    Database-->>Interface: Message object
    ChatService->>ChatService: Sync workflow.messages
    ChatService-->>AIService: Message stored
    
    AIService->>ChatService: progressLogFinish(opId, True)
    AIService-->>Route: Final result with PDF
    
    Route->>Route: Format response
    Route-->>Client: JSON with PDF document

Conclusion

The Services layer is the backbone of the Gateway application's business logic, providing:

Modularity: Each service handles a specific domain
Reusability: Services can be used across features and routes
Composability: Services build on each other for complex workflows
Maintainability: Clear boundaries and responsibilities
Testability: Well-defined interfaces and dependencies

By following the patterns and best practices outlined in this documentation, developers can effectively use and extend the services layer to build robust, scalable features.

Additional Resources

Services API Reference - Complete API documentation with method signatures and examples
Architecture Overview - High-level system architecture
Security Component - Security and authentication
Real Estate Feature Guide - Example feature implementation
Data Models Documentation - Data model reference

Document Version: 1.0
Last Updated: 2025-01-25
Status: Complete

46 KiB Raw Blame History

Services Component Documentation

Table of Contents

Overview

Key Characteristics

What is a Service?

Service vs Feature vs Interface

Service Architecture

Service Container Pattern

Service Initialization Flow

Service Initialization

Services Class Structure

PublicService Wrapper

Core Services

Service Overview Diagram

1. AI Service

Key Responsibilities

AI Service Architecture

Core Capabilities

Special Features

2. Chat Service

Key Responsibilities

Chat Service Data Flow

Document Reference System

Core Capabilities

3. Extraction Service

Key Responsibilities

Extraction Service Architecture

Extraction Pipeline

Supported Formats

4. Generation Service

Key Responsibilities

Generation Service Architecture

Document Generation Flow

Supported Output Formats

5. Neutralization Service

Key Responsibilities

Neutralization Service Architecture

Neutralization Flow

PII Detection Patterns

6. SharePoint Service

Key Responsibilities

Core Capabilities

7. Ticket Service

Key Responsibilities

Ticket Service Architecture

Core Capabilities

8. Utils Service

Key Responsibilities

Core Capabilities

Service Usage Patterns

Pattern 1: Direct Service Call from Route

Pattern 2: Feature Using Multiple Services

Pattern 3: Service Composition

Pattern 4: Progress Tracking

Service Lifecycle

Lifecycle Stages

Request Lifecycle Example

Service Initialization Order

Service Destruction

Best Practices

1. Service Design Principles

Keep Services Stateless

Single Responsibility

Clear Interface Boundaries

2. Error Handling

3. Service Composition

4. Progress Tracking Best Practices

5. Testing Services

6. Documentation Standards

Service Interaction Patterns

Complete Request Flow with Multiple Services

Conclusion

Additional Resources

46 KiB

Raw Blame History