# Services Component Documentation

Comprehensive documentation of the Services layer in the Gateway application, explaining the architecture, patterns, and implementation details of all service modules.

> 📚 **Looking for detailed API reference?** See the [Services API Reference](./services-api-reference.md) for complete method signatures, parameters, return types, and usage examples.

## Table of Contents

1. [Overview](#overview)
2. [What is a Service?](#what-is-a-service)
3. [Service Architecture](#service-architecture)
4. [Service Initialization](#service-initialization)
5. [Core Services](#core-services)
6. [Service Usage Patterns](#service-usage-patterns)
7. [Service Lifecycle](#service-lifecycle)
8. [Best Practices](#best-practices)

---

## Overview

The **Services Layer** is a critical component of the Gateway application's business logic layer. Services provide reusable, composable functionality that can be used by features, routes, and other services. They encapsulate complex operations and act as orchestrators between the API layer and the data access layer.

```mermaid
graph TB
    subgraph "Application Layers"
        Routes[Routes Layer<br/>API Endpoints]
        Features[Features Layer<br/>Business Logic]
        Services[Services Layer<br/>Reusable Components]
        Interfaces[Interfaces Layer<br/>Data Access]
        Connectors[Connectors Layer<br/>External Systems]
    end
    
    Routes --> Features
    Routes --> Services
    Features --> Services
    Services --> Interfaces
    Services --> Services
    Interfaces --> Connectors
    
    style Services fill:#e8f5e9,stroke:#1b5e20,stroke-width:3px
```

### Key Characteristics

- **Reusable**: Services provide functionality that can be used across multiple features and routes
- **Composable**: Services can call other services to build complex workflows
- **Stateless**: Services operate without maintaining session state
- **User-Context Aware**: All services receive user context through the Services container
- **Interface-Dependent**: Services use interfaces to access data and external systems

---

## What is a Service?

A **Service** is a Python class that encapsulates a specific domain of functionality within the business logic layer. Services are designed to:

1. **Abstract Complexity**: Hide implementation details behind clean, well-defined APIs
2. **Promote Reusability**: Provide operations that can be used in multiple contexts
3. **Orchestrate Operations**: Coordinate between multiple interfaces and other services
4. **Process Data**: Transform, validate, and enrich data between layers
5. **Enforce Business Rules**: Implement domain-specific logic and constraints

### Service vs Feature vs Interface

```mermaid
graph LR
    subgraph "Comparison"
        A[Route] -->|Delegates to| B[Feature]
        A -->|Can call directly| C[Service]
        B -->|Uses| C
        C -->|Calls| D[Interface]
        C -->|Calls| E[Other Services]
        D -->|Uses| F[Connector]
    end
    
    style A fill:#f3e5f5,stroke:#4a148c
    style B fill:#e8f5e9,stroke:#1b5e20
    style C fill:#e8f5e9,stroke:#1b5e20
    style D fill:#fff3e0,stroke:#e65100
    style F fill:#fce4ec,stroke:#880e4f
```

| Aspect | Feature | Service | Interface |
|--------|---------|---------|-----------|
| **Purpose** | Domain-specific business logic | Cross-cutting, reusable functionality | Data access abstraction |
| **Scope** | Single use case or domain | Multiple use cases | Database/API operations |
| **Reusability** | Low (domain-specific) | High (cross-domain) | High (data operations) |
| **Dependencies** | Can use services | Can use other services and interfaces | Can use connectors |
| **Examples** | Real Estate workflow | AI processing, Document extraction | Database queries, API calls |

---

## Service Architecture

The Services layer follows a **Service-Oriented Architecture (SOA)** pattern with a centralized service container.

### Service Container Pattern

All services are initialized and accessed through a central `Services` container that provides:

- **User Context**: Every service has access to the current user
- **Workflow Context**: Access to the current workflow state
- **Interface Access**: Direct access to all data interfaces
- **Service Composition**: Services can call other services via `self.services`

```mermaid
graph TB
    subgraph "Services Container"
        SC[Services Instance]
        
        SC -->|Holds| User[User Context]
        SC -->|Holds| Workflow[Workflow Context]
        SC -->|Provides| IntChat[Chat Interface]
        SC -->|Provides| IntApp[App Interface]
        SC -->|Provides| IntComp[Component Interface]
        
        SC -->|Initializes| SvcAI[AI Service]
        SC -->|Initializes| SvcChat[Chat Service]
        SC -->|Initializes| SvcExtract[Extraction Service]
        SC -->|Initializes| SvcGen[Generation Service]
        SC -->|Initializes| SvcNeut[Neutralization Service]
        SC -->|Initializes| SvcUtils[Utils Service]
        SC -->|Initializes| SvcTicket[Ticket Service]
        SC -->|Initializes| SvcSP[SharePoint Service]
        
        SvcAI -.->|Uses| SC
        SvcChat -.->|Uses| SC
        SvcExtract -.->|Uses| SC
    end
    
    style SC fill:#e1f5ff,stroke:#01579b,stroke-width:3px
    style User fill:#fff3e0,stroke:#e65100
    style Workflow fill:#fff3e0,stroke:#e65100
```

### Service Initialization Flow

```mermaid
sequenceDiagram
    participant Route
    participant Services Container
    participant Service
    participant Interface
    participant Connector
    
    Route->>Services Container: Create(user, workflow)
    Services Container->>Interface: Initialize Interfaces
    Interface-->>Services Container: Ready
    Services Container->>Service: Initialize Service(serviceCenter)
    Service->>Service: Store serviceCenter reference
    Service-->>Services Container: Ready
    Services Container-->>Route: Services Ready
    
    Route->>Services Container: services.ai.someMethod()
    Services Container->>Service: someMethod()
    Service->>Services Container: self.services.chat.getContext()
    Services Container->>Service: Context Data
    Service->>Interface: Query Data
    Interface->>Connector: Execute Query
    Connector-->>Interface: Results
    Interface-->>Service: Domain Objects
    Service-->>Services Container: Result
    Services Container-->>Route: Result
```

---

## Service Initialization

Services are initialized through the `Services` class in `modules/services/__init__.py`.

### Services Class Structure

```python
class Services:
    def __init__(self, user: User, workflow: ChatWorkflow = None):
        # Core Context
        self.user: User = user
        self.workflow: ChatWorkflow = workflow
        self.currentUserPrompt: str = ""
        self.rawUserPrompt: str = ""
        
        # Initialize Interfaces
        self.interfaceDbChat = getChatInterface(user)
        self.interfaceDbApp = getAppInterface(user)
        self.interfaceDbComponent = getComponentInterface(user)
        
        # Initialize Services
        self.extraction = PublicService(ExtractionService(self))
        self.generation = PublicService(GenerationService(self))
        self.neutralization = PublicService(NeutralizationService(self))
        self.sharepoint = PublicService(SharepointService(self))
        self.ai = PublicService(AiService(self), functionsOnly=False)
        self.ticket = PublicService(TicketService(self))
        self.chat = PublicService(ChatService(self))
        self.utils = PublicService(UtilsService(self))
```

### PublicService Wrapper

The `PublicService` wrapper provides:

- **Method Access Control**: Exposes only public methods (non-underscore prefixed)
- **Attribute Protection**: Prevents direct attribute access from outside
- **Clean API Surface**: Provides a well-defined public interface

```python
class PublicService:
    """Wrapper that exposes only public methods of a service"""
    
    def __init__(self, service_instance, functionsOnly=True):
        self._service = service_instance
        self._functionsOnly = functionsOnly
    
    def __getattr__(self, name):
        # Only expose public methods/attributes
        if name.startswith('_'):
            raise AttributeError(f"Access to private attributes is not allowed")
        return getattr(self._service, name)
```

---

## Core Services

The Gateway application includes 8 core services, each handling a specific domain of functionality.

### Service Overview Diagram

```mermaid
graph TB
    subgraph "Core Services"
        AI[AI Service<br/>AI model operations]
        Chat[Chat Service<br/>Workflow & documents]
        Extract[Extraction Service<br/>Content extraction]
        Gen[Generation Service<br/>Document generation]
        Neut[Neutralization Service<br/>Data anonymization]
        SP[SharePoint Service<br/>SharePoint integration]
        Ticket[Ticket Service<br/>Ticket system integration]
        Utils[Utils Service<br/>Common utilities]
    end
    
    subgraph "Service Dependencies"
        AI -.->|Uses| Extract
        Gen -.->|Uses| AI
        Chat -.->|Uses| Utils
    end
    
    style AI fill:#bbdefb,stroke:#1976d2,stroke-width:2px
    style Chat fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
    style Extract fill:#fff9c4,stroke:#fbc02d,stroke-width:2px
    style Gen fill:#f8bbd0,stroke:#c2185b,stroke-width:2px
    style Neut fill:#d1c4e9,stroke:#512da8,stroke-width:2px
    style SP fill:#ffccbc,stroke:#e64a19,stroke-width:2px
    style Ticket fill:#b2dfdb,stroke:#00796b,stroke-width:2px
    style Utils fill:#e0e0e0,stroke:#616161,stroke-width:2px
```

---

### 1. AI Service

**Location**: `modules/services/serviceAi/mainServiceAi.py`

**Purpose**: Orchestrates all AI model operations including text generation, document processing, image generation, web search, and web crawling.

#### Key Responsibilities

- AI model selection and routing
- Prompt building and placeholder replacement
- Iterative generation with continuation support
- Document processing with chunking
- Model-aware content handling
- Progress tracking and statistics

#### AI Service Architecture

```mermaid
graph TB
    subgraph "AI Service Components"
        Main[AiService Main]
        Main --> Planning[Planning Call]
        Main --> Documents[Documents Call]
        Main --> Text[Text Call]
        
        Documents --> ImageGen[Image Generation]
        Documents --> WebOps[Web Operations]
        Documents --> DocGen[Document Generation]
        Documents --> TextProc[Text Processing]
        
        Main --> Helpers[Helper Methods]
        Helpers --> Placeholders[Placeholder Building]
        Helpers --> Analysis[Prompt Analysis]
        Helpers --> Looping[Iterative Generation]
        Helpers --> Parsing[Response Parsing]
    end
    
    Main -.->|Uses| ExtractSvc[Extraction Service]
    Main -.->|Uses| AiObjects[AI Objects Interface]
    
    style Main fill:#bbdefb,stroke:#1976d2,stroke-width:3px
```

#### Core Capabilities

**Planning Operations**
- Task and action planning with optimized parameters
- Intent analysis and workflow design
- Returns structured JSON plans

**Document Operations**
- Multi-format document processing
- Document generation (PDF, DOCX, XLSX, HTML, etc.)
- Image generation
- Web search and crawling

**Text Processing**
- Content extraction and summarization
- Data analysis and insights
- Model-aware chunking for large documents

#### Special Features

1. **Iterative Generation**: Supports multi-iteration generation with continuation context
2. **Repair-Based Looping**: Automatically repairs broken JSON and continues generation
3. **Model-Aware Processing**: Automatically chunks content based on model limits
4. **Progress Tracking**: Provides granular progress updates for long operations
5. **Statistics Collection**: Tracks costs, processing time, and token usage

> 📖 **See [Services API Reference](./services-api-reference.md#ai-service-api)** for detailed method signatures, parameters, and examples.

---

### 2. Chat Service

**Location**: `modules/services/serviceChat/mainServiceChat.py`

**Purpose**: Manages workflow operations, message handling, document resolution, and chat context management.

#### Key Responsibilities

- Workflow CRUD operations
- Message and document management
- Document reference resolution
- Connection management
- Progress logging
- Workflow statistics

#### Chat Service Data Flow

```mermaid
sequenceDiagram
    participant Feature
    participant ChatService
    participant Interface
    participant Database
    
    Feature->>ChatService: createWorkflow(data)
    ChatService->>Interface: createWorkflow(data)
    Interface->>Database: INSERT workflow
    Database-->>Interface: Workflow object
    Interface-->>ChatService: Workflow
    ChatService-->>Feature: Workflow
    
    Feature->>ChatService: storeMessageWithDocuments(workflow, message, docs)
    ChatService->>Interface: createMessage(messageData)
    Interface->>Database: INSERT message
    Database-->>Interface: Message object
    Interface->>Database: INSERT documents
    Database-->>Interface: Document objects
    ChatService->>ChatService: Sync in-memory workflow.messages
    ChatService-->>Feature: ChatMessage with documents
    
    Feature->>ChatService: getChatDocumentsFromDocumentList(refs)
    ChatService->>ChatService: Parse document references
    ChatService->>ChatService: Resolve from workflow.messages
    ChatService-->>Feature: List[ChatDocument]
```

#### Document Reference System

The Chat Service implements a sophisticated document reference system with three formats:

1. **docItem**: Single document by ID - `docItem:<id>:<filename>`
2. **docList with message ID**: Documents from specific message - `docList:<messageId>:<label>`
3. **docList by label**: Documents by label (newest) - `docList:<label>`

#### Core Capabilities

**Workflow Management**
- Create, update, retrieve workflows
- Context tracking (round, task, action numbers)
- Workflow statistics and metadata

**Document Operations**
- Resolve document references to ChatDocument objects
- Store messages with attached documents
- Track document lineage through workflow

**Progress Tracking**
- Start, update, finish operation tracking
- Real-time progress reporting
- Success/failure status

**Connection Management**
- User connection references
- Token refresh and management
- Connection status tracking

> 📖 **See [Services API Reference](./services-api-reference.md#chat-service-api)** for detailed method signatures, parameters, and examples.

---

### 3. Extraction Service

**Location**: `modules/services/serviceExtraction/mainServiceExtraction.py`

**Purpose**: Extracts and processes content from various document formats (PDF, DOCX, XLSX, images, etc.) with intelligent chunking and merging strategies.

#### Key Responsibilities

- Multi-format document extraction
- Content chunking for large documents
- Intelligent content merging
- Type-specific processing (text, tables, images, structures)
- Model-aware processing with parallel execution

#### Extraction Service Architecture

```mermaid
graph TB
    subgraph "Extraction Service"
        Main[ExtractionService Main]
        Main --> Extract[extractContent]
        Main --> Merge[mergeAiResults]
        Main --> Process[processDocumentsPerChunk]
        
        Extract --> Registry[Extractor Registry]
        Registry --> PDF[PDF Extractor]
        Registry --> DOCX[DOCX Extractor]
        Registry --> XLSX[XLSX Extractor]
        Registry --> IMG[Image Extractor]
        Registry --> TXT[Text Extractor]
        Registry --> CSV[CSV Extractor]
        Registry --> JSON[JSON Extractor]
        Registry --> XML[XML Extractor]
        
        Extract --> Chunkers[Chunker Registry]
        Chunkers --> TextChunk[Text Chunker]
        Chunkers --> TableChunk[Table Chunker]
        Chunkers --> ImgChunk[Image Chunker]
        Chunkers --> StructChunk[Structure Chunker]
    end
    
    Main -.->|Uses| AiObjects[AI Objects]
    
    style Main fill:#fff9c4,stroke:#fbc02d,stroke-width:3px
```

#### Extraction Pipeline

```mermaid
sequenceDiagram
    participant Client
    participant ExtractSvc as Extraction Service
    participant Extractor
    participant Chunker
    participant Merger
    
    Client->>ExtractSvc: extractContent(documents, options)
    
    loop For each document
        ExtractSvc->>Extractor: Extract by MIME type
        Extractor-->>ExtractSvc: ContentParts
        
        ExtractSvc->>Chunker: Chunk if needed
        Chunker-->>ExtractSvc: Chunked ContentParts
    end
    
    ExtractSvc-->>Client: List[ContentExtracted]
    
    Client->>ExtractSvc: processDocumentsPerChunk(docs, prompt)
    ExtractSvc->>ExtractSvc: Extract content
    
    loop For each content part
        ExtractSvc->>AiObjects: Process part with AI
        AiObjects-->>ExtractSvc: AI result
    end
    
    ExtractSvc->>Merger: Merge part results
    Merger-->>ExtractSvc: Merged content
    ExtractSvc-->>Client: Final merged text
```

#### Supported Formats

| Format | Extractor | Chunking | Special Handling |
|--------|-----------|----------|------------------|
| PDF | extractorPdf.py | ✅ | Page-aware extraction |
| DOCX | extractorDocx.py | ✅ | Paragraph structure |
| XLSX | extractorXlsx.py | ✅ | Sheet-based processing |
| PPTX | extractorPptx.py | ✅ | Slide-based extraction |
| CSV | extractorCsv.py | ✅ | Row/column chunking |
| JSON | extractorJson.py | ✅ | Tree-based chunking |
| XML | extractorXml.py | ✅ | Element-based chunking |
| HTML | extractorHtml.py | ✅ | DOM-based extraction |
| Images | extractorImage.py | ✅ | OCR + metadata |
| Text | extractorText.py | ✅ | Paragraph chunking |

> 📖 **See [Services API Reference](./services-api-reference.md#extraction-service-api)** for detailed method signatures, parameters, and examples.

---

### 4. Generation Service

**Location**: `modules/services/serviceGeneration/mainServiceGeneration.py`

**Purpose**: Generates documents in various formats from structured content (HTML, PDF, DOCX, XLSX, etc.).

#### Key Responsibilities

- Document rendering in multiple formats
- Action result document processing
- File creation and storage
- Workflow context attachment
- MIME type detection and normalization

#### Generation Service Architecture

```mermaid
graph TB
    subgraph "Generation Service"
        Main[GenerationService Main]
        Main --> Process[Process Action Results]
        Main --> Render[renderReport]
        Main --> Create[Create Documents]
        
        Render --> RendererRegistry[Renderer Registry]
        RendererRegistry --> HTML[HTML Renderer]
        RendererRegistry --> PDF[PDF Renderer]
        RendererRegistry --> DOCX[DOCX Renderer]
        RendererRegistry --> XLSX[XLSX Renderer]
        RendererRegistry --> CSV[CSV Renderer]
        RendererRegistry --> MD[Markdown Renderer]
        RendererRegistry --> JSON[JSON Renderer]
        RendererRegistry --> TXT[Text Renderer]
        RendererRegistry --> IMG[Image Renderer]
    end
    
    Main -.->|Uses| ComponentInterface[Component Interface]
    Main -.->|Uses| ChatInterface[Chat Interface]
    
    style Main fill:#f8bbd0,stroke:#c2185b,stroke-width:3px
```

#### Document Generation Flow

```mermaid
sequenceDiagram
    participant Action
    participant GenSvc as Generation Service
    participant Renderer
    participant CompInterface as Component Interface
    participant Storage
    
    Action->>GenSvc: processActionResultDocuments(result, action)
    GenSvc->>GenSvc: Extract document data
    GenSvc->>GenSvc: Detect MIME type
    GenSvc-->>Action: Processed documents list
    
    Action->>GenSvc: createDocumentsFromActionResult(result, action, workflow)
    
    loop For each document
        GenSvc->>GenSvc: Normalize filename & extension
        GenSvc->>GenSvc: Convert to string content
        GenSvc->>CompInterface: createFile(name, mimeType, content)
        CompInterface->>Storage: Store file
        Storage-->>CompInterface: File object
        GenSvc->>GenSvc: Create ChatDocument
        GenSvc->>GenSvc: Set workflow context
    end
    
    GenSvc-->>Action: List[ChatDocument]
    
    Action->>GenSvc: renderReport(content, format, title)
    GenSvc->>Renderer: Get renderer for format
    Renderer->>Renderer: render(content, title)
    Renderer-->>GenSvc: (rendered_content, mime_type)
    GenSvc-->>Action: Rendered document
```

#### Supported Output Formats

| Format | Renderer | Use Case | Special Features |
|--------|----------|----------|------------------|
| HTML | rendererHtml.py | Web display | CSS styling, responsive |
| PDF | rendererPdf.py | Print-ready docs | Page layout, headers/footers |
| DOCX | rendererDocx.py | Editable documents | Styles, tables, images |
| XLSX | rendererXlsx.py | Spreadsheets | Multiple sheets, formulas |
| PPTX | rendererPptx.py | Presentations | Slides, layouts, transitions |
| CSV | rendererCsv.py | Data export | Tabular data |
| JSON | rendererJson.py | Data exchange | Structured data |
| Markdown | rendererMarkdown.py | Documentation | GitHub-flavored markdown |
| Text | rendererText.py | Plain text | Simple formatting |
| Image | rendererImage.py | Charts, diagrams | PNG, JPEG |

> 📖 **See [Services API Reference](./services-api-reference.md#generation-service-api)** for detailed method signatures, parameters, and examples.

---

### 5. Neutralization Service

**Location**: `modules/services/serviceNeutralization/mainServiceNeutralization.py`

**Purpose**: Anonymizes sensitive data in documents for GDPR compliance, replacing personal information with placeholders.

#### Key Responsibilities

- PII detection and replacement
- Multi-language support (DE, EN, FR, IT)
- Format-specific processing (text, JSON, CSV, XML)
- Attribute tracking and resolution
- Configuration management

#### Neutralization Service Architecture

```mermaid
graph TB
    subgraph "Neutralization Service"
        Main[NeutralizationService Main]
        Main --> Config[Configuration Mgmt]
        Main --> Process[Process Text/File]
        Main --> Resolve[Resolve Placeholders]
        Main --> Attributes[Attribute Management]
        
        Process --> Detection[Pattern Detection]
        Detection --> HeaderPat[Header Patterns]
        Detection --> DataPat[Data Patterns]
        Detection --> TablePat[Table Patterns]
        
        Process --> Processors[Content Processors]
        Processors --> TextProc[Text Processor]
        Processors --> ListProc[List Processor]
        Processors --> BinaryProc[Binary Processor]
        
        TextProc --> PlainText[Plain Text Handler]
        ListProc --> CSV[CSV Handler]
        ListProc --> JSON[JSON Handler]
        ListProc --> XML[XML Handler]
        ListProc --> TableData[Table Data Handler]
    end
    
    Main -.->|Uses| AppInterface[App Interface]
    Main -.->|Uses| ComponentInterface[Component Interface]
    
    style Main fill:#d1c4e9,stroke:#512da8,stroke-width:3px
```

#### Neutralization Flow

```mermaid
sequenceDiagram
    participant Client
    participant NeutSvc as Neutralization Service
    participant Processor
    participant Patterns
    participant AppInterface as App Interface
    participant Database
    
    Client->>NeutSvc: processText(text)
    NeutSvc->>NeutSvc: _reloadNamesFromConfig()
    NeutSvc->>AppInterface: getNeutralizationConfig()
    AppInterface->>Database: Query config
    Database-->>AppInterface: Config with names
    AppInterface-->>NeutSvc: Config
    
    NeutSvc->>NeutSvc: Auto-detect content type
    
    alt Text Content
        NeutSvc->>Processor: processTextContent(text)
        Processor->>Patterns: Match PII patterns
        Patterns-->>Processor: Matches
        Processor->>Processor: Replace with placeholders
    else JSON/CSV/XML
        NeutSvc->>Processor: processJsonContent(text)
        Processor->>Processor: Parse structure
        Processor->>Patterns: Match in fields
        Processor->>Processor: Replace in-place
    end
    
    Processor-->>NeutSvc: (data, mapping, replacements, info)
    
    NeutSvc->>AppInterface: createNeutralizationAttributes(mapping)
    AppInterface->>Database: Store attributes
    
    NeutSvc-->>Client: NeutralizationResult
```

#### PII Detection Patterns

The service detects and neutralizes:

- **Names**: First names, last names, full names
- **Email Addresses**: Standard email formats
- **Phone Numbers**: International formats
- **Addresses**: Street addresses, postal codes
- **IDs**: SSN, passport numbers, employee IDs
- **Financial**: Credit cards, bank accounts, IBAN
- **Dates**: Birth dates, dates of service
- **Custom**: User-configured patterns

> 📖 **See [Services API Reference](./services-api-reference.md#neutralization-service-api)** for detailed method signatures, parameters, and examples.

---

### 6. SharePoint Service

**Location**: `modules/services/serviceSharepoint/mainServiceSharepoint.py`

**Purpose**: Integrates with Microsoft SharePoint for document management and synchronization.

#### Key Responsibilities

- SharePoint authentication
- Document upload/download
- Folder management
- Metadata handling
- Permission management

#### Core Capabilities

- Connect to SharePoint sites with OAuth
- Upload and download documents
- List and create folders
- Manage document metadata
- Handle permissions

> 📖 **See [Services API Reference](./services-api-reference.md#sharepoint-service-api)** for detailed method signatures, parameters, and examples.

---

### 7. Ticket Service

**Location**: `modules/services/serviceTicket/mainServiceTicket.py`

**Purpose**: Integrates with ticket management systems (Jira, ClickUp, ServiceNow, etc.).

#### Key Responsibilities

- Ticket system authentication
- Ticket CRUD operations
- Field mapping and synchronization
- Status tracking
- Comment management

#### Ticket Service Architecture

```mermaid
graph TB
    subgraph "Ticket Service"
        Main[TicketService Main]
        Main --> Connect[connectTicket]
        
        Connect --> TicketInterface[Ticket Interface Factory]
        TicketInterface --> Jira[Jira Interface]
        TicketInterface --> ClickUp[ClickUp Interface]
        TicketInterface --> ServiceNow[ServiceNow Interface]
    end
    
    Main -.->|Uses| Connectors[Ticket Connectors]
    
    style Main fill:#b2dfdb,stroke:#00796b,stroke-width:3px
```

#### Core Capabilities

- Connect to multiple ticket systems
- Create and update tickets
- Map fields between systems
- Track ticket status
- Manage comments and attachments

> 📖 **See [Services API Reference](./services-api-reference.md#ticket-service-api)** for detailed method signatures, parameters, and examples.

---

### 8. Utils Service

**Location**: `modules/services/serviceUtils/mainServiceUtils.py`

**Purpose**: Provides common utility functions used across the application.

#### Key Responsibilities

- Configuration access
- Event scheduling (cron, interval jobs)
- Timestamp utilities
- Debug logging
- Prompt sanitization
- JSON utilities
- Enum mapping

#### Core Capabilities

**Event Management**
- Schedule cron jobs (daily, weekly, custom)
- Schedule interval jobs (every N seconds/minutes/hours)
- Remove scheduled jobs

**Configuration**
- Get/set configuration values
- User-specific configurations
- Default value support

**Time Utilities**
- UTC timestamp generation
- Timezone conversion

**Debug Tools**
- Write debug files
- Log to debug file
- Store debug artifacts

**Content Sanitization**
- Sanitize user input for prompts
- Escape special characters
- Prevent injection attacks

**JSON Utilities**
- Extract JSON from text
- Parse with error recovery
- Normalize JSON text

> 📖 **See [Services API Reference](./services-api-reference.md#utils-service-api)** for detailed method signatures, parameters, and examples.

---

## Service Usage Patterns

### Pattern 1: Direct Service Call from Route

```mermaid
sequenceDiagram
    participant Client
    participant Route
    participant Services
    participant Service
    participant Interface
    
    Client->>Route: POST /api/neutralize
    Route->>Services: Create(user, workflow)
    Services-->>Route: services instance
    Route->>Service: services.neutralization.processText(text)
    Service->>Interface: Query/Update data
    Interface-->>Service: Result
    Service-->>Route: Neutralized text
    Route-->>Client: JSON Response
```

Example:
```python
@router.post("/neutralize-text")
async def neutralizeText(
    request: NeutralizeRequest,
    currentUser: User = Depends(getCurrentUser)
):
    # Create services container
    services = Services(user=currentUser)
    
    # Call service directly
    result = services.neutralization.processText(request.text)
    
    return {
        "success": True,
        "data": result
    }
```

### Pattern 2: Feature Using Multiple Services

```mermaid
sequenceDiagram
    participant Route
    participant Feature
    participant Services
    participant AIService
    participant ChatService
    participant GenService
    
    Route->>Feature: execute(services)
    Feature->>Services: services.ai.callAiDocuments()
    Services->>AIService: Process documents
    AIService->>ChatService: Get document context
    ChatService-->>AIService: Context
    AIService-->>Feature: AI result
    
    Feature->>Services: services.generation.renderReport()
    Services->>GenService: Render to PDF
    GenService-->>Feature: PDF document
    
    Feature->>Services: services.chat.storeMessageWithDocuments()
    Services->>ChatService: Store result
    ChatService-->>Feature: Message stored
    
    Feature-->>Route: Feature result
```

Example:
```python
class RealEstateFeature:
    async def execute(self, services: Services):
        # Use AI service
        analysis = await services.ai.callAiDocuments(
            prompt=self.prompt,
            documents=self.documents,
            options=self.options
        )
        
        # Use generation service
        report, mime = await services.generation.renderReport(
            extractedContent=analysis,
            outputFormat="pdf",
            title="Real Estate Analysis"
        )
        
        # Use chat service
        message = services.chat.storeMessageWithDocuments(
            workflow=services.workflow,
            messageData=message_data,
            documents=[report_document]
        )
        
        return message
```

### Pattern 3: Service Composition

```mermaid
sequenceDiagram
    participant Feature
    participant AIService
    participant ExtractService
    participant ChatService
    participant Interface
    
    Feature->>AIService: callAiText(documents, prompt)
    AIService->>ExtractService: processDocumentsPerChunk(docs, prompt)
    ExtractService->>ExtractService: extractContent(documents)
    ExtractService->>Interface: Get file data
    Interface-->>ExtractService: Raw bytes
    ExtractService->>ExtractService: Extract & chunk
    
    loop For each chunk
        ExtractService->>AIService: call(request)
        AIService-->>ExtractService: AI result
    end
    
    ExtractService->>ExtractService: Merge results
    ExtractService-->>AIService: Merged text
    
    AIService->>ChatService: storeWorkflowStat()
    ChatService-->>AIService: Stat stored
    
    AIService-->>Feature: Final result
```

Example:
```python
class AiService:
    async def callAiText(self, prompt, documents, options):
        # AI service uses extraction service internally
        result = await self.extractionService.processDocumentsPerChunk(
            documents=documents,
            prompt=prompt,
            aiObjects=self.aiObjects,
            options=options
        )
        
        # Also uses chat service for statistics
        self.services.chat.storeWorkflowStat(
            self.services.workflow,
            response,
            "ai.text_call"
        )
        
        return result
```

### Pattern 4: Progress Tracking

```mermaid
sequenceDiagram
    participant Feature
    participant AIService
    participant ChatService
    participant ProgressLogger
    participant Database
    
    Feature->>AIService: callAiDocuments(prompt, docs)
    AIService->>ChatService: progressLogStart(opId, "AI Call", "Processing")
    ChatService->>ProgressLogger: startOperation(opId, details)
    ProgressLogger->>Database: Store progress log
    
    AIService->>AIService: Process step 1
    AIService->>ChatService: progressLogUpdate(opId, 0.3, "Extracting")
    ChatService->>ProgressLogger: updateOperation(opId, 0.3)
    ProgressLogger->>Database: Update progress
    
    AIService->>AIService: Process step 2
    AIService->>ChatService: progressLogUpdate(opId, 0.6, "Generating")
    ChatService->>ProgressLogger: updateOperation(opId, 0.6)
    
    AIService->>AIService: Complete
    AIService->>ChatService: progressLogFinish(opId, True)
    ChatService->>ProgressLogger: finishOperation(opId, True)
    ProgressLogger->>Database: Mark complete
    
    AIService-->>Feature: Result
```

Example:
```python
# In AI Service
async def callAiDocuments(self, prompt, documents, ...):
    # Create operation ID
    operationId = f"ai_docs_{workflow_id}_{timestamp}"
    
    # Start progress tracking
    self.services.chat.progressLogStart(
        operationId,
        "AI Service",
        "Document Processing",
        f"Processing {len(documents)} documents"
    )
    
    try:
        # Step 1
        self.services.chat.progressLogUpdate(
            operationId, 0.2, "Extracting content"
        )
        extracted = await self.extract(documents)
        
        # Step 2
        self.services.chat.progressLogUpdate(
            operationId, 0.5, "Calling AI model"
        )
        result = await self.callAI(extracted)
        
        # Step 3
        self.services.chat.progressLogUpdate(
            operationId, 0.9, "Finalizing result"
        )
        final = self.finalize(result)
        
        # Complete
        self.services.chat.progressLogFinish(operationId, True)
        return final
        
    except Exception as e:
        self.services.chat.progressLogFinish(operationId, False)
        raise
```

---

## Service Lifecycle

### Lifecycle Stages

```mermaid
stateDiagram-v2
    [*] --> Created: Services(user, workflow)
    Created --> Initialized: Interface setup
    Initialized --> Ready: Services ready
    Ready --> InUse: Method calls
    InUse --> Ready: Method completes
    InUse --> Error: Exception
    Error --> Ready: Error handled
    Ready --> Disposed: Request complete
    Disposed --> [*]
    
    note right of Created
        Services container created
        User context set
        Workflow context set
    end note
    
    note right of Initialized
        All interfaces connected
        All services instantiated
        Dependencies resolved
    end note
    
    note right of InUse
        Service methods executing
        Can call other services
        Progress tracked
    end note
```

### Request Lifecycle Example

```mermaid
sequenceDiagram
    participant Client
    participant FastAPI
    participant Route
    participant Services
    participant Service
    
    Client->>FastAPI: HTTP Request
    FastAPI->>FastAPI: Authentication
    FastAPI->>Route: route_handler(user=currentUser)
    
    Note over Route: Request Start
    Route->>Services: Create Services(user)
    Services->>Services: Initialize interfaces
    Services->>Services: Initialize services
    Services-->>Route: services instance
    
    Note over Route,Service: Service Usage
    Route->>Service: services.ai.callAiDocuments()
    Service->>Service: Process
    Service->>Service: self.services.chat.progressLog()
    Service-->>Route: Result
    
    Note over Route: Request End
    Route->>Route: Format response
    Route-->>FastAPI: JSON response
    FastAPI->>FastAPI: services destroyed
    FastAPI-->>Client: HTTP Response
```

### Service Initialization Order

1. **User Context**: User object retrieved from authentication
2. **Workflow Context** (optional): Workflow loaded if ID provided
3. **Interfaces**: Database interfaces initialized with user context
4. **Services**: All services instantiated with Services container reference
5. **Dependencies**: Services can now call other services

### Service Destruction

Services are automatically destroyed at the end of each request:

- No explicit cleanup required
- Database connections returned to pool
- Memory released
- No persistent state maintained

---

## Best Practices

### 1. Service Design Principles

#### Keep Services Stateless

```python
# ❌ BAD: Storing state in service
class BadService:
    def __init__(self, services):
        self.services = services
        self.cached_results = {}  # State!
        
    def process(self, data):
        if data.id in self.cached_results:
            return self.cached_results[data.id]
        # ...

# ✅ GOOD: Stateless service
class GoodService:
    def __init__(self, services):
        self.services = services
        
    def process(self, data):
        # Use database or pass state explicitly
        cached = self.services.interfaceDbApp.getCache(data.id)
        if cached:
            return cached
        # ...
```

#### Single Responsibility

```python
# ❌ BAD: Service doing too much
class BadService:
    def processDocuments(self, docs):
        # Extract content
        # Call AI
        # Generate report
        # Send email
        # Update database
        pass

# ✅ GOOD: Focused services
class ExtractionService:
    def extractContent(self, docs):
        # Only extraction logic
        pass

class GenerationService:
    def renderReport(self, content, format):
        # Only generation logic
        pass

class EmailService:
    def sendReport(self, report, recipient):
        # Only email logic
        pass
```

#### Clear Interface Boundaries

```python
# ✅ GOOD: Well-defined public API
class GoodService:
    # Public API
    def processDocument(self, doc: ChatDocument) -> str:
        """Process a document and return result."""
        # Validate input
        self._validateDocument(doc)
        # Process
        content = self._extractContent(doc)
        result = self._transformContent(content)
        return result
    
    # Private helpers
    def _validateDocument(self, doc):
        # Internal validation
        pass
    
    def _extractContent(self, doc):
        # Internal extraction
        pass
    
    def _transformContent(self, content):
        # Internal transformation
        pass
```

### 2. Error Handling

```python
import logging

logger = logging.getLogger(__name__)

class MyService:
    def processData(self, data):
        try:
            # Process data
            result = self._process(data)
            return result
            
        except ValidationError as e:
            # Expected errors - log and re-raise
            logger.warning(f"Validation error: {str(e)}")
            raise
            
        except Exception as e:
            # Unexpected errors - log with details and re-raise
            logger.error(f"Unexpected error in processData: {str(e)}", 
                        exc_info=True)
            raise ServiceException("Failed to process data") from e
```

### 3. Service Composition

```python
class CompositeService:
    def __init__(self, services):
        self.services = services
    
    async def complexOperation(self, input_data):
        # Use multiple services in sequence
        
        # Step 1: Extract
        extracted = await self.services.extraction.extractContent(
            input_data.documents
        )
        
        # Step 2: Process with AI
        processed = await self.services.ai.callAiText(
            prompt=input_data.prompt,
            documents=extracted
        )
        
        # Step 3: Generate output
        result = await self.services.generation.renderReport(
            content=processed,
            format=input_data.format
        )
        
        # Step 4: Store result
        message = self.services.chat.storeMessageWithDocuments(
            workflow=self.services.workflow,
            messageData={"content": "Result"},
            documents=[result]
        )
        
        return message
```

### 4. Progress Tracking Best Practices

```python
class ProgressAwareService:
    async def longRunningOperation(self, data, operation_id=None):
        # Create operation ID if not provided
        if not operation_id:
            operation_id = f"operation_{uuid.uuid4()}"
        
        # Always use try-finally for progress tracking
        try:
            # Start
            self.services.chat.progressLogStart(
                operation_id,
                "My Service",
                "Long Operation",
                f"Processing {len(data)} items"
            )
            
            # Process with granular updates
            total = len(data)
            for i, item in enumerate(data):
                # Process item
                self._processItem(item)
                
                # Update progress (0.0 to 1.0)
                progress = (i + 1) / total
                self.services.chat.progressLogUpdate(
                    operation_id,
                    progress,
                    f"Processed {i + 1}/{total} items"
                )
            
            # Success
            self.services.chat.progressLogFinish(operation_id, True)
            return results
            
        except Exception as e:
            # Failure
            self.services.chat.progressLogFinish(operation_id, False)
            raise
```

### 5. Testing Services

```python
import pytest
from unittest.mock import Mock, MagicMock
from modules.services import Services
from modules.datamodels.datamodelUam import User

@pytest.fixture
def mock_services():
    """Create a mock services container for testing"""
    user = User(id="test-user", email="test@example.com")
    services = Services(user=user)
    
    # Mock interfaces
    services.interfaceDbChat = MagicMock()
    services.interfaceDbApp = MagicMock()
    services.interfaceDbComponent = MagicMock()
    
    return services

@pytest.mark.asyncio
async def test_service_method(mock_services):
    """Test a service method"""
    # Setup
    mock_services.interfaceDbChat.getWorkflow.return_value = Mock(
        id="workflow-123",
        messages=[]
    )
    
    # Execute
    result = await mock_services.ai.callAiPlanning(
        prompt="Test prompt"
    )
    
    # Assert
    assert result is not None
    mock_services.interfaceDbChat.getWorkflow.assert_called_once()
```

### 6. Documentation Standards

Every service method should include:

```python
def serviceMethod(
    self,
    required_param: str,
    optional_param: Optional[int] = None
) -> Dict[str, Any]:
    """
    Short one-line description of what the method does.
    
    Longer description providing more details about the method's
    purpose, behavior, and any important considerations.
    
    Args:
        required_param: Description of required parameter
        optional_param: Description of optional parameter (default: None)
        
    Returns:
        Dict containing:
            - field1: Description of field1
            - field2: Description of field2
            
    Raises:
        ValueError: When required_param is invalid
        ServiceException: When operation fails
        
    Example:
        ```python
        result = services.myservice.serviceMethod(
            required_param="value",
            optional_param=42
        )
        ```
    """
    # Implementation
    pass
```

---

## Service Interaction Patterns

### Complete Request Flow with Multiple Services

```mermaid
sequenceDiagram
    participant Client
    participant Route
    participant Services
    participant AIService
    participant ExtractionService
    participant ChatService
    participant GenerationService
    participant Interface
    participant Database
    
    Client->>Route: POST /api/analyze-documents
    Route->>Services: Create Services(user, workflow)
    Services-->>Route: services instance
    
    Route->>AIService: services.ai.callAiDocuments(prompt, docs, format="pdf")
    
    Note over AIService,ExtractionService: Document Processing Phase
    AIService->>ChatService: progressLogStart(opId, "AI Service", "Processing")
    AIService->>ExtractionService: processDocumentsPerChunk(docs, prompt)
    ExtractionService->>Interface: getFileData(fileId)
    Interface->>Database: SELECT file_data
    Database-->>Interface: Raw bytes
    Interface-->>ExtractionService: File data
    
    ExtractionService->>ExtractionService: Extract & chunk content
    ExtractionService->>ChatService: storeWorkflowStat(extraction stats)
    
    loop For each chunk
        ExtractionService->>AIService: call(chunk, prompt)
        AIService-->>ExtractionService: AI result
        AIService->>ChatService: storeWorkflowStat(ai stats)
    end
    
    ExtractionService->>ExtractionService: Merge results
    ExtractionService-->>AIService: Merged content
    
    Note over AIService,GenerationService: Generation Phase
    AIService->>ChatService: progressLogUpdate(opId, 0.7, "Generating")
    AIService->>GenerationService: renderReport(content, "pdf", title)
    GenerationService->>GenerationService: Render to PDF
    GenerationService->>Interface: createFile(name, mimeType, content)
    Interface->>Database: INSERT file
    Database-->>Interface: File object
    Interface-->>GenerationService: File stored
    GenerationService-->>AIService: (pdf_content, mime_type)
    
    Note over AIService,ChatService: Storage Phase
    AIService->>ChatService: progressLogUpdate(opId, 0.9, "Storing")
    AIService->>ChatService: storeMessageWithDocuments(workflow, msg, [doc])
    ChatService->>Interface: createMessage(messageData)
    Interface->>Database: INSERT message
    Database-->>Interface: Message object
    ChatService->>ChatService: Sync workflow.messages
    ChatService-->>AIService: Message stored
    
    AIService->>ChatService: progressLogFinish(opId, True)
    AIService-->>Route: Final result with PDF
    
    Route->>Route: Format response
    Route-->>Client: JSON with PDF document
```

---

## Conclusion

The Services layer is the backbone of the Gateway application's business logic, providing:

- **Modularity**: Each service handles a specific domain
- **Reusability**: Services can be used across features and routes
- **Composability**: Services build on each other for complex workflows
- **Maintainability**: Clear boundaries and responsibilities
- **Testability**: Well-defined interfaces and dependencies

By following the patterns and best practices outlined in this documentation, developers can effectively use and extend the services layer to build robust, scalable features.

---

## Additional Resources

- [Services API Reference](./services-api-reference.md) - Complete API documentation with method signatures and examples
- [Architecture Overview](./architecture-overview.md) - High-level system architecture
- [Security Component](./security-component.md) - Security and authentication
- [Real Estate Feature Guide](../real-estate-feature-integration-guide/README.md) - Example feature implementation
- [Data Models Documentation](../../modules/datamodels/README.md) - Data model reference

---

**Document Version**: 1.0  
**Last Updated**: 2025-01-25  
**Status**: Complete