refactored actions handlers
This commit is contained in:
parent
4af9e5fc87
commit
56d6ecf978
78 changed files with 9858 additions and 350 deletions
|
|
@ -1,292 +0,0 @@
|
||||||
# Module Dependencies Analysis
|
|
||||||
|
|
||||||
This document provides a comprehensive analysis of import dependencies between modules in the `modules` directory.
|
|
||||||
|
|
||||||
## Overview
|
|
||||||
|
|
||||||
The codebase is organized into the following top-level modules:
|
|
||||||
- **aicore** - AI core functionality and model management
|
|
||||||
- **auth** - High-level authentication and token management
|
|
||||||
- **connectors** - External service connectors
|
|
||||||
- **datamodels** - Data models and schemas
|
|
||||||
- **features** - Feature modules (workflow, dynamicOptions, etc.)
|
|
||||||
- **interfaces** - Database and service interfaces
|
|
||||||
- **routes** - API route handlers
|
|
||||||
- **security** - Low-level core security (RBAC and root access)
|
|
||||||
- **services** - Business logic services
|
|
||||||
- **shared** - Shared utilities and helpers
|
|
||||||
- **workflows** - Workflow processing and management
|
|
||||||
|
|
||||||
## Bidirectional Dependency Matrix
|
|
||||||
|
|
||||||
This table shows all module pairs with dependencies, displaying imports in both directions.
|
|
||||||
|
|
||||||
| Module X | Module Y | X → Y | Y → X | Total |
|
|
||||||
|----------|----------|-------|-------|-------|
|
|
||||||
| aicore | connectors | 1 | 0 | 1 |
|
|
||||||
| aicore | datamodels | 13 | 0 | 13 |
|
|
||||||
| aicore | interfaces | 0 | 2 | 2 |
|
|
||||||
| aicore | security | 2 | 0 | 2 |
|
|
||||||
| aicore | services | 0 | 2 | 2 |
|
|
||||||
| aicore | shared | 5 | 0 | 5 |
|
|
||||||
| auth | datamodels | 5 | 0 | 5 |
|
|
||||||
| auth | interfaces | 4 | 0 | 4 |
|
|
||||||
| auth | routes | 0 | 32 | 32 |
|
|
||||||
| auth | security | 4 | 0 | 4 |
|
|
||||||
| auth | services | 0 | 1 | 1 |
|
|
||||||
| auth | shared | 8 | 0 | 8 |
|
|
||||||
| connectors | datamodels | 4 | 0 | 4 |
|
|
||||||
| connectors | interfaces | 0 | 10 | 10 |
|
|
||||||
| connectors | shared | 5 | 0 | 5 |
|
|
||||||
| datamodels | features | 0 | 6 | 6 |
|
|
||||||
| datamodels | interfaces | 0 | 27 | 27 |
|
|
||||||
| datamodels | routes | 0 | 48 | 48 |
|
|
||||||
| datamodels | security | 0 | 5 | 5 |
|
|
||||||
| datamodels | services | 0 | 52 | 52 |
|
|
||||||
| datamodels | shared | 19 | 0 | 19 |
|
|
||||||
| datamodels | workflows | 0 | 72 | 72 |
|
|
||||||
| features | interfaces | 0 | 0 | 0 |
|
|
||||||
| features | routes | 0 | 6 | 6 |
|
|
||||||
| features | services | 4 | 0 | 4 |
|
|
||||||
| features | shared | 3 | 0 | 3 |
|
|
||||||
| features | workflows | 1 | 0 | 1 |
|
|
||||||
| interfaces | routes | 0 | 29 | 29 |
|
|
||||||
| interfaces | security | 9 | 0 | 9 |
|
|
||||||
| interfaces | services | 0 | 8 | 8 |
|
|
||||||
| interfaces | shared | 11 | 0 | 11 |
|
|
||||||
| routes | interfaces | 29 | 0 | 29 |
|
|
||||||
| routes | services | 5 | 0 | 5 |
|
|
||||||
| routes | shared | 21 | 0 | 21 |
|
|
||||||
| security | connectors | 2 | 0 | 2 |
|
|
||||||
| security | datamodels | 5 | 0 | 5 |
|
|
||||||
| services | shared | 16 | 0 | 16 |
|
|
||||||
| services | workflows | 0 | 1 | 1 |
|
|
||||||
| shared | workflows | 0 | 9 | 9 |
|
|
||||||
|
|
||||||
**Legend:**
|
|
||||||
- **X → Y**: Number of imports from Module X to Module Y
|
|
||||||
- **Y → X**: Number of imports from Module Y to Module X
|
|
||||||
- **Total**: Sum of imports in both directions
|
|
||||||
|
|
||||||
## Bidirectional Dependencies Only (Circular Dependencies)
|
|
||||||
|
|
||||||
This table shows only module pairs where imports exist in **both directions**, indicating potential circular dependencies that should be monitored.
|
|
||||||
|
|
||||||
| Module X | Module Y | X → Y | Y → X | Total |
|
|
||||||
|----------|----------|-------|-------|-------|
|
|
||||||
|
|
||||||
**Total bidirectional dependencies: 0**
|
|
||||||
|
|
||||||
**Note:** All circular dependencies have been eliminated. The architecture now has clean one-way dependencies.
|
|
||||||
|
|
||||||
**Key Improvements:**
|
|
||||||
1. **Eliminated `connectors ↔ security` circular dependency**: After moving RBAC logic from `connectorDbPostgre.py` to `interfaces/interfaceRbac.py`, connectors no longer import from security. Security still imports from connectors (for `rootAccess` to create `DatabaseConnector` instances), but this is a one-way dependency (security → connectors: 2, connectors → security: 0).
|
|
||||||
2. **Eliminated `shared ↔ security` circular dependency**: Moved `rbacHelpers.py` from `shared` to `security` module since it was only used in `aicore` and `aicore` already imports from `security`. This eliminates the architectural violation where `shared` imported from `security`.
|
|
||||||
3. **Eliminated `datamodels ↔ shared` circular dependency**: `shared` no longer has any static imports from `datamodels`. The only reference is a dynamic import in `attributeUtils.py` using `importlib.import_module()` for runtime model discovery, which is not detected by static analysis. This is acceptable as it's a runtime-only dependency.
|
|
||||||
4. **New `interfaces/interfaceRbac.py` module**: Created to handle RBAC filtering for interfaces, importing from both `security` and `connectors`. This maintains proper architectural layering where connectors remain generic.
|
|
||||||
5. **Updated dependency counts**:
|
|
||||||
- `interfaces` → `connectors`: increased from 9 to 10 (interfaceRbac imports connectorDbPostgre)
|
|
||||||
- `interfaces` → `security`: increased from 7 to 9 (interfaceRbac imports rbac and rootAccess)
|
|
||||||
- `features` → `interfaces`: increased from 1 to 2 (mainWorkflow imports interfaceRbac)
|
|
||||||
- `routes` → `interfaces`: increased from 28 to 29 (routeWorkflows imports interfaceRbac)
|
|
||||||
- `aicore` → `security`: increased from 1 to 2 (now imports rbacHelpers from security)
|
|
||||||
- `security` → `datamodels`: increased from 3 to 5 (rbacHelpers adds datamodel imports)
|
|
||||||
|
|
||||||
## Dependency Graph (Mermaid)
|
|
||||||
|
|
||||||
```mermaid
|
|
||||||
graph TD
|
|
||||||
aicore[aicore]
|
|
||||||
auth[auth]
|
|
||||||
connectors[connectors]
|
|
||||||
datamodels[datamodels]
|
|
||||||
features[features]
|
|
||||||
interfaces[interfaces]
|
|
||||||
routes[routes]
|
|
||||||
security[security]
|
|
||||||
services[services]
|
|
||||||
shared[shared]
|
|
||||||
workflows[workflows]
|
|
||||||
|
|
||||||
aicore -->|13| datamodels
|
|
||||||
aicore -->|1| connectors
|
|
||||||
aicore -->|2| security
|
|
||||||
aicore -->|5| shared
|
|
||||||
|
|
||||||
auth -->|5| datamodels
|
|
||||||
auth -->|4| interfaces
|
|
||||||
auth -->|4| security
|
|
||||||
auth -->|8| shared
|
|
||||||
|
|
||||||
connectors -->|4| datamodels
|
|
||||||
connectors -->|5| shared
|
|
||||||
|
|
||||||
datamodels -->|19| shared
|
|
||||||
|
|
||||||
features -->|6| datamodels
|
|
||||||
features -->|0| interfaces
|
|
||||||
features -->|4| services
|
|
||||||
features -->|3| shared
|
|
||||||
features -->|1| workflows
|
|
||||||
|
|
||||||
interfaces -->|29| datamodels
|
|
||||||
interfaces -->|10| connectors
|
|
||||||
interfaces -->|2| aicore
|
|
||||||
interfaces -->|9| security
|
|
||||||
interfaces -->|11| shared
|
|
||||||
|
|
||||||
routes -->|48| datamodels
|
|
||||||
routes -->|29| interfaces
|
|
||||||
routes -->|32| auth
|
|
||||||
routes -->|21| shared
|
|
||||||
routes -->|6| features
|
|
||||||
routes -->|5| services
|
|
||||||
|
|
||||||
security -->|5| datamodels
|
|
||||||
security -->|2| connectors
|
|
||||||
security -->|1| shared
|
|
||||||
|
|
||||||
services -->|52| datamodels
|
|
||||||
services -->|8| interfaces
|
|
||||||
services -->|2| aicore
|
|
||||||
services -->|1| auth
|
|
||||||
services -->|16| shared
|
|
||||||
|
|
||||||
|
|
||||||
workflows -->|72| datamodels
|
|
||||||
workflows -->|1| services
|
|
||||||
workflows -->|9| shared
|
|
||||||
```
|
|
||||||
|
|
||||||
## Detailed Module Dependencies
|
|
||||||
|
|
||||||
### aicore
|
|
||||||
**Imports from:**
|
|
||||||
- `connectors` (1 import)
|
|
||||||
- `datamodels` (13 imports)
|
|
||||||
- `security` (2 imports: rbac, rbacHelpers)
|
|
||||||
- `shared` (4 imports)
|
|
||||||
|
|
||||||
**Dependencies:** Low-level AI functionality, depends on data models and connectors.
|
|
||||||
|
|
||||||
### auth
|
|
||||||
**Imports from:**
|
|
||||||
- `datamodels` (5 imports)
|
|
||||||
- `interfaces` (4 imports)
|
|
||||||
- `security` (4 imports)
|
|
||||||
- `shared` (8 imports)
|
|
||||||
|
|
||||||
**Dependencies:** High-level authentication and token management, used by routes and services.
|
|
||||||
|
|
||||||
### connectors
|
|
||||||
**Imports from:**
|
|
||||||
- `datamodels` (4 imports)
|
|
||||||
- `shared` (5 imports)
|
|
||||||
|
|
||||||
**Dependencies:** External service connectors, minimal dependencies. No longer imports from security or interfaces. Connectors are now fully generic and do not depend on security modules.
|
|
||||||
|
|
||||||
### datamodels
|
|
||||||
**Imports from:**
|
|
||||||
- `shared` (19 imports)
|
|
||||||
|
|
||||||
**Dependencies:** Core data models, only depends on shared utilities.
|
|
||||||
|
|
||||||
### features
|
|
||||||
**Imports from:**
|
|
||||||
- `datamodels` (6 imports)
|
|
||||||
- `services` (4 imports)
|
|
||||||
- `shared` (3 imports)
|
|
||||||
- `workflows` (1 import)
|
|
||||||
|
|
||||||
**Dependencies:** Feature modules that orchestrate workflows and services. Features now use services exclusively, not interfaces directly, maintaining proper architectural layering.
|
|
||||||
|
|
||||||
### interfaces
|
|
||||||
**Imports from:**
|
|
||||||
- `aicore` (2 imports)
|
|
||||||
- `connectors` (10 imports)
|
|
||||||
- `datamodels` (29 imports)
|
|
||||||
- `security` (9 imports)
|
|
||||||
- `shared` (11 imports)
|
|
||||||
|
|
||||||
**Dependencies:** Database and service interfaces, heavily depends on data models. Includes `interfaceRbac.py` which handles RBAC filtering for all interfaces. No longer creates circular dependency with connectors.
|
|
||||||
|
|
||||||
### routes
|
|
||||||
**Imports from:**
|
|
||||||
- `auth` (32 imports)
|
|
||||||
- `datamodels` (48 imports)
|
|
||||||
- `features` (6 imports)
|
|
||||||
- `interfaces` (29 imports)
|
|
||||||
- `services` (5 imports)
|
|
||||||
- `shared` (21 imports)
|
|
||||||
|
|
||||||
**Dependencies:** API endpoints, highest dependency count, orchestrates all layers. Now imports from `auth` instead of `security` for authentication. Increased use of services (from 2 to 5 imports) after architectural refactoring to use services instead of direct interface access in features.
|
|
||||||
|
|
||||||
### security
|
|
||||||
**Imports from:**
|
|
||||||
- `connectors` (2 imports)
|
|
||||||
- `datamodels` (5 imports: rbac uses 3, rbacHelpers uses 2)
|
|
||||||
- `shared` (1 import: rootAccess uses configuration)
|
|
||||||
|
|
||||||
**Dependencies:** Low-level core security (RBAC, root access, and RBAC helper functions). Used by interfaces (including `interfaceRbac.py`), auth, and aicore. The `rbacHelpers` module was moved from `shared` to `security` to eliminate the architectural violation where `shared` imported from `security`. Security imports from connectors only for `rootAccess` to create `DatabaseConnector` instances - this is acceptable as it's a one-way dependency (security → connectors).
|
|
||||||
|
|
||||||
### services
|
|
||||||
**Imports from:**
|
|
||||||
- `aicore` (2 imports)
|
|
||||||
- `auth` (1 import)
|
|
||||||
- `datamodels` (52 imports)
|
|
||||||
- `interfaces` (8 imports)
|
|
||||||
- `shared` (16 imports)
|
|
||||||
|
|
||||||
**Dependencies:** Business logic services, heavily depends on data models.
|
|
||||||
|
|
||||||
### shared
|
|
||||||
**Imports from:**
|
|
||||||
- None (0 imports)
|
|
||||||
|
|
||||||
**Dependencies:** Shared utilities, completely self-contained with no dependencies on other modules. No longer imports from security (rbacHelpers was moved to security module) or datamodels (only uses dynamic imports at runtime for model discovery in `attributeUtils.py`), maintaining proper architectural layering.
|
|
||||||
|
|
||||||
### workflows
|
|
||||||
**Imports from:**
|
|
||||||
- `datamodels` (72 imports)
|
|
||||||
- `services` (1 import)
|
|
||||||
- `shared` (9 imports)
|
|
||||||
|
|
||||||
**Dependencies:** Workflow processing, heavily depends on data models (highest count). Reduced from 74 to 72 imports after removing unused imports from `contentValidator.py`.
|
|
||||||
|
|
||||||
## Key Observations
|
|
||||||
|
|
||||||
1. **datamodels** is the most imported module (used by 9 out of 11 modules)
|
|
||||||
2. **shared** is widely used but has minimal dependencies (good design)
|
|
||||||
3. **routes** has the most diverse dependencies (imports from 6 different modules)
|
|
||||||
4. **workflows** has the highest number of imports from datamodels (72)
|
|
||||||
5. **auth** is now a separate module, used exclusively by routes and services
|
|
||||||
6. **security** is now a low-level module, used by interfaces (including `interfaceRbac.py`)
|
|
||||||
7. **connectors** are now fully generic - no dependencies on security or interfaces
|
|
||||||
8. **Circular dependencies eliminated**: Reduced from 3 to 0 after RBAC refactoring and `rbacHelpers` move (eliminated `connectors ↔ security`, `shared ↔ security`, and `datamodels ↔ shared`)
|
|
||||||
9. **New `interfaceRbac.py` module** centralizes RBAC filtering logic for all interfaces
|
|
||||||
10. **`shared` module is now completely self-contained** - no static imports from any other module
|
|
||||||
11. **Features architectural improvements**: Features no longer import directly from interfaces (reduced from 2 to 0). All features now use services exclusively, maintaining proper layering: Features → Services → Interfaces → Connectors
|
|
||||||
12. **Routes increased services usage**: Routes now import from services 5 times (up from 2) after refactoring features to use services instead of direct interface access
|
|
||||||
|
|
||||||
## Dependency Layers
|
|
||||||
|
|
||||||
Based on the analysis, the architecture follows these layers:
|
|
||||||
|
|
||||||
1. **Foundation Layer**: `shared`, `datamodels`
|
|
||||||
2. **Core Layer**: `aicore`, `connectors`, `security`
|
|
||||||
3. **Interface Layer**: `interfaces`
|
|
||||||
4. **Authentication Layer**: `auth`
|
|
||||||
5. **Business Logic Layer**: `services`, `workflows`
|
|
||||||
6. **Feature Layer**: `features`
|
|
||||||
7. **API Layer**: `routes`
|
|
||||||
|
|
||||||
## Recommendations
|
|
||||||
|
|
||||||
1. **datamodels** should remain stable as it's a core dependency
|
|
||||||
2. **shared** is excellently designed - completely self-contained with zero dependencies (perfect foundation layer)
|
|
||||||
3. **security** split and RBAC refactoring were successful - eliminated all circular dependencies (`connectors ↔ security`, `shared ↔ security`)
|
|
||||||
4. **connectors** are now fully generic and maintainable - keep them free of security/interface dependencies
|
|
||||||
5. **interfaceRbac.py** successfully centralizes RBAC logic - consider this pattern for other cross-cutting concerns
|
|
||||||
6. Consider breaking down **workflows** if it continues to grow
|
|
||||||
7. **routes** could benefit from further abstraction to reduce direct dependencies
|
|
||||||
8. **Architecture is now clean** - no circular dependencies remain, maintaining clear separation of concerns
|
|
||||||
88
modules/datamodels/datamodelWorkflowActions.py
Normal file
88
modules/datamodels/datamodelWorkflowActions.py
Normal file
|
|
@ -0,0 +1,88 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
"""Workflow Action models: WorkflowActionParameter, WorkflowActionDefinition."""
|
||||||
|
|
||||||
|
from typing import Optional, Any, Union, List, Dict, Callable, Awaitable
|
||||||
|
from pydantic import BaseModel, Field
|
||||||
|
from modules.datamodels.datamodelChat import ActionResult
|
||||||
|
from modules.shared.frontendTypes import FrontendType
|
||||||
|
from modules.shared.attributeUtils import registerModelLabels
|
||||||
|
|
||||||
|
|
||||||
|
class WorkflowActionParameter(BaseModel):
|
||||||
|
"""
|
||||||
|
Parameter schema definition for a workflow action.
|
||||||
|
|
||||||
|
This defines the structure and UI rendering for a single action parameter,
|
||||||
|
NOT the actual parameter values (those are in ActionDefinition.parameters).
|
||||||
|
"""
|
||||||
|
name: str = Field(description="Parameter name")
|
||||||
|
type: str = Field(description="Python type as string: 'str', 'int', 'bool', 'List[str]', etc.")
|
||||||
|
frontendType: FrontendType = Field(description="UI rendering type (from global FrontendType enum)")
|
||||||
|
frontendOptions: Optional[Union[str, List[Dict[str, Any]]]] = Field(
|
||||||
|
None,
|
||||||
|
description="Options for select/multiselect/custom types. String reference (e.g., 'user.connection') or static list. For custom types, this is automatically set to the API endpoint."
|
||||||
|
)
|
||||||
|
required: bool = Field(False, description="Whether parameter is required")
|
||||||
|
default: Optional[Any] = Field(None, description="Default value")
|
||||||
|
description: str = Field("", description="Parameter description")
|
||||||
|
validation: Optional[Dict[str, Any]] = Field(
|
||||||
|
None,
|
||||||
|
description="Validation rules (e.g., {'min': 1, 'max': 100})"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class WorkflowActionDefinition(BaseModel):
|
||||||
|
"""
|
||||||
|
Complete schema definition of a workflow action.
|
||||||
|
|
||||||
|
This defines the metadata, parameters, and execution function for an action.
|
||||||
|
This is different from datamodelWorkflow.ActionDefinition which contains
|
||||||
|
actual execution values (action, actionObjective, parameters with values).
|
||||||
|
|
||||||
|
This class defines the ACTION SCHEMA, not the execution plan.
|
||||||
|
"""
|
||||||
|
actionId: str = Field(
|
||||||
|
description="Unique action identifier for RBAC (format: 'module.actionName', e.g., 'outlook.readEmails')"
|
||||||
|
)
|
||||||
|
description: str = Field(description="Action description")
|
||||||
|
parameters: Dict[str, WorkflowActionParameter] = Field(
|
||||||
|
default_factory=dict,
|
||||||
|
description="Parameter schema definitions"
|
||||||
|
)
|
||||||
|
execute: Optional[Callable] = Field(
|
||||||
|
None,
|
||||||
|
description="Execution function - async function that takes parameters dict and returns ActionResult. Set dynamically."
|
||||||
|
)
|
||||||
|
category: Optional[str] = Field(None, description="Action category for grouping")
|
||||||
|
tags: List[str] = Field(default_factory=list, description="Tags for search/filtering")
|
||||||
|
|
||||||
|
|
||||||
|
# Register model labels for UI
|
||||||
|
registerModelLabels(
|
||||||
|
"WorkflowActionDefinition",
|
||||||
|
{"en": "Workflow Action Definition", "fr": "Définition d'action de workflow"},
|
||||||
|
{
|
||||||
|
"actionId": {"en": "Action ID", "fr": "ID d'action"},
|
||||||
|
"description": {"en": "Description", "fr": "Description"},
|
||||||
|
"parameters": {"en": "Parameters", "fr": "Paramètres"},
|
||||||
|
"category": {"en": "Category", "fr": "Catégorie"},
|
||||||
|
"tags": {"en": "Tags", "fr": "Étiquettes"},
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
registerModelLabels(
|
||||||
|
"WorkflowActionParameter",
|
||||||
|
{"en": "Workflow Action Parameter", "fr": "Paramètre d'action de workflow"},
|
||||||
|
{
|
||||||
|
"name": {"en": "Name", "fr": "Nom"},
|
||||||
|
"type": {"en": "Type", "fr": "Type"},
|
||||||
|
"frontendType": {"en": "Frontend Type", "fr": "Type frontend"},
|
||||||
|
"frontendOptions": {"en": "Frontend Options", "fr": "Options frontend"},
|
||||||
|
"required": {"en": "Required", "fr": "Requis"},
|
||||||
|
"default": {"en": "Default", "fr": "Par défaut"},
|
||||||
|
"description": {"en": "Description", "fr": "Description"},
|
||||||
|
"validation": {"en": "Validation", "fr": "Validation"},
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
@ -233,6 +233,9 @@ def initRbacRules(db: DatabaseConnector) -> None:
|
||||||
# Create RESOURCE context rules
|
# Create RESOURCE context rules
|
||||||
createResourceContextRules(db)
|
createResourceContextRules(db)
|
||||||
|
|
||||||
|
# Create Action-specific RBAC rules
|
||||||
|
createActionRules(db)
|
||||||
|
|
||||||
logger.info("RBAC rules initialization completed")
|
logger.info("RBAC rules initialization completed")
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -785,6 +788,108 @@ def createResourceContextRules(db: DatabaseConnector) -> None:
|
||||||
logger.info(f"Created {len(resourceRules)} RESOURCE context rules")
|
logger.info(f"Created {len(resourceRules)} RESOURCE context rules")
|
||||||
|
|
||||||
|
|
||||||
|
def createActionRules(db: DatabaseConnector) -> None:
|
||||||
|
"""
|
||||||
|
Create default RBAC rules for workflow actions.
|
||||||
|
|
||||||
|
This function dynamically discovers all available actions from all methods
|
||||||
|
and creates RBAC rules for them. Actions are protected via RESOURCE context
|
||||||
|
with actionId as the item identifier (format: 'module.actionName').
|
||||||
|
|
||||||
|
Args:
|
||||||
|
db: Database connector instance
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
# Import method discovery to get all actions
|
||||||
|
from modules.workflows.processing.shared.methodDiscovery import discoverMethods
|
||||||
|
from modules.services import getInterface as getServices
|
||||||
|
from modules.datamodels.datamodelUam import User
|
||||||
|
|
||||||
|
# Create a temporary user context for discovery (will be filtered by RBAC later)
|
||||||
|
# We need to discover methods, but we'll use a minimal user context
|
||||||
|
# In production, this should use a system user or admin user
|
||||||
|
try:
|
||||||
|
# Try to get an admin user for discovery
|
||||||
|
adminUsers = db.getRecordset("User", recordFilter={"roleLabel": "sysadmin"}, limit=1)
|
||||||
|
if adminUsers:
|
||||||
|
tempUser = User(**adminUsers[0])
|
||||||
|
else:
|
||||||
|
# Fallback: create minimal user context
|
||||||
|
tempUser = User(id="system", roleLabel="sysadmin")
|
||||||
|
except:
|
||||||
|
# Fallback: create minimal user context
|
||||||
|
tempUser = User(id="system", roleLabel="sysadmin")
|
||||||
|
|
||||||
|
# Get services and discover methods
|
||||||
|
services = getServices(tempUser, None)
|
||||||
|
discoverMethods(services)
|
||||||
|
|
||||||
|
# Import methods catalog
|
||||||
|
from modules.workflows.processing.shared.methodDiscovery import methods
|
||||||
|
|
||||||
|
# Collect all action IDs
|
||||||
|
allActionIds = []
|
||||||
|
for methodName, methodInfo in methods.items():
|
||||||
|
# Skip duplicate entries (same method stored with full and short name)
|
||||||
|
if methodName.startswith('Method'):
|
||||||
|
continue
|
||||||
|
|
||||||
|
methodInstance = methodInfo['instance']
|
||||||
|
methodActions = methodInstance.actions
|
||||||
|
|
||||||
|
for actionName in methodActions.keys():
|
||||||
|
actionId = f"{methodInstance.name}.{actionName}"
|
||||||
|
allActionIds.append(actionId)
|
||||||
|
|
||||||
|
logger.info(f"Discovered {len(allActionIds)} actions for RBAC rule creation")
|
||||||
|
|
||||||
|
# Define default action access by role
|
||||||
|
# SysAdmin and Admin: Access to all actions
|
||||||
|
# User: Access to common actions (read, search, process, etc.)
|
||||||
|
# Viewer: Read-only actions
|
||||||
|
|
||||||
|
actionRules = []
|
||||||
|
|
||||||
|
# All roles: Generic access to all actions
|
||||||
|
# Using item=None grants access to all resources (all actions) in RESOURCE context
|
||||||
|
|
||||||
|
# SysAdmin: Access to all actions
|
||||||
|
actionRules.append(AccessRule(
|
||||||
|
roleLabel="sysadmin",
|
||||||
|
context=AccessRuleContext.RESOURCE,
|
||||||
|
item=None, # All resources (covers all actions)
|
||||||
|
view=True
|
||||||
|
))
|
||||||
|
|
||||||
|
# Admin: Access to all actions
|
||||||
|
actionRules.append(AccessRule(
|
||||||
|
roleLabel="admin",
|
||||||
|
context=AccessRuleContext.RESOURCE,
|
||||||
|
item=None, # All resources (covers all actions)
|
||||||
|
view=True
|
||||||
|
))
|
||||||
|
|
||||||
|
# User: Access to all actions (generic rights)
|
||||||
|
actionRules.append(AccessRule(
|
||||||
|
roleLabel="user",
|
||||||
|
context=AccessRuleContext.RESOURCE,
|
||||||
|
item=None, # All resources (covers all actions)
|
||||||
|
view=True
|
||||||
|
))
|
||||||
|
|
||||||
|
|
||||||
|
# Create all action rules
|
||||||
|
for rule in actionRules:
|
||||||
|
db.recordCreate(AccessRule, rule)
|
||||||
|
|
||||||
|
logger.info(f"Created {len(actionRules)} action RBAC rules")
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error creating action RBAC rules: {str(e)}", exc_info=True)
|
||||||
|
# Don't fail bootstrap if action rules can't be created
|
||||||
|
# They can be created manually or via migration script
|
||||||
|
|
||||||
|
|
||||||
def _addMissingTableRules(db: DatabaseConnector, existingRules: List[Dict[str, Any]]) -> None:
|
def _addMissingTableRules(db: DatabaseConnector, existingRules: List[Dict[str, Any]]) -> None:
|
||||||
"""
|
"""
|
||||||
Add missing RBAC rules for tables that were added after initial bootstrap.
|
Add missing RBAC rules for tables that were added after initial bootstrap.
|
||||||
|
|
|
||||||
|
|
@ -1574,18 +1574,21 @@ class AppObjects:
|
||||||
self,
|
self,
|
||||||
roleLabel: Optional[str] = None,
|
roleLabel: Optional[str] = None,
|
||||||
context: Optional[AccessRuleContext] = None,
|
context: Optional[AccessRuleContext] = None,
|
||||||
item: Optional[str] = None
|
item: Optional[str] = None,
|
||||||
) -> List[AccessRule]:
|
pagination: Optional[PaginationParams] = None
|
||||||
|
) -> Union[List[AccessRule], PaginatedResult]:
|
||||||
"""
|
"""
|
||||||
Get access rules with optional filters.
|
Get access rules with optional filters and pagination.
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
roleLabel: Optional role label filter
|
roleLabel: Optional role label filter
|
||||||
context: Optional context filter
|
context: Optional context filter
|
||||||
item: Optional item filter
|
item: Optional item filter
|
||||||
|
pagination: Optional pagination parameters. If None, returns all items.
|
||||||
|
|
||||||
Returns:
|
Returns:
|
||||||
List of AccessRule objects
|
If pagination is None: List[AccessRule]
|
||||||
|
If pagination is provided: PaginatedResult with items and metadata
|
||||||
"""
|
"""
|
||||||
try:
|
try:
|
||||||
recordFilter = {}
|
recordFilter = {}
|
||||||
|
|
@ -1596,11 +1599,55 @@ class AppObjects:
|
||||||
if item:
|
if item:
|
||||||
recordFilter["item"] = item
|
recordFilter["item"] = item
|
||||||
|
|
||||||
rules = self.db.getRecordset(AccessRule, recordFilter=recordFilter if recordFilter else None)
|
# Use RBAC filtering
|
||||||
return [AccessRule(**rule) for rule in rules]
|
rules = getRecordsetWithRBAC(
|
||||||
|
self.db,
|
||||||
|
AccessRule,
|
||||||
|
self.currentUser,
|
||||||
|
recordFilter=recordFilter if recordFilter else None
|
||||||
|
)
|
||||||
|
|
||||||
|
# Filter out database-specific fields
|
||||||
|
filteredRules = []
|
||||||
|
for rule in rules:
|
||||||
|
cleanedRule = {k: v for k, v in rule.items() if not k.startswith("_")}
|
||||||
|
filteredRules.append(cleanedRule)
|
||||||
|
|
||||||
|
# If no pagination requested, return all items
|
||||||
|
if pagination is None:
|
||||||
|
return [AccessRule(**rule) for rule in filteredRules]
|
||||||
|
|
||||||
|
# Apply filtering (if filters provided)
|
||||||
|
if pagination.filters:
|
||||||
|
filteredRules = self._applyFilters(filteredRules, pagination.filters)
|
||||||
|
|
||||||
|
# Apply sorting (in order of sortFields)
|
||||||
|
if pagination.sort:
|
||||||
|
filteredRules = self._applySorting(filteredRules, pagination.sort)
|
||||||
|
|
||||||
|
# Count total items after filters
|
||||||
|
totalItems = len(filteredRules)
|
||||||
|
totalPages = math.ceil(totalItems / pagination.pageSize) if totalItems > 0 else 0
|
||||||
|
|
||||||
|
# Apply pagination (skip/limit)
|
||||||
|
startIdx = (pagination.page - 1) * pagination.pageSize
|
||||||
|
endIdx = startIdx + pagination.pageSize
|
||||||
|
pagedRules = filteredRules[startIdx:endIdx]
|
||||||
|
|
||||||
|
# Convert to model objects
|
||||||
|
items = [AccessRule(**rule) for rule in pagedRules]
|
||||||
|
|
||||||
|
return PaginatedResult(
|
||||||
|
items=items,
|
||||||
|
totalItems=totalItems,
|
||||||
|
totalPages=totalPages
|
||||||
|
)
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.error(f"Error getting access rules: {str(e)}")
|
logger.error(f"Error getting access rules: {str(e)}")
|
||||||
return []
|
if pagination is None:
|
||||||
|
return []
|
||||||
|
else:
|
||||||
|
return PaginatedResult(items=[], totalItems=0, totalPages=0)
|
||||||
|
|
||||||
def getAccessRulesForRoles(
|
def getAccessRulesForRoles(
|
||||||
self,
|
self,
|
||||||
|
|
@ -1701,19 +1748,62 @@ class AppObjects:
|
||||||
logger.error(f"Error getting role by label {roleLabel}: {str(e)}")
|
logger.error(f"Error getting role by label {roleLabel}: {str(e)}")
|
||||||
return None
|
return None
|
||||||
|
|
||||||
def getAllRoles(self) -> List[Role]:
|
def getAllRoles(self, pagination: Optional[PaginationParams] = None) -> Union[List[Role], PaginatedResult]:
|
||||||
"""
|
"""
|
||||||
Get all roles.
|
Get all roles with optional pagination, sorting, and filtering.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
pagination: Optional pagination parameters. If None, returns all items.
|
||||||
|
|
||||||
Returns:
|
Returns:
|
||||||
List of Role objects
|
If pagination is None: List[Role]
|
||||||
|
If pagination is provided: PaginatedResult with items and metadata
|
||||||
"""
|
"""
|
||||||
try:
|
try:
|
||||||
|
# Get all roles from database
|
||||||
roles = self.db.getRecordset(Role)
|
roles = self.db.getRecordset(Role)
|
||||||
return [Role(**role) for role in roles]
|
|
||||||
|
# Filter out database-specific fields
|
||||||
|
filteredRoles = []
|
||||||
|
for role in roles:
|
||||||
|
cleanedRole = {k: v for k, v in role.items() if not k.startswith("_")}
|
||||||
|
filteredRoles.append(cleanedRole)
|
||||||
|
|
||||||
|
# If no pagination requested, return all items
|
||||||
|
if pagination is None:
|
||||||
|
return [Role(**role) for role in filteredRoles]
|
||||||
|
|
||||||
|
# Apply filtering (if filters provided)
|
||||||
|
if pagination.filters:
|
||||||
|
filteredRoles = self._applyFilters(filteredRoles, pagination.filters)
|
||||||
|
|
||||||
|
# Apply sorting (in order of sortFields)
|
||||||
|
if pagination.sort:
|
||||||
|
filteredRoles = self._applySorting(filteredRoles, pagination.sort)
|
||||||
|
|
||||||
|
# Count total items after filters
|
||||||
|
totalItems = len(filteredRoles)
|
||||||
|
totalPages = math.ceil(totalItems / pagination.pageSize) if totalItems > 0 else 0
|
||||||
|
|
||||||
|
# Apply pagination (skip/limit)
|
||||||
|
startIdx = (pagination.page - 1) * pagination.pageSize
|
||||||
|
endIdx = startIdx + pagination.pageSize
|
||||||
|
pagedRoles = filteredRoles[startIdx:endIdx]
|
||||||
|
|
||||||
|
# Convert to model objects
|
||||||
|
items = [Role(**role) for role in pagedRoles]
|
||||||
|
|
||||||
|
return PaginatedResult(
|
||||||
|
items=items,
|
||||||
|
totalItems=totalItems,
|
||||||
|
totalPages=totalPages
|
||||||
|
)
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.error(f"Error getting all roles: {str(e)}")
|
logger.error(f"Error getting all roles: {str(e)}")
|
||||||
return []
|
if pagination is None:
|
||||||
|
return []
|
||||||
|
else:
|
||||||
|
return PaginatedResult(items=[], totalItems=0, totalPages=0)
|
||||||
|
|
||||||
def updateRole(self, roleId: str, role: Role) -> Role:
|
def updateRole(self, roleId: str, role: Role) -> Role:
|
||||||
"""
|
"""
|
||||||
|
|
|
||||||
|
|
@ -8,10 +8,13 @@ Implements endpoints for role-based access control permissions.
|
||||||
from fastapi import APIRouter, HTTPException, Depends, Query, Body, Path, Request
|
from fastapi import APIRouter, HTTPException, Depends, Query, Body, Path, Request
|
||||||
from typing import Optional, List, Dict, Any
|
from typing import Optional, List, Dict, Any
|
||||||
import logging
|
import logging
|
||||||
|
import json
|
||||||
|
import math
|
||||||
|
|
||||||
from modules.auth import getCurrentUser, limiter
|
from modules.auth import getCurrentUser, limiter
|
||||||
from modules.datamodels.datamodelUam import User, UserPermissions, AccessLevel
|
from modules.datamodels.datamodelUam import User, UserPermissions, AccessLevel
|
||||||
from modules.datamodels.datamodelRbac import AccessRuleContext, AccessRule, Role
|
from modules.datamodels.datamodelRbac import AccessRuleContext, AccessRule, Role
|
||||||
|
from modules.datamodels.datamodelPagination import PaginationParams, PaginatedResponse, PaginationMetadata
|
||||||
from modules.interfaces.interfaceDbAppObjects import getInterface
|
from modules.interfaces.interfaceDbAppObjects import getInterface
|
||||||
|
|
||||||
# Configure logger
|
# Configure logger
|
||||||
|
|
@ -86,15 +89,16 @@ async def getPermissions(
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@router.get("/rules", response_model=list)
|
@router.get("/rules", response_model=PaginatedResponse)
|
||||||
@limiter.limit("30/minute")
|
@limiter.limit("30/minute")
|
||||||
async def getAccessRules(
|
async def getAccessRules(
|
||||||
request: Request,
|
request: Request,
|
||||||
roleLabel: Optional[str] = Query(None, description="Filter by role label"),
|
roleLabel: Optional[str] = Query(None, description="Filter by role label"),
|
||||||
context: Optional[str] = Query(None, description="Filter by context (DATA, UI, RESOURCE)"),
|
context: Optional[str] = Query(None, description="Filter by context (DATA, UI, RESOURCE)"),
|
||||||
item: Optional[str] = Query(None, description="Filter by item identifier"),
|
item: Optional[str] = Query(None, description="Filter by item identifier"),
|
||||||
|
pagination: Optional[str] = Query(None, description="JSON-encoded PaginationParams object"),
|
||||||
currentUser: User = Depends(getCurrentUser)
|
currentUser: User = Depends(getCurrentUser)
|
||||||
) -> list:
|
) -> PaginatedResponse:
|
||||||
"""
|
"""
|
||||||
Get access rules with optional filters.
|
Get access rules with optional filters.
|
||||||
Only returns rules that the current user has permission to view.
|
Only returns rules that the current user has permission to view.
|
||||||
|
|
@ -143,15 +147,45 @@ async def getAccessRules(
|
||||||
detail=f"Invalid context '{context}'. Must be one of: DATA, UI, RESOURCE"
|
detail=f"Invalid context '{context}'. Must be one of: DATA, UI, RESOURCE"
|
||||||
)
|
)
|
||||||
|
|
||||||
# Get rules
|
# Parse pagination parameter
|
||||||
rules = interface.getAccessRules(
|
paginationParams = None
|
||||||
|
if pagination:
|
||||||
|
try:
|
||||||
|
paginationDict = json.loads(pagination)
|
||||||
|
paginationParams = PaginationParams(**paginationDict) if paginationDict else None
|
||||||
|
except (json.JSONDecodeError, ValueError) as e:
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=400,
|
||||||
|
detail=f"Invalid pagination parameter: {str(e)}"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Get rules with optional pagination
|
||||||
|
result = interface.getAccessRules(
|
||||||
roleLabel=roleLabel,
|
roleLabel=roleLabel,
|
||||||
context=accessContext,
|
context=accessContext,
|
||||||
item=item
|
item=item,
|
||||||
|
pagination=paginationParams
|
||||||
)
|
)
|
||||||
|
|
||||||
# Convert to dict for JSON serialization
|
# If pagination was requested, result is PaginatedResult
|
||||||
return [rule.model_dump() for rule in rules]
|
# If no pagination, result is List[AccessRule]
|
||||||
|
if paginationParams:
|
||||||
|
return PaginatedResponse(
|
||||||
|
items=[rule.model_dump() for rule in result.items],
|
||||||
|
pagination=PaginationMetadata(
|
||||||
|
currentPage=paginationParams.page,
|
||||||
|
pageSize=paginationParams.pageSize,
|
||||||
|
totalItems=result.totalItems,
|
||||||
|
totalPages=result.totalPages,
|
||||||
|
sort=paginationParams.sort,
|
||||||
|
filters=paginationParams.filters
|
||||||
|
)
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
return PaginatedResponse(
|
||||||
|
items=[rule.model_dump() for rule in result],
|
||||||
|
pagination=None
|
||||||
|
)
|
||||||
|
|
||||||
except HTTPException:
|
except HTTPException:
|
||||||
raise
|
raise
|
||||||
|
|
@ -489,12 +523,13 @@ def _ensureAdminAccess(currentUser: User) -> None:
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@router.get("/roles", response_model=List[Dict[str, Any]])
|
@router.get("/roles", response_model=PaginatedResponse)
|
||||||
@limiter.limit("60/minute")
|
@limiter.limit("60/minute")
|
||||||
async def listRoles(
|
async def listRoles(
|
||||||
request: Request,
|
request: Request,
|
||||||
|
pagination: Optional[str] = Query(None, description="JSON-encoded PaginationParams object"),
|
||||||
currentUser: User = Depends(getCurrentUser)
|
currentUser: User = Depends(getCurrentUser)
|
||||||
) -> List[Dict[str, Any]]:
|
) -> PaginatedResponse:
|
||||||
"""
|
"""
|
||||||
Get list of all available roles with metadata.
|
Get list of all available roles with metadata.
|
||||||
|
|
||||||
|
|
@ -506,14 +541,27 @@ async def listRoles(
|
||||||
|
|
||||||
interface = getInterface(currentUser)
|
interface = getInterface(currentUser)
|
||||||
|
|
||||||
# Get all roles from database
|
# Parse pagination parameter
|
||||||
dbRoles = interface.getAllRoles()
|
paginationParams = None
|
||||||
|
if pagination:
|
||||||
|
try:
|
||||||
|
paginationDict = json.loads(pagination)
|
||||||
|
paginationParams = PaginationParams(**paginationDict) if paginationDict else None
|
||||||
|
except (json.JSONDecodeError, ValueError) as e:
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=400,
|
||||||
|
detail=f"Invalid pagination parameter: {str(e)}"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Get all roles from database (without pagination) to enrich with user counts and add custom roles
|
||||||
|
# Note: We get all roles first because we need to add custom roles before pagination
|
||||||
|
dbRoles = interface.getAllRoles(pagination=None)
|
||||||
|
|
||||||
# Get all users to count role assignments
|
# Get all users to count role assignments
|
||||||
# Since _ensureAdminAccess ensures user is sysadmin or admin,
|
# Since _ensureAdminAccess ensures user is sysadmin or admin,
|
||||||
# and getUsersByMandate returns all users for sysadmin regardless of mandateId,
|
# and getUsersByMandate returns all users for sysadmin regardless of mandateId,
|
||||||
# we can pass the current user's mandateId (for sysadmin it will be ignored by RBAC)
|
# we can pass the current user's mandateId (for sysadmin it will be ignored by RBAC)
|
||||||
allUsers = interface.getUsersByMandate(currentUser.mandateId or "")
|
allUsers = interface.getUsersByMandate(currentUser.mandateId or "", pagination=None)
|
||||||
|
|
||||||
# Count users per role
|
# Count users per role
|
||||||
roleCounts: Dict[str, int] = {}
|
roleCounts: Dict[str, int] = {}
|
||||||
|
|
@ -544,7 +592,45 @@ async def listRoles(
|
||||||
"isSystemRole": False
|
"isSystemRole": False
|
||||||
})
|
})
|
||||||
|
|
||||||
return result
|
# Apply filtering and sorting if pagination requested
|
||||||
|
if paginationParams:
|
||||||
|
# Apply filtering (if filters provided)
|
||||||
|
if paginationParams.filters:
|
||||||
|
# Use the interface's filter method
|
||||||
|
filteredResult = interface._applyFilters(result, paginationParams.filters)
|
||||||
|
else:
|
||||||
|
filteredResult = result
|
||||||
|
|
||||||
|
# Apply sorting (in order of sortFields)
|
||||||
|
if paginationParams.sort:
|
||||||
|
sortedResult = interface._applySorting(filteredResult, paginationParams.sort)
|
||||||
|
else:
|
||||||
|
sortedResult = filteredResult
|
||||||
|
|
||||||
|
# Apply pagination
|
||||||
|
totalItems = len(sortedResult)
|
||||||
|
totalPages = math.ceil(totalItems / paginationParams.pageSize) if totalItems > 0 else 0
|
||||||
|
startIdx = (paginationParams.page - 1) * paginationParams.pageSize
|
||||||
|
endIdx = startIdx + paginationParams.pageSize
|
||||||
|
paginatedResult = sortedResult[startIdx:endIdx]
|
||||||
|
|
||||||
|
return PaginatedResponse(
|
||||||
|
items=paginatedResult,
|
||||||
|
pagination=PaginationMetadata(
|
||||||
|
currentPage=paginationParams.page,
|
||||||
|
pageSize=paginationParams.pageSize,
|
||||||
|
totalItems=totalItems,
|
||||||
|
totalPages=totalPages,
|
||||||
|
sort=paginationParams.sort,
|
||||||
|
filters=paginationParams.filters
|
||||||
|
)
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
# No pagination - return all roles
|
||||||
|
return PaginatedResponse(
|
||||||
|
items=result,
|
||||||
|
pagination=None
|
||||||
|
)
|
||||||
|
|
||||||
except HTTPException:
|
except HTTPException:
|
||||||
raise
|
raise
|
||||||
|
|
|
||||||
|
|
@ -572,3 +572,247 @@ async def delete_file_from_message(
|
||||||
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
|
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
|
||||||
detail=f"Error deleting file reference: {str(e)}"
|
detail=f"Error deleting file reference: {str(e)}"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# Action Discovery Endpoints
|
||||||
|
|
||||||
|
@router.get("/actions", response_model=Dict[str, Any])
|
||||||
|
@limiter.limit("120/minute")
|
||||||
|
async def get_all_actions(
|
||||||
|
request: Request,
|
||||||
|
currentUser: User = Depends(getCurrentUser)
|
||||||
|
) -> Dict[str, Any]:
|
||||||
|
"""
|
||||||
|
Get all available workflow actions for the current user (filtered by RBAC).
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
- Dictionary with actions grouped by module, filtered by RBAC permissions
|
||||||
|
|
||||||
|
Example response:
|
||||||
|
{
|
||||||
|
"actions": [
|
||||||
|
{
|
||||||
|
"module": "outlook",
|
||||||
|
"actionId": "outlook.readEmails",
|
||||||
|
"name": "readEmails",
|
||||||
|
"description": "Read emails and metadata from a mailbox folder",
|
||||||
|
"parameters": {...}
|
||||||
|
},
|
||||||
|
...
|
||||||
|
]
|
||||||
|
}
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
from modules.services import getInterface as getServices
|
||||||
|
from modules.workflows.processing.shared.methodDiscovery import discoverMethods
|
||||||
|
|
||||||
|
# Get services and discover methods
|
||||||
|
services = getServices(currentUser, None)
|
||||||
|
discoverMethods(services)
|
||||||
|
|
||||||
|
# Import methods catalog
|
||||||
|
from modules.workflows.processing.shared.methodDiscovery import methods
|
||||||
|
|
||||||
|
# Collect all actions from all methods
|
||||||
|
allActions = []
|
||||||
|
for methodName, methodInfo in methods.items():
|
||||||
|
# Skip duplicate entries (same method stored with full and short name)
|
||||||
|
if methodName.startswith('Method'):
|
||||||
|
continue
|
||||||
|
|
||||||
|
methodInstance = methodInfo['instance']
|
||||||
|
methodActions = methodInstance.actions
|
||||||
|
|
||||||
|
for actionName, actionInfo in methodActions.items():
|
||||||
|
# Build action response
|
||||||
|
actionResponse = {
|
||||||
|
"module": methodInstance.name,
|
||||||
|
"actionId": f"{methodInstance.name}.{actionName}",
|
||||||
|
"name": actionName,
|
||||||
|
"description": actionInfo.get('description', ''),
|
||||||
|
"parameters": actionInfo.get('parameters', {})
|
||||||
|
}
|
||||||
|
allActions.append(actionResponse)
|
||||||
|
|
||||||
|
return {
|
||||||
|
"actions": allActions
|
||||||
|
}
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error getting all actions: {str(e)}", exc_info=True)
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
|
||||||
|
detail=f"Failed to get actions: {str(e)}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/actions/{method}", response_model=Dict[str, Any])
|
||||||
|
@limiter.limit("120/minute")
|
||||||
|
async def get_method_actions(
|
||||||
|
request: Request,
|
||||||
|
method: str = Path(..., description="Method name (e.g., 'outlook', 'sharepoint')"),
|
||||||
|
currentUser: User = Depends(getCurrentUser)
|
||||||
|
) -> Dict[str, Any]:
|
||||||
|
"""
|
||||||
|
Get all available actions for a specific method (filtered by RBAC).
|
||||||
|
|
||||||
|
Path Parameters:
|
||||||
|
- method: Method name (e.g., 'outlook', 'sharepoint', 'ai')
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
- Dictionary with actions for the specified method
|
||||||
|
|
||||||
|
Example response:
|
||||||
|
{
|
||||||
|
"module": "outlook",
|
||||||
|
"actions": [
|
||||||
|
{
|
||||||
|
"actionId": "outlook.readEmails",
|
||||||
|
"name": "readEmails",
|
||||||
|
"description": "Read emails and metadata from a mailbox folder",
|
||||||
|
"parameters": {...}
|
||||||
|
},
|
||||||
|
...
|
||||||
|
]
|
||||||
|
}
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
from modules.services import getInterface as getServices
|
||||||
|
from modules.workflows.processing.shared.methodDiscovery import discoverMethods
|
||||||
|
|
||||||
|
# Get services and discover methods
|
||||||
|
services = getServices(currentUser, None)
|
||||||
|
discoverMethods(services)
|
||||||
|
|
||||||
|
# Import methods catalog
|
||||||
|
from modules.workflows.processing.shared.methodDiscovery import methods
|
||||||
|
|
||||||
|
# Find method instance
|
||||||
|
methodInstance = None
|
||||||
|
for methodName, methodInfo in methods.items():
|
||||||
|
if methodInfo['instance'].name == method:
|
||||||
|
methodInstance = methodInfo['instance']
|
||||||
|
break
|
||||||
|
|
||||||
|
if not methodInstance:
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=status.HTTP_404_NOT_FOUND,
|
||||||
|
detail=f"Method '{method}' not found"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Collect actions for this method
|
||||||
|
actions = []
|
||||||
|
methodActions = methodInstance.actions
|
||||||
|
|
||||||
|
for actionName, actionInfo in methodActions.items():
|
||||||
|
actionResponse = {
|
||||||
|
"actionId": f"{methodInstance.name}.{actionName}",
|
||||||
|
"name": actionName,
|
||||||
|
"description": actionInfo.get('description', ''),
|
||||||
|
"parameters": actionInfo.get('parameters', {})
|
||||||
|
}
|
||||||
|
actions.append(actionResponse)
|
||||||
|
|
||||||
|
return {
|
||||||
|
"module": methodInstance.name,
|
||||||
|
"description": methodInstance.description,
|
||||||
|
"actions": actions
|
||||||
|
}
|
||||||
|
|
||||||
|
except HTTPException:
|
||||||
|
raise
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error getting actions for method {method}: {str(e)}", exc_info=True)
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
|
||||||
|
detail=f"Failed to get actions for method {method}: {str(e)}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/actions/{method}/{action}", response_model=Dict[str, Any])
|
||||||
|
@limiter.limit("120/minute")
|
||||||
|
async def get_action_schema(
|
||||||
|
request: Request,
|
||||||
|
method: str = Path(..., description="Method name (e.g., 'outlook', 'sharepoint')"),
|
||||||
|
action: str = Path(..., description="Action name (e.g., 'readEmails', 'uploadDocument')"),
|
||||||
|
currentUser: User = Depends(getCurrentUser)
|
||||||
|
) -> Dict[str, Any]:
|
||||||
|
"""
|
||||||
|
Get action schema with parameter definitions for a specific action.
|
||||||
|
|
||||||
|
Path Parameters:
|
||||||
|
- method: Method name (e.g., 'outlook', 'sharepoint', 'ai')
|
||||||
|
- action: Action name (e.g., 'readEmails', 'uploadDocument')
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
- Action schema with full parameter definitions
|
||||||
|
|
||||||
|
Example response:
|
||||||
|
{
|
||||||
|
"method": "outlook",
|
||||||
|
"action": "readEmails",
|
||||||
|
"actionId": "outlook.readEmails",
|
||||||
|
"description": "Read emails and metadata from a mailbox folder",
|
||||||
|
"parameters": {
|
||||||
|
"connectionReference": {
|
||||||
|
"name": "connectionReference",
|
||||||
|
"type": "str",
|
||||||
|
"frontendType": "userConnection",
|
||||||
|
"frontendOptions": "user.connection",
|
||||||
|
"required": true,
|
||||||
|
"description": "Microsoft connection label"
|
||||||
|
},
|
||||||
|
...
|
||||||
|
}
|
||||||
|
}
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
from modules.services import getInterface as getServices
|
||||||
|
from modules.workflows.processing.shared.methodDiscovery import discoverMethods
|
||||||
|
|
||||||
|
# Get services and discover methods
|
||||||
|
services = getServices(currentUser, None)
|
||||||
|
discoverMethods(services)
|
||||||
|
|
||||||
|
# Import methods catalog
|
||||||
|
from modules.workflows.processing.shared.methodDiscovery import methods
|
||||||
|
|
||||||
|
# Find method instance
|
||||||
|
methodInstance = None
|
||||||
|
for methodName, methodInfo in methods.items():
|
||||||
|
if methodInfo['instance'].name == method:
|
||||||
|
methodInstance = methodInfo['instance']
|
||||||
|
break
|
||||||
|
|
||||||
|
if not methodInstance:
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=status.HTTP_404_NOT_FOUND,
|
||||||
|
detail=f"Method '{method}' not found"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Get action
|
||||||
|
methodActions = methodInstance.actions
|
||||||
|
if action not in methodActions:
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=status.HTTP_404_NOT_FOUND,
|
||||||
|
detail=f"Action '{action}' not found in method '{method}'"
|
||||||
|
)
|
||||||
|
|
||||||
|
actionInfo = methodActions[action]
|
||||||
|
|
||||||
|
return {
|
||||||
|
"method": methodInstance.name,
|
||||||
|
"action": action,
|
||||||
|
"actionId": f"{methodInstance.name}.{action}",
|
||||||
|
"description": actionInfo.get('description', ''),
|
||||||
|
"parameters": actionInfo.get('parameters', {})
|
||||||
|
}
|
||||||
|
|
||||||
|
except HTTPException:
|
||||||
|
raise
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error getting action schema for {method}.{action}: {str(e)}", exc_info=True)
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
|
||||||
|
detail=f"Failed to get action schema: {str(e)}"
|
||||||
|
)
|
||||||
7
modules/workflows/methods/methodAi/__init__.py
Normal file
7
modules/workflows/methods/methodAi/__init__.py
Normal file
|
|
@ -0,0 +1,7 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
from .methodAi import MethodAi
|
||||||
|
|
||||||
|
__all__ = ['MethodAi']
|
||||||
|
|
||||||
26
modules/workflows/methods/methodAi/actions/__init__.py
Normal file
26
modules/workflows/methods/methodAi/actions/__init__.py
Normal file
|
|
@ -0,0 +1,26 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
"""Action modules for AI operations."""
|
||||||
|
|
||||||
|
# Export all actions
|
||||||
|
from .process import process
|
||||||
|
from .webResearch import webResearch
|
||||||
|
from .summarizeDocument import summarizeDocument
|
||||||
|
from .translateDocument import translateDocument
|
||||||
|
from .convert import convert
|
||||||
|
from .convertDocument import convertDocument
|
||||||
|
from .extractData import extractData
|
||||||
|
from .generateDocument import generateDocument
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
'process',
|
||||||
|
'webResearch',
|
||||||
|
'summarizeDocument',
|
||||||
|
'translateDocument',
|
||||||
|
'convert',
|
||||||
|
'convertDocument',
|
||||||
|
'extractData',
|
||||||
|
'generateDocument',
|
||||||
|
]
|
||||||
|
|
||||||
157
modules/workflows/methods/methodAi/actions/convert.py
Normal file
157
modules/workflows/methods/methodAi/actions/convert.py
Normal file
|
|
@ -0,0 +1,157 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
"""
|
||||||
|
Convert action for AI operations.
|
||||||
|
Converts documents/data between different formats with specific formatting options.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import json
|
||||||
|
from typing import Dict, Any
|
||||||
|
from modules.workflows.methods.methodBase import action
|
||||||
|
from modules.datamodels.datamodelChat import ActionResult, ActionDocument
|
||||||
|
from modules.datamodels.datamodelDocref import DocumentReferenceList
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
@action
|
||||||
|
async def convert(self, parameters: Dict[str, Any]) -> ActionResult:
|
||||||
|
"""
|
||||||
|
GENERAL:
|
||||||
|
- Purpose: Convert documents/data between different formats with specific formatting options (e.g., JSON→CSV with custom columns, delimiters).
|
||||||
|
- Input requirements: documentList (required); inputFormat and outputFormat (required).
|
||||||
|
- Output format: Document in target format with specified formatting options.
|
||||||
|
- CRITICAL: If input is already in standardized JSON format, uses automatic rendering system (no AI call needed).
|
||||||
|
|
||||||
|
Parameters:
|
||||||
|
- documentList (list, required): Document reference(s) to convert.
|
||||||
|
- inputFormat (str, required): Source format (json, csv, xlsx, txt, etc.).
|
||||||
|
- outputFormat (str, required): Target format (csv, json, xlsx, txt, etc.).
|
||||||
|
- columnsPerRow (int, optional): For CSV output, number of columns per row. Default: auto-detect.
|
||||||
|
- delimiter (str, optional): For CSV output, delimiter character. Default: comma (,).
|
||||||
|
- includeHeader (bool, optional): For CSV output, whether to include header row. Default: True.
|
||||||
|
- language (str, optional): Language for output (e.g., 'de', 'en', 'fr'). Default: 'en'.
|
||||||
|
"""
|
||||||
|
documentList = parameters.get("documentList", [])
|
||||||
|
if not documentList:
|
||||||
|
return ActionResult.isFailure(error="documentList is required")
|
||||||
|
|
||||||
|
inputFormat = parameters.get("inputFormat")
|
||||||
|
outputFormat = parameters.get("outputFormat")
|
||||||
|
if not inputFormat or not outputFormat:
|
||||||
|
return ActionResult.isFailure(error="inputFormat and outputFormat are required")
|
||||||
|
|
||||||
|
# Normalize formats (remove leading dot if present)
|
||||||
|
normalizedInputFormat = inputFormat.strip().lstrip('.').lower()
|
||||||
|
normalizedOutputFormat = outputFormat.strip().lstrip('.').lower()
|
||||||
|
|
||||||
|
# Get documents
|
||||||
|
if isinstance(documentList, DocumentReferenceList):
|
||||||
|
docRefList = documentList
|
||||||
|
elif isinstance(documentList, list):
|
||||||
|
docRefList = DocumentReferenceList.from_string_list(documentList)
|
||||||
|
else:
|
||||||
|
docRefList = DocumentReferenceList.from_string_list([documentList])
|
||||||
|
|
||||||
|
chatDocuments = self.services.chat.getChatDocumentsFromDocumentList(docRefList)
|
||||||
|
if not chatDocuments:
|
||||||
|
return ActionResult.isFailure(error="No documents found in documentList")
|
||||||
|
|
||||||
|
# Check if input is standardized JSON format - if so, use direct rendering
|
||||||
|
if normalizedInputFormat == "json" and len(chatDocuments) == 1:
|
||||||
|
try:
|
||||||
|
doc = chatDocuments[0]
|
||||||
|
# ChatDocument doesn't have documentData - need to load file content using fileId
|
||||||
|
docBytes = self.services.chat.getFileData(doc.fileId)
|
||||||
|
if not docBytes:
|
||||||
|
raise ValueError(f"No file data found for fileId={doc.fileId}")
|
||||||
|
|
||||||
|
# Decode bytes to string
|
||||||
|
docData = docBytes.decode('utf-8')
|
||||||
|
|
||||||
|
# Try to parse as JSON
|
||||||
|
if isinstance(docData, str):
|
||||||
|
jsonData = json.loads(docData)
|
||||||
|
elif isinstance(docData, dict):
|
||||||
|
jsonData = docData
|
||||||
|
else:
|
||||||
|
jsonData = None
|
||||||
|
|
||||||
|
# Check if it's standardized JSON format (has "documents" or "sections")
|
||||||
|
if jsonData and (isinstance(jsonData, dict) and ("documents" in jsonData or "sections" in jsonData)):
|
||||||
|
# Use direct rendering - no AI call needed!
|
||||||
|
from modules.services.serviceGeneration.mainServiceGeneration import GenerationService
|
||||||
|
generationService = GenerationService(self.services)
|
||||||
|
|
||||||
|
# Ensure format is "documents" array
|
||||||
|
if "documents" not in jsonData:
|
||||||
|
jsonData = {"documents": [{"sections": jsonData.get("sections", []), "metadata": jsonData.get("metadata", {})}]}
|
||||||
|
|
||||||
|
# Get title
|
||||||
|
title = jsonData.get("metadata", {}).get("title", doc.documentName or "Converted Document")
|
||||||
|
|
||||||
|
# Render with options
|
||||||
|
renderOptions = {}
|
||||||
|
if normalizedOutputFormat == "csv":
|
||||||
|
renderOptions["delimiter"] = parameters.get("delimiter", ",")
|
||||||
|
renderOptions["columnsPerRow"] = parameters.get("columnsPerRow")
|
||||||
|
renderOptions["includeHeader"] = parameters.get("includeHeader", True)
|
||||||
|
|
||||||
|
rendered_content, mime_type = await generationService.renderReport(
|
||||||
|
jsonData, normalizedOutputFormat, title, None, None
|
||||||
|
)
|
||||||
|
|
||||||
|
# Apply CSV options if needed (renderer will handle them)
|
||||||
|
if normalizedOutputFormat == "csv" and renderOptions:
|
||||||
|
rendered_content = self.csvProcessing.applyCsvOptions(rendered_content, renderOptions)
|
||||||
|
|
||||||
|
validationMetadata = {
|
||||||
|
"actionType": "ai.convert",
|
||||||
|
"inputFormat": normalizedInputFormat,
|
||||||
|
"outputFormat": normalizedOutputFormat,
|
||||||
|
"hasSourceJson": True,
|
||||||
|
"conversionType": "direct_rendering"
|
||||||
|
}
|
||||||
|
actionDoc = ActionDocument(
|
||||||
|
documentName=f"{doc.documentName.rsplit('.', 1)[0] if '.' in doc.documentName else doc.documentName}.{normalizedOutputFormat}",
|
||||||
|
documentData=rendered_content,
|
||||||
|
mimeType=mime_type,
|
||||||
|
sourceJson=jsonData, # Preserve source JSON for structure validation
|
||||||
|
validationMetadata=validationMetadata
|
||||||
|
)
|
||||||
|
|
||||||
|
return ActionResult.isSuccess(documents=[actionDoc])
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Direct rendering failed, falling back to AI conversion: {str(e)}")
|
||||||
|
# Fall through to AI-based conversion
|
||||||
|
|
||||||
|
# Fallback: Use AI for conversion (for non-JSON inputs or complex conversions)
|
||||||
|
columnsPerRow = parameters.get("columnsPerRow")
|
||||||
|
delimiter = parameters.get("delimiter", ",")
|
||||||
|
includeHeader = parameters.get("includeHeader", True)
|
||||||
|
language = parameters.get("language", "en")
|
||||||
|
|
||||||
|
aiPrompt = f"Convert the provided document(s) from {normalizedInputFormat.upper()} format to {normalizedOutputFormat.upper()} format."
|
||||||
|
|
||||||
|
if normalizedOutputFormat == "csv":
|
||||||
|
aiPrompt += f" Use '{delimiter}' as the delimiter character."
|
||||||
|
if columnsPerRow:
|
||||||
|
aiPrompt += f" Format the output with {columnsPerRow} columns per row."
|
||||||
|
if not includeHeader:
|
||||||
|
aiPrompt += " Do not include a header row."
|
||||||
|
else:
|
||||||
|
aiPrompt += " Include a header row with column names."
|
||||||
|
|
||||||
|
if language and language != "en":
|
||||||
|
aiPrompt += f" Use language: {language}."
|
||||||
|
|
||||||
|
aiPrompt += " Preserve all data and ensure accurate conversion. Maintain data integrity and structure."
|
||||||
|
|
||||||
|
return await self.process({
|
||||||
|
"aiPrompt": aiPrompt,
|
||||||
|
"documentList": documentList,
|
||||||
|
"resultType": normalizedOutputFormat
|
||||||
|
})
|
||||||
|
|
||||||
|
|
@ -0,0 +1,52 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
"""
|
||||||
|
Convert Document action for AI operations.
|
||||||
|
Converts documents between different formats (PDF→Word, Excel→CSV, etc.).
|
||||||
|
"""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
from typing import Dict, Any
|
||||||
|
from modules.workflows.methods.methodBase import action
|
||||||
|
from modules.datamodels.datamodelChat import ActionResult
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
@action
|
||||||
|
async def convertDocument(self, parameters: Dict[str, Any]) -> ActionResult:
|
||||||
|
"""
|
||||||
|
GENERAL:
|
||||||
|
- Purpose: Convert documents between different formats (PDF→Word, Excel→CSV, etc.).
|
||||||
|
- Input requirements: documentList (required); targetFormat (required).
|
||||||
|
- Output format: Document in target format.
|
||||||
|
|
||||||
|
Parameters:
|
||||||
|
- documentList (list, required): Document reference(s) to convert.
|
||||||
|
- targetFormat (str, required): Target format extension (docx, pdf, xlsx, csv, txt, html, json, md, etc.).
|
||||||
|
- preserveStructure (bool, optional): Whether to preserve document structure (headings, tables, etc.). Default: True.
|
||||||
|
"""
|
||||||
|
documentList = parameters.get("documentList", [])
|
||||||
|
if not documentList:
|
||||||
|
return ActionResult.isFailure(error="documentList is required")
|
||||||
|
|
||||||
|
targetFormat = parameters.get("targetFormat")
|
||||||
|
if not targetFormat:
|
||||||
|
return ActionResult.isFailure(error="targetFormat is required")
|
||||||
|
|
||||||
|
preserveStructure = parameters.get("preserveStructure", True)
|
||||||
|
|
||||||
|
# Normalize format (remove leading dot if present)
|
||||||
|
normalizedFormat = targetFormat.strip().lstrip('.').lower()
|
||||||
|
|
||||||
|
aiPrompt = f"Convert the provided document(s) to {normalizedFormat.upper()} format."
|
||||||
|
if preserveStructure:
|
||||||
|
aiPrompt += " Preserve all document structure including headings, tables, formatting, lists, and layout."
|
||||||
|
aiPrompt += " Ensure the converted document maintains the same content and information as the original."
|
||||||
|
|
||||||
|
return await self.process({
|
||||||
|
"aiPrompt": aiPrompt,
|
||||||
|
"documentList": documentList,
|
||||||
|
"resultType": normalizedFormat
|
||||||
|
})
|
||||||
|
|
||||||
59
modules/workflows/methods/methodAi/actions/extractData.py
Normal file
59
modules/workflows/methods/methodAi/actions/extractData.py
Normal file
|
|
@ -0,0 +1,59 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
"""
|
||||||
|
Extract Data action for AI operations.
|
||||||
|
Extracts structured data from documents (key-value pairs, entities, facts, etc.).
|
||||||
|
"""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
from typing import Dict, Any
|
||||||
|
from modules.workflows.methods.methodBase import action
|
||||||
|
from modules.datamodels.datamodelChat import ActionResult
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
@action
|
||||||
|
async def extractData(self, parameters: Dict[str, Any]) -> ActionResult:
|
||||||
|
"""
|
||||||
|
GENERAL:
|
||||||
|
- Purpose: Extract structured data from documents (key-value pairs, entities, facts, etc.).
|
||||||
|
- Input requirements: documentList (required); optional dataStructure, fields.
|
||||||
|
- Output format: JSON by default, or specified resultType.
|
||||||
|
|
||||||
|
Parameters:
|
||||||
|
- documentList (list, required): Document reference(s) to extract data from.
|
||||||
|
- dataStructure (str, optional): Desired data structure - flat, nested, or list. Default: nested.
|
||||||
|
- fields (list, optional): Specific fields/properties to extract (e.g., ["name", "date", "amount"]).
|
||||||
|
- resultType (str, optional): Output format (json, csv, xlsx, etc.). Default: json.
|
||||||
|
"""
|
||||||
|
documentList = parameters.get("documentList", [])
|
||||||
|
if not documentList:
|
||||||
|
return ActionResult.isFailure(error="documentList is required")
|
||||||
|
|
||||||
|
dataStructure = parameters.get("dataStructure", "nested")
|
||||||
|
fields = parameters.get("fields", [])
|
||||||
|
resultType = parameters.get("resultType", "json")
|
||||||
|
|
||||||
|
aiPrompt = "Extract structured data from the provided document(s)."
|
||||||
|
if fields:
|
||||||
|
fieldsStr = ", ".join(fields)
|
||||||
|
aiPrompt += f" Extract the following specific fields: {fieldsStr}."
|
||||||
|
else:
|
||||||
|
aiPrompt += " Extract all relevant data including names, dates, amounts, entities, and key information."
|
||||||
|
|
||||||
|
structureInstructions = {
|
||||||
|
"flat": "Use a flat key-value structure with simple properties.",
|
||||||
|
"nested": "Use a nested JSON structure with logical grouping of related data.",
|
||||||
|
"list": "Structure the data as a list/array of objects, one per entity or record."
|
||||||
|
}
|
||||||
|
aiPrompt += f" {structureInstructions.get(dataStructure.lower(), structureInstructions['nested'])}"
|
||||||
|
|
||||||
|
aiPrompt += " Ensure all extracted data is accurate and complete."
|
||||||
|
|
||||||
|
return await self.process({
|
||||||
|
"aiPrompt": aiPrompt,
|
||||||
|
"documentList": documentList,
|
||||||
|
"resultType": resultType
|
||||||
|
})
|
||||||
|
|
||||||
|
|
@ -0,0 +1,53 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
"""
|
||||||
|
Generate Document action for AI operations.
|
||||||
|
Generates documents from scratch or based on templates/inputs.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
from typing import Dict, Any
|
||||||
|
from modules.workflows.methods.methodBase import action
|
||||||
|
from modules.datamodels.datamodelChat import ActionResult
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
@action
|
||||||
|
async def generateDocument(self, parameters: Dict[str, Any]) -> ActionResult:
|
||||||
|
"""
|
||||||
|
GENERAL:
|
||||||
|
- Purpose: Generate documents from scratch or based on templates/inputs.
|
||||||
|
- Input requirements: prompt or description (required); optional documentList (for templates/references).
|
||||||
|
- Output format: Document in specified format (default: docx).
|
||||||
|
|
||||||
|
Parameters:
|
||||||
|
- prompt (str, required): Description of the document to generate.
|
||||||
|
- documentList (list, optional): Template documents or reference documents to use as a guide.
|
||||||
|
- documentType (str, optional): Type of document - letter, memo, proposal, contract, etc.
|
||||||
|
- resultType (str, optional): Output format (docx, pdf, txt, md, etc.). Default: docx.
|
||||||
|
"""
|
||||||
|
prompt = parameters.get("prompt")
|
||||||
|
if not prompt:
|
||||||
|
return ActionResult.isFailure(error="prompt is required")
|
||||||
|
|
||||||
|
documentList = parameters.get("documentList", [])
|
||||||
|
documentType = parameters.get("documentType")
|
||||||
|
resultType = parameters.get("resultType", "docx")
|
||||||
|
|
||||||
|
aiPrompt = f"Generate a document based on the following requirements: {prompt}"
|
||||||
|
if documentType:
|
||||||
|
aiPrompt += f" Document type: {documentType}."
|
||||||
|
if documentList:
|
||||||
|
aiPrompt += " Use the provided template/reference documents as a guide for structure, format, and style."
|
||||||
|
aiPrompt += " Create a professional, well-structured document with appropriate formatting and organization."
|
||||||
|
|
||||||
|
processParams = {
|
||||||
|
"aiPrompt": aiPrompt,
|
||||||
|
"resultType": resultType
|
||||||
|
}
|
||||||
|
if documentList:
|
||||||
|
processParams["documentList"] = documentList
|
||||||
|
|
||||||
|
return await self.process(processParams)
|
||||||
|
|
||||||
219
modules/workflows/methods/methodAi/actions/process.py
Normal file
219
modules/workflows/methods/methodAi/actions/process.py
Normal file
|
|
@ -0,0 +1,219 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
"""
|
||||||
|
Process action for AI operations.
|
||||||
|
Universal AI document processing action.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import time
|
||||||
|
from typing import Dict, Any, List, Optional
|
||||||
|
from modules.workflows.methods.methodBase import action
|
||||||
|
from modules.datamodels.datamodelChat import ActionResult, ActionDocument
|
||||||
|
from modules.datamodels.datamodelAi import AiCallOptions
|
||||||
|
from modules.datamodels.datamodelExtraction import ExtractionOptions, MergeStrategy, ContentPart
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
@action
|
||||||
|
async def process(self, parameters: Dict[str, Any]) -> ActionResult:
|
||||||
|
"""
|
||||||
|
GENERAL:
|
||||||
|
- Purpose: Universal AI document processing action - accepts MULTIPLE input documents in ANY format (docx, pdf, json, txt, xlsx, html, images, etc.) and processes them together with a prompt to produce MULTIPLE output documents in ANY specified format (via resultType). Use for document generation, format conversion, content transformation, analysis, summarization, translation, extraction, comparison, and any AI-powered document manipulation.
|
||||||
|
- Input requirements: aiPrompt (required); optional documentList (can contain multiple documents in any format).
|
||||||
|
- Output format: Multiple documents in the same format per call (via resultType: txt, json, pdf, docx, xlsx, pptx, png, jpg, etc.). The AI can generate multiple files based on the prompt (e.g., "create separate documents for each section"). Default: txt.
|
||||||
|
- Key capabilities: Can process any number of input documents together, extract data from mixed formats, combine information, generate multiple output files, transform between formats, perform analysis/comparison/summarization on document sets.
|
||||||
|
|
||||||
|
Parameters:
|
||||||
|
- aiPrompt (str, required): Instruction for the AI describing what processing to perform.
|
||||||
|
- documentList (list, optional): Document reference(s) in any format to use as input/context.
|
||||||
|
- resultType (str, optional): Output file extension (txt, json, md, csv, xml, html, pdf, docx, xlsx, png, etc.). All output documents will use this format. Default: txt.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
# Init progress logger
|
||||||
|
workflowId = self.services.workflow.id if self.services.workflow else f"no-workflow-{int(time.time())}"
|
||||||
|
operationId = f"ai_process_{workflowId}_{int(time.time())}"
|
||||||
|
|
||||||
|
# Start progress tracking
|
||||||
|
parentOperationId = parameters.get('parentOperationId')
|
||||||
|
self.services.chat.progressLogStart(
|
||||||
|
operationId,
|
||||||
|
"Generate",
|
||||||
|
"AI Processing",
|
||||||
|
f"Format: {parameters.get('resultType', 'txt')}",
|
||||||
|
parentOperationId=parentOperationId
|
||||||
|
)
|
||||||
|
|
||||||
|
aiPrompt = parameters.get("aiPrompt")
|
||||||
|
logger.info(f"aiPrompt extracted: '{aiPrompt}' (type: {type(aiPrompt)})")
|
||||||
|
|
||||||
|
# Update progress - preparing parameters
|
||||||
|
self.services.chat.progressLogUpdate(operationId, 0.2, "Preparing parameters")
|
||||||
|
|
||||||
|
from modules.datamodels.datamodelDocref import DocumentReferenceList
|
||||||
|
|
||||||
|
documentListParam = parameters.get("documentList")
|
||||||
|
# Convert to DocumentReferenceList if needed
|
||||||
|
if documentListParam is None:
|
||||||
|
documentList = DocumentReferenceList(references=[])
|
||||||
|
elif isinstance(documentListParam, DocumentReferenceList):
|
||||||
|
documentList = documentListParam
|
||||||
|
elif isinstance(documentListParam, str):
|
||||||
|
documentList = DocumentReferenceList.from_string_list([documentListParam])
|
||||||
|
elif isinstance(documentListParam, list):
|
||||||
|
documentList = DocumentReferenceList.from_string_list(documentListParam)
|
||||||
|
else:
|
||||||
|
logger.error(f"Invalid documentList type: {type(documentListParam)}")
|
||||||
|
documentList = DocumentReferenceList(references=[])
|
||||||
|
|
||||||
|
resultType = parameters.get("resultType", "txt")
|
||||||
|
|
||||||
|
|
||||||
|
if not aiPrompt:
|
||||||
|
logger.error(f"aiPrompt is missing or empty. Parameters: {parameters}")
|
||||||
|
return ActionResult.isFailure(
|
||||||
|
error="AI prompt is required"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Determine output extension and default MIME type without duplicating service logic
|
||||||
|
normalized_result_type = (str(resultType).strip().lstrip('.').lower() or "txt")
|
||||||
|
output_extension = f".{normalized_result_type}"
|
||||||
|
output_mime_type = "application/octet-stream" # Prefer service-provided mimeType when available
|
||||||
|
logger.info(f"Using result type: {resultType} -> {output_extension}")
|
||||||
|
|
||||||
|
# Phase 7.3: Extract content first if documents provided, then use contentParts
|
||||||
|
# Check if contentParts are already provided (preferred path)
|
||||||
|
contentParts: Optional[List[ContentPart]] = None
|
||||||
|
if "contentParts" in parameters:
|
||||||
|
contentParts = parameters.get("contentParts")
|
||||||
|
if contentParts and not isinstance(contentParts, list):
|
||||||
|
# Try to extract from ContentExtracted if it's an ActionDocument
|
||||||
|
if hasattr(contentParts, 'parts'):
|
||||||
|
contentParts = contentParts.parts
|
||||||
|
else:
|
||||||
|
logger.warning(f"Invalid contentParts type: {type(contentParts)}, treating as empty")
|
||||||
|
contentParts = None
|
||||||
|
|
||||||
|
# If contentParts not provided but documentList is, extract content first
|
||||||
|
if not contentParts and documentList.references:
|
||||||
|
self.services.chat.progressLogUpdate(operationId, 0.3, "Extracting content from documents")
|
||||||
|
|
||||||
|
# Get ChatDocuments
|
||||||
|
chatDocuments = self.services.chat.getChatDocumentsFromDocumentList(documentList)
|
||||||
|
if not chatDocuments:
|
||||||
|
logger.warning("No documents found in documentList")
|
||||||
|
else:
|
||||||
|
logger.info(f"Extracting content from {len(chatDocuments)} documents")
|
||||||
|
|
||||||
|
# Prepare extraction options (use defaults if not provided)
|
||||||
|
extractionOptions = parameters.get("extractionOptions")
|
||||||
|
if not extractionOptions:
|
||||||
|
extractionOptions = ExtractionOptions(
|
||||||
|
prompt="Extract all content from the document",
|
||||||
|
mergeStrategy=MergeStrategy(
|
||||||
|
mergeType="concatenate",
|
||||||
|
groupBy="typeGroup",
|
||||||
|
orderBy="id"
|
||||||
|
),
|
||||||
|
processDocumentsIndividually=True
|
||||||
|
)
|
||||||
|
|
||||||
|
# Extract content using extraction service with hierarchical progress logging
|
||||||
|
# Pass operationId for per-document progress tracking
|
||||||
|
extractedResults = self.services.extraction.extractContent(chatDocuments, extractionOptions, operationId=operationId)
|
||||||
|
|
||||||
|
# Combine all ContentParts from all extracted results
|
||||||
|
contentParts = []
|
||||||
|
for extracted in extractedResults:
|
||||||
|
if extracted.parts:
|
||||||
|
contentParts.extend(extracted.parts)
|
||||||
|
|
||||||
|
logger.info(f"Extracted {len(contentParts)} content parts from {len(extractedResults)} documents")
|
||||||
|
|
||||||
|
# Update progress - preparing AI call
|
||||||
|
self.services.chat.progressLogUpdate(operationId, 0.4, "Preparing AI call")
|
||||||
|
|
||||||
|
# Build options with only resultFormat - let service layer handle all other parameters
|
||||||
|
output_format = output_extension.replace('.', '') or 'txt'
|
||||||
|
options = AiCallOptions(
|
||||||
|
resultFormat=output_format
|
||||||
|
# Removed all model parameters - service layer will analyze prompt and determine optimal parameters
|
||||||
|
)
|
||||||
|
|
||||||
|
# Update progress - calling AI
|
||||||
|
self.services.chat.progressLogUpdate(operationId, 0.6, "Calling AI")
|
||||||
|
|
||||||
|
# Use unified callAiContent method with contentParts (extraction is now separate)
|
||||||
|
aiResponse = await self.services.ai.callAiContent(
|
||||||
|
prompt=aiPrompt,
|
||||||
|
options=options,
|
||||||
|
contentParts=contentParts, # Already extracted (or None if no documents)
|
||||||
|
outputFormat=output_format,
|
||||||
|
parentOperationId=operationId
|
||||||
|
)
|
||||||
|
|
||||||
|
# Update progress - processing result
|
||||||
|
self.services.chat.progressLogUpdate(operationId, 0.8, "Processing result")
|
||||||
|
|
||||||
|
# Extract documents from AiResponse
|
||||||
|
if aiResponse.documents and len(aiResponse.documents) > 0:
|
||||||
|
action_documents = []
|
||||||
|
for doc in aiResponse.documents:
|
||||||
|
validationMetadata = {
|
||||||
|
"actionType": "ai.process",
|
||||||
|
"resultType": normalized_result_type,
|
||||||
|
"outputFormat": output_format,
|
||||||
|
"hasDocuments": True,
|
||||||
|
"documentCount": len(aiResponse.documents)
|
||||||
|
}
|
||||||
|
action_documents.append(ActionDocument(
|
||||||
|
documentName=doc.documentName,
|
||||||
|
documentData=doc.documentData,
|
||||||
|
mimeType=doc.mimeType or output_mime_type,
|
||||||
|
sourceJson=getattr(doc, 'sourceJson', None), # Preserve source JSON for structure validation
|
||||||
|
validationMetadata=validationMetadata
|
||||||
|
))
|
||||||
|
|
||||||
|
final_documents = action_documents
|
||||||
|
else:
|
||||||
|
# Text response - create document from content
|
||||||
|
extension = output_extension.lstrip('.')
|
||||||
|
meaningful_name = self._generateMeaningfulFileName(
|
||||||
|
base_name="ai",
|
||||||
|
extension=extension,
|
||||||
|
action_name="result"
|
||||||
|
)
|
||||||
|
validationMetadata = {
|
||||||
|
"actionType": "ai.process",
|
||||||
|
"resultType": normalized_result_type,
|
||||||
|
"outputFormat": output_format,
|
||||||
|
"hasDocuments": False,
|
||||||
|
"contentType": "text"
|
||||||
|
}
|
||||||
|
action_document = ActionDocument(
|
||||||
|
documentName=meaningful_name,
|
||||||
|
documentData=aiResponse.content,
|
||||||
|
mimeType=output_mime_type,
|
||||||
|
validationMetadata=validationMetadata
|
||||||
|
)
|
||||||
|
final_documents = [action_document]
|
||||||
|
|
||||||
|
# Complete progress tracking
|
||||||
|
self.services.chat.progressLogFinish(operationId, True)
|
||||||
|
|
||||||
|
return ActionResult.isSuccess(documents=final_documents)
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error in AI processing: {str(e)}")
|
||||||
|
|
||||||
|
# Complete progress tracking with failure
|
||||||
|
try:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
except:
|
||||||
|
pass # Don't fail on progress logging errors
|
||||||
|
|
||||||
|
return ActionResult.isFailure(
|
||||||
|
error=str(e)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
@ -0,0 +1,55 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
"""
|
||||||
|
Summarize Document action for AI operations.
|
||||||
|
Summarizes one or more documents, extracting key points and main ideas.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
from typing import Dict, Any
|
||||||
|
from modules.workflows.methods.methodBase import action
|
||||||
|
from modules.datamodels.datamodelChat import ActionResult
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
@action
|
||||||
|
async def summarizeDocument(self, parameters: Dict[str, Any]) -> ActionResult:
|
||||||
|
"""
|
||||||
|
GENERAL:
|
||||||
|
- Purpose: Summarize one or more documents, extracting key points and main ideas.
|
||||||
|
- Input requirements: documentList (required); optional summaryLength, focus.
|
||||||
|
- Output format: Text document with summary (default: txt, can be overridden with resultType).
|
||||||
|
|
||||||
|
Parameters:
|
||||||
|
- documentList (list, required): Document reference(s) to summarize.
|
||||||
|
- summaryLength (str, optional): Desired summary length - brief, medium, or detailed. Default: medium.
|
||||||
|
- focus (str, optional): Specific aspect to focus on in the summary (e.g., "financial data", "key decisions").
|
||||||
|
- resultType (str, optional): Output file extension (txt, md, docx, etc.). Default: txt.
|
||||||
|
"""
|
||||||
|
documentList = parameters.get("documentList", [])
|
||||||
|
if not documentList:
|
||||||
|
return ActionResult.isFailure(error="documentList is required")
|
||||||
|
|
||||||
|
summaryLength = parameters.get("summaryLength", "medium")
|
||||||
|
focus = parameters.get("focus")
|
||||||
|
resultType = parameters.get("resultType", "txt")
|
||||||
|
|
||||||
|
lengthInstructions = {
|
||||||
|
"brief": "Create a brief summary (2-3 paragraphs)",
|
||||||
|
"medium": "Create a medium-length summary (comprehensive but concise)",
|
||||||
|
"detailed": "Create a detailed summary covering all major points"
|
||||||
|
}
|
||||||
|
lengthInstruction = lengthInstructions.get(summaryLength.lower(), lengthInstructions["medium"])
|
||||||
|
|
||||||
|
aiPrompt = f"Summarize the provided document(s). {lengthInstruction}."
|
||||||
|
if focus:
|
||||||
|
aiPrompt += f" Focus specifically on: {focus}."
|
||||||
|
aiPrompt += " Extract and present the key points, main ideas, and important information in a clear, well-structured format."
|
||||||
|
|
||||||
|
return await self.process({
|
||||||
|
"aiPrompt": aiPrompt,
|
||||||
|
"documentList": documentList,
|
||||||
|
"resultType": resultType
|
||||||
|
})
|
||||||
|
|
||||||
|
|
@ -0,0 +1,60 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
"""
|
||||||
|
Translate Document action for AI operations.
|
||||||
|
Translates documents to a target language while preserving formatting and structure.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
from typing import Dict, Any
|
||||||
|
from modules.workflows.methods.methodBase import action
|
||||||
|
from modules.datamodels.datamodelChat import ActionResult
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
@action
|
||||||
|
async def translateDocument(self, parameters: Dict[str, Any]) -> ActionResult:
|
||||||
|
"""
|
||||||
|
GENERAL:
|
||||||
|
- Purpose: Translate documents to a target language while preserving formatting and structure.
|
||||||
|
- Input requirements: documentList (required); targetLanguage (required).
|
||||||
|
- Output format: Translated document in same format as input (default) or specified resultType.
|
||||||
|
|
||||||
|
Parameters:
|
||||||
|
- documentList (list, required): Document reference(s) to translate.
|
||||||
|
- targetLanguage (str, required): Target language code or name (e.g., "de", "German", "French", "es").
|
||||||
|
- sourceLanguage (str, optional): Source language if known (e.g., "en", "English"). If not provided, AI will detect.
|
||||||
|
- preserveFormatting (bool, optional): Whether to preserve original formatting. Default: True.
|
||||||
|
- resultType (str, optional): Output file extension. If not specified, uses same format as input.
|
||||||
|
"""
|
||||||
|
documentList = parameters.get("documentList", [])
|
||||||
|
if not documentList:
|
||||||
|
return ActionResult.isFailure(error="documentList is required")
|
||||||
|
|
||||||
|
targetLanguage = parameters.get("targetLanguage")
|
||||||
|
if not targetLanguage:
|
||||||
|
return ActionResult.isFailure(error="targetLanguage is required")
|
||||||
|
|
||||||
|
sourceLanguage = parameters.get("sourceLanguage")
|
||||||
|
preserveFormatting = parameters.get("preserveFormatting", True)
|
||||||
|
resultType = parameters.get("resultType")
|
||||||
|
|
||||||
|
aiPrompt = f"Translate the provided document(s) to {targetLanguage}."
|
||||||
|
if sourceLanguage:
|
||||||
|
aiPrompt += f" The source language is {sourceLanguage}."
|
||||||
|
if preserveFormatting:
|
||||||
|
aiPrompt += " Preserve all formatting, structure, tables, and layout exactly as they appear in the original document."
|
||||||
|
else:
|
||||||
|
aiPrompt += " Focus on accurate translation of content."
|
||||||
|
aiPrompt += " Maintain the same document structure, headings, and organization."
|
||||||
|
|
||||||
|
processParams = {
|
||||||
|
"aiPrompt": aiPrompt,
|
||||||
|
"documentList": documentList
|
||||||
|
}
|
||||||
|
if resultType:
|
||||||
|
processParams["resultType"] = resultType
|
||||||
|
|
||||||
|
return await self.process(processParams)
|
||||||
|
|
||||||
117
modules/workflows/methods/methodAi/actions/webResearch.py
Normal file
117
modules/workflows/methods/methodAi/actions/webResearch.py
Normal file
|
|
@ -0,0 +1,117 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
"""
|
||||||
|
Web Research action for AI operations.
|
||||||
|
Web research with two-step process: search for URLs, then crawl content.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import time
|
||||||
|
import re
|
||||||
|
from typing import Dict, Any
|
||||||
|
from modules.workflows.methods.methodBase import action
|
||||||
|
from modules.datamodels.datamodelChat import ActionResult, ActionDocument
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
@action
|
||||||
|
async def webResearch(self, parameters: Dict[str, Any]) -> ActionResult:
|
||||||
|
"""
|
||||||
|
GENERAL:
|
||||||
|
- Purpose: Web research with two-step process: search for URLs, then crawl content.
|
||||||
|
- Input requirements: prompt (required); optional list(url), country, language, researchDepth.
|
||||||
|
- Output format: JSON with research results including URLs and content.
|
||||||
|
|
||||||
|
Parameters:
|
||||||
|
- prompt (str, required): Natural language research instruction.
|
||||||
|
- urlList (list, optional): Specific URLs to crawl, if needed.
|
||||||
|
- country (str, optional): Two-digit country code (lowercase, e.g., ch, us, de).
|
||||||
|
- language (str, optional): Language code (lowercase, e.g., de, en, fr).
|
||||||
|
- researchDepth (str, optional): Research depth - fast, general, or deep. Default: general.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
prompt = parameters.get("prompt")
|
||||||
|
if not prompt:
|
||||||
|
return ActionResult.isFailure(error="Research prompt is required")
|
||||||
|
|
||||||
|
# Init progress logger
|
||||||
|
workflowId = self.services.workflow.id if self.services.workflow else f"no-workflow-{int(time.time())}"
|
||||||
|
operationId = f"web_research_{workflowId}_{int(time.time())}"
|
||||||
|
|
||||||
|
# Start progress tracking
|
||||||
|
parentOperationId = parameters.get('parentOperationId')
|
||||||
|
self.services.chat.progressLogStart(
|
||||||
|
operationId,
|
||||||
|
"Web Research",
|
||||||
|
"Searching and Crawling",
|
||||||
|
"Extracting URLs and Content",
|
||||||
|
parentOperationId=parentOperationId
|
||||||
|
)
|
||||||
|
|
||||||
|
# Call webcrawl service - service handles all AI intention analysis and processing
|
||||||
|
result = await self.services.web.performWebResearch(
|
||||||
|
prompt=prompt,
|
||||||
|
urls=parameters.get("urlList", []),
|
||||||
|
country=parameters.get("country"),
|
||||||
|
language=parameters.get("language"),
|
||||||
|
researchDepth=parameters.get("researchDepth", "general"),
|
||||||
|
operationId=operationId
|
||||||
|
)
|
||||||
|
|
||||||
|
# Complete progress tracking
|
||||||
|
self.services.chat.progressLogFinish(operationId, True)
|
||||||
|
|
||||||
|
# Get meaningful filename from research result (generated by intent analyzer)
|
||||||
|
suggestedFilename = result.get("suggested_filename")
|
||||||
|
if suggestedFilename:
|
||||||
|
# Clean and validate filename
|
||||||
|
cleaned = suggestedFilename.strip().strip('"\'')
|
||||||
|
cleaned = cleaned.replace('\n', ' ').replace('\r', ' ').strip()
|
||||||
|
# Ensure it doesn't already have extension
|
||||||
|
if cleaned.lower().endswith('.json'):
|
||||||
|
cleaned = cleaned[:-5]
|
||||||
|
# Validate: should be reasonable length and contain only safe characters
|
||||||
|
if cleaned and len(cleaned) <= 60 and re.match(r'^[a-zA-Z0-9_\-]+$', cleaned):
|
||||||
|
meaningfulName = f"{cleaned}.json"
|
||||||
|
else:
|
||||||
|
# Fallback to generic meaningful filename
|
||||||
|
meaningfulName = self._generateMeaningfulFileName(
|
||||||
|
base_name="web_research",
|
||||||
|
extension="json",
|
||||||
|
action_name="research"
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
# Fallback to generic meaningful filename
|
||||||
|
meaningfulName = self._generateMeaningfulFileName(
|
||||||
|
base_name="web_research",
|
||||||
|
extension="json",
|
||||||
|
action_name="research"
|
||||||
|
)
|
||||||
|
|
||||||
|
validationMetadata = {
|
||||||
|
"actionType": "ai.webResearch",
|
||||||
|
"prompt": prompt,
|
||||||
|
"urlList": parameters.get("urlList", []),
|
||||||
|
"country": parameters.get("country"),
|
||||||
|
"language": parameters.get("language"),
|
||||||
|
"researchDepth": parameters.get("researchDepth", "general"),
|
||||||
|
"resultFormat": "json"
|
||||||
|
}
|
||||||
|
actionDocument = ActionDocument(
|
||||||
|
documentName=meaningfulName,
|
||||||
|
documentData=result,
|
||||||
|
mimeType="application/json",
|
||||||
|
validationMetadata=validationMetadata
|
||||||
|
)
|
||||||
|
|
||||||
|
return ActionResult.isSuccess(documents=[actionDocument])
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error in web research: {str(e)}")
|
||||||
|
try:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
except:
|
||||||
|
pass
|
||||||
|
return ActionResult.isFailure(error=str(e))
|
||||||
|
|
||||||
5
modules/workflows/methods/methodAi/helpers/__init__.py
Normal file
5
modules/workflows/methods/methodAi/helpers/__init__.py
Normal file
|
|
@ -0,0 +1,5 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
"""Helper modules for AI method operations."""
|
||||||
|
|
||||||
59
modules/workflows/methods/methodAi/helpers/csvProcessing.py
Normal file
59
modules/workflows/methods/methodAi/helpers/csvProcessing.py
Normal file
|
|
@ -0,0 +1,59 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
"""
|
||||||
|
CSV Processing helper for AI operations.
|
||||||
|
Handles CSV content processing with options.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
from typing import Dict, Any
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
class CsvProcessingHelper:
|
||||||
|
"""Helper for CSV processing operations"""
|
||||||
|
|
||||||
|
def __init__(self, methodInstance):
|
||||||
|
"""
|
||||||
|
Initialize CSV processing helper.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
methodInstance: Instance of MethodAi (for access to services)
|
||||||
|
"""
|
||||||
|
self.method = methodInstance
|
||||||
|
self.services = methodInstance.services
|
||||||
|
|
||||||
|
def applyCsvOptions(self, csvContent: str, options: Dict[str, Any]) -> str:
|
||||||
|
"""
|
||||||
|
Apply CSV processing options to CSV content.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
csvContent: CSV content as string
|
||||||
|
options: Dictionary with CSV processing options
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Processed CSV content as string
|
||||||
|
"""
|
||||||
|
if not csvContent:
|
||||||
|
return csvContent
|
||||||
|
|
||||||
|
# Apply options if provided
|
||||||
|
if options:
|
||||||
|
# Handle delimiter option
|
||||||
|
if "delimiter" in options:
|
||||||
|
delimiter = options["delimiter"]
|
||||||
|
# Replace delimiter in content (simple approach)
|
||||||
|
# Note: This is a basic implementation, may need enhancement
|
||||||
|
if delimiter != ",":
|
||||||
|
csvContent = csvContent.replace(",", delimiter)
|
||||||
|
|
||||||
|
# Handle quote character option
|
||||||
|
if "quotechar" in options:
|
||||||
|
quotechar = options["quotechar"]
|
||||||
|
# Replace quote character (simple approach)
|
||||||
|
if quotechar != '"':
|
||||||
|
csvContent = csvContent.replace('"', quotechar)
|
||||||
|
|
||||||
|
return csvContent
|
||||||
|
|
||||||
383
modules/workflows/methods/methodAi/methodAi.py
Normal file
383
modules/workflows/methods/methodAi/methodAi.py
Normal file
|
|
@ -0,0 +1,383 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
import logging
|
||||||
|
from datetime import datetime, UTC
|
||||||
|
from modules.workflows.methods.methodBase import MethodBase
|
||||||
|
from modules.datamodels.datamodelWorkflowActions import WorkflowActionDefinition, WorkflowActionParameter
|
||||||
|
from modules.shared.frontendTypes import FrontendType
|
||||||
|
|
||||||
|
# Import helpers
|
||||||
|
from .helpers.csvProcessing import CsvProcessingHelper
|
||||||
|
|
||||||
|
# Import actions
|
||||||
|
from .actions.process import process
|
||||||
|
from .actions.webResearch import webResearch
|
||||||
|
from .actions.summarizeDocument import summarizeDocument
|
||||||
|
from .actions.translateDocument import translateDocument
|
||||||
|
from .actions.convert import convert
|
||||||
|
from .actions.convertDocument import convertDocument
|
||||||
|
from .actions.extractData import extractData
|
||||||
|
from .actions.generateDocument import generateDocument
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
class MethodAi(MethodBase):
|
||||||
|
"""AI processing methods."""
|
||||||
|
|
||||||
|
def __init__(self, services):
|
||||||
|
super().__init__(services)
|
||||||
|
self.name = "ai"
|
||||||
|
self.description = "AI processing methods"
|
||||||
|
|
||||||
|
# Initialize helper modules
|
||||||
|
self.csvProcessing = CsvProcessingHelper(self)
|
||||||
|
|
||||||
|
# RBAC-Integration: Action-Definitionen mit actionId
|
||||||
|
self._actions = {
|
||||||
|
"process": WorkflowActionDefinition(
|
||||||
|
actionId="ai.process",
|
||||||
|
description="Universal AI document processing action - accepts multiple input documents in any format and processes them together with a prompt",
|
||||||
|
parameters={
|
||||||
|
"aiPrompt": WorkflowActionParameter(
|
||||||
|
name="aiPrompt",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.TEXTAREA,
|
||||||
|
required=True,
|
||||||
|
description="Instruction for the AI describing what processing to perform"
|
||||||
|
),
|
||||||
|
"documentList": WorkflowActionParameter(
|
||||||
|
name="documentList",
|
||||||
|
type="List[str]",
|
||||||
|
frontendType=FrontendType.DOCUMENT_REFERENCE,
|
||||||
|
required=False,
|
||||||
|
description="Document reference(s) in any format to use as input/context"
|
||||||
|
),
|
||||||
|
"resultType": WorkflowActionParameter(
|
||||||
|
name="resultType",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.SELECT,
|
||||||
|
frontendOptions=["txt", "json", "md", "csv", "xml", "html", "pdf", "docx", "xlsx", "pptx", "png", "jpg"],
|
||||||
|
required=False,
|
||||||
|
default="txt",
|
||||||
|
description="Output file extension. All output documents will use this format"
|
||||||
|
)
|
||||||
|
},
|
||||||
|
execute=process.__get__(self, self.__class__)
|
||||||
|
),
|
||||||
|
"webResearch": WorkflowActionDefinition(
|
||||||
|
actionId="ai.webResearch",
|
||||||
|
description="Web research with two-step process: search for URLs, then crawl content",
|
||||||
|
parameters={
|
||||||
|
"prompt": WorkflowActionParameter(
|
||||||
|
name="prompt",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.TEXTAREA,
|
||||||
|
required=True,
|
||||||
|
description="Natural language research instruction"
|
||||||
|
),
|
||||||
|
"urlList": WorkflowActionParameter(
|
||||||
|
name="urlList",
|
||||||
|
type="List[str]",
|
||||||
|
frontendType=FrontendType.MULTISELECT,
|
||||||
|
required=False,
|
||||||
|
description="Specific URLs to crawl, if needed"
|
||||||
|
),
|
||||||
|
"country": WorkflowActionParameter(
|
||||||
|
name="country",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.TEXT,
|
||||||
|
required=False,
|
||||||
|
description="Two-digit country code (lowercase, e.g., ch, us, de)"
|
||||||
|
),
|
||||||
|
"language": WorkflowActionParameter(
|
||||||
|
name="language",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.SELECT,
|
||||||
|
frontendOptions=["de", "en", "fr", "it", "es"],
|
||||||
|
required=False,
|
||||||
|
description="Language code (lowercase, e.g., de, en, fr)"
|
||||||
|
),
|
||||||
|
"researchDepth": WorkflowActionParameter(
|
||||||
|
name="researchDepth",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.SELECT,
|
||||||
|
frontendOptions=["fast", "general", "deep"],
|
||||||
|
required=False,
|
||||||
|
default="general",
|
||||||
|
description="Research depth"
|
||||||
|
)
|
||||||
|
},
|
||||||
|
execute=webResearch.__get__(self, self.__class__)
|
||||||
|
),
|
||||||
|
"summarizeDocument": WorkflowActionDefinition(
|
||||||
|
actionId="ai.summarizeDocument",
|
||||||
|
description="Summarize one or more documents, extracting key points and main ideas",
|
||||||
|
parameters={
|
||||||
|
"documentList": WorkflowActionParameter(
|
||||||
|
name="documentList",
|
||||||
|
type="List[str]",
|
||||||
|
frontendType=FrontendType.DOCUMENT_REFERENCE,
|
||||||
|
required=True,
|
||||||
|
description="Document reference(s) to summarize"
|
||||||
|
),
|
||||||
|
"summaryLength": WorkflowActionParameter(
|
||||||
|
name="summaryLength",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.SELECT,
|
||||||
|
frontendOptions=["brief", "medium", "detailed"],
|
||||||
|
required=False,
|
||||||
|
default="medium",
|
||||||
|
description="Desired summary length"
|
||||||
|
),
|
||||||
|
"focus": WorkflowActionParameter(
|
||||||
|
name="focus",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.TEXT,
|
||||||
|
required=False,
|
||||||
|
description="Specific aspect to focus on in the summary (e.g., financial data, key decisions)"
|
||||||
|
),
|
||||||
|
"resultType": WorkflowActionParameter(
|
||||||
|
name="resultType",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.SELECT,
|
||||||
|
frontendOptions=["txt", "md", "docx"],
|
||||||
|
required=False,
|
||||||
|
default="txt",
|
||||||
|
description="Output file extension"
|
||||||
|
)
|
||||||
|
},
|
||||||
|
execute=summarizeDocument.__get__(self, self.__class__)
|
||||||
|
),
|
||||||
|
"translateDocument": WorkflowActionDefinition(
|
||||||
|
actionId="ai.translateDocument",
|
||||||
|
description="Translate documents to a target language while preserving formatting and structure",
|
||||||
|
parameters={
|
||||||
|
"documentList": WorkflowActionParameter(
|
||||||
|
name="documentList",
|
||||||
|
type="List[str]",
|
||||||
|
frontendType=FrontendType.DOCUMENT_REFERENCE,
|
||||||
|
required=True,
|
||||||
|
description="Document reference(s) to translate"
|
||||||
|
),
|
||||||
|
"targetLanguage": WorkflowActionParameter(
|
||||||
|
name="targetLanguage",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.TEXT,
|
||||||
|
required=True,
|
||||||
|
description="Target language code or name (e.g., de, German, French, es)"
|
||||||
|
),
|
||||||
|
"sourceLanguage": WorkflowActionParameter(
|
||||||
|
name="sourceLanguage",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.TEXT,
|
||||||
|
required=False,
|
||||||
|
description="Source language if known (e.g., en, English). If not provided, AI will detect"
|
||||||
|
),
|
||||||
|
"preserveFormatting": WorkflowActionParameter(
|
||||||
|
name="preserveFormatting",
|
||||||
|
type="bool",
|
||||||
|
frontendType=FrontendType.CHECKBOX,
|
||||||
|
required=False,
|
||||||
|
default=True,
|
||||||
|
description="Whether to preserve original formatting"
|
||||||
|
),
|
||||||
|
"resultType": WorkflowActionParameter(
|
||||||
|
name="resultType",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.TEXT,
|
||||||
|
required=False,
|
||||||
|
description="Output file extension. If not specified, uses same format as input"
|
||||||
|
)
|
||||||
|
},
|
||||||
|
execute=translateDocument.__get__(self, self.__class__)
|
||||||
|
),
|
||||||
|
"convert": WorkflowActionDefinition(
|
||||||
|
actionId="ai.convert",
|
||||||
|
description="Convert documents/data between different formats with specific formatting options",
|
||||||
|
parameters={
|
||||||
|
"documentList": WorkflowActionParameter(
|
||||||
|
name="documentList",
|
||||||
|
type="List[str]",
|
||||||
|
frontendType=FrontendType.DOCUMENT_REFERENCE,
|
||||||
|
required=True,
|
||||||
|
description="Document reference(s) to convert"
|
||||||
|
),
|
||||||
|
"inputFormat": WorkflowActionParameter(
|
||||||
|
name="inputFormat",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.SELECT,
|
||||||
|
frontendOptions=["json", "csv", "xlsx", "txt"],
|
||||||
|
required=True,
|
||||||
|
description="Source format"
|
||||||
|
),
|
||||||
|
"outputFormat": WorkflowActionParameter(
|
||||||
|
name="outputFormat",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.SELECT,
|
||||||
|
frontendOptions=["csv", "json", "xlsx", "txt"],
|
||||||
|
required=True,
|
||||||
|
description="Target format"
|
||||||
|
),
|
||||||
|
"columnsPerRow": WorkflowActionParameter(
|
||||||
|
name="columnsPerRow",
|
||||||
|
type="int",
|
||||||
|
frontendType=FrontendType.NUMBER,
|
||||||
|
required=False,
|
||||||
|
description="For CSV output, number of columns per row. Default: auto-detect",
|
||||||
|
validation={"min": 1, "max": 100}
|
||||||
|
),
|
||||||
|
"delimiter": WorkflowActionParameter(
|
||||||
|
name="delimiter",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.TEXT,
|
||||||
|
required=False,
|
||||||
|
default=",",
|
||||||
|
description="For CSV output, delimiter character"
|
||||||
|
),
|
||||||
|
"includeHeader": WorkflowActionParameter(
|
||||||
|
name="includeHeader",
|
||||||
|
type="bool",
|
||||||
|
frontendType=FrontendType.CHECKBOX,
|
||||||
|
required=False,
|
||||||
|
default=True,
|
||||||
|
description="For CSV output, whether to include header row"
|
||||||
|
),
|
||||||
|
"language": WorkflowActionParameter(
|
||||||
|
name="language",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.SELECT,
|
||||||
|
frontendOptions=["de", "en", "fr"],
|
||||||
|
required=False,
|
||||||
|
default="en",
|
||||||
|
description="Language for output"
|
||||||
|
)
|
||||||
|
},
|
||||||
|
execute=convert.__get__(self, self.__class__)
|
||||||
|
),
|
||||||
|
"convertDocument": WorkflowActionDefinition(
|
||||||
|
actionId="ai.convertDocument",
|
||||||
|
description="Convert documents between different formats (PDF→Word, Excel→CSV, etc.)",
|
||||||
|
parameters={
|
||||||
|
"documentList": WorkflowActionParameter(
|
||||||
|
name="documentList",
|
||||||
|
type="List[str]",
|
||||||
|
frontendType=FrontendType.DOCUMENT_REFERENCE,
|
||||||
|
required=True,
|
||||||
|
description="Document reference(s) to convert"
|
||||||
|
),
|
||||||
|
"targetFormat": WorkflowActionParameter(
|
||||||
|
name="targetFormat",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.SELECT,
|
||||||
|
frontendOptions=["docx", "pdf", "xlsx", "csv", "txt", "html", "json", "md"],
|
||||||
|
required=True,
|
||||||
|
description="Target format extension"
|
||||||
|
),
|
||||||
|
"preserveStructure": WorkflowActionParameter(
|
||||||
|
name="preserveStructure",
|
||||||
|
type="bool",
|
||||||
|
frontendType=FrontendType.CHECKBOX,
|
||||||
|
required=False,
|
||||||
|
default=True,
|
||||||
|
description="Whether to preserve document structure (headings, tables, etc.)"
|
||||||
|
)
|
||||||
|
},
|
||||||
|
execute=convertDocument.__get__(self, self.__class__)
|
||||||
|
),
|
||||||
|
"extractData": WorkflowActionDefinition(
|
||||||
|
actionId="ai.extractData",
|
||||||
|
description="Extract structured data from documents (key-value pairs, entities, facts, etc.)",
|
||||||
|
parameters={
|
||||||
|
"documentList": WorkflowActionParameter(
|
||||||
|
name="documentList",
|
||||||
|
type="List[str]",
|
||||||
|
frontendType=FrontendType.DOCUMENT_REFERENCE,
|
||||||
|
required=True,
|
||||||
|
description="Document reference(s) to extract data from"
|
||||||
|
),
|
||||||
|
"dataStructure": WorkflowActionParameter(
|
||||||
|
name="dataStructure",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.SELECT,
|
||||||
|
frontendOptions=["flat", "nested", "list"],
|
||||||
|
required=False,
|
||||||
|
default="nested",
|
||||||
|
description="Desired data structure"
|
||||||
|
),
|
||||||
|
"fields": WorkflowActionParameter(
|
||||||
|
name="fields",
|
||||||
|
type="List[str]",
|
||||||
|
frontendType=FrontendType.MULTISELECT,
|
||||||
|
required=False,
|
||||||
|
description="Specific fields/properties to extract (e.g., [name, date, amount])"
|
||||||
|
),
|
||||||
|
"resultType": WorkflowActionParameter(
|
||||||
|
name="resultType",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.SELECT,
|
||||||
|
frontendOptions=["json", "csv", "xlsx"],
|
||||||
|
required=False,
|
||||||
|
default="json",
|
||||||
|
description="Output format"
|
||||||
|
)
|
||||||
|
},
|
||||||
|
execute=extractData.__get__(self, self.__class__)
|
||||||
|
),
|
||||||
|
"generateDocument": WorkflowActionDefinition(
|
||||||
|
actionId="ai.generateDocument",
|
||||||
|
description="Generate documents from scratch or based on templates/inputs",
|
||||||
|
parameters={
|
||||||
|
"prompt": WorkflowActionParameter(
|
||||||
|
name="prompt",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.TEXTAREA,
|
||||||
|
required=True,
|
||||||
|
description="Description of the document to generate"
|
||||||
|
),
|
||||||
|
"documentList": WorkflowActionParameter(
|
||||||
|
name="documentList",
|
||||||
|
type="List[str]",
|
||||||
|
frontendType=FrontendType.DOCUMENT_REFERENCE,
|
||||||
|
required=False,
|
||||||
|
description="Template documents or reference documents to use as a guide"
|
||||||
|
),
|
||||||
|
"documentType": WorkflowActionParameter(
|
||||||
|
name="documentType",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.SELECT,
|
||||||
|
frontendOptions=["letter", "memo", "proposal", "contract", "report", "email"],
|
||||||
|
required=False,
|
||||||
|
description="Type of document"
|
||||||
|
),
|
||||||
|
"resultType": WorkflowActionParameter(
|
||||||
|
name="resultType",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.SELECT,
|
||||||
|
frontendOptions=["docx", "pdf", "txt", "md"],
|
||||||
|
required=False,
|
||||||
|
default="docx",
|
||||||
|
description="Output format"
|
||||||
|
)
|
||||||
|
},
|
||||||
|
execute=generateDocument.__get__(self, self.__class__)
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
# Validate actions after definition
|
||||||
|
self._validateActions()
|
||||||
|
|
||||||
|
# Register actions as methods (optional, für direkten Zugriff)
|
||||||
|
self.process = process.__get__(self, self.__class__)
|
||||||
|
self.webResearch = webResearch.__get__(self, self.__class__)
|
||||||
|
self.summarizeDocument = summarizeDocument.__get__(self, self.__class__)
|
||||||
|
self.translateDocument = translateDocument.__get__(self, self.__class__)
|
||||||
|
self.convert = convert.__get__(self, self.__class__)
|
||||||
|
self.convertDocument = convertDocument.__get__(self, self.__class__)
|
||||||
|
self.extractData = extractData.__get__(self, self.__class__)
|
||||||
|
self.generateDocument = generateDocument.__get__(self, self.__class__)
|
||||||
|
|
||||||
|
def _format_timestamp_for_filename(self) -> str:
|
||||||
|
"""Format current timestamp as YYYYMMDD-hhmmss for filenames."""
|
||||||
|
return datetime.now(UTC).strftime("%Y%m%d-%H%M%S")
|
||||||
|
|
||||||
|
|
@ -7,6 +7,9 @@ import logging
|
||||||
from functools import wraps
|
from functools import wraps
|
||||||
import inspect
|
import inspect
|
||||||
|
|
||||||
|
from modules.datamodels.datamodelWorkflowActions import WorkflowActionDefinition, WorkflowActionParameter
|
||||||
|
from modules.datamodels.datamodelRbac import AccessRuleContext
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
def action(func):
|
def action(func):
|
||||||
|
|
@ -57,37 +60,194 @@ class MethodBase:
|
||||||
self.description: str
|
self.description: str
|
||||||
self.logger = logging.getLogger(f"{__name__}.{self.__class__.__name__}")
|
self.logger = logging.getLogger(f"{__name__}.{self.__class__.__name__}")
|
||||||
|
|
||||||
|
# Actions MÜSSEN als Dictionary definiert sein
|
||||||
|
# Jede Method-Klasse muss _actions Dictionary in __init__ definieren
|
||||||
|
self._actions: Dict[str, WorkflowActionDefinition] = {}
|
||||||
|
|
||||||
|
# Nach Initialisierung: Actions validieren (wird überschrieben, wenn _actions gesetzt wird)
|
||||||
|
# Validierung erfolgt erst nach vollständiger Initialisierung der Subklasse
|
||||||
|
|
||||||
|
def _validateActions(self):
|
||||||
|
"""Validate that _actions dictionary is properly defined"""
|
||||||
|
if not hasattr(self, '_actions') or not isinstance(self._actions, dict):
|
||||||
|
raise ValueError(f"Method {self.name} must define _actions dictionary in __init__")
|
||||||
|
|
||||||
|
for actionName, actionDef in self._actions.items():
|
||||||
|
if not isinstance(actionDef, WorkflowActionDefinition):
|
||||||
|
raise ValueError(f"Action '{actionName}' in {self.name} must be WorkflowActionDefinition instance")
|
||||||
|
|
||||||
|
if not actionDef.actionId:
|
||||||
|
raise ValueError(f"Action '{actionName}' in {self.name} must have actionId")
|
||||||
|
|
||||||
|
if not actionDef.execute:
|
||||||
|
raise ValueError(f"Action '{actionName}' in {self.name} must have execute function")
|
||||||
|
|
||||||
@property
|
@property
|
||||||
def actions(self) -> Dict[str, Dict[str, Any]]:
|
def actions(self) -> Dict[str, Dict[str, Any]]:
|
||||||
"""Dynamically collect all actions decorated with @action in the class."""
|
"""
|
||||||
actions = {}
|
Dynamically collect all actions from _actions dictionary.
|
||||||
for attr_name in dir(self):
|
Returns format for API/UI consumption.
|
||||||
# Skip the actions property itself to avoid recursion
|
|
||||||
if attr_name == 'actions':
|
REQUIREMENT: Alle Actions müssen in _actions Dictionary definiert sein.
|
||||||
continue
|
Actions ohne _actions Definition sind nicht verfügbar.
|
||||||
try:
|
"""
|
||||||
attr = getattr(self, attr_name)
|
result = {}
|
||||||
if callable(attr) and getattr(attr, 'is_action', False):
|
|
||||||
sig = inspect.signature(attr)
|
# Actions müssen in _actions Dictionary definiert sein
|
||||||
params = {}
|
if not hasattr(self, '_actions') or not self._actions:
|
||||||
for param_name, param in sig.parameters.items():
|
self.logger.error(f"Method {self.name} has no _actions dictionary defined. Actions will not be available.")
|
||||||
if param_name not in ['self', 'parameters']:
|
return result
|
||||||
param_type = param.annotation if param.annotation != param.empty else Any
|
|
||||||
params[param_name] = {
|
for actionName, actionDef in self._actions.items():
|
||||||
'type': param_type,
|
# RBAC-Check: Prüfe ob Action für aktuellen User verfügbar ist
|
||||||
'required': param.default == param.empty,
|
if not self._checkActionPermission(actionDef.actionId):
|
||||||
'description': None,
|
continue # Skip if user doesn't have permission
|
||||||
'default': param.default if param.default != param.empty else None
|
|
||||||
}
|
# Konvertiere WorkflowActionDefinition zu System-Format
|
||||||
actions[attr_name] = {
|
result[actionName] = {
|
||||||
'description': attr.__doc__ or '',
|
'description': actionDef.description,
|
||||||
'parameters': params,
|
'parameters': self._convertParametersToSystemFormat(actionDef.parameters),
|
||||||
'method': attr
|
'method': self._createActionWrapper(actionDef)
|
||||||
}
|
}
|
||||||
except (AttributeError, RecursionError):
|
|
||||||
# Skip attributes that cause issues
|
return result
|
||||||
continue
|
|
||||||
return actions
|
def _checkActionPermission(self, actionId: str) -> bool:
|
||||||
|
"""
|
||||||
|
Check if current user has permission to execute this action.
|
||||||
|
Uses RBAC RESOURCE context.
|
||||||
|
|
||||||
|
REQUIREMENT: RBAC-Service muss verfügbar sein.
|
||||||
|
"""
|
||||||
|
if not hasattr(self.services, 'rbac') or not self.services.rbac:
|
||||||
|
self.logger.error(f"RBAC service not available. Action {actionId} will be denied.")
|
||||||
|
return False
|
||||||
|
|
||||||
|
currentUser = self.services.chat.getCurrentUser()
|
||||||
|
if not currentUser:
|
||||||
|
self.logger.warning(f"No current user found. Action {actionId} will be denied.")
|
||||||
|
return False
|
||||||
|
|
||||||
|
# RBAC-Check: RESOURCE context, item = actionId
|
||||||
|
permissions = self.services.rbac.getUserPermissions(
|
||||||
|
user=currentUser,
|
||||||
|
context=AccessRuleContext.RESOURCE,
|
||||||
|
item=actionId
|
||||||
|
)
|
||||||
|
|
||||||
|
return permissions.view
|
||||||
|
|
||||||
|
def _convertParametersToSystemFormat(self, parameters: Dict[str, WorkflowActionParameter]) -> Dict[str, Dict[str, Any]]:
|
||||||
|
"""Convert WorkflowActionParameter dict to system format for API/UI consumption"""
|
||||||
|
result = {}
|
||||||
|
for paramName, param in parameters.items():
|
||||||
|
result[paramName] = {
|
||||||
|
'type': param.type,
|
||||||
|
'required': param.required,
|
||||||
|
'description': param.description,
|
||||||
|
'default': param.default,
|
||||||
|
'frontendType': param.frontendType.value,
|
||||||
|
'frontendOptions': param.frontendOptions,
|
||||||
|
'validation': param.validation
|
||||||
|
}
|
||||||
|
return result
|
||||||
|
|
||||||
|
def _createActionWrapper(self, actionDef: WorkflowActionDefinition):
|
||||||
|
"""Create wrapper function for action execution with parameter validation"""
|
||||||
|
async def wrapper(parameters: Dict[str, Any], *args, **kwargs):
|
||||||
|
# Parameter-Validierung basierend auf WorkflowActionParameter definitions
|
||||||
|
validatedParams = self._validateParameters(parameters, actionDef.parameters)
|
||||||
|
|
||||||
|
# Execute action
|
||||||
|
return await actionDef.execute(validatedParams, *args, **kwargs)
|
||||||
|
|
||||||
|
wrapper.is_action = True
|
||||||
|
return wrapper
|
||||||
|
|
||||||
|
def _validateParameters(self, parameters: Dict[str, Any], paramDefs: Dict[str, WorkflowActionParameter]) -> Dict[str, Any]:
|
||||||
|
"""Validate parameters against definitions"""
|
||||||
|
validated = {}
|
||||||
|
|
||||||
|
for paramName, paramDef in paramDefs.items():
|
||||||
|
value = parameters.get(paramName)
|
||||||
|
|
||||||
|
# Check required
|
||||||
|
if paramDef.required and value is None:
|
||||||
|
raise ValueError(f"Required parameter '{paramName}' is missing")
|
||||||
|
|
||||||
|
# Use default if not provided
|
||||||
|
if value is None and paramDef.default is not None:
|
||||||
|
value = paramDef.default
|
||||||
|
|
||||||
|
# Type validation
|
||||||
|
if value is not None:
|
||||||
|
value = self._validateType(value, paramDef.type)
|
||||||
|
|
||||||
|
# Custom validation rules
|
||||||
|
if paramDef.validation and value is not None:
|
||||||
|
self._applyValidationRules(value, paramDef.validation)
|
||||||
|
|
||||||
|
validated[paramName] = value
|
||||||
|
|
||||||
|
return validated
|
||||||
|
|
||||||
|
def _validateType(self, value: Any, expectedType: str) -> Any:
|
||||||
|
"""Validate and convert value to expected type"""
|
||||||
|
# Type validation logic
|
||||||
|
typeMap = {
|
||||||
|
'str': str,
|
||||||
|
'int': int,
|
||||||
|
'float': float,
|
||||||
|
'bool': bool,
|
||||||
|
'list': list,
|
||||||
|
'dict': dict,
|
||||||
|
}
|
||||||
|
|
||||||
|
# Handle List[str], List[int], etc.
|
||||||
|
if expectedType.startswith('List['):
|
||||||
|
if not isinstance(value, list):
|
||||||
|
raise ValueError(f"Expected list for type '{expectedType}', got {type(value).__name__}")
|
||||||
|
# Extract inner type
|
||||||
|
innerType = expectedType[5:-1].strip() # Remove "List[" and "]"
|
||||||
|
if innerType in typeMap:
|
||||||
|
return [typeMap[innerType](v) for v in value]
|
||||||
|
return value
|
||||||
|
|
||||||
|
# Handle Dict[str, Any], etc.
|
||||||
|
if expectedType.startswith('Dict['):
|
||||||
|
if not isinstance(value, dict):
|
||||||
|
raise ValueError(f"Expected dict for type '{expectedType}', got {type(value).__name__}")
|
||||||
|
return value
|
||||||
|
|
||||||
|
# Handle simple types
|
||||||
|
if expectedType in typeMap:
|
||||||
|
expectedTypeClass = typeMap[expectedType]
|
||||||
|
if not isinstance(value, expectedTypeClass):
|
||||||
|
try:
|
||||||
|
return expectedTypeClass(value)
|
||||||
|
except (ValueError, TypeError) as e:
|
||||||
|
raise ValueError(f"Cannot convert {value} to {expectedType}: {str(e)}")
|
||||||
|
|
||||||
|
return value
|
||||||
|
|
||||||
|
def _applyValidationRules(self, value: Any, rules: Dict[str, Any]):
|
||||||
|
"""Apply custom validation rules"""
|
||||||
|
if 'min' in rules:
|
||||||
|
if isinstance(value, (int, float)) and value < rules['min']:
|
||||||
|
raise ValueError(f"Value must be >= {rules['min']}")
|
||||||
|
elif isinstance(value, str) and len(value) < rules['min']:
|
||||||
|
raise ValueError(f"String length must be >= {rules['min']}")
|
||||||
|
|
||||||
|
if 'max' in rules:
|
||||||
|
if isinstance(value, (int, float)) and value > rules['max']:
|
||||||
|
raise ValueError(f"Value must be <= {rules['max']}")
|
||||||
|
elif isinstance(value, str) and len(value) > rules['max']:
|
||||||
|
raise ValueError(f"String length must be <= {rules['max']}")
|
||||||
|
|
||||||
|
if 'pattern' in rules:
|
||||||
|
import re
|
||||||
|
if not re.match(rules['pattern'], str(value)):
|
||||||
|
raise ValueError(f"Value does not match required pattern: {rules['pattern']}")
|
||||||
|
|
||||||
def getActionSignature(self, actionName: str) -> str:
|
def getActionSignature(self, actionName: str) -> str:
|
||||||
"""Get formatted action signature for AI prompt generation (detailed version)"""
|
"""Get formatted action signature for AI prompt generation (detailed version)"""
|
||||||
|
|
|
||||||
7
modules/workflows/methods/methodContext/__init__.py
Normal file
7
modules/workflows/methods/methodContext/__init__.py
Normal file
|
|
@ -0,0 +1,7 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
from .methodContext import MethodContext
|
||||||
|
|
||||||
|
__all__ = ['MethodContext']
|
||||||
|
|
||||||
16
modules/workflows/methods/methodContext/actions/__init__.py
Normal file
16
modules/workflows/methods/methodContext/actions/__init__.py
Normal file
|
|
@ -0,0 +1,16 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
"""Action modules for Context operations."""
|
||||||
|
|
||||||
|
# Export all actions
|
||||||
|
from .getDocumentIndex import getDocumentIndex
|
||||||
|
from .extractContent import extractContent
|
||||||
|
from .triggerPreprocessingServer import triggerPreprocessingServer
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
'getDocumentIndex',
|
||||||
|
'extractContent',
|
||||||
|
'triggerPreprocessingServer',
|
||||||
|
]
|
||||||
|
|
||||||
|
|
@ -0,0 +1,156 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
"""
|
||||||
|
Extract Content action for Context operations.
|
||||||
|
Extracts content from documents (separate from AI calls).
|
||||||
|
"""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import time
|
||||||
|
from typing import Dict, Any
|
||||||
|
from modules.workflows.methods.methodBase import action
|
||||||
|
from modules.datamodels.datamodelChat import ActionResult, ActionDocument
|
||||||
|
from modules.datamodels.datamodelDocref import DocumentReferenceList
|
||||||
|
from modules.datamodels.datamodelExtraction import ExtractionOptions, MergeStrategy
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
@action
|
||||||
|
async def extractContent(self, parameters: Dict[str, Any]) -> ActionResult:
|
||||||
|
"""
|
||||||
|
Extract content from documents (separate from AI calls).
|
||||||
|
|
||||||
|
This action performs pure content extraction without AI processing.
|
||||||
|
The extracted ContentParts can then be used by subsequent AI processing actions.
|
||||||
|
|
||||||
|
Parameters:
|
||||||
|
- documentList (list, required): Document reference(s) to extract content from.
|
||||||
|
- extractionOptions (dict, optional): Extraction options (if not provided, defaults are used).
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
- ActionResult with ActionDocument containing ContentExtracted objects
|
||||||
|
- ContentExtracted.parts contains List[ContentPart] (already chunked if needed)
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
# Init progress logger
|
||||||
|
workflowId = self.services.workflow.id if self.services.workflow else f"no-workflow-{int(time.time())}"
|
||||||
|
operationId = f"context_extract_{workflowId}_{int(time.time())}"
|
||||||
|
|
||||||
|
# Extract documentList from parameters dict
|
||||||
|
documentListParam = parameters.get("documentList")
|
||||||
|
if not documentListParam:
|
||||||
|
return ActionResult.isFailure(error="documentList is required")
|
||||||
|
|
||||||
|
# Convert to DocumentReferenceList if needed
|
||||||
|
if isinstance(documentListParam, DocumentReferenceList):
|
||||||
|
documentList = documentListParam
|
||||||
|
elif isinstance(documentListParam, str):
|
||||||
|
documentList = DocumentReferenceList.from_string_list([documentListParam])
|
||||||
|
elif isinstance(documentListParam, list):
|
||||||
|
documentList = DocumentReferenceList.from_string_list(documentListParam)
|
||||||
|
else:
|
||||||
|
return ActionResult.isFailure(error=f"Invalid documentList type: {type(documentListParam)}")
|
||||||
|
|
||||||
|
# Start progress tracking
|
||||||
|
parentOperationId = parameters.get('parentOperationId')
|
||||||
|
self.services.chat.progressLogStart(
|
||||||
|
operationId,
|
||||||
|
"Extracting content from documents",
|
||||||
|
"Content Extraction",
|
||||||
|
f"Documents: {len(documentList.references)}",
|
||||||
|
parentOperationId=parentOperationId
|
||||||
|
)
|
||||||
|
|
||||||
|
# Get ChatDocuments from documentList
|
||||||
|
self.services.chat.progressLogUpdate(operationId, 0.2, "Loading documents")
|
||||||
|
chatDocuments = self.services.chat.getChatDocumentsFromDocumentList(documentList)
|
||||||
|
|
||||||
|
if not chatDocuments:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
return ActionResult.isFailure(error="No documents found in documentList")
|
||||||
|
|
||||||
|
logger.info(f"Extracting content from {len(chatDocuments)} documents")
|
||||||
|
|
||||||
|
# Prepare extraction options
|
||||||
|
self.services.chat.progressLogUpdate(operationId, 0.3, "Preparing extraction options")
|
||||||
|
extractionOptionsParam = parameters.get("extractionOptions")
|
||||||
|
|
||||||
|
# Convert dict to ExtractionOptions object if needed, or create defaults
|
||||||
|
if extractionOptionsParam:
|
||||||
|
if isinstance(extractionOptionsParam, dict):
|
||||||
|
# Convert dict to ExtractionOptions object
|
||||||
|
extractionOptions = ExtractionOptions(**extractionOptionsParam)
|
||||||
|
elif isinstance(extractionOptionsParam, ExtractionOptions):
|
||||||
|
extractionOptions = extractionOptionsParam
|
||||||
|
else:
|
||||||
|
# Invalid type, use defaults
|
||||||
|
extractionOptions = None
|
||||||
|
else:
|
||||||
|
extractionOptions = None
|
||||||
|
|
||||||
|
# If extractionOptions not provided, create defaults
|
||||||
|
if not extractionOptions:
|
||||||
|
# Default extraction options for pure content extraction (no AI processing)
|
||||||
|
extractionOptions = ExtractionOptions(
|
||||||
|
prompt="Extract all content from the document",
|
||||||
|
mergeStrategy=MergeStrategy(
|
||||||
|
mergeType="concatenate",
|
||||||
|
groupBy="typeGroup",
|
||||||
|
orderBy="id"
|
||||||
|
),
|
||||||
|
processDocumentsIndividually=True
|
||||||
|
)
|
||||||
|
|
||||||
|
# Call extraction service with hierarchical progress logging
|
||||||
|
self.services.chat.progressLogUpdate(operationId, 0.4, "Initiating")
|
||||||
|
self.services.chat.progressLogUpdate(operationId, 0.5, f"Extracting content from {len(chatDocuments)} documents")
|
||||||
|
# Pass operationId for hierarchical per-document progress logging
|
||||||
|
extractedResults = self.services.extraction.extractContent(chatDocuments, extractionOptions, operationId=operationId)
|
||||||
|
|
||||||
|
# Build ActionDocuments from ContentExtracted results
|
||||||
|
self.services.chat.progressLogUpdate(operationId, 0.8, "Building result documents")
|
||||||
|
actionDocuments = []
|
||||||
|
# Map extracted results back to original documents by index (results are in same order)
|
||||||
|
for i, extracted in enumerate(extractedResults):
|
||||||
|
# Get original document name if available
|
||||||
|
originalDoc = chatDocuments[i] if i < len(chatDocuments) else None
|
||||||
|
if originalDoc and hasattr(originalDoc, 'fileName') and originalDoc.fileName:
|
||||||
|
# Use original filename with "extracted_" prefix
|
||||||
|
baseName = originalDoc.fileName.rsplit('.', 1)[0] if '.' in originalDoc.fileName else originalDoc.fileName
|
||||||
|
documentName = f"{baseName}_extracted_{extracted.id}.json"
|
||||||
|
else:
|
||||||
|
# Fallback to generic name with index
|
||||||
|
documentName = f"document_{i+1:03d}_extracted_{extracted.id}.json"
|
||||||
|
|
||||||
|
# Store ContentExtracted object in ActionDocument.documentData
|
||||||
|
validationMetadata = {
|
||||||
|
"actionType": "context.extractContent",
|
||||||
|
"documentIndex": i,
|
||||||
|
"extractedId": extracted.id,
|
||||||
|
"partCount": len(extracted.parts) if extracted.parts else 0,
|
||||||
|
"originalFileName": originalDoc.fileName if originalDoc and hasattr(originalDoc, 'fileName') else None
|
||||||
|
}
|
||||||
|
actionDoc = ActionDocument(
|
||||||
|
documentName=documentName,
|
||||||
|
documentData=extracted, # ContentExtracted object
|
||||||
|
mimeType="application/json",
|
||||||
|
validationMetadata=validationMetadata
|
||||||
|
)
|
||||||
|
actionDocuments.append(actionDoc)
|
||||||
|
|
||||||
|
self.services.chat.progressLogFinish(operationId, True)
|
||||||
|
|
||||||
|
return ActionResult.isSuccess(documents=actionDocuments)
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error in content extraction: {str(e)}")
|
||||||
|
|
||||||
|
# Complete progress tracking with failure
|
||||||
|
try:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
except:
|
||||||
|
pass # Don't fail on progress logging errors
|
||||||
|
|
||||||
|
return ActionResult.isFailure(error=str(e))
|
||||||
|
|
||||||
|
|
@ -0,0 +1,94 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
"""
|
||||||
|
Get Document Index action for Context operations.
|
||||||
|
Generates a comprehensive index of all documents available in the current workflow.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import json
|
||||||
|
from typing import Dict, Any
|
||||||
|
from modules.workflows.methods.methodBase import action
|
||||||
|
from modules.datamodels.datamodelChat import ActionResult, ActionDocument
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
@action
|
||||||
|
async def getDocumentIndex(self, parameters: Dict[str, Any]) -> ActionResult:
|
||||||
|
"""
|
||||||
|
GENERAL:
|
||||||
|
- Purpose: Generate a comprehensive index of all documents available in the current workflow, including documents from all rounds and tasks.
|
||||||
|
- Input requirements: No input documents required. Optional resultType parameter.
|
||||||
|
- Output format: Structured document index in JSON format (default) or text format, listing all documents with their references, metadata, and organization by rounds/tasks.
|
||||||
|
|
||||||
|
Parameters:
|
||||||
|
- resultType (str, optional): Output format (json, txt, md). Default: json.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
workflow = self.services.workflow
|
||||||
|
if not workflow:
|
||||||
|
return ActionResult.isFailure(
|
||||||
|
error="No workflow available"
|
||||||
|
)
|
||||||
|
|
||||||
|
resultType = parameters.get("resultType", "json").lower().strip().lstrip('.')
|
||||||
|
|
||||||
|
# Get available documents index from chat service
|
||||||
|
documentsIndex = self.services.chat.getAvailableDocuments(workflow)
|
||||||
|
|
||||||
|
if not documentsIndex or documentsIndex == "No documents available" or documentsIndex == "NO DOCUMENTS AVAILABLE - This workflow has no documents to process.":
|
||||||
|
# Return empty index structure
|
||||||
|
if resultType == "json":
|
||||||
|
indexData = {
|
||||||
|
"workflowId": getattr(workflow, 'id', 'unknown'),
|
||||||
|
"totalDocuments": 0,
|
||||||
|
"rounds": [],
|
||||||
|
"documentReferences": []
|
||||||
|
}
|
||||||
|
indexContent = json.dumps(indexData, indent=2, ensure_ascii=False)
|
||||||
|
else:
|
||||||
|
indexContent = "Document Index\n==============\n\nNo documents available in this workflow.\n"
|
||||||
|
else:
|
||||||
|
# Parse the document index string to extract structured information
|
||||||
|
indexData = self.documentIndex.parseDocumentIndex(documentsIndex, workflow)
|
||||||
|
|
||||||
|
if resultType == "json":
|
||||||
|
indexContent = json.dumps(indexData, indent=2, ensure_ascii=False)
|
||||||
|
elif resultType == "md":
|
||||||
|
indexContent = self.formatting.formatAsMarkdown(indexData)
|
||||||
|
else: # txt
|
||||||
|
indexContent = self.formatting.formatAsText(indexData, documentsIndex)
|
||||||
|
|
||||||
|
# Generate meaningful filename
|
||||||
|
workflowContext = self.services.chat.getWorkflowContext()
|
||||||
|
filename = self._generateMeaningfulFileName(
|
||||||
|
"document_index",
|
||||||
|
resultType if resultType in ["json", "txt", "md"] else "json",
|
||||||
|
workflowContext,
|
||||||
|
"getDocumentIndex"
|
||||||
|
)
|
||||||
|
|
||||||
|
validationMetadata = {
|
||||||
|
"actionType": "context.getDocumentIndex",
|
||||||
|
"resultType": resultType,
|
||||||
|
"workflowId": getattr(workflow, 'id', 'unknown'),
|
||||||
|
"totalDocuments": indexData.get("totalDocuments", 0) if isinstance(indexData, dict) else 0
|
||||||
|
}
|
||||||
|
|
||||||
|
# Create ActionDocument
|
||||||
|
document = ActionDocument(
|
||||||
|
documentName=filename,
|
||||||
|
documentData=indexContent,
|
||||||
|
mimeType="application/json" if resultType == "json" else "text/plain",
|
||||||
|
validationMetadata=validationMetadata
|
||||||
|
)
|
||||||
|
|
||||||
|
return ActionResult.isSuccess(documents=[document])
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error generating document index: {str(e)}")
|
||||||
|
return ActionResult.isFailure(
|
||||||
|
error=f"Failed to generate document index: {str(e)}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
@ -0,0 +1,121 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
"""
|
||||||
|
Trigger Preprocessing Server action for Context operations.
|
||||||
|
Triggers preprocessing server at customer tenant to update database with configuration.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import json
|
||||||
|
import aiohttp
|
||||||
|
from typing import Dict, Any
|
||||||
|
from modules.workflows.methods.methodBase import action
|
||||||
|
from modules.datamodels.datamodelChat import ActionResult, ActionDocument
|
||||||
|
from modules.shared.configuration import APP_CONFIG
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
@action
|
||||||
|
async def triggerPreprocessingServer(self, parameters: Dict[str, Any]) -> ActionResult:
|
||||||
|
"""
|
||||||
|
Trigger preprocessing server at customer tenant to update database with configuration.
|
||||||
|
|
||||||
|
This action makes a POST request to the preprocessing server endpoint with the provided
|
||||||
|
configuration JSON. The authorization secret is retrieved from APP_CONFIG using the provided config key.
|
||||||
|
|
||||||
|
Parameters:
|
||||||
|
- endpoint (str, required): The full URL endpoint for the preprocessing server API.
|
||||||
|
- configJson (dict or str, required): Configuration JSON object to send to the preprocessing server. Can be provided as a dict or as a JSON string that will be parsed.
|
||||||
|
- authSecretConfigKey (str, required): The APP_CONFIG key name to retrieve the authorization secret from.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
- ActionResult with ActionDocument containing "ok" on success, or error message on failure.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
endpoint = parameters.get("endpoint")
|
||||||
|
if not endpoint:
|
||||||
|
return ActionResult.isFailure(error="endpoint parameter is required")
|
||||||
|
|
||||||
|
configJsonParam = parameters.get("configJson")
|
||||||
|
if not configJsonParam:
|
||||||
|
return ActionResult.isFailure(error="configJson parameter is required")
|
||||||
|
|
||||||
|
authSecretConfigKey = parameters.get("authSecretConfigKey")
|
||||||
|
if not authSecretConfigKey:
|
||||||
|
return ActionResult.isFailure(error="authSecretConfigKey parameter is required")
|
||||||
|
|
||||||
|
# Handle configJson as either dict or JSON string
|
||||||
|
if isinstance(configJsonParam, str):
|
||||||
|
try:
|
||||||
|
configJson = json.loads(configJsonParam)
|
||||||
|
except json.JSONDecodeError as e:
|
||||||
|
return ActionResult.isFailure(error=f"configJson is not valid JSON: {str(e)}")
|
||||||
|
elif isinstance(configJsonParam, dict):
|
||||||
|
configJson = configJsonParam
|
||||||
|
else:
|
||||||
|
return ActionResult.isFailure(error=f"configJson must be a dict or JSON string, got {type(configJsonParam)}")
|
||||||
|
|
||||||
|
# Get authorization secret from APP_CONFIG using the provided config key
|
||||||
|
authSecret = APP_CONFIG.get(authSecretConfigKey)
|
||||||
|
if not authSecret:
|
||||||
|
errorMsg = f"{authSecretConfigKey} not found in APP_CONFIG"
|
||||||
|
logger.error(errorMsg)
|
||||||
|
return ActionResult.isFailure(error=errorMsg)
|
||||||
|
|
||||||
|
# Prepare headers with authorization (default headers as in original function)
|
||||||
|
headers = {
|
||||||
|
"X-PP-API-Key": authSecret,
|
||||||
|
"Content-Type": "application/json"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Make POST request
|
||||||
|
timeout = aiohttp.ClientTimeout(total=60)
|
||||||
|
async with aiohttp.ClientSession(timeout=timeout) as session:
|
||||||
|
async with session.post(
|
||||||
|
endpoint,
|
||||||
|
headers=headers,
|
||||||
|
json=configJson
|
||||||
|
) as response:
|
||||||
|
if response.status in [200, 201]:
|
||||||
|
responseText = await response.text()
|
||||||
|
logger.info(f"Preprocessing server trigger successful: {response.status}")
|
||||||
|
logger.debug(f"Response: {responseText}")
|
||||||
|
|
||||||
|
# Generate meaningful filename
|
||||||
|
workflowContext = self.services.chat.getWorkflowContext() if hasattr(self.services, 'chat') else None
|
||||||
|
filename = self._generateMeaningfulFileName(
|
||||||
|
"preprocessing_result",
|
||||||
|
"txt",
|
||||||
|
workflowContext,
|
||||||
|
"triggerPreprocessingServer"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Create validation metadata
|
||||||
|
validationMetadata = self._createValidationMetadata(
|
||||||
|
"triggerPreprocessingServer",
|
||||||
|
endpoint=endpoint,
|
||||||
|
statusCode=response.status,
|
||||||
|
responseText=responseText
|
||||||
|
)
|
||||||
|
|
||||||
|
# Return success with "ok" document
|
||||||
|
document = ActionDocument(
|
||||||
|
documentName=filename,
|
||||||
|
documentData="ok",
|
||||||
|
mimeType="text/plain",
|
||||||
|
validationMetadata=validationMetadata
|
||||||
|
)
|
||||||
|
|
||||||
|
return ActionResult.isSuccess(documents=[document])
|
||||||
|
else:
|
||||||
|
errorText = await response.text()
|
||||||
|
errorMsg = f"Preprocessing server trigger failed: {response.status} - {errorText}"
|
||||||
|
logger.error(errorMsg)
|
||||||
|
return ActionResult.isFailure(error=errorMsg)
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
errorMsg = f"Error triggering preprocessing server: {str(e)}"
|
||||||
|
logger.error(errorMsg)
|
||||||
|
return ActionResult.isFailure(error=errorMsg)
|
||||||
|
|
||||||
|
|
@ -0,0 +1,5 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
"""Helper modules for Context method operations."""
|
||||||
|
|
||||||
|
|
@ -0,0 +1,89 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
"""
|
||||||
|
Document Index helper for Context operations.
|
||||||
|
Handles parsing and formatting of document indexes.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
from typing import Dict, Any
|
||||||
|
from datetime import datetime, UTC
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
class DocumentIndexHelper:
|
||||||
|
"""Helper for document index operations"""
|
||||||
|
|
||||||
|
def __init__(self, methodInstance):
|
||||||
|
"""
|
||||||
|
Initialize document index helper.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
methodInstance: Instance of MethodContext (for access to services)
|
||||||
|
"""
|
||||||
|
self.method = methodInstance
|
||||||
|
self.services = methodInstance.services
|
||||||
|
|
||||||
|
def parseDocumentIndex(self, documentsIndex: str, workflow: Any) -> Dict[str, Any]:
|
||||||
|
"""Parse the document index string into structured data."""
|
||||||
|
try:
|
||||||
|
indexData = {
|
||||||
|
"workflowId": getattr(workflow, 'id', 'unknown'),
|
||||||
|
"generatedAt": datetime.now(UTC).isoformat(),
|
||||||
|
"totalDocuments": 0,
|
||||||
|
"rounds": [],
|
||||||
|
"documentReferences": []
|
||||||
|
}
|
||||||
|
|
||||||
|
# Extract document references from the index string
|
||||||
|
lines = documentsIndex.split('\n')
|
||||||
|
currentRound = None
|
||||||
|
currentDocList = None
|
||||||
|
|
||||||
|
for line in lines:
|
||||||
|
line = line.strip()
|
||||||
|
if not line:
|
||||||
|
continue
|
||||||
|
|
||||||
|
# Check for round headers
|
||||||
|
if "Current round documents:" in line:
|
||||||
|
currentRound = "current"
|
||||||
|
continue
|
||||||
|
elif "Past rounds documents:" in line:
|
||||||
|
currentRound = "past"
|
||||||
|
continue
|
||||||
|
|
||||||
|
# Check for document list references (docList:...)
|
||||||
|
if line.startswith("- docList:"):
|
||||||
|
docListRef = line.replace("- docList:", "").strip()
|
||||||
|
currentDocList = {
|
||||||
|
"reference": docListRef,
|
||||||
|
"round": currentRound,
|
||||||
|
"documents": []
|
||||||
|
}
|
||||||
|
indexData["rounds"].append(currentDocList)
|
||||||
|
continue
|
||||||
|
|
||||||
|
# Check for individual document references (docItem:...)
|
||||||
|
if line.startswith(" - docItem:") or line.startswith("- docItem:"):
|
||||||
|
docItemRef = line.replace(" - docItem:", "").replace("- docItem:", "").strip()
|
||||||
|
indexData["documentReferences"].append({
|
||||||
|
"reference": docItemRef,
|
||||||
|
"round": currentRound,
|
||||||
|
"docList": currentDocList["reference"] if currentDocList else None
|
||||||
|
})
|
||||||
|
indexData["totalDocuments"] += 1
|
||||||
|
if currentDocList:
|
||||||
|
currentDocList["documents"].append(docItemRef)
|
||||||
|
|
||||||
|
return indexData
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error parsing document index: {str(e)}")
|
||||||
|
return {
|
||||||
|
"workflowId": getattr(workflow, 'id', 'unknown'),
|
||||||
|
"error": f"Failed to parse document index: {str(e)}",
|
||||||
|
"rawIndex": documentsIndex
|
||||||
|
}
|
||||||
|
|
||||||
|
|
@ -0,0 +1,75 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
"""
|
||||||
|
Formatting helper for Context operations.
|
||||||
|
Handles formatting of document indexes in different formats.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
from typing import Dict, Any
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
class FormattingHelper:
|
||||||
|
"""Helper for formatting operations"""
|
||||||
|
|
||||||
|
def __init__(self, methodInstance):
|
||||||
|
"""
|
||||||
|
Initialize formatting helper.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
methodInstance: Instance of MethodContext (for access to services)
|
||||||
|
"""
|
||||||
|
self.method = methodInstance
|
||||||
|
self.services = methodInstance.services
|
||||||
|
|
||||||
|
def formatAsMarkdown(self, indexData: Dict[str, Any]) -> str:
|
||||||
|
"""Format document index as Markdown."""
|
||||||
|
try:
|
||||||
|
md = f"# Document Index\n\n"
|
||||||
|
md += f"**Workflow ID:** {indexData.get('workflowId', 'unknown')}\n\n"
|
||||||
|
md += f"**Generated At:** {indexData.get('generatedAt', 'unknown')}\n\n"
|
||||||
|
md += f"**Total Documents:** {indexData.get('totalDocuments', 0)}\n\n"
|
||||||
|
|
||||||
|
if indexData.get('rounds'):
|
||||||
|
md += "## Documents by Round\n\n"
|
||||||
|
for roundInfo in indexData['rounds']:
|
||||||
|
roundLabel = roundInfo.get('round', 'unknown').title()
|
||||||
|
md += f"### {roundLabel} Round\n\n"
|
||||||
|
md += f"**Document List:** `{roundInfo.get('reference', 'unknown')}`\n\n"
|
||||||
|
if roundInfo.get('documents'):
|
||||||
|
md += "**Documents:**\n\n"
|
||||||
|
for docRef in roundInfo['documents']:
|
||||||
|
md += f"- `{docRef}`\n"
|
||||||
|
md += "\n"
|
||||||
|
|
||||||
|
if indexData.get('documentReferences'):
|
||||||
|
md += "## All Document References\n\n"
|
||||||
|
for docRef in indexData['documentReferences']:
|
||||||
|
md += f"- `{docRef.get('reference', 'unknown')}`\n"
|
||||||
|
|
||||||
|
return md
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error formatting as Markdown: {str(e)}")
|
||||||
|
return f"# Document Index\n\nError formatting index: {str(e)}\n"
|
||||||
|
|
||||||
|
def formatAsText(self, indexData: Dict[str, Any], rawIndex: str) -> str:
|
||||||
|
"""Format document index as plain text."""
|
||||||
|
try:
|
||||||
|
text = "Document Index\n"
|
||||||
|
text += "=" * 50 + "\n\n"
|
||||||
|
text += f"Workflow ID: {indexData.get('workflowId', 'unknown')}\n"
|
||||||
|
text += f"Generated At: {indexData.get('generatedAt', 'unknown')}\n"
|
||||||
|
text += f"Total Documents: {indexData.get('totalDocuments', 0)}\n\n"
|
||||||
|
|
||||||
|
# Include the raw formatted index for readability
|
||||||
|
text += rawIndex
|
||||||
|
|
||||||
|
return text
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error formatting as text: {str(e)}")
|
||||||
|
return f"Document Index\n\nError formatting index: {str(e)}\n\nRaw index:\n{rawIndex}\n"
|
||||||
|
|
||||||
108
modules/workflows/methods/methodContext/methodContext.py
Normal file
108
modules/workflows/methods/methodContext/methodContext.py
Normal file
|
|
@ -0,0 +1,108 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
import logging
|
||||||
|
from modules.workflows.methods.methodBase import MethodBase
|
||||||
|
from modules.datamodels.datamodelWorkflowActions import WorkflowActionDefinition, WorkflowActionParameter
|
||||||
|
from modules.shared.frontendTypes import FrontendType
|
||||||
|
|
||||||
|
# Import helpers
|
||||||
|
from .helpers.documentIndex import DocumentIndexHelper
|
||||||
|
from .helpers.formatting import FormattingHelper
|
||||||
|
|
||||||
|
# Import actions
|
||||||
|
from .actions.getDocumentIndex import getDocumentIndex
|
||||||
|
from .actions.extractContent import extractContent
|
||||||
|
from .actions.triggerPreprocessingServer import triggerPreprocessingServer
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
class MethodContext(MethodBase):
|
||||||
|
"""Context and workflow information methods."""
|
||||||
|
|
||||||
|
def __init__(self, services):
|
||||||
|
super().__init__(services)
|
||||||
|
self.name = "context"
|
||||||
|
self.description = "Context and workflow information methods"
|
||||||
|
|
||||||
|
# Initialize helper modules
|
||||||
|
self.documentIndex = DocumentIndexHelper(self)
|
||||||
|
self.formatting = FormattingHelper(self)
|
||||||
|
|
||||||
|
# RBAC-Integration: Action-Definitionen mit actionId
|
||||||
|
self._actions = {
|
||||||
|
"getDocumentIndex": WorkflowActionDefinition(
|
||||||
|
actionId="context.getDocumentIndex",
|
||||||
|
description="Generate a comprehensive index of all documents available in the current workflow",
|
||||||
|
parameters={
|
||||||
|
"resultType": WorkflowActionParameter(
|
||||||
|
name="resultType",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.SELECT,
|
||||||
|
frontendOptions=["json", "txt", "md"],
|
||||||
|
required=False,
|
||||||
|
default="json",
|
||||||
|
description="Output format"
|
||||||
|
)
|
||||||
|
},
|
||||||
|
execute=getDocumentIndex.__get__(self, self.__class__)
|
||||||
|
),
|
||||||
|
"extractContent": WorkflowActionDefinition(
|
||||||
|
actionId="context.extractContent",
|
||||||
|
description="Extract content from documents (separate from AI calls)",
|
||||||
|
parameters={
|
||||||
|
"documentList": WorkflowActionParameter(
|
||||||
|
name="documentList",
|
||||||
|
type="List[str]",
|
||||||
|
frontendType=FrontendType.DOCUMENT_REFERENCE,
|
||||||
|
required=True,
|
||||||
|
description="Document reference(s) to extract content from"
|
||||||
|
),
|
||||||
|
"extractionOptions": WorkflowActionParameter(
|
||||||
|
name="extractionOptions",
|
||||||
|
type="dict",
|
||||||
|
frontendType=FrontendType.JSON,
|
||||||
|
required=False,
|
||||||
|
description="Extraction options (if not provided, defaults are used)"
|
||||||
|
)
|
||||||
|
},
|
||||||
|
execute=extractContent.__get__(self, self.__class__)
|
||||||
|
),
|
||||||
|
"triggerPreprocessingServer": WorkflowActionDefinition(
|
||||||
|
actionId="context.triggerPreprocessingServer",
|
||||||
|
description="Trigger preprocessing server at customer tenant to update database with configuration",
|
||||||
|
parameters={
|
||||||
|
"endpoint": WorkflowActionParameter(
|
||||||
|
name="endpoint",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.TEXT,
|
||||||
|
required=True,
|
||||||
|
description="The full URL endpoint for the preprocessing server API"
|
||||||
|
),
|
||||||
|
"configJson": WorkflowActionParameter(
|
||||||
|
name="configJson",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.JSON,
|
||||||
|
required=True,
|
||||||
|
description="Configuration JSON object to send to the preprocessing server. Can be provided as a dict or as a JSON string"
|
||||||
|
),
|
||||||
|
"authSecretConfigKey": WorkflowActionParameter(
|
||||||
|
name="authSecretConfigKey",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.TEXT,
|
||||||
|
required=True,
|
||||||
|
description="The APP_CONFIG key name to retrieve the authorization secret from"
|
||||||
|
)
|
||||||
|
},
|
||||||
|
execute=triggerPreprocessingServer.__get__(self, self.__class__)
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
# Validate actions after definition
|
||||||
|
self._validateActions()
|
||||||
|
|
||||||
|
# Register actions as methods (optional, für direkten Zugriff)
|
||||||
|
self.getDocumentIndex = getDocumentIndex.__get__(self, self.__class__)
|
||||||
|
self.extractContent = extractContent.__get__(self, self.__class__)
|
||||||
|
self.triggerPreprocessingServer = triggerPreprocessingServer.__get__(self, self.__class__)
|
||||||
|
|
||||||
7
modules/workflows/methods/methodJira/__init__.py
Normal file
7
modules/workflows/methods/methodJira/__init__.py
Normal file
|
|
@ -0,0 +1,7 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
from .methodJira import MethodJira
|
||||||
|
|
||||||
|
__all__ = ['MethodJira']
|
||||||
|
|
||||||
26
modules/workflows/methods/methodJira/actions/__init__.py
Normal file
26
modules/workflows/methods/methodJira/actions/__init__.py
Normal file
|
|
@ -0,0 +1,26 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
"""Action modules for JIRA operations."""
|
||||||
|
|
||||||
|
# Export all actions
|
||||||
|
from .connectJira import connectJira
|
||||||
|
from .exportTicketsAsJson import exportTicketsAsJson
|
||||||
|
from .importTicketsFromJson import importTicketsFromJson
|
||||||
|
from .mergeTicketData import mergeTicketData
|
||||||
|
from .parseCsvContent import parseCsvContent
|
||||||
|
from .parseExcelContent import parseExcelContent
|
||||||
|
from .createCsvContent import createCsvContent
|
||||||
|
from .createExcelContent import createExcelContent
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
'connectJira',
|
||||||
|
'exportTicketsAsJson',
|
||||||
|
'importTicketsFromJson',
|
||||||
|
'mergeTicketData',
|
||||||
|
'parseCsvContent',
|
||||||
|
'parseExcelContent',
|
||||||
|
'createCsvContent',
|
||||||
|
'createExcelContent',
|
||||||
|
]
|
||||||
|
|
||||||
139
modules/workflows/methods/methodJira/actions/connectJira.py
Normal file
139
modules/workflows/methods/methodJira/actions/connectJira.py
Normal file
|
|
@ -0,0 +1,139 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
"""
|
||||||
|
Connect JIRA action for JIRA operations.
|
||||||
|
Connects to JIRA instance and creates ticket interface.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import json
|
||||||
|
import uuid
|
||||||
|
from typing import Dict, Any
|
||||||
|
from modules.workflows.methods.methodBase import action
|
||||||
|
from modules.datamodels.datamodelChat import ActionResult, ActionDocument
|
||||||
|
from modules.shared.configuration import APP_CONFIG
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
@action
|
||||||
|
async def connectJira(self, parameters: Dict[str, Any]) -> ActionResult:
|
||||||
|
"""
|
||||||
|
Connect to JIRA instance and create ticket interface.
|
||||||
|
|
||||||
|
Parameters:
|
||||||
|
- apiUsername (str, required): JIRA API username/email
|
||||||
|
- apiTokenConfigKey (str, required): APP_CONFIG key name for JIRA API token
|
||||||
|
- apiUrl (str, required): JIRA instance URL (e.g., https://example.atlassian.net)
|
||||||
|
- projectCode (str, required): JIRA project code (e.g., "DCS")
|
||||||
|
- issueType (str, required): JIRA issue type (e.g., "Task")
|
||||||
|
- taskSyncDefinition (str or dict, required): Field mapping definition as JSON string or dict
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
- ActionResult with ActionDocument containing connection ID
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
apiUsername = parameters.get("apiUsername")
|
||||||
|
if not apiUsername:
|
||||||
|
return ActionResult.isFailure(error="apiUsername parameter is required")
|
||||||
|
|
||||||
|
apiTokenConfigKey = parameters.get("apiTokenConfigKey")
|
||||||
|
if not apiTokenConfigKey:
|
||||||
|
return ActionResult.isFailure(error="apiTokenConfigKey parameter is required")
|
||||||
|
|
||||||
|
apiUrl = parameters.get("apiUrl")
|
||||||
|
if not apiUrl:
|
||||||
|
return ActionResult.isFailure(error="apiUrl parameter is required")
|
||||||
|
|
||||||
|
projectCode = parameters.get("projectCode")
|
||||||
|
if not projectCode:
|
||||||
|
return ActionResult.isFailure(error="projectCode parameter is required")
|
||||||
|
|
||||||
|
issueType = parameters.get("issueType")
|
||||||
|
if not issueType:
|
||||||
|
return ActionResult.isFailure(error="issueType parameter is required")
|
||||||
|
|
||||||
|
taskSyncDefinitionParam = parameters.get("taskSyncDefinition")
|
||||||
|
if not taskSyncDefinitionParam:
|
||||||
|
return ActionResult.isFailure(error="taskSyncDefinition parameter is required")
|
||||||
|
|
||||||
|
# Parse taskSyncDefinition
|
||||||
|
if isinstance(taskSyncDefinitionParam, str):
|
||||||
|
try:
|
||||||
|
taskSyncDefinition = json.loads(taskSyncDefinitionParam)
|
||||||
|
except json.JSONDecodeError as e:
|
||||||
|
return ActionResult.isFailure(error=f"taskSyncDefinition is not valid JSON: {str(e)}")
|
||||||
|
elif isinstance(taskSyncDefinitionParam, dict):
|
||||||
|
taskSyncDefinition = taskSyncDefinitionParam
|
||||||
|
else:
|
||||||
|
return ActionResult.isFailure(error=f"taskSyncDefinition must be a dict or JSON string, got {type(taskSyncDefinitionParam)}")
|
||||||
|
|
||||||
|
# Get API token from APP_CONFIG
|
||||||
|
apiToken = APP_CONFIG.get(apiTokenConfigKey)
|
||||||
|
if not apiToken:
|
||||||
|
errorMsg = f"{apiTokenConfigKey} not found in APP_CONFIG"
|
||||||
|
logger.error(errorMsg)
|
||||||
|
return ActionResult.isFailure(error=errorMsg)
|
||||||
|
|
||||||
|
# Create ticket interface
|
||||||
|
syncInterface = await self.services.ticket.connectTicket(
|
||||||
|
taskSyncDefinition=taskSyncDefinition,
|
||||||
|
connectorType="Jira",
|
||||||
|
connectorParams={
|
||||||
|
"apiUsername": apiUsername,
|
||||||
|
"apiToken": apiToken,
|
||||||
|
"apiUrl": apiUrl,
|
||||||
|
"projectCode": projectCode,
|
||||||
|
"ticketType": issueType,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
# Store connection with unique ID
|
||||||
|
connectionId = str(uuid.uuid4())
|
||||||
|
self._connections[connectionId] = {
|
||||||
|
"interface": syncInterface,
|
||||||
|
"taskSyncDefinition": taskSyncDefinition,
|
||||||
|
"apiUrl": apiUrl,
|
||||||
|
"projectCode": projectCode,
|
||||||
|
}
|
||||||
|
|
||||||
|
logger.info(f"JIRA connection established: {connectionId} (Project: {projectCode})")
|
||||||
|
|
||||||
|
# Generate filename
|
||||||
|
workflowContext = self.services.chat.getWorkflowContext() if hasattr(self.services, 'chat') else None
|
||||||
|
filename = self._generateMeaningfulFileName(
|
||||||
|
"jira_connection",
|
||||||
|
"json",
|
||||||
|
workflowContext,
|
||||||
|
"connectJira"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Create connection info document
|
||||||
|
connectionInfo = {
|
||||||
|
"connectionId": connectionId,
|
||||||
|
"apiUrl": apiUrl,
|
||||||
|
"projectCode": projectCode,
|
||||||
|
"issueType": issueType,
|
||||||
|
}
|
||||||
|
|
||||||
|
validationMetadata = self._createValidationMetadata(
|
||||||
|
"connectJira",
|
||||||
|
connectionId=connectionId,
|
||||||
|
apiUrl=apiUrl,
|
||||||
|
projectCode=projectCode
|
||||||
|
)
|
||||||
|
|
||||||
|
document = ActionDocument(
|
||||||
|
documentName=filename,
|
||||||
|
documentData=json.dumps(connectionInfo, indent=2),
|
||||||
|
mimeType="application/json",
|
||||||
|
validationMetadata=validationMetadata
|
||||||
|
)
|
||||||
|
|
||||||
|
return ActionResult.isSuccess(documents=[document])
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
errorMsg = f"Error connecting to JIRA: {str(e)}"
|
||||||
|
logger.error(errorMsg)
|
||||||
|
return ActionResult.isFailure(error=errorMsg)
|
||||||
|
|
||||||
157
modules/workflows/methods/methodJira/actions/createCsvContent.py
Normal file
157
modules/workflows/methods/methodJira/actions/createCsvContent.py
Normal file
|
|
@ -0,0 +1,157 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
"""
|
||||||
|
Create CSV Content action for JIRA operations.
|
||||||
|
Creates CSV content with custom headers.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import json
|
||||||
|
import base64
|
||||||
|
import pandas as pd
|
||||||
|
import csv as csv_module
|
||||||
|
from io import StringIO
|
||||||
|
from datetime import datetime, UTC
|
||||||
|
from typing import Dict, Any
|
||||||
|
from modules.workflows.methods.methodBase import action
|
||||||
|
from modules.datamodels.datamodelChat import ActionResult, ActionDocument
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
@action
|
||||||
|
async def createCsvContent(self, parameters: Dict[str, Any]) -> ActionResult:
|
||||||
|
"""
|
||||||
|
Create CSV content with custom headers.
|
||||||
|
|
||||||
|
Parameters:
|
||||||
|
- data (str, required): Document reference containing data as JSON (with "data" field from mergeTicketData)
|
||||||
|
- headers (str, optional): Document reference containing headers JSON (from parseCsvContent/parseExcelContent)
|
||||||
|
- columns (str or list, optional): List of column names (if not provided, extracted from taskSyncDefinition or data)
|
||||||
|
- taskSyncDefinition (str or dict, optional): Field mapping definition (used to extract column names if columns not provided)
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
- ActionResult with ActionDocument containing CSV content as bytes
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
dataParam = parameters.get("data")
|
||||||
|
if not dataParam:
|
||||||
|
return ActionResult.isFailure(error="data parameter is required")
|
||||||
|
|
||||||
|
headersParam = parameters.get("headers")
|
||||||
|
columnsParam = parameters.get("columns")
|
||||||
|
taskSyncDefinitionParam = parameters.get("taskSyncDefinition")
|
||||||
|
|
||||||
|
# Get data from document
|
||||||
|
dataJson = self.documentParsing.parseJsonFromDocument(dataParam)
|
||||||
|
if dataJson is None:
|
||||||
|
return ActionResult.isFailure(error="Could not parse data from document reference")
|
||||||
|
|
||||||
|
# Extract data array if wrapped in object
|
||||||
|
if isinstance(dataJson, dict) and "data" in dataJson:
|
||||||
|
dataList = dataJson["data"]
|
||||||
|
elif isinstance(dataJson, list):
|
||||||
|
dataList = dataJson
|
||||||
|
else:
|
||||||
|
return ActionResult.isFailure(error="Data must be a JSON array or object with 'data' field")
|
||||||
|
|
||||||
|
# Get headers
|
||||||
|
headers = {"header1": "Header 1", "header2": "Header 2"}
|
||||||
|
if headersParam:
|
||||||
|
headersJson = self.documentParsing.parseJsonFromDocument(headersParam)
|
||||||
|
if headersJson and isinstance(headersJson, dict) and "headers" in headersJson:
|
||||||
|
headers = headersJson["headers"]
|
||||||
|
elif headersJson and isinstance(headersJson, dict):
|
||||||
|
headers = headersJson
|
||||||
|
|
||||||
|
# Get columns
|
||||||
|
if columnsParam:
|
||||||
|
if isinstance(columnsParam, str):
|
||||||
|
try:
|
||||||
|
columns = json.loads(columnsParam) if columnsParam.startswith('[') or columnsParam.startswith('{') else columnsParam.split(',')
|
||||||
|
except:
|
||||||
|
columns = columnsParam.split(',')
|
||||||
|
elif isinstance(columnsParam, list):
|
||||||
|
columns = columnsParam
|
||||||
|
else:
|
||||||
|
columns = None
|
||||||
|
elif taskSyncDefinitionParam:
|
||||||
|
# Extract columns from taskSyncDefinition
|
||||||
|
if isinstance(taskSyncDefinitionParam, str):
|
||||||
|
taskSyncDefinition = json.loads(taskSyncDefinitionParam)
|
||||||
|
else:
|
||||||
|
taskSyncDefinition = taskSyncDefinitionParam
|
||||||
|
columns = list(taskSyncDefinition.keys())
|
||||||
|
elif dataList and len(dataList) > 0:
|
||||||
|
columns = list(dataList[0].keys())
|
||||||
|
else:
|
||||||
|
columns = []
|
||||||
|
|
||||||
|
# Create DataFrame
|
||||||
|
if not dataList:
|
||||||
|
df = pd.DataFrame(columns=columns)
|
||||||
|
else:
|
||||||
|
df = pd.DataFrame(dataList)
|
||||||
|
# Ensure all columns exist
|
||||||
|
for col in columns:
|
||||||
|
if col not in df.columns:
|
||||||
|
df[col] = ""
|
||||||
|
# Reorder columns
|
||||||
|
df = df[columns]
|
||||||
|
|
||||||
|
# Clean data
|
||||||
|
for column in df.columns:
|
||||||
|
df[column] = df[column].astype("object").fillna("")
|
||||||
|
df[column] = df[column].astype(str).str.replace('\n', '\\n', regex=False).str.replace('"', '""', regex=False)
|
||||||
|
|
||||||
|
# Create headers with timestamp
|
||||||
|
timestamp = datetime.fromtimestamp(self.services.utils.timestampGetUtc(), UTC).strftime("%Y-%m-%d %H:%M:%S UTC")
|
||||||
|
header1Row = next(csv_module.reader([headers.get("header1", "Header 1")]), [])
|
||||||
|
header2Row = next(csv_module.reader([headers.get("header2", "Header 2")]), [])
|
||||||
|
if len(header2Row) > 1:
|
||||||
|
header2Row[1] = timestamp
|
||||||
|
|
||||||
|
headerRow1 = pd.DataFrame([header1Row + [""] * (len(df.columns) - len(header1Row))], columns=df.columns)
|
||||||
|
headerRow2 = pd.DataFrame([header2Row + [""] * (len(df.columns) - len(header2Row))], columns=df.columns)
|
||||||
|
tableHeaders = pd.DataFrame([df.columns.tolist()], columns=df.columns)
|
||||||
|
finalDf = pd.concat([headerRow1, headerRow2, tableHeaders, df], ignore_index=True)
|
||||||
|
|
||||||
|
# Convert to CSV bytes
|
||||||
|
out = StringIO()
|
||||||
|
finalDf.to_csv(out, index=False, header=False, quoting=1, escapechar='\\')
|
||||||
|
csvBytes = out.getvalue().encode('utf-8')
|
||||||
|
|
||||||
|
logger.info(f"Created CSV content: {len(dataList)} rows, {len(columns)} columns")
|
||||||
|
|
||||||
|
# Generate filename
|
||||||
|
workflowContext = self.services.chat.getWorkflowContext() if hasattr(self.services, 'chat') else None
|
||||||
|
filename = self._generateMeaningfulFileName(
|
||||||
|
"ticket_sync",
|
||||||
|
"csv",
|
||||||
|
workflowContext,
|
||||||
|
"createCsvContent"
|
||||||
|
)
|
||||||
|
|
||||||
|
validationMetadata = self._createValidationMetadata(
|
||||||
|
"createCsvContent",
|
||||||
|
rowCount=len(dataList),
|
||||||
|
columnCount=len(columns)
|
||||||
|
)
|
||||||
|
|
||||||
|
# Store as base64 for document
|
||||||
|
csvBase64 = base64.b64encode(csvBytes).decode('utf-8')
|
||||||
|
|
||||||
|
document = ActionDocument(
|
||||||
|
documentName=filename,
|
||||||
|
documentData=csvBase64,
|
||||||
|
mimeType="application/octet-stream",
|
||||||
|
validationMetadata=validationMetadata
|
||||||
|
)
|
||||||
|
|
||||||
|
return ActionResult.isSuccess(documents=[document])
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
errorMsg = f"Error creating CSV content: {str(e)}"
|
||||||
|
logger.error(errorMsg)
|
||||||
|
return ActionResult.isFailure(error=errorMsg)
|
||||||
|
|
||||||
|
|
@ -0,0 +1,157 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
"""
|
||||||
|
Create Excel Content action for JIRA operations.
|
||||||
|
Creates Excel content with custom headers.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import json
|
||||||
|
import base64
|
||||||
|
import pandas as pd
|
||||||
|
import csv as csv_module
|
||||||
|
from io import BytesIO
|
||||||
|
from datetime import datetime, UTC
|
||||||
|
from typing import Dict, Any
|
||||||
|
from modules.workflows.methods.methodBase import action
|
||||||
|
from modules.datamodels.datamodelChat import ActionResult, ActionDocument
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
@action
|
||||||
|
async def createExcelContent(self, parameters: Dict[str, Any]) -> ActionResult:
|
||||||
|
"""
|
||||||
|
Create Excel content with custom headers.
|
||||||
|
|
||||||
|
Parameters:
|
||||||
|
- data (str, required): Document reference containing data as JSON (with "data" field from mergeTicketData)
|
||||||
|
- headers (str, optional): Document reference containing headers JSON (from parseExcelContent)
|
||||||
|
- columns (str or list, optional): List of column names (if not provided, extracted from taskSyncDefinition or data)
|
||||||
|
- taskSyncDefinition (str or dict, optional): Field mapping definition (used to extract column names if columns not provided)
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
- ActionResult with ActionDocument containing Excel content as bytes
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
dataParam = parameters.get("data")
|
||||||
|
if not dataParam:
|
||||||
|
return ActionResult.isFailure(error="data parameter is required")
|
||||||
|
|
||||||
|
headersParam = parameters.get("headers")
|
||||||
|
columnsParam = parameters.get("columns")
|
||||||
|
taskSyncDefinitionParam = parameters.get("taskSyncDefinition")
|
||||||
|
|
||||||
|
# Get data from document
|
||||||
|
dataJson = self.documentParsing.parseJsonFromDocument(dataParam)
|
||||||
|
if dataJson is None:
|
||||||
|
return ActionResult.isFailure(error="Could not parse data from document reference")
|
||||||
|
|
||||||
|
# Extract data array if wrapped in object
|
||||||
|
if isinstance(dataJson, dict) and "data" in dataJson:
|
||||||
|
dataList = dataJson["data"]
|
||||||
|
elif isinstance(dataJson, list):
|
||||||
|
dataList = dataJson
|
||||||
|
else:
|
||||||
|
return ActionResult.isFailure(error="Data must be a JSON array or object with 'data' field")
|
||||||
|
|
||||||
|
# Get headers
|
||||||
|
headers = {"header1": "Header 1", "header2": "Header 2"}
|
||||||
|
if headersParam:
|
||||||
|
headersJson = self.documentParsing.parseJsonFromDocument(headersParam)
|
||||||
|
if headersJson and isinstance(headersJson, dict) and "headers" in headersJson:
|
||||||
|
headers = headersJson["headers"]
|
||||||
|
elif headersJson and isinstance(headersJson, dict):
|
||||||
|
headers = headersJson
|
||||||
|
|
||||||
|
# Get columns
|
||||||
|
if columnsParam:
|
||||||
|
if isinstance(columnsParam, str):
|
||||||
|
try:
|
||||||
|
columns = json.loads(columnsParam) if columnsParam.startswith('[') or columnsParam.startswith('{') else columnsParam.split(',')
|
||||||
|
except:
|
||||||
|
columns = columnsParam.split(',')
|
||||||
|
elif isinstance(columnsParam, list):
|
||||||
|
columns = columnsParam
|
||||||
|
else:
|
||||||
|
columns = None
|
||||||
|
elif taskSyncDefinitionParam:
|
||||||
|
# Extract columns from taskSyncDefinition
|
||||||
|
if isinstance(taskSyncDefinitionParam, str):
|
||||||
|
taskSyncDefinition = json.loads(taskSyncDefinitionParam)
|
||||||
|
else:
|
||||||
|
taskSyncDefinition = taskSyncDefinitionParam
|
||||||
|
columns = list(taskSyncDefinition.keys())
|
||||||
|
elif dataList and len(dataList) > 0:
|
||||||
|
columns = list(dataList[0].keys())
|
||||||
|
else:
|
||||||
|
columns = []
|
||||||
|
|
||||||
|
# Create DataFrame
|
||||||
|
if not dataList:
|
||||||
|
df = pd.DataFrame(columns=columns)
|
||||||
|
else:
|
||||||
|
df = pd.DataFrame(dataList)
|
||||||
|
# Ensure all columns exist
|
||||||
|
for col in columns:
|
||||||
|
if col not in df.columns:
|
||||||
|
df[col] = ""
|
||||||
|
# Reorder columns
|
||||||
|
df = df[columns]
|
||||||
|
|
||||||
|
# Clean data
|
||||||
|
for column in df.columns:
|
||||||
|
df[column] = df[column].astype("object").fillna("")
|
||||||
|
df[column] = df[column].astype(str).str.replace('\n', '\\n', regex=False).str.replace('"', '""', regex=False)
|
||||||
|
|
||||||
|
# Create headers with timestamp
|
||||||
|
timestamp = datetime.fromtimestamp(self.services.utils.timestampGetUtc(), UTC).strftime("%Y-%m-%d %H:%M:%S UTC")
|
||||||
|
header1Row = next(csv_module.reader([headers.get("header1", "Header 1")]), [])
|
||||||
|
header2Row = next(csv_module.reader([headers.get("header2", "Header 2")]), [])
|
||||||
|
if len(header2Row) > 1:
|
||||||
|
header2Row[1] = timestamp
|
||||||
|
|
||||||
|
headerRow1 = pd.DataFrame([header1Row + [""] * (len(df.columns) - len(header1Row))], columns=df.columns)
|
||||||
|
headerRow2 = pd.DataFrame([header2Row + [""] * (len(df.columns) - len(header2Row))], columns=df.columns)
|
||||||
|
tableHeaders = pd.DataFrame([df.columns.tolist()], columns=df.columns)
|
||||||
|
finalDf = pd.concat([headerRow1, headerRow2, tableHeaders, df], ignore_index=True)
|
||||||
|
|
||||||
|
# Convert to Excel bytes
|
||||||
|
buf = BytesIO()
|
||||||
|
finalDf.to_excel(buf, index=False, header=False, engine='openpyxl')
|
||||||
|
excelBytes = buf.getvalue()
|
||||||
|
|
||||||
|
logger.info(f"Created Excel content: {len(dataList)} rows, {len(columns)} columns")
|
||||||
|
|
||||||
|
# Generate filename
|
||||||
|
workflowContext = self.services.chat.getWorkflowContext() if hasattr(self.services, 'chat') else None
|
||||||
|
filename = self._generateMeaningfulFileName(
|
||||||
|
"ticket_sync",
|
||||||
|
"xlsx",
|
||||||
|
workflowContext,
|
||||||
|
"createExcelContent"
|
||||||
|
)
|
||||||
|
|
||||||
|
validationMetadata = self._createValidationMetadata(
|
||||||
|
"createExcelContent",
|
||||||
|
rowCount=len(dataList),
|
||||||
|
columnCount=len(columns)
|
||||||
|
)
|
||||||
|
|
||||||
|
# Store as base64 for document
|
||||||
|
excelBase64 = base64.b64encode(excelBytes).decode('utf-8')
|
||||||
|
|
||||||
|
document = ActionDocument(
|
||||||
|
documentName=filename,
|
||||||
|
documentData=excelBase64,
|
||||||
|
mimeType="application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
|
||||||
|
validationMetadata=validationMetadata
|
||||||
|
)
|
||||||
|
|
||||||
|
return ActionResult.isSuccess(documents=[document])
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
errorMsg = f"Error creating Excel content: {str(e)}"
|
||||||
|
logger.error(errorMsg)
|
||||||
|
return ActionResult.isFailure(error=errorMsg)
|
||||||
|
|
||||||
|
|
@ -0,0 +1,84 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
"""
|
||||||
|
Export Tickets As JSON action for JIRA operations.
|
||||||
|
Exports tickets from JIRA as JSON list.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import json
|
||||||
|
from typing import Dict, Any
|
||||||
|
from modules.workflows.methods.methodBase import action
|
||||||
|
from modules.datamodels.datamodelChat import ActionResult, ActionDocument
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
@action
|
||||||
|
async def exportTicketsAsJson(self, parameters: Dict[str, Any]) -> ActionResult:
|
||||||
|
"""
|
||||||
|
Export tickets from JIRA as JSON list.
|
||||||
|
|
||||||
|
Parameters:
|
||||||
|
- connectionId (str, required): Connection ID from connectJira action result
|
||||||
|
- taskSyncDefinition (str or dict, optional): Field mapping definition (if not provided, uses stored definition)
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
- ActionResult with ActionDocument containing list of tickets as JSON
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
connectionIdParam = parameters.get("connectionId")
|
||||||
|
if not connectionIdParam:
|
||||||
|
return ActionResult.isFailure(error="connectionId parameter is required")
|
||||||
|
|
||||||
|
# Get connection ID from document if it's a reference
|
||||||
|
connectionId = None
|
||||||
|
if isinstance(connectionIdParam, str):
|
||||||
|
# Try to parse from document reference
|
||||||
|
connectionInfo = self.documentParsing.parseJsonFromDocument(connectionIdParam)
|
||||||
|
if connectionInfo and "connectionId" in connectionInfo:
|
||||||
|
connectionId = connectionInfo["connectionId"]
|
||||||
|
else:
|
||||||
|
# Assume it's the connection ID directly
|
||||||
|
connectionId = connectionIdParam
|
||||||
|
|
||||||
|
if not connectionId or connectionId not in self._connections:
|
||||||
|
return ActionResult.isFailure(error=f"Connection ID {connectionIdParam} not found. Ensure connectJira was called first.")
|
||||||
|
|
||||||
|
connection = self._connections[connectionId]
|
||||||
|
syncInterface = connection["interface"]
|
||||||
|
|
||||||
|
# Export tickets
|
||||||
|
dataList = await syncInterface.exportTicketsAsList()
|
||||||
|
|
||||||
|
logger.info(f"Exported {len(dataList)} tickets from JIRA")
|
||||||
|
|
||||||
|
# Generate filename
|
||||||
|
workflowContext = self.services.chat.getWorkflowContext() if hasattr(self.services, 'chat') else None
|
||||||
|
filename = self._generateMeaningfulFileName(
|
||||||
|
"jira_tickets_export",
|
||||||
|
"json",
|
||||||
|
workflowContext,
|
||||||
|
"exportTicketsAsJson"
|
||||||
|
)
|
||||||
|
|
||||||
|
validationMetadata = self._createValidationMetadata(
|
||||||
|
"exportTicketsAsJson",
|
||||||
|
connectionId=connectionId,
|
||||||
|
ticketCount=len(dataList)
|
||||||
|
)
|
||||||
|
|
||||||
|
document = ActionDocument(
|
||||||
|
documentName=filename,
|
||||||
|
documentData=json.dumps(dataList, indent=2, ensure_ascii=False),
|
||||||
|
mimeType="application/json",
|
||||||
|
validationMetadata=validationMetadata
|
||||||
|
)
|
||||||
|
|
||||||
|
return ActionResult.isSuccess(documents=[document])
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
errorMsg = f"Error exporting tickets from JIRA: {str(e)}"
|
||||||
|
logger.error(errorMsg)
|
||||||
|
return ActionResult.isFailure(error=errorMsg)
|
||||||
|
|
||||||
|
|
@ -0,0 +1,101 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
"""
|
||||||
|
Import Tickets From JSON action for JIRA operations.
|
||||||
|
Imports ticket data from JSON back to JIRA.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import json
|
||||||
|
from typing import Dict, Any
|
||||||
|
from modules.workflows.methods.methodBase import action
|
||||||
|
from modules.datamodels.datamodelChat import ActionResult, ActionDocument
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
@action
|
||||||
|
async def importTicketsFromJson(self, parameters: Dict[str, Any]) -> ActionResult:
|
||||||
|
"""
|
||||||
|
Import ticket data from JSON back to JIRA.
|
||||||
|
|
||||||
|
Parameters:
|
||||||
|
- connectionId (str, required): Connection ID from connectJira action result
|
||||||
|
- ticketData (str, required): Document reference containing ticket data as JSON
|
||||||
|
- taskSyncDefinition (str or dict, optional): Field mapping definition (if not provided, uses stored definition)
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
- ActionResult with ActionDocument containing import result with counts
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
connectionIdParam = parameters.get("connectionId")
|
||||||
|
if not connectionIdParam:
|
||||||
|
return ActionResult.isFailure(error="connectionId parameter is required")
|
||||||
|
|
||||||
|
ticketDataParam = parameters.get("ticketData")
|
||||||
|
if not ticketDataParam:
|
||||||
|
return ActionResult.isFailure(error="ticketData parameter is required")
|
||||||
|
|
||||||
|
# Get connection ID from document if it's a reference
|
||||||
|
connectionId = None
|
||||||
|
if isinstance(connectionIdParam, str):
|
||||||
|
connectionInfo = self.documentParsing.parseJsonFromDocument(connectionIdParam)
|
||||||
|
if connectionInfo and "connectionId" in connectionInfo:
|
||||||
|
connectionId = connectionInfo["connectionId"]
|
||||||
|
else:
|
||||||
|
connectionId = connectionIdParam
|
||||||
|
|
||||||
|
if not connectionId or connectionId not in self._connections:
|
||||||
|
return ActionResult.isFailure(error=f"Connection ID {connectionIdParam} not found. Ensure connectJira was called first.")
|
||||||
|
|
||||||
|
connection = self._connections[connectionId]
|
||||||
|
syncInterface = connection["interface"]
|
||||||
|
|
||||||
|
# Get ticket data from document
|
||||||
|
ticketDataJson = self.documentParsing.parseJsonFromDocument(ticketDataParam)
|
||||||
|
if ticketDataJson is None:
|
||||||
|
return ActionResult.isFailure(error="Could not parse ticket data from document reference")
|
||||||
|
|
||||||
|
# Ensure it's a list
|
||||||
|
if not isinstance(ticketDataJson, list):
|
||||||
|
return ActionResult.isFailure(error="ticketData must be a JSON array")
|
||||||
|
|
||||||
|
# Import tickets
|
||||||
|
await syncInterface.importListToTickets(ticketDataJson)
|
||||||
|
|
||||||
|
logger.info(f"Imported {len(ticketDataJson)} tickets to JIRA")
|
||||||
|
|
||||||
|
# Generate filename
|
||||||
|
workflowContext = self.services.chat.getWorkflowContext() if hasattr(self.services, 'chat') else None
|
||||||
|
filename = self._generateMeaningfulFileName(
|
||||||
|
"jira_import_result",
|
||||||
|
"json",
|
||||||
|
workflowContext,
|
||||||
|
"importTicketsFromJson"
|
||||||
|
)
|
||||||
|
|
||||||
|
importResult = {
|
||||||
|
"imported": len(ticketDataJson),
|
||||||
|
"connectionId": connectionId,
|
||||||
|
}
|
||||||
|
|
||||||
|
validationMetadata = self._createValidationMetadata(
|
||||||
|
"importTicketsFromJson",
|
||||||
|
connectionId=connectionId,
|
||||||
|
importedCount=len(ticketDataJson)
|
||||||
|
)
|
||||||
|
|
||||||
|
document = ActionDocument(
|
||||||
|
documentName=filename,
|
||||||
|
documentData=json.dumps(importResult, indent=2),
|
||||||
|
mimeType="application/json",
|
||||||
|
validationMetadata=validationMetadata
|
||||||
|
)
|
||||||
|
|
||||||
|
return ActionResult.isSuccess(documents=[document])
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
errorMsg = f"Error importing tickets to JIRA: {str(e)}"
|
||||||
|
logger.error(errorMsg)
|
||||||
|
return ActionResult.isFailure(error=errorMsg)
|
||||||
|
|
||||||
157
modules/workflows/methods/methodJira/actions/mergeTicketData.py
Normal file
157
modules/workflows/methods/methodJira/actions/mergeTicketData.py
Normal file
|
|
@ -0,0 +1,157 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
"""
|
||||||
|
Merge Ticket Data action for JIRA operations.
|
||||||
|
Merges JIRA export data with existing SharePoint data.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import json
|
||||||
|
from typing import Dict, Any, List
|
||||||
|
from modules.workflows.methods.methodBase import action
|
||||||
|
from modules.datamodels.datamodelChat import ActionResult, ActionDocument
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
@action
|
||||||
|
async def mergeTicketData(self, parameters: Dict[str, Any]) -> ActionResult:
|
||||||
|
"""
|
||||||
|
Merge JIRA export data with existing SharePoint data.
|
||||||
|
|
||||||
|
Parameters:
|
||||||
|
- jiraData (str, required): Document reference containing JIRA ticket data as JSON array
|
||||||
|
- existingData (str, required): Document reference containing existing SharePoint data as JSON array
|
||||||
|
- taskSyncDefinition (str or dict, required): Field mapping definition
|
||||||
|
- idField (str, optional): Field name to use as ID for merging (default: "ID")
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
- ActionResult with ActionDocument containing merged data and merge details
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
jiraDataParam = parameters.get("jiraData")
|
||||||
|
if not jiraDataParam:
|
||||||
|
return ActionResult.isFailure(error="jiraData parameter is required")
|
||||||
|
|
||||||
|
existingDataParam = parameters.get("existingData")
|
||||||
|
if not existingDataParam:
|
||||||
|
return ActionResult.isFailure(error="existingData parameter is required")
|
||||||
|
|
||||||
|
taskSyncDefinitionParam = parameters.get("taskSyncDefinition")
|
||||||
|
if not taskSyncDefinitionParam:
|
||||||
|
return ActionResult.isFailure(error="taskSyncDefinition parameter is required")
|
||||||
|
|
||||||
|
idField = parameters.get("idField", "ID")
|
||||||
|
|
||||||
|
# Parse taskSyncDefinition
|
||||||
|
if isinstance(taskSyncDefinitionParam, str):
|
||||||
|
try:
|
||||||
|
taskSyncDefinition = json.loads(taskSyncDefinitionParam)
|
||||||
|
except json.JSONDecodeError as e:
|
||||||
|
return ActionResult.isFailure(error=f"taskSyncDefinition is not valid JSON: {str(e)}")
|
||||||
|
elif isinstance(taskSyncDefinitionParam, dict):
|
||||||
|
taskSyncDefinition = taskSyncDefinitionParam
|
||||||
|
else:
|
||||||
|
return ActionResult.isFailure(error=f"taskSyncDefinition must be a dict or JSON string, got {type(taskSyncDefinitionParam)}")
|
||||||
|
|
||||||
|
# Get data from documents
|
||||||
|
jiraDataJson = self.documentParsing.parseJsonFromDocument(jiraDataParam)
|
||||||
|
if jiraDataJson is None or not isinstance(jiraDataJson, list):
|
||||||
|
return ActionResult.isFailure(error="Could not parse jiraData as JSON array")
|
||||||
|
|
||||||
|
existingDataJson = self.documentParsing.parseJsonFromDocument(existingDataParam)
|
||||||
|
if existingDataJson is None or not isinstance(existingDataJson, list):
|
||||||
|
# Empty existing data is OK
|
||||||
|
existingDataJson = []
|
||||||
|
|
||||||
|
# Perform merge
|
||||||
|
existingLookup = {row.get(idField): row for row in existingDataJson if row.get(idField)}
|
||||||
|
mergedData: List[dict] = []
|
||||||
|
changes: List[str] = []
|
||||||
|
updatedCount = addedCount = unchangedCount = 0
|
||||||
|
|
||||||
|
for jiraRow in jiraDataJson:
|
||||||
|
jiraId = jiraRow.get(idField)
|
||||||
|
if jiraId and jiraId in existingLookup:
|
||||||
|
existingRow = existingLookup[jiraId].copy()
|
||||||
|
rowChanges: List[str] = []
|
||||||
|
|
||||||
|
for fieldName, fieldConfig in taskSyncDefinition.items():
|
||||||
|
if fieldConfig[0] == 'get':
|
||||||
|
oldValue = "" if existingRow.get(fieldName) is None else str(existingRow.get(fieldName))
|
||||||
|
newValue = "" if jiraRow.get(fieldName) is None else str(jiraRow.get(fieldName))
|
||||||
|
|
||||||
|
# Convert ADF data to readable text for logging
|
||||||
|
if isinstance(newValue, dict) and newValue.get("type") == "doc":
|
||||||
|
newValueReadable = self.adfConverter.convertAdfToText(newValue)
|
||||||
|
if oldValue != newValueReadable:
|
||||||
|
rowChanges.append(f"{fieldName}: '{oldValue[:100]}...' -> '{newValueReadable[:100]}...'")
|
||||||
|
elif oldValue != newValue:
|
||||||
|
# Truncate long values for logging
|
||||||
|
oldTruncated = oldValue[:100] + "..." if len(oldValue) > 100 else oldValue
|
||||||
|
newTruncated = newValue[:100] + "..." if len(newValue) > 100 else newValue
|
||||||
|
rowChanges.append(f"{fieldName}: '{oldTruncated}' -> '{newTruncated}'")
|
||||||
|
|
||||||
|
existingRow[fieldName] = jiraRow.get(fieldName)
|
||||||
|
|
||||||
|
mergedData.append(existingRow)
|
||||||
|
if rowChanges:
|
||||||
|
updatedCount += 1
|
||||||
|
changes.append(f"Row ID {jiraId} updated: {', '.join(rowChanges)}")
|
||||||
|
else:
|
||||||
|
unchangedCount += 1
|
||||||
|
del existingLookup[jiraId]
|
||||||
|
else:
|
||||||
|
mergedData.append(jiraRow)
|
||||||
|
addedCount += 1
|
||||||
|
changes.append(f"Row ID {jiraId} added as new record")
|
||||||
|
|
||||||
|
# Add remaining existing rows
|
||||||
|
for remaining in existingLookup.values():
|
||||||
|
mergedData.append(remaining)
|
||||||
|
unchangedCount += 1
|
||||||
|
|
||||||
|
mergeDetails = {
|
||||||
|
"updated": updatedCount,
|
||||||
|
"added": addedCount,
|
||||||
|
"unchanged": unchangedCount,
|
||||||
|
"changes": changes
|
||||||
|
}
|
||||||
|
|
||||||
|
logger.info(f"Merged ticket data: {updatedCount} updated, {addedCount} added, {unchangedCount} unchanged")
|
||||||
|
|
||||||
|
# Generate filename
|
||||||
|
workflowContext = self.services.chat.getWorkflowContext() if hasattr(self.services, 'chat') else None
|
||||||
|
filename = self._generateMeaningfulFileName(
|
||||||
|
"merged_ticket_data",
|
||||||
|
"json",
|
||||||
|
workflowContext,
|
||||||
|
"mergeTicketData"
|
||||||
|
)
|
||||||
|
|
||||||
|
result = {
|
||||||
|
"data": mergedData,
|
||||||
|
"mergeDetails": mergeDetails
|
||||||
|
}
|
||||||
|
|
||||||
|
validationMetadata = self._createValidationMetadata(
|
||||||
|
"mergeTicketData",
|
||||||
|
updated=updatedCount,
|
||||||
|
added=addedCount,
|
||||||
|
unchanged=unchangedCount
|
||||||
|
)
|
||||||
|
|
||||||
|
document = ActionDocument(
|
||||||
|
documentName=filename,
|
||||||
|
documentData=json.dumps(result, indent=2, ensure_ascii=False),
|
||||||
|
mimeType="application/json",
|
||||||
|
validationMetadata=validationMetadata
|
||||||
|
)
|
||||||
|
|
||||||
|
return ActionResult.isSuccess(documents=[document])
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
errorMsg = f"Error merging ticket data: {str(e)}"
|
||||||
|
logger.error(errorMsg)
|
||||||
|
return ActionResult.isFailure(error=errorMsg)
|
||||||
|
|
||||||
112
modules/workflows/methods/methodJira/actions/parseCsvContent.py
Normal file
112
modules/workflows/methods/methodJira/actions/parseCsvContent.py
Normal file
|
|
@ -0,0 +1,112 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
"""
|
||||||
|
Parse CSV Content action for JIRA operations.
|
||||||
|
Parses CSV content with custom headers.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import json
|
||||||
|
import io
|
||||||
|
import pandas as pd
|
||||||
|
from typing import Dict, Any
|
||||||
|
from modules.workflows.methods.methodBase import action
|
||||||
|
from modules.datamodels.datamodelChat import ActionResult, ActionDocument
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
@action
|
||||||
|
async def parseCsvContent(self, parameters: Dict[str, Any]) -> ActionResult:
|
||||||
|
"""
|
||||||
|
Parse CSV content with custom headers.
|
||||||
|
|
||||||
|
Parameters:
|
||||||
|
- csvContent (str, required): Document reference containing CSV file content as bytes
|
||||||
|
- skipRows (int, optional): Number of header rows to skip (default: 2)
|
||||||
|
- hasCustomHeaders (bool, optional): Whether CSV has custom header rows (default: true)
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
- ActionResult with ActionDocument containing parsed data and headers as JSON
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
csvContentParam = parameters.get("csvContent")
|
||||||
|
if not csvContentParam:
|
||||||
|
return ActionResult.isFailure(error="csvContent parameter is required")
|
||||||
|
|
||||||
|
skipRows = parameters.get("skipRows", 2)
|
||||||
|
hasCustomHeaders = parameters.get("hasCustomHeaders", True)
|
||||||
|
|
||||||
|
# Get CSV content from document
|
||||||
|
csvBytes = self.documentParsing.getDocumentData(csvContentParam)
|
||||||
|
if csvBytes is None:
|
||||||
|
return ActionResult.isFailure(error="Could not get CSV content from document reference")
|
||||||
|
|
||||||
|
# Convert to bytes if needed
|
||||||
|
if isinstance(csvBytes, str):
|
||||||
|
csvBytes = csvBytes.encode('utf-8')
|
||||||
|
elif not isinstance(csvBytes, bytes):
|
||||||
|
return ActionResult.isFailure(error="CSV content must be bytes or string")
|
||||||
|
|
||||||
|
# Parse headers if hasCustomHeaders
|
||||||
|
headers = {"header1": "Header 1", "header2": "Header 2"}
|
||||||
|
if hasCustomHeaders:
|
||||||
|
csvLines = csvBytes.decode('utf-8').split('\n')
|
||||||
|
if len(csvLines) >= 2:
|
||||||
|
headers["header1"] = csvLines[0].rstrip('\r\n')
|
||||||
|
headers["header2"] = csvLines[1].rstrip('\r\n')
|
||||||
|
|
||||||
|
# Parse CSV data
|
||||||
|
df = pd.read_csv(
|
||||||
|
io.BytesIO(csvBytes),
|
||||||
|
skiprows=skipRows,
|
||||||
|
quoting=1,
|
||||||
|
escapechar='\\',
|
||||||
|
on_bad_lines='skip',
|
||||||
|
engine='python'
|
||||||
|
)
|
||||||
|
|
||||||
|
# Convert to dict records
|
||||||
|
for column in df.columns:
|
||||||
|
df[column] = df[column].astype('object').fillna('')
|
||||||
|
data = df.to_dict(orient='records')
|
||||||
|
|
||||||
|
logger.info(f"Parsed CSV: {len(data)} rows, {len(df.columns)} columns")
|
||||||
|
|
||||||
|
# Generate filename
|
||||||
|
workflowContext = self.services.chat.getWorkflowContext() if hasattr(self.services, 'chat') else None
|
||||||
|
filename = self._generateMeaningfulFileName(
|
||||||
|
"parsed_csv_data",
|
||||||
|
"json",
|
||||||
|
workflowContext,
|
||||||
|
"parseCsvContent"
|
||||||
|
)
|
||||||
|
|
||||||
|
result = {
|
||||||
|
"data": data,
|
||||||
|
"headers": headers,
|
||||||
|
"rowCount": len(data),
|
||||||
|
"columnCount": len(df.columns)
|
||||||
|
}
|
||||||
|
|
||||||
|
validationMetadata = self._createValidationMetadata(
|
||||||
|
"parseCsvContent",
|
||||||
|
rowCount=len(data),
|
||||||
|
columnCount=len(df.columns),
|
||||||
|
skipRows=skipRows
|
||||||
|
)
|
||||||
|
|
||||||
|
document = ActionDocument(
|
||||||
|
documentName=filename,
|
||||||
|
documentData=json.dumps(result, indent=2, ensure_ascii=False),
|
||||||
|
mimeType="application/json",
|
||||||
|
validationMetadata=validationMetadata
|
||||||
|
)
|
||||||
|
|
||||||
|
return ActionResult.isSuccess(documents=[document])
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
errorMsg = f"Error parsing CSV content: {str(e)}"
|
||||||
|
logger.error(errorMsg)
|
||||||
|
return ActionResult.isFailure(error=errorMsg)
|
||||||
|
|
||||||
|
|
@ -0,0 +1,121 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
"""
|
||||||
|
Parse Excel Content action for JIRA operations.
|
||||||
|
Parses Excel content with custom headers.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import json
|
||||||
|
import pandas as pd
|
||||||
|
from io import BytesIO
|
||||||
|
from typing import Dict, Any
|
||||||
|
from modules.workflows.methods.methodBase import action
|
||||||
|
from modules.datamodels.datamodelChat import ActionResult, ActionDocument
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
@action
|
||||||
|
async def parseExcelContent(self, parameters: Dict[str, Any]) -> ActionResult:
|
||||||
|
"""
|
||||||
|
Parse Excel content with custom headers.
|
||||||
|
|
||||||
|
Parameters:
|
||||||
|
- excelContent (str, required): Document reference containing Excel file content as bytes
|
||||||
|
- skipRows (int, optional): Number of header rows to skip (default: 3)
|
||||||
|
- hasCustomHeaders (bool, optional): Whether Excel has custom header rows (default: true)
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
- ActionResult with ActionDocument containing parsed data and headers as JSON
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
excelContentParam = parameters.get("excelContent")
|
||||||
|
if not excelContentParam:
|
||||||
|
return ActionResult.isFailure(error="excelContent parameter is required")
|
||||||
|
|
||||||
|
skipRows = parameters.get("skipRows", 3)
|
||||||
|
hasCustomHeaders = parameters.get("hasCustomHeaders", True)
|
||||||
|
|
||||||
|
# Get Excel content from document
|
||||||
|
excelBytes = self.documentParsing.getDocumentData(excelContentParam)
|
||||||
|
if excelBytes is None:
|
||||||
|
return ActionResult.isFailure(error="Could not get Excel content from document reference")
|
||||||
|
|
||||||
|
# Convert to bytes if needed
|
||||||
|
if isinstance(excelBytes, str):
|
||||||
|
excelBytes = excelBytes.encode('latin-1') # Excel might have binary data
|
||||||
|
elif not isinstance(excelBytes, bytes):
|
||||||
|
return ActionResult.isFailure(error="Excel content must be bytes or string")
|
||||||
|
|
||||||
|
# Parse Excel
|
||||||
|
df = pd.read_excel(BytesIO(excelBytes), engine='openpyxl', header=None)
|
||||||
|
|
||||||
|
# Extract headers if hasCustomHeaders
|
||||||
|
headers = {"header1": "Header 1", "header2": "Header 2"}
|
||||||
|
if hasCustomHeaders and len(df) >= 3:
|
||||||
|
headerRow1 = df.iloc[0:1].copy()
|
||||||
|
headerRow2 = df.iloc[1:2].copy()
|
||||||
|
tableHeaders = df.iloc[2:3].copy()
|
||||||
|
dfData = df.iloc[skipRows:].copy()
|
||||||
|
dfData.columns = tableHeaders.iloc[0]
|
||||||
|
|
||||||
|
headers = {
|
||||||
|
"header1": ",".join([str(x) if pd.notna(x) else "" for x in headerRow1.iloc[0].tolist()]),
|
||||||
|
"header2": ",".join([str(x) if pd.notna(x) else "" for x in headerRow2.iloc[0].tolist()]),
|
||||||
|
}
|
||||||
|
else:
|
||||||
|
# No custom headers, use standard parsing
|
||||||
|
if skipRows > 0:
|
||||||
|
dfData = df.iloc[skipRows:].copy()
|
||||||
|
if len(df) > skipRows:
|
||||||
|
dfData.columns = df.iloc[skipRows-1]
|
||||||
|
else:
|
||||||
|
dfData = df.copy()
|
||||||
|
|
||||||
|
# Reset index and clean data
|
||||||
|
dfData = dfData.reset_index(drop=True)
|
||||||
|
for column in dfData.columns:
|
||||||
|
dfData[column] = dfData[column].astype('object').fillna('')
|
||||||
|
|
||||||
|
data = dfData.to_dict(orient='records')
|
||||||
|
|
||||||
|
logger.info(f"Parsed Excel: {len(data)} rows, {len(dfData.columns)} columns")
|
||||||
|
|
||||||
|
# Generate filename
|
||||||
|
workflowContext = self.services.chat.getWorkflowContext() if hasattr(self.services, 'chat') else None
|
||||||
|
filename = self._generateMeaningfulFileName(
|
||||||
|
"parsed_excel_data",
|
||||||
|
"json",
|
||||||
|
workflowContext,
|
||||||
|
"parseExcelContent"
|
||||||
|
)
|
||||||
|
|
||||||
|
result = {
|
||||||
|
"data": data,
|
||||||
|
"headers": headers,
|
||||||
|
"rowCount": len(data),
|
||||||
|
"columnCount": len(dfData.columns)
|
||||||
|
}
|
||||||
|
|
||||||
|
validationMetadata = self._createValidationMetadata(
|
||||||
|
"parseExcelContent",
|
||||||
|
rowCount=len(data),
|
||||||
|
columnCount=len(dfData.columns),
|
||||||
|
skipRows=skipRows
|
||||||
|
)
|
||||||
|
|
||||||
|
document = ActionDocument(
|
||||||
|
documentName=filename,
|
||||||
|
documentData=json.dumps(result, indent=2, ensure_ascii=False),
|
||||||
|
mimeType="application/json",
|
||||||
|
validationMetadata=validationMetadata
|
||||||
|
)
|
||||||
|
|
||||||
|
return ActionResult.isSuccess(documents=[document])
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
errorMsg = f"Error parsing Excel content: {str(e)}"
|
||||||
|
logger.error(errorMsg)
|
||||||
|
return ActionResult.isFailure(error=errorMsg)
|
||||||
|
|
||||||
5
modules/workflows/methods/methodJira/helpers/__init__.py
Normal file
5
modules/workflows/methods/methodJira/helpers/__init__.py
Normal file
|
|
@ -0,0 +1,5 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
"""Helper modules for JIRA method operations."""
|
||||||
|
|
||||||
180
modules/workflows/methods/methodJira/helpers/adfConverter.py
Normal file
180
modules/workflows/methods/methodJira/helpers/adfConverter.py
Normal file
|
|
@ -0,0 +1,180 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
"""
|
||||||
|
ADF Converter helper for JIRA operations.
|
||||||
|
Handles conversion of Atlassian Document Format (ADF) to plain text.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
class AdfConverterHelper:
|
||||||
|
"""Helper for ADF conversion operations"""
|
||||||
|
|
||||||
|
def __init__(self, methodInstance):
|
||||||
|
"""
|
||||||
|
Initialize ADF converter helper.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
methodInstance: Instance of MethodJira (for access to services)
|
||||||
|
"""
|
||||||
|
self.method = methodInstance
|
||||||
|
self.services = methodInstance.services
|
||||||
|
|
||||||
|
def convertAdfToText(self, adfData):
|
||||||
|
"""Convert Atlassian Document Format (ADF) to plain text.
|
||||||
|
|
||||||
|
Based on Atlassian Document Format specification for JIRA fields.
|
||||||
|
Handles paragraphs, lists, text formatting, and other ADF node types.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
adfData: ADF object or None
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
str: Plain text content, or empty string if None/invalid
|
||||||
|
"""
|
||||||
|
if not adfData or not isinstance(adfData, dict):
|
||||||
|
return ""
|
||||||
|
|
||||||
|
if adfData.get("type") != "doc":
|
||||||
|
return str(adfData) if adfData else ""
|
||||||
|
|
||||||
|
content = adfData.get("content", [])
|
||||||
|
if not isinstance(content, list):
|
||||||
|
return ""
|
||||||
|
|
||||||
|
def extractTextFromContent(contentList, listLevel=0):
|
||||||
|
"""Recursively extract text from ADF content with proper formatting."""
|
||||||
|
textParts = []
|
||||||
|
listCounter = 1
|
||||||
|
|
||||||
|
for item in contentList:
|
||||||
|
if not isinstance(item, dict):
|
||||||
|
continue
|
||||||
|
|
||||||
|
itemType = item.get("type", "")
|
||||||
|
|
||||||
|
if itemType == "text":
|
||||||
|
# Extract text content, preserving formatting
|
||||||
|
text = item.get("text", "")
|
||||||
|
marks = item.get("marks", [])
|
||||||
|
|
||||||
|
# Handle text formatting (bold, italic, etc.)
|
||||||
|
if marks:
|
||||||
|
for mark in marks:
|
||||||
|
if mark.get("type") == "strong":
|
||||||
|
text = f"**{text}**"
|
||||||
|
elif mark.get("type") == "em":
|
||||||
|
text = f"*{text}*"
|
||||||
|
elif mark.get("type") == "code":
|
||||||
|
text = f"`{text}`"
|
||||||
|
elif mark.get("type") == "link":
|
||||||
|
attrs = mark.get("attrs", {})
|
||||||
|
href = attrs.get("href", "")
|
||||||
|
if href:
|
||||||
|
text = f"[{text}]({href})"
|
||||||
|
|
||||||
|
textParts.append(text)
|
||||||
|
|
||||||
|
elif itemType == "hardBreak":
|
||||||
|
textParts.append("\n")
|
||||||
|
|
||||||
|
elif itemType == "paragraph":
|
||||||
|
paragraphContent = item.get("content", [])
|
||||||
|
if paragraphContent:
|
||||||
|
paragraphText = extractTextFromContent(paragraphContent, listLevel)
|
||||||
|
if paragraphText.strip():
|
||||||
|
textParts.append(paragraphText)
|
||||||
|
|
||||||
|
elif itemType == "bulletList":
|
||||||
|
listContent = item.get("content", [])
|
||||||
|
if listContent:
|
||||||
|
listText = extractTextFromContent(listContent, listLevel + 1)
|
||||||
|
if listText.strip():
|
||||||
|
textParts.append(listText)
|
||||||
|
|
||||||
|
elif itemType == "orderedList":
|
||||||
|
listContent = item.get("content", [])
|
||||||
|
if listContent:
|
||||||
|
listText = extractTextFromContent(listContent, listLevel + 1)
|
||||||
|
if listText.strip():
|
||||||
|
textParts.append(listText)
|
||||||
|
|
||||||
|
elif itemType == "listItem":
|
||||||
|
itemContent = item.get("content", [])
|
||||||
|
if itemContent:
|
||||||
|
indent = " " * listLevel
|
||||||
|
itemText = extractTextFromContent(itemContent, listLevel)
|
||||||
|
if itemText.strip():
|
||||||
|
prefix = f"{indent}- " if listLevel > 0 else "- "
|
||||||
|
textParts.append(f"{prefix}{itemText}")
|
||||||
|
|
||||||
|
elif itemType == "heading":
|
||||||
|
level = item.get("attrs", {}).get("level", 1)
|
||||||
|
headingContent = item.get("content", [])
|
||||||
|
if headingContent:
|
||||||
|
headingText = extractTextFromContent(headingContent, listLevel)
|
||||||
|
if headingText.strip():
|
||||||
|
prefix = "#" * level + " "
|
||||||
|
textParts.append(f"{prefix}{headingText}")
|
||||||
|
|
||||||
|
elif itemType == "codeBlock":
|
||||||
|
codeContent = item.get("content", [])
|
||||||
|
if codeContent:
|
||||||
|
codeText = extractTextFromContent(codeContent, listLevel)
|
||||||
|
if codeText.strip():
|
||||||
|
textParts.append(f"```\n{codeText}\n```")
|
||||||
|
|
||||||
|
elif itemType == "blockquote":
|
||||||
|
quoteContent = item.get("content", [])
|
||||||
|
if quoteContent:
|
||||||
|
quoteText = extractTextFromContent(quoteContent, listLevel)
|
||||||
|
if quoteText.strip():
|
||||||
|
textParts.append(f"> {quoteText}")
|
||||||
|
|
||||||
|
elif itemType == "table":
|
||||||
|
tableContent = item.get("content", [])
|
||||||
|
if tableContent:
|
||||||
|
tableText = extractTextFromContent(tableContent, listLevel)
|
||||||
|
if tableText.strip():
|
||||||
|
textParts.append(tableText)
|
||||||
|
|
||||||
|
elif itemType == "tableRow":
|
||||||
|
rowContent = item.get("content", [])
|
||||||
|
if rowContent:
|
||||||
|
rowText = extractTextFromContent(rowContent, listLevel)
|
||||||
|
if rowText.strip():
|
||||||
|
textParts.append(rowText)
|
||||||
|
|
||||||
|
elif itemType == "tableCell":
|
||||||
|
cellContent = item.get("content", [])
|
||||||
|
if cellContent:
|
||||||
|
cellText = extractTextFromContent(cellContent, listLevel)
|
||||||
|
if cellText.strip():
|
||||||
|
textParts.append(cellText)
|
||||||
|
|
||||||
|
elif itemType == "mediaGroup":
|
||||||
|
# Skip media groups for now
|
||||||
|
pass
|
||||||
|
|
||||||
|
elif itemType == "media":
|
||||||
|
# Skip media for now
|
||||||
|
pass
|
||||||
|
|
||||||
|
else:
|
||||||
|
# Unknown type - try to extract content if available
|
||||||
|
if "content" in item:
|
||||||
|
unknownContent = item.get("content", [])
|
||||||
|
if unknownContent:
|
||||||
|
unknownText = extractTextFromContent(unknownContent, listLevel)
|
||||||
|
if unknownText.strip():
|
||||||
|
textParts.append(unknownText)
|
||||||
|
|
||||||
|
return "".join(textParts)
|
||||||
|
|
||||||
|
result = extractTextFromContent(content)
|
||||||
|
return result.strip() if result else ""
|
||||||
|
|
||||||
|
|
@ -0,0 +1,81 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
"""
|
||||||
|
Document Parsing helper for JIRA operations.
|
||||||
|
Handles parsing of document references and JSON content.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import json
|
||||||
|
from typing import Any, Optional, Dict
|
||||||
|
from modules.datamodels.datamodelDocref import DocumentReferenceList
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
class DocumentParsingHelper:
|
||||||
|
"""Helper for document parsing operations"""
|
||||||
|
|
||||||
|
def __init__(self, methodInstance):
|
||||||
|
"""
|
||||||
|
Initialize document parsing helper.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
methodInstance: Instance of MethodJira (for access to services)
|
||||||
|
"""
|
||||||
|
self.method = methodInstance
|
||||||
|
self.services = methodInstance.services
|
||||||
|
|
||||||
|
def getDocumentData(self, documentReference: str) -> Any:
|
||||||
|
"""
|
||||||
|
Get document data from a document reference.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
documentReference: Document reference string
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Document data (bytes, str, or None)
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
docList = DocumentReferenceList.from_string_list([documentReference])
|
||||||
|
chatDocuments = self.services.chat.getChatDocumentsFromDocumentList(docList)
|
||||||
|
if not chatDocuments:
|
||||||
|
return None
|
||||||
|
|
||||||
|
doc = chatDocuments[0]
|
||||||
|
fileId = getattr(doc, 'fileId', None)
|
||||||
|
if not fileId:
|
||||||
|
return None
|
||||||
|
|
||||||
|
return self.services.chat.getFileData(fileId)
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error getting document data: {str(e)}")
|
||||||
|
return None
|
||||||
|
|
||||||
|
def parseJsonFromDocument(self, documentReference: str) -> Optional[Dict[str, Any]]:
|
||||||
|
"""
|
||||||
|
Parse JSON content from a document reference.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
documentReference: Document reference string
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Parsed JSON dictionary or None
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
fileData = self.getDocumentData(documentReference)
|
||||||
|
if not fileData:
|
||||||
|
return None
|
||||||
|
|
||||||
|
# Handle bytes
|
||||||
|
if isinstance(fileData, bytes):
|
||||||
|
jsonStr = fileData.decode('utf-8')
|
||||||
|
else:
|
||||||
|
jsonStr = str(fileData)
|
||||||
|
|
||||||
|
# Parse JSON
|
||||||
|
return json.loads(jsonStr)
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error parsing JSON from document: {str(e)}")
|
||||||
|
return None
|
||||||
|
|
||||||
322
modules/workflows/methods/methodJira/methodJira.py
Normal file
322
modules/workflows/methods/methodJira/methodJira.py
Normal file
|
|
@ -0,0 +1,322 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
import logging
|
||||||
|
from typing import Dict, Any
|
||||||
|
from modules.workflows.methods.methodBase import MethodBase
|
||||||
|
from modules.datamodels.datamodelWorkflowActions import WorkflowActionDefinition, WorkflowActionParameter
|
||||||
|
from modules.shared.frontendTypes import FrontendType
|
||||||
|
|
||||||
|
# Import helpers
|
||||||
|
from .helpers.adfConverter import AdfConverterHelper
|
||||||
|
from .helpers.documentParsing import DocumentParsingHelper
|
||||||
|
|
||||||
|
# Import actions
|
||||||
|
from .actions.connectJira import connectJira
|
||||||
|
from .actions.exportTicketsAsJson import exportTicketsAsJson
|
||||||
|
from .actions.importTicketsFromJson import importTicketsFromJson
|
||||||
|
from .actions.mergeTicketData import mergeTicketData
|
||||||
|
from .actions.parseCsvContent import parseCsvContent
|
||||||
|
from .actions.parseExcelContent import parseExcelContent
|
||||||
|
from .actions.createCsvContent import createCsvContent
|
||||||
|
from .actions.createExcelContent import createExcelContent
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
class MethodJira(MethodBase):
|
||||||
|
"""JIRA operations methods."""
|
||||||
|
|
||||||
|
def __init__(self, services):
|
||||||
|
super().__init__(services)
|
||||||
|
self.name = "jira"
|
||||||
|
self.description = "JIRA operations methods"
|
||||||
|
# Store connections in memory (keyed by connectionId)
|
||||||
|
self._connections: Dict[str, Any] = {}
|
||||||
|
|
||||||
|
# Initialize helper modules
|
||||||
|
self.adfConverter = AdfConverterHelper(self)
|
||||||
|
self.documentParsing = DocumentParsingHelper(self)
|
||||||
|
|
||||||
|
# RBAC-Integration: Action-Definitionen mit actionId
|
||||||
|
self._actions = {
|
||||||
|
"connectJira": WorkflowActionDefinition(
|
||||||
|
actionId="jira.connectJira",
|
||||||
|
description="Connect to JIRA instance and create ticket interface",
|
||||||
|
parameters={
|
||||||
|
"apiUsername": WorkflowActionParameter(
|
||||||
|
name="apiUsername",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.EMAIL,
|
||||||
|
required=True,
|
||||||
|
description="JIRA API username/email"
|
||||||
|
),
|
||||||
|
"apiTokenConfigKey": WorkflowActionParameter(
|
||||||
|
name="apiTokenConfigKey",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.TEXT,
|
||||||
|
required=True,
|
||||||
|
description="APP_CONFIG key name for JIRA API token"
|
||||||
|
),
|
||||||
|
"apiUrl": WorkflowActionParameter(
|
||||||
|
name="apiUrl",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.TEXT,
|
||||||
|
required=True,
|
||||||
|
description="JIRA instance URL (e.g., https://example.atlassian.net)"
|
||||||
|
),
|
||||||
|
"projectCode": WorkflowActionParameter(
|
||||||
|
name="projectCode",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.TEXT,
|
||||||
|
required=True,
|
||||||
|
description="JIRA project code (e.g., DCS)"
|
||||||
|
),
|
||||||
|
"issueType": WorkflowActionParameter(
|
||||||
|
name="issueType",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.TEXT,
|
||||||
|
required=True,
|
||||||
|
description="JIRA issue type (e.g., Task)"
|
||||||
|
),
|
||||||
|
"taskSyncDefinition": WorkflowActionParameter(
|
||||||
|
name="taskSyncDefinition",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.TEXTAREA,
|
||||||
|
required=True,
|
||||||
|
description="Field mapping definition as JSON string or dict"
|
||||||
|
)
|
||||||
|
},
|
||||||
|
execute=connectJira.__get__(self, self.__class__)
|
||||||
|
),
|
||||||
|
"exportTicketsAsJson": WorkflowActionDefinition(
|
||||||
|
actionId="jira.exportTicketsAsJson",
|
||||||
|
description="Export tickets from JIRA as JSON list",
|
||||||
|
parameters={
|
||||||
|
"connectionId": WorkflowActionParameter(
|
||||||
|
name="connectionId",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.TEXT,
|
||||||
|
required=True,
|
||||||
|
description="Connection ID from connectJira action result"
|
||||||
|
),
|
||||||
|
"taskSyncDefinition": WorkflowActionParameter(
|
||||||
|
name="taskSyncDefinition",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.TEXTAREA,
|
||||||
|
required=False,
|
||||||
|
description="Field mapping definition (if not provided, uses stored definition)"
|
||||||
|
)
|
||||||
|
},
|
||||||
|
execute=exportTicketsAsJson.__get__(self, self.__class__)
|
||||||
|
),
|
||||||
|
"importTicketsFromJson": WorkflowActionDefinition(
|
||||||
|
actionId="jira.importTicketsFromJson",
|
||||||
|
description="Import ticket data from JSON back to JIRA",
|
||||||
|
parameters={
|
||||||
|
"connectionId": WorkflowActionParameter(
|
||||||
|
name="connectionId",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.TEXT,
|
||||||
|
required=True,
|
||||||
|
description="Connection ID from connectJira action result"
|
||||||
|
),
|
||||||
|
"ticketData": WorkflowActionParameter(
|
||||||
|
name="ticketData",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.DOCUMENT_REFERENCE,
|
||||||
|
required=True,
|
||||||
|
description="Document reference containing ticket data as JSON"
|
||||||
|
),
|
||||||
|
"taskSyncDefinition": WorkflowActionParameter(
|
||||||
|
name="taskSyncDefinition",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.TEXTAREA,
|
||||||
|
required=False,
|
||||||
|
description="Field mapping definition (if not provided, uses stored definition)"
|
||||||
|
)
|
||||||
|
},
|
||||||
|
execute=importTicketsFromJson.__get__(self, self.__class__)
|
||||||
|
),
|
||||||
|
"mergeTicketData": WorkflowActionDefinition(
|
||||||
|
actionId="jira.mergeTicketData",
|
||||||
|
description="Merge JIRA export data with existing SharePoint data",
|
||||||
|
parameters={
|
||||||
|
"jiraData": WorkflowActionParameter(
|
||||||
|
name="jiraData",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.DOCUMENT_REFERENCE,
|
||||||
|
required=True,
|
||||||
|
description="Document reference containing JIRA ticket data as JSON array"
|
||||||
|
),
|
||||||
|
"existingData": WorkflowActionParameter(
|
||||||
|
name="existingData",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.DOCUMENT_REFERENCE,
|
||||||
|
required=True,
|
||||||
|
description="Document reference containing existing SharePoint data as JSON array"
|
||||||
|
),
|
||||||
|
"taskSyncDefinition": WorkflowActionParameter(
|
||||||
|
name="taskSyncDefinition",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.TEXTAREA,
|
||||||
|
required=True,
|
||||||
|
description="Field mapping definition"
|
||||||
|
),
|
||||||
|
"idField": WorkflowActionParameter(
|
||||||
|
name="idField",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.TEXT,
|
||||||
|
required=False,
|
||||||
|
default="ID",
|
||||||
|
description="Field name to use as ID for merging"
|
||||||
|
)
|
||||||
|
},
|
||||||
|
execute=mergeTicketData.__get__(self, self.__class__)
|
||||||
|
),
|
||||||
|
"parseCsvContent": WorkflowActionDefinition(
|
||||||
|
actionId="jira.parseCsvContent",
|
||||||
|
description="Parse CSV content with custom headers",
|
||||||
|
parameters={
|
||||||
|
"csvContent": WorkflowActionParameter(
|
||||||
|
name="csvContent",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.DOCUMENT_REFERENCE,
|
||||||
|
required=True,
|
||||||
|
description="Document reference containing CSV file content as bytes"
|
||||||
|
),
|
||||||
|
"skipRows": WorkflowActionParameter(
|
||||||
|
name="skipRows",
|
||||||
|
type="int",
|
||||||
|
frontendType=FrontendType.NUMBER,
|
||||||
|
required=False,
|
||||||
|
default=2,
|
||||||
|
description="Number of header rows to skip",
|
||||||
|
validation={"min": 0, "max": 100}
|
||||||
|
),
|
||||||
|
"hasCustomHeaders": WorkflowActionParameter(
|
||||||
|
name="hasCustomHeaders",
|
||||||
|
type="bool",
|
||||||
|
frontendType=FrontendType.CHECKBOX,
|
||||||
|
required=False,
|
||||||
|
default=True,
|
||||||
|
description="Whether CSV has custom header rows"
|
||||||
|
)
|
||||||
|
},
|
||||||
|
execute=parseCsvContent.__get__(self, self.__class__)
|
||||||
|
),
|
||||||
|
"parseExcelContent": WorkflowActionDefinition(
|
||||||
|
actionId="jira.parseExcelContent",
|
||||||
|
description="Parse Excel content with custom headers",
|
||||||
|
parameters={
|
||||||
|
"excelContent": WorkflowActionParameter(
|
||||||
|
name="excelContent",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.DOCUMENT_REFERENCE,
|
||||||
|
required=True,
|
||||||
|
description="Document reference containing Excel file content as bytes"
|
||||||
|
),
|
||||||
|
"skipRows": WorkflowActionParameter(
|
||||||
|
name="skipRows",
|
||||||
|
type="int",
|
||||||
|
frontendType=FrontendType.NUMBER,
|
||||||
|
required=False,
|
||||||
|
default=3,
|
||||||
|
description="Number of header rows to skip",
|
||||||
|
validation={"min": 0, "max": 100}
|
||||||
|
),
|
||||||
|
"hasCustomHeaders": WorkflowActionParameter(
|
||||||
|
name="hasCustomHeaders",
|
||||||
|
type="bool",
|
||||||
|
frontendType=FrontendType.CHECKBOX,
|
||||||
|
required=False,
|
||||||
|
default=True,
|
||||||
|
description="Whether Excel has custom header rows"
|
||||||
|
)
|
||||||
|
},
|
||||||
|
execute=parseExcelContent.__get__(self, self.__class__)
|
||||||
|
),
|
||||||
|
"createCsvContent": WorkflowActionDefinition(
|
||||||
|
actionId="jira.createCsvContent",
|
||||||
|
description="Create CSV content with custom headers",
|
||||||
|
parameters={
|
||||||
|
"data": WorkflowActionParameter(
|
||||||
|
name="data",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.DOCUMENT_REFERENCE,
|
||||||
|
required=True,
|
||||||
|
description="Document reference containing data as JSON (with data field from mergeTicketData)"
|
||||||
|
),
|
||||||
|
"headers": WorkflowActionParameter(
|
||||||
|
name="headers",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.DOCUMENT_REFERENCE,
|
||||||
|
required=False,
|
||||||
|
description="Document reference containing headers JSON (from parseCsvContent/parseExcelContent)"
|
||||||
|
),
|
||||||
|
"columns": WorkflowActionParameter(
|
||||||
|
name="columns",
|
||||||
|
type="List[str]",
|
||||||
|
frontendType=FrontendType.MULTISELECT,
|
||||||
|
required=False,
|
||||||
|
description="List of column names (if not provided, extracted from taskSyncDefinition or data)"
|
||||||
|
),
|
||||||
|
"taskSyncDefinition": WorkflowActionParameter(
|
||||||
|
name="taskSyncDefinition",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.TEXTAREA,
|
||||||
|
required=False,
|
||||||
|
description="Field mapping definition (used to extract column names if columns not provided)"
|
||||||
|
)
|
||||||
|
},
|
||||||
|
execute=createCsvContent.__get__(self, self.__class__)
|
||||||
|
),
|
||||||
|
"createExcelContent": WorkflowActionDefinition(
|
||||||
|
actionId="jira.createExcelContent",
|
||||||
|
description="Create Excel content with custom headers",
|
||||||
|
parameters={
|
||||||
|
"data": WorkflowActionParameter(
|
||||||
|
name="data",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.DOCUMENT_REFERENCE,
|
||||||
|
required=True,
|
||||||
|
description="Document reference containing data as JSON (with data field from mergeTicketData)"
|
||||||
|
),
|
||||||
|
"headers": WorkflowActionParameter(
|
||||||
|
name="headers",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.DOCUMENT_REFERENCE,
|
||||||
|
required=False,
|
||||||
|
description="Document reference containing headers JSON (from parseExcelContent)"
|
||||||
|
),
|
||||||
|
"columns": WorkflowActionParameter(
|
||||||
|
name="columns",
|
||||||
|
type="List[str]",
|
||||||
|
frontendType=FrontendType.MULTISELECT,
|
||||||
|
required=False,
|
||||||
|
description="List of column names (if not provided, extracted from taskSyncDefinition or data)"
|
||||||
|
),
|
||||||
|
"taskSyncDefinition": WorkflowActionParameter(
|
||||||
|
name="taskSyncDefinition",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.TEXTAREA,
|
||||||
|
required=False,
|
||||||
|
description="Field mapping definition (used to extract column names if columns not provided)"
|
||||||
|
)
|
||||||
|
},
|
||||||
|
execute=createExcelContent.__get__(self, self.__class__)
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
# Validate actions after definition
|
||||||
|
self._validateActions()
|
||||||
|
|
||||||
|
# Register actions as methods (optional, für direkten Zugriff)
|
||||||
|
self.connectJira = connectJira.__get__(self, self.__class__)
|
||||||
|
self.exportTicketsAsJson = exportTicketsAsJson.__get__(self, self.__class__)
|
||||||
|
self.importTicketsFromJson = importTicketsFromJson.__get__(self, self.__class__)
|
||||||
|
self.mergeTicketData = mergeTicketData.__get__(self, self.__class__)
|
||||||
|
self.parseCsvContent = parseCsvContent.__get__(self, self.__class__)
|
||||||
|
self.parseExcelContent = parseExcelContent.__get__(self, self.__class__)
|
||||||
|
self.createCsvContent = createCsvContent.__get__(self, self.__class__)
|
||||||
|
self.createExcelContent = createExcelContent.__get__(self, self.__class__)
|
||||||
|
|
||||||
7
modules/workflows/methods/methodOutlook/__init__.py
Normal file
7
modules/workflows/methods/methodOutlook/__init__.py
Normal file
|
|
@ -0,0 +1,7 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
from .methodOutlook import MethodOutlook
|
||||||
|
|
||||||
|
__all__ = ['MethodOutlook']
|
||||||
|
|
||||||
18
modules/workflows/methods/methodOutlook/actions/__init__.py
Normal file
18
modules/workflows/methods/methodOutlook/actions/__init__.py
Normal file
|
|
@ -0,0 +1,18 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
"""Action modules for Outlook operations."""
|
||||||
|
|
||||||
|
# Export all actions
|
||||||
|
from .readEmails import readEmails
|
||||||
|
from .searchEmails import searchEmails
|
||||||
|
from .composeAndDraftEmailWithContext import composeAndDraftEmailWithContext
|
||||||
|
from .sendDraftEmail import sendDraftEmail
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
'readEmails',
|
||||||
|
'searchEmails',
|
||||||
|
'composeAndDraftEmailWithContext',
|
||||||
|
'sendDraftEmail',
|
||||||
|
]
|
||||||
|
|
||||||
|
|
@ -0,0 +1,362 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
"""
|
||||||
|
Compose And Draft Email With Context action for Outlook operations.
|
||||||
|
Composes email content using AI from context and optional documents, then creates a draft.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import json
|
||||||
|
import base64
|
||||||
|
import requests
|
||||||
|
from typing import Dict, Any
|
||||||
|
from modules.workflows.methods.methodBase import action
|
||||||
|
from modules.datamodels.datamodelChat import ActionResult, ActionDocument
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
@action
|
||||||
|
async def composeAndDraftEmailWithContext(self, parameters: Dict[str, Any]) -> ActionResult:
|
||||||
|
"""
|
||||||
|
GENERAL:
|
||||||
|
- Purpose: Compose email content using AI from context and optional documents, then create a draft.
|
||||||
|
- Input requirements: connectionReference (required); to (required); context (required); optional documentList, cc, bcc, emailStyle, maxLength.
|
||||||
|
- Output format: JSON confirmation with AI-generated draft metadata.
|
||||||
|
|
||||||
|
Parameters:
|
||||||
|
- connectionReference (str, required): Microsoft connection label.
|
||||||
|
- to (list, required): Recipient email addresses.
|
||||||
|
- context (str, required): Detailled context for composing the email.
|
||||||
|
- documentList (list, optional): Document references for context/attachments.
|
||||||
|
- cc (list, optional): CC recipients.
|
||||||
|
- bcc (list, optional): BCC recipients.
|
||||||
|
- emailStyle (str, optional): formal | casual | business. Default: business.
|
||||||
|
- maxLength (int, optional): Maximum length for generated content. Default: 1000.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
connectionReference = parameters.get("connectionReference")
|
||||||
|
to = parameters.get("to")
|
||||||
|
context = parameters.get("context")
|
||||||
|
documentList = parameters.get("documentList", [])
|
||||||
|
cc = parameters.get("cc", [])
|
||||||
|
bcc = parameters.get("bcc", [])
|
||||||
|
emailStyle = parameters.get("emailStyle", "business")
|
||||||
|
maxLength = parameters.get("maxLength", 1000)
|
||||||
|
|
||||||
|
if not connectionReference or not to or not context:
|
||||||
|
return ActionResult.isFailure(error="connectionReference, to, and context are required")
|
||||||
|
|
||||||
|
# Convert single values to lists for all recipient parameters
|
||||||
|
if isinstance(to, str):
|
||||||
|
to = [to]
|
||||||
|
if isinstance(cc, str):
|
||||||
|
cc = [cc]
|
||||||
|
if isinstance(bcc, str):
|
||||||
|
bcc = [bcc]
|
||||||
|
if isinstance(documentList, str):
|
||||||
|
documentList = [documentList]
|
||||||
|
|
||||||
|
# Get Microsoft connection
|
||||||
|
connection = self.connection.getMicrosoftConnection(connectionReference)
|
||||||
|
if not connection:
|
||||||
|
return ActionResult.isFailure(error="No valid Microsoft connection found")
|
||||||
|
|
||||||
|
# Check permissions
|
||||||
|
permissions_ok = await self.connection.checkPermissions(connection)
|
||||||
|
if not permissions_ok:
|
||||||
|
return ActionResult.isFailure(error="Connection lacks necessary permissions for Outlook operations")
|
||||||
|
|
||||||
|
# Prepare documents for AI processing
|
||||||
|
from modules.datamodels.datamodelDocref import DocumentReferenceList
|
||||||
|
chatDocuments = []
|
||||||
|
if documentList:
|
||||||
|
# Convert to DocumentReferenceList if needed
|
||||||
|
if isinstance(documentList, DocumentReferenceList):
|
||||||
|
docRefList = documentList
|
||||||
|
elif isinstance(documentList, list):
|
||||||
|
docRefList = DocumentReferenceList.from_string_list(documentList)
|
||||||
|
elif isinstance(documentList, str):
|
||||||
|
docRefList = DocumentReferenceList.from_string_list([documentList])
|
||||||
|
else:
|
||||||
|
docRefList = DocumentReferenceList(references=[])
|
||||||
|
chatDocuments = self.services.chat.getChatDocumentsFromDocumentList(docRefList)
|
||||||
|
|
||||||
|
# Create AI prompt for email composition
|
||||||
|
# Build document reference list for AI with expanded list contents when possible
|
||||||
|
doc_references = documentList
|
||||||
|
doc_list_text = ""
|
||||||
|
if doc_references:
|
||||||
|
lines = ["Available_Document_References:"]
|
||||||
|
for ref in doc_references:
|
||||||
|
# Each item is a label: resolve to its document list and render contained items
|
||||||
|
from modules.datamodels.datamodelDocref import DocumentReferenceList
|
||||||
|
list_docs = self.services.chat.getChatDocumentsFromDocumentList(DocumentReferenceList.from_string_list([ref])) or []
|
||||||
|
if list_docs:
|
||||||
|
for d in list_docs:
|
||||||
|
doc_ref_label = self.services.chat.getDocumentReferenceFromChatDocument(d)
|
||||||
|
lines.append(f"- {doc_ref_label}")
|
||||||
|
else:
|
||||||
|
lines.append(" - (no documents)")
|
||||||
|
doc_list_text = "\n" + "\n".join(lines)
|
||||||
|
else:
|
||||||
|
doc_list_text = "Available_Document_References: (No documents available for attachment)"
|
||||||
|
|
||||||
|
# Escape only the user-controlled context to prevent prompt injection
|
||||||
|
escaped_context = context.replace('"', '\\"').replace('\n', '\\n').replace('\r', '\\r')
|
||||||
|
|
||||||
|
ai_prompt = f"""Compose an email based on this context:
|
||||||
|
-------
|
||||||
|
{escaped_context}
|
||||||
|
-------
|
||||||
|
|
||||||
|
Recipients: {to}
|
||||||
|
Style: {emailStyle}
|
||||||
|
Max length: {maxLength} characters
|
||||||
|
{doc_list_text}
|
||||||
|
|
||||||
|
Based on the context, decide which documents to attach.
|
||||||
|
|
||||||
|
CRITICAL: Use EXACT document references from Available_Document_References above. For individual documents: ALWAYS use docItem:<documentId>:<filename> format (include filename)
|
||||||
|
|
||||||
|
Return JSON:
|
||||||
|
{{
|
||||||
|
"subject": "subject line",
|
||||||
|
"body": "email body (HTML allowed)",
|
||||||
|
"attachments": ["docItem:<documentId>:<filename>"]
|
||||||
|
}}
|
||||||
|
"""
|
||||||
|
|
||||||
|
# Call AI service to generate email content
|
||||||
|
try:
|
||||||
|
ai_response = await self.services.ai.callAiPlanning(
|
||||||
|
prompt=ai_prompt,
|
||||||
|
placeholders=None,
|
||||||
|
debugType="email_composition"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Parse AI response
|
||||||
|
try:
|
||||||
|
ai_content = ai_response
|
||||||
|
# Extract JSON from AI response
|
||||||
|
if "```json" in ai_content:
|
||||||
|
json_start = ai_content.find("```json") + 7
|
||||||
|
json_end = ai_content.find("```", json_start)
|
||||||
|
json_content = ai_content[json_start:json_end].strip()
|
||||||
|
elif "{" in ai_content and "}" in ai_content:
|
||||||
|
json_start = ai_content.find("{")
|
||||||
|
json_end = ai_content.rfind("}") + 1
|
||||||
|
json_content = ai_content[json_start:json_end]
|
||||||
|
else:
|
||||||
|
json_content = ai_content
|
||||||
|
|
||||||
|
email_data = json.loads(json_content)
|
||||||
|
subject = email_data.get("subject", "")
|
||||||
|
body = email_data.get("body", "")
|
||||||
|
ai_attachments = email_data.get("attachments", [])
|
||||||
|
|
||||||
|
if not subject or not body:
|
||||||
|
return ActionResult.isFailure(error="AI did not generate valid subject and body")
|
||||||
|
|
||||||
|
# Use AI-selected attachments if provided, otherwise use all documents
|
||||||
|
normalized_ai_attachments = []
|
||||||
|
if documentList:
|
||||||
|
try:
|
||||||
|
available_refs = [documentList] if isinstance(documentList, str) else documentList
|
||||||
|
from modules.datamodels.datamodelDocref import DocumentReferenceList
|
||||||
|
available_docs = self.services.chat.getChatDocumentsFromDocumentList(DocumentReferenceList.from_string_list(available_refs)) or []
|
||||||
|
except Exception:
|
||||||
|
available_docs = []
|
||||||
|
|
||||||
|
# Normalize AI attachments to a list of strings
|
||||||
|
if isinstance(ai_attachments, str):
|
||||||
|
ai_attachments = [ai_attachments]
|
||||||
|
elif isinstance(ai_attachments, list):
|
||||||
|
ai_attachments = [a for a in ai_attachments if isinstance(a, str)]
|
||||||
|
|
||||||
|
if ai_attachments:
|
||||||
|
try:
|
||||||
|
ai_refs = [ai_attachments] if isinstance(ai_attachments, str) else ai_attachments
|
||||||
|
from modules.datamodels.datamodelDocref import DocumentReferenceList
|
||||||
|
ai_docs = self.services.chat.getChatDocumentsFromDocumentList(DocumentReferenceList.from_string_list(ai_refs)) or []
|
||||||
|
except Exception:
|
||||||
|
ai_docs = []
|
||||||
|
|
||||||
|
# Intersect by document id
|
||||||
|
available_ids = {getattr(d, 'id', None) for d in available_docs}
|
||||||
|
selected_docs = [d for d in ai_docs if getattr(d, 'id', None) in available_ids]
|
||||||
|
|
||||||
|
if selected_docs:
|
||||||
|
# Map selected ChatDocuments back to docItem references (with full filename)
|
||||||
|
documentList = [self.services.chat.getDocumentReferenceFromChatDocument(d) for d in selected_docs]
|
||||||
|
# Normalize ai_attachments to full format for storage
|
||||||
|
normalized_ai_attachments = documentList.copy()
|
||||||
|
logger.info(f"AI selected {len(documentList)} documents for attachment (resolved via ChatDocuments)")
|
||||||
|
else:
|
||||||
|
# No intersection; use all available documents
|
||||||
|
documentList = [self.services.chat.getDocumentReferenceFromChatDocument(d) for d in available_docs]
|
||||||
|
normalized_ai_attachments = documentList.copy()
|
||||||
|
logger.warning("AI selected attachments not found in available documents, using all documents")
|
||||||
|
else:
|
||||||
|
# No AI selection; use all available documents
|
||||||
|
documentList = [self.services.chat.getDocumentReferenceFromChatDocument(d) for d in available_docs]
|
||||||
|
normalized_ai_attachments = documentList.copy()
|
||||||
|
logger.warning("AI did not specify attachments, using all available documents")
|
||||||
|
else:
|
||||||
|
logger.info("No documents provided in documentList; skipping attachment processing")
|
||||||
|
|
||||||
|
except json.JSONDecodeError as e:
|
||||||
|
logger.error(f"Failed to parse AI response as JSON: {str(e)}")
|
||||||
|
logger.error(f"AI response content: {ai_response}")
|
||||||
|
return ActionResult.isFailure(error="AI response was not valid JSON format")
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error calling AI service: {str(e)}")
|
||||||
|
return ActionResult.isFailure(error=f"Failed to generate email content: {str(e)}")
|
||||||
|
|
||||||
|
# Now create the email with AI-generated content
|
||||||
|
try:
|
||||||
|
graph_url = "https://graph.microsoft.com/v1.0"
|
||||||
|
headers = {
|
||||||
|
"Authorization": f"Bearer {connection['accessToken']}",
|
||||||
|
"Content-Type": "application/json"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Clean and format body content
|
||||||
|
cleaned_body = body.strip()
|
||||||
|
|
||||||
|
# Check if body is already HTML
|
||||||
|
if cleaned_body.startswith('<html>') or cleaned_body.startswith('<body>') or '<br>' in cleaned_body:
|
||||||
|
html_body = cleaned_body
|
||||||
|
else:
|
||||||
|
# Convert plain text to proper HTML formatting
|
||||||
|
html_body = cleaned_body.replace('\n', '<br>')
|
||||||
|
html_body = f"<html><body>{html_body}</body></html>"
|
||||||
|
|
||||||
|
# Build the email message
|
||||||
|
message = {
|
||||||
|
"subject": subject,
|
||||||
|
"body": {
|
||||||
|
"contentType": "HTML",
|
||||||
|
"content": html_body
|
||||||
|
},
|
||||||
|
"toRecipients": [{"emailAddress": {"address": email}} for email in to],
|
||||||
|
"ccRecipients": [{"emailAddress": {"address": email}} for email in cc] if cc else [],
|
||||||
|
"bccRecipients": [{"emailAddress": {"address": email}} for email in bcc] if bcc else []
|
||||||
|
}
|
||||||
|
|
||||||
|
# Add documents as attachments if provided
|
||||||
|
if documentList:
|
||||||
|
message["attachments"] = []
|
||||||
|
for attachment_ref in documentList:
|
||||||
|
# Get attachment document from service center
|
||||||
|
from modules.datamodels.datamodelDocref import DocumentReferenceList
|
||||||
|
attachment_docs = self.services.chat.getChatDocumentsFromDocumentList(DocumentReferenceList.from_string_list([attachment_ref]))
|
||||||
|
if attachment_docs:
|
||||||
|
for doc in attachment_docs:
|
||||||
|
file_id = getattr(doc, 'fileId', None)
|
||||||
|
if file_id:
|
||||||
|
try:
|
||||||
|
file_content = self.services.chat.getFileData(file_id)
|
||||||
|
if file_content:
|
||||||
|
if isinstance(file_content, bytes):
|
||||||
|
content_bytes = file_content
|
||||||
|
else:
|
||||||
|
content_bytes = str(file_content).encode('utf-8')
|
||||||
|
|
||||||
|
base64_content = base64.b64encode(content_bytes).decode('utf-8')
|
||||||
|
|
||||||
|
attachment = {
|
||||||
|
"@odata.type": "#microsoft.graph.fileAttachment",
|
||||||
|
"name": doc.fileName,
|
||||||
|
"contentType": doc.mimeType or "application/octet-stream",
|
||||||
|
"contentBytes": base64_content
|
||||||
|
}
|
||||||
|
message["attachments"].append(attachment)
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error reading attachment file {doc.fileName}: {str(e)}")
|
||||||
|
|
||||||
|
# Create the draft message
|
||||||
|
drafts_folder_id = self.folderManagement.getFolderId("Drafts", connection)
|
||||||
|
|
||||||
|
if drafts_folder_id:
|
||||||
|
api_url = f"{graph_url}/me/mailFolders/{drafts_folder_id}/messages"
|
||||||
|
else:
|
||||||
|
api_url = f"{graph_url}/me/messages"
|
||||||
|
logger.warning("Could not find Drafts folder, creating draft in default location")
|
||||||
|
|
||||||
|
response = requests.post(api_url, headers=headers, json=message)
|
||||||
|
|
||||||
|
if response.status_code in [200, 201]:
|
||||||
|
draft_data = response.json()
|
||||||
|
draft_id = draft_data.get("id", "Unknown")
|
||||||
|
|
||||||
|
# Create draft result data with full draft information
|
||||||
|
draftResultData = {
|
||||||
|
"status": "draft",
|
||||||
|
"message": "Email draft created successfully with AI-generated content",
|
||||||
|
"draftId": draft_id,
|
||||||
|
"folder": "Drafts (Entwürfe)",
|
||||||
|
"mailbox": connection.get('userEmail', 'Unknown'),
|
||||||
|
"subject": subject,
|
||||||
|
"body": body,
|
||||||
|
"recipients": to,
|
||||||
|
"cc": cc,
|
||||||
|
"bcc": bcc,
|
||||||
|
"attachments": len(documentList) if documentList else 0,
|
||||||
|
"aiSelectedAttachments": normalized_ai_attachments if normalized_ai_attachments else "all documents",
|
||||||
|
"aiGenerated": True,
|
||||||
|
"context": context,
|
||||||
|
"emailStyle": emailStyle,
|
||||||
|
"timestamp": self.services.utils.timestampGetUtc(),
|
||||||
|
"draftData": draft_data
|
||||||
|
}
|
||||||
|
|
||||||
|
# Extract attachment filenames for validation metadata
|
||||||
|
attachmentFilenames = []
|
||||||
|
attachmentReferences = []
|
||||||
|
if documentList:
|
||||||
|
try:
|
||||||
|
from modules.datamodels.datamodelDocref import DocumentReferenceList
|
||||||
|
attached_docs = self.services.chat.getChatDocumentsFromDocumentList(DocumentReferenceList.from_string_list(documentList)) or []
|
||||||
|
attachmentFilenames = [getattr(doc, 'fileName', '') for doc in attached_docs if getattr(doc, 'fileName', None)]
|
||||||
|
# Store normalized document references (with filenames) - use normalized_ai_attachments if available
|
||||||
|
attachmentReferences = normalized_ai_attachments if normalized_ai_attachments else [self.services.chat.getDocumentReferenceFromChatDocument(d) for d in attached_docs]
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
# Create validation metadata for content validator
|
||||||
|
validationMetadata = {
|
||||||
|
"actionType": "outlook.composeAndDraftEmailWithContext",
|
||||||
|
"emailRecipients": to,
|
||||||
|
"emailCc": cc,
|
||||||
|
"emailBcc": bcc,
|
||||||
|
"emailSubject": subject,
|
||||||
|
"emailAttachments": attachmentFilenames,
|
||||||
|
"emailAttachmentReferences": attachmentReferences,
|
||||||
|
"emailAttachmentCount": len(attachmentFilenames),
|
||||||
|
"emailStyle": emailStyle,
|
||||||
|
"hasAttachments": len(attachmentFilenames) > 0
|
||||||
|
}
|
||||||
|
|
||||||
|
return ActionResult(
|
||||||
|
success=True,
|
||||||
|
documents=[ActionDocument(
|
||||||
|
documentName=f"ai_generated_email_draft_{self._format_timestamp_for_filename()}.json",
|
||||||
|
documentData=json.dumps(draftResultData, indent=2),
|
||||||
|
mimeType="application/json",
|
||||||
|
validationMetadata=validationMetadata
|
||||||
|
)]
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
logger.error(f"Failed to create draft. Status: {response.status_code}, Response: {response.text}")
|
||||||
|
return ActionResult.isFailure(error=f"Failed to create email draft: {response.status_code} - {response.text}")
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error creating email via Microsoft Graph API: {str(e)}")
|
||||||
|
return ActionResult.isFailure(error=f"Failed to create email: {str(e)}")
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error in composeAndDraftEmailWithContext: {str(e)}")
|
||||||
|
return ActionResult.isFailure(error=str(e))
|
||||||
|
|
||||||
245
modules/workflows/methods/methodOutlook/actions/readEmails.py
Normal file
245
modules/workflows/methods/methodOutlook/actions/readEmails.py
Normal file
|
|
@ -0,0 +1,245 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
"""
|
||||||
|
Read Emails action for Outlook operations.
|
||||||
|
Reads emails and metadata from a mailbox folder.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import time
|
||||||
|
import json
|
||||||
|
import requests
|
||||||
|
from typing import Dict, Any
|
||||||
|
from modules.workflows.methods.methodBase import action
|
||||||
|
from modules.datamodels.datamodelChat import ActionResult, ActionDocument
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
@action
|
||||||
|
async def readEmails(self, parameters: Dict[str, Any]) -> ActionResult:
|
||||||
|
"""
|
||||||
|
GENERAL:
|
||||||
|
- Purpose: Read emails and metadata from a mailbox folder.
|
||||||
|
- Input requirements: connectionReference (required); optional folder, limit, filter, outputMimeType.
|
||||||
|
- Output format: JSON with emails and metadata.
|
||||||
|
|
||||||
|
Parameters:
|
||||||
|
- connectionReference (str, required): Microsoft connection label.
|
||||||
|
- folder (str, optional): Folder to read from. Default: Inbox.
|
||||||
|
- limit (int, optional): Maximum items to return. Must be > 0. Default: 1000.
|
||||||
|
- filter (str, optional): Sender, query operators, or subject text.
|
||||||
|
- outputMimeType (str, optional): MIME type for output file. Options: "application/json" (default), "text/plain", "text/csv". Default: "application/json".
|
||||||
|
"""
|
||||||
|
operationId = None
|
||||||
|
try:
|
||||||
|
# Init progress logger
|
||||||
|
workflowId = self.services.workflow.id if self.services.workflow else f"no-workflow-{int(time.time())}"
|
||||||
|
operationId = f"outlook_read_{workflowId}_{int(time.time())}"
|
||||||
|
|
||||||
|
# Start progress tracking
|
||||||
|
parentOperationId = parameters.get('parentOperationId')
|
||||||
|
self.services.chat.progressLogStart(
|
||||||
|
operationId,
|
||||||
|
"Read Emails",
|
||||||
|
"Outlook Email Reading",
|
||||||
|
f"Folder: {parameters.get('folder', 'Inbox')}",
|
||||||
|
parentOperationId=parentOperationId
|
||||||
|
)
|
||||||
|
|
||||||
|
connectionReference = parameters.get("connectionReference")
|
||||||
|
folder = parameters.get("folder", "Inbox")
|
||||||
|
limit = parameters.get("limit", 10)
|
||||||
|
filter = parameters.get("filter")
|
||||||
|
outputMimeType = parameters.get("outputMimeType", "application/json")
|
||||||
|
|
||||||
|
if not connectionReference:
|
||||||
|
if operationId:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
return ActionResult.isFailure(error="Connection reference is required")
|
||||||
|
|
||||||
|
self.services.chat.progressLogUpdate(operationId, 0.2, "Validating parameters")
|
||||||
|
|
||||||
|
# Validate limit parameter
|
||||||
|
if limit <= 0:
|
||||||
|
limit = 1000
|
||||||
|
logger.warning(f"Invalid limit value ({limit}), using default value 1000")
|
||||||
|
|
||||||
|
# Validate filter parameter if provided
|
||||||
|
if filter:
|
||||||
|
# Remove any potentially dangerous characters that could break the filter
|
||||||
|
filter = filter.strip()
|
||||||
|
if len(filter) > 100:
|
||||||
|
logger.warning(f"Filter too long ({len(filter)} chars), truncating to 100 characters")
|
||||||
|
filter = filter[:100]
|
||||||
|
|
||||||
|
|
||||||
|
# Get Microsoft connection
|
||||||
|
self.services.chat.progressLogUpdate(operationId, 0.3, "Getting Microsoft connection")
|
||||||
|
connection = self.connection.getMicrosoftConnection(connectionReference)
|
||||||
|
if not connection:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
return ActionResult.isFailure(error="No valid Microsoft connection found for the provided connection reference")
|
||||||
|
|
||||||
|
# Read emails using Microsoft Graph API
|
||||||
|
self.services.chat.progressLogUpdate(operationId, 0.4, "Reading emails from Microsoft Graph API")
|
||||||
|
try:
|
||||||
|
# Microsoft Graph API endpoint for messages
|
||||||
|
graph_url = "https://graph.microsoft.com/v1.0"
|
||||||
|
headers = {
|
||||||
|
"Authorization": f"Bearer {connection['accessToken']}",
|
||||||
|
"Content-Type": "application/json"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Get the folder ID for the specified folder
|
||||||
|
folder_id = self.folderManagement.getFolderId(folder, connection)
|
||||||
|
|
||||||
|
if folder_id:
|
||||||
|
# Build the API request with folder ID
|
||||||
|
api_url = f"{graph_url}/me/mailFolders/{folder_id}/messages"
|
||||||
|
else:
|
||||||
|
# Fallback: use folder name directly (for well-known folders like "Inbox")
|
||||||
|
api_url = f"{graph_url}/me/mailFolders/{folder}/messages"
|
||||||
|
logger.warning(f"Could not find folder ID for '{folder}', using folder name directly")
|
||||||
|
params = {
|
||||||
|
"$top": limit,
|
||||||
|
"$orderby": "receivedDateTime desc"
|
||||||
|
}
|
||||||
|
|
||||||
|
if filter:
|
||||||
|
# Build proper Graph API filter parameters
|
||||||
|
filter_params = self.emailProcessing.buildGraphFilter(filter)
|
||||||
|
params.update(filter_params)
|
||||||
|
|
||||||
|
# If using $search, remove $orderby as they can't be combined
|
||||||
|
if "$search" in params:
|
||||||
|
params.pop("$orderby", None)
|
||||||
|
|
||||||
|
# If using $filter with contains(), remove $orderby as they can't be combined
|
||||||
|
# Microsoft Graph API doesn't support contains() with orderby
|
||||||
|
if "$filter" in params and "contains(" in params["$filter"].lower():
|
||||||
|
params.pop("$orderby", None)
|
||||||
|
|
||||||
|
# Filter applied
|
||||||
|
|
||||||
|
# Make the API call
|
||||||
|
|
||||||
|
|
||||||
|
response = requests.get(api_url, headers=headers, params=params)
|
||||||
|
|
||||||
|
if response.status_code != 200:
|
||||||
|
logger.error(f"Graph API error: {response.status_code} - {response.text}")
|
||||||
|
logger.error(f"Request URL: {response.url}")
|
||||||
|
logger.error(f"Request headers: {headers}")
|
||||||
|
logger.error(f"Request params: {params}")
|
||||||
|
|
||||||
|
response.raise_for_status()
|
||||||
|
|
||||||
|
self.services.chat.progressLogUpdate(operationId, 0.7, "Processing email data")
|
||||||
|
emails_data = response.json()
|
||||||
|
email_data = {
|
||||||
|
"emails": emails_data.get("value", []),
|
||||||
|
"count": len(emails_data.get("value", [])),
|
||||||
|
"folder": folder,
|
||||||
|
"filter": filter,
|
||||||
|
"apiMetadata": {
|
||||||
|
"@odata.context": emails_data.get("@odata.context"),
|
||||||
|
"@odata.count": emails_data.get("@odata.count"),
|
||||||
|
"@odata.nextLink": emails_data.get("@odata.nextLink")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
except ImportError:
|
||||||
|
logger.error("requests module not available")
|
||||||
|
if operationId:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
return ActionResult.isFailure(error="requests module not available")
|
||||||
|
except requests.exceptions.HTTPError as e:
|
||||||
|
if e.response.status_code == 400:
|
||||||
|
logger.error(f"Bad Request (400) - Invalid filter or parameter: {e.response.text}")
|
||||||
|
if operationId:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
return ActionResult.isFailure(error=f"Invalid filter syntax. Please check your filter parameter. Error: {e.response.text}")
|
||||||
|
elif e.response.status_code == 401:
|
||||||
|
logger.error("Unauthorized (401) - Access token may be expired or invalid")
|
||||||
|
if operationId:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
return ActionResult.isFailure(error="Authentication failed. Please check your connection and try again.")
|
||||||
|
elif e.response.status_code == 403:
|
||||||
|
logger.error("Forbidden (403) - Insufficient permissions to access emails")
|
||||||
|
if operationId:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
return ActionResult.isFailure(error="Insufficient permissions to read emails from this folder.")
|
||||||
|
else:
|
||||||
|
logger.error(f"HTTP Error {e.response.status_code}: {e.response.text}")
|
||||||
|
if operationId:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
return ActionResult.isFailure(error=f"HTTP Error {e.response.status_code}: {e.response.text}")
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error reading emails from Microsoft Graph API: {str(e)}")
|
||||||
|
if operationId:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
return ActionResult.isFailure(error=f"Failed to read emails: {str(e)}")
|
||||||
|
|
||||||
|
# Determine output format based on MIME type
|
||||||
|
mime_type_mapping = {
|
||||||
|
"application/json": ".json",
|
||||||
|
"text/plain": ".txt",
|
||||||
|
"text/csv": ".csv"
|
||||||
|
}
|
||||||
|
output_extension = mime_type_mapping.get(outputMimeType, ".json")
|
||||||
|
output_mime_type = outputMimeType
|
||||||
|
logger.info(f"Using output format: {output_extension} ({output_mime_type})")
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
# Create result data as JSON string
|
||||||
|
result_data = {
|
||||||
|
"connectionReference": connectionReference,
|
||||||
|
"folder": folder,
|
||||||
|
"limit": limit,
|
||||||
|
"filter": filter,
|
||||||
|
"emails": email_data,
|
||||||
|
"connection": {
|
||||||
|
"id": connection["id"],
|
||||||
|
"authority": "microsoft",
|
||||||
|
"reference": connectionReference
|
||||||
|
},
|
||||||
|
"timestamp": self.services.utils.timestampGetUtc()
|
||||||
|
}
|
||||||
|
|
||||||
|
validationMetadata = {
|
||||||
|
"actionType": "outlook.readEmails",
|
||||||
|
"connectionReference": connectionReference,
|
||||||
|
"folder": folder,
|
||||||
|
"limit": limit,
|
||||||
|
"filter": filter,
|
||||||
|
"emailCount": email_data.get("count", 0),
|
||||||
|
"outputMimeType": outputMimeType
|
||||||
|
}
|
||||||
|
|
||||||
|
self.services.chat.progressLogUpdate(operationId, 0.9, f"Found {email_data.get('count', 0)} emails")
|
||||||
|
self.services.chat.progressLogFinish(operationId, True)
|
||||||
|
|
||||||
|
return ActionResult.isSuccess(
|
||||||
|
documents=[ActionDocument(
|
||||||
|
documentName=f"outlook_emails_{self._format_timestamp_for_filename()}.json",
|
||||||
|
documentData=json.dumps(result_data, indent=2),
|
||||||
|
mimeType="application/json",
|
||||||
|
validationMetadata=validationMetadata
|
||||||
|
)]
|
||||||
|
)
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error reading emails: {str(e)}")
|
||||||
|
if operationId:
|
||||||
|
try:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
except:
|
||||||
|
pass # Don't fail on progress logging errors
|
||||||
|
return ActionResult.isFailure(
|
||||||
|
error=str(e)
|
||||||
|
)
|
||||||
|
|
||||||
257
modules/workflows/methods/methodOutlook/actions/searchEmails.py
Normal file
257
modules/workflows/methods/methodOutlook/actions/searchEmails.py
Normal file
|
|
@ -0,0 +1,257 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
"""
|
||||||
|
Search Emails action for Outlook operations.
|
||||||
|
Searches emails by query and returns matching items with metadata.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import json
|
||||||
|
import requests
|
||||||
|
from typing import Dict, Any
|
||||||
|
from modules.workflows.methods.methodBase import action
|
||||||
|
from modules.datamodels.datamodelChat import ActionResult, ActionDocument
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
@action
|
||||||
|
async def searchEmails(self, parameters: Dict[str, Any]) -> ActionResult:
|
||||||
|
"""
|
||||||
|
GENERAL:
|
||||||
|
- Purpose: Search emails by query and return matching items with metadata.
|
||||||
|
- Input requirements: connectionReference (required); query (required); optional folder, limit, outputMimeType.
|
||||||
|
- Output format: JSON with search results and metadata.
|
||||||
|
|
||||||
|
Parameters:
|
||||||
|
- connectionReference (str, required): Microsoft connection label.
|
||||||
|
- query (str, required): Search expression.
|
||||||
|
- folder (str, optional): Folder scope or All. Default: All.
|
||||||
|
- limit (int, optional): Maximum items to return. Must be > 0. Default: 1000.
|
||||||
|
- outputMimeType (str, optional): MIME type for output file. Options: "application/json" (default), "text/plain", "text/csv". Default: "application/json".
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
connectionReference = parameters.get("connectionReference")
|
||||||
|
query = parameters.get("query")
|
||||||
|
folder = parameters.get("folder", "All")
|
||||||
|
limit = parameters.get("limit", 1000)
|
||||||
|
outputMimeType = parameters.get("outputMimeType", "application/json")
|
||||||
|
|
||||||
|
# Validate parameters
|
||||||
|
if not connectionReference:
|
||||||
|
return ActionResult.isFailure(error="Connection reference is required")
|
||||||
|
|
||||||
|
# Validate limit parameter
|
||||||
|
if limit <= 0:
|
||||||
|
limit = 1000
|
||||||
|
logger.warning(f"Invalid limit value ({limit}), using default value 1000")
|
||||||
|
|
||||||
|
if not query or not query.strip():
|
||||||
|
return ActionResult.isFailure(error="Search query is required and cannot be empty")
|
||||||
|
|
||||||
|
# Check if this is a folder specification query
|
||||||
|
if query.strip().lower().startswith('folder:'):
|
||||||
|
folder_name = query.strip()[7:].strip() # Remove "folder:" prefix
|
||||||
|
if not folder_name:
|
||||||
|
return ActionResult.isFailure(error="Invalid folder specification. Use format 'folder:FolderName'")
|
||||||
|
logger.info(f"Search query is a folder specification: {folder_name}")
|
||||||
|
|
||||||
|
# Validate limit
|
||||||
|
try:
|
||||||
|
limit = int(limit)
|
||||||
|
if limit <= 0:
|
||||||
|
limit = 1000
|
||||||
|
logger.warning(f"Invalid limit value (<=0), using default value 1000")
|
||||||
|
elif limit > 1000: # Microsoft Graph API has limits
|
||||||
|
limit = 1000
|
||||||
|
logger.warning(f"Limit {limit} exceeds maximum (1000), using 1000")
|
||||||
|
except (ValueError, TypeError):
|
||||||
|
limit = 1000
|
||||||
|
logger.warning(f"Invalid limit value, using default value 1000")
|
||||||
|
|
||||||
|
# Get Microsoft connection
|
||||||
|
connection = self.connection.getMicrosoftConnection(connectionReference)
|
||||||
|
if not connection:
|
||||||
|
return ActionResult.isFailure(error="No valid Microsoft connection found for the provided connection reference")
|
||||||
|
|
||||||
|
# Search emails using Microsoft Graph API
|
||||||
|
try:
|
||||||
|
# Microsoft Graph API endpoint for searching messages
|
||||||
|
graph_url = "https://graph.microsoft.com/v1.0"
|
||||||
|
headers = {
|
||||||
|
"Authorization": f"Bearer {connection['accessToken']}",
|
||||||
|
"Content-Type": "application/json"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Get the folder ID for the specified folder if needed
|
||||||
|
folder_id = None
|
||||||
|
if folder and folder.lower() != "all":
|
||||||
|
folder_id = self.folderManagement.getFolderId(folder, connection)
|
||||||
|
if folder_id:
|
||||||
|
logger.debug(f"Found folder ID for '{folder}': {folder_id}")
|
||||||
|
else:
|
||||||
|
logger.warning(f"Could not find folder ID for '{folder}', using folder name directly")
|
||||||
|
|
||||||
|
# Build the search API request
|
||||||
|
api_url = f"{graph_url}/me/messages"
|
||||||
|
params = self.emailProcessing.buildSearchParameters(query, folder_id or folder, limit)
|
||||||
|
|
||||||
|
# Log search parameters for debugging
|
||||||
|
logger.debug(f"Search query: '{query}'")
|
||||||
|
logger.debug(f"Search folder: '{folder}'")
|
||||||
|
logger.debug(f"Search parameters: {params}")
|
||||||
|
logger.debug(f"API URL: {api_url}")
|
||||||
|
|
||||||
|
# Make the API call
|
||||||
|
response = requests.get(api_url, headers=headers, params=params)
|
||||||
|
|
||||||
|
# Log response details for debugging
|
||||||
|
|
||||||
|
|
||||||
|
if response.status_code != 200:
|
||||||
|
# Log detailed error information
|
||||||
|
try:
|
||||||
|
error_data = response.json()
|
||||||
|
logger.error(f"Microsoft Graph API error: {response.status_code} - {error_data}")
|
||||||
|
except:
|
||||||
|
logger.error(f"Microsoft Graph API error: {response.status_code} - {response.text}")
|
||||||
|
|
||||||
|
# Check for specific error types and provide helpful messages
|
||||||
|
if response.status_code == 400:
|
||||||
|
logger.error("Bad Request (400) - Check search query format and parameters")
|
||||||
|
logger.error(f"Search query: '{query}'")
|
||||||
|
logger.error(f"Search parameters: {params}")
|
||||||
|
logger.error(f"API URL: {api_url}")
|
||||||
|
elif response.status_code == 401:
|
||||||
|
logger.error("Unauthorized (401) - Check access token and permissions")
|
||||||
|
elif response.status_code == 403:
|
||||||
|
logger.error("Forbidden (403) - Check API permissions and scopes")
|
||||||
|
elif response.status_code == 429:
|
||||||
|
logger.error("Too Many Requests (429) - Rate limit exceeded")
|
||||||
|
|
||||||
|
raise Exception(f"Microsoft Graph API returned {response.status_code}: {response.text}")
|
||||||
|
|
||||||
|
response.raise_for_status()
|
||||||
|
|
||||||
|
search_data = response.json()
|
||||||
|
emails = search_data.get("value", [])
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
# Apply folder filtering if needed and we used $search
|
||||||
|
if folder and folder.lower() != "all" and "$search" in params:
|
||||||
|
# Get the actual folder ID for proper filtering
|
||||||
|
folder_id = self.folderManagement.getFolderId(folder, connection)
|
||||||
|
|
||||||
|
if folder_id:
|
||||||
|
# Filter results by folder ID
|
||||||
|
filtered_emails = []
|
||||||
|
for email in emails:
|
||||||
|
if email.get("parentFolderId") == folder_id:
|
||||||
|
filtered_emails.append(email)
|
||||||
|
emails = filtered_emails
|
||||||
|
logger.debug(f"Applied folder filtering: {len(filtered_emails)} emails found in folder {folder}")
|
||||||
|
else:
|
||||||
|
# Fallback: try to filter by folder name (less reliable)
|
||||||
|
filtered_emails = []
|
||||||
|
for email in emails:
|
||||||
|
# Check if email has folder information
|
||||||
|
if hasattr(email, 'parentFolderId') and email.get('parentFolderId'):
|
||||||
|
if email.get('parentFolderId') == folder:
|
||||||
|
filtered_emails.append(email)
|
||||||
|
else:
|
||||||
|
# If no folder info, include the email (less strict filtering)
|
||||||
|
filtered_emails.append(email)
|
||||||
|
|
||||||
|
emails = filtered_emails
|
||||||
|
logger.debug(f"Applied fallback folder filtering: {len(filtered_emails)} emails found in folder {folder}")
|
||||||
|
|
||||||
|
# Special handling for folder specification queries
|
||||||
|
if query.strip().lower().startswith('folder:'):
|
||||||
|
folder_name = query.strip()[7:].strip()
|
||||||
|
folder_id = self.folderManagement.getFolderId(folder_name, connection)
|
||||||
|
if folder_id:
|
||||||
|
# Filter results to only include emails from the specified folder
|
||||||
|
filtered_emails = []
|
||||||
|
for email in emails:
|
||||||
|
if email.get("parentFolderId") == folder_id:
|
||||||
|
filtered_emails.append(email)
|
||||||
|
emails = filtered_emails
|
||||||
|
logger.debug(f"Applied folder specification filtering: {len(filtered_emails)} emails found in folder {folder_name}")
|
||||||
|
else:
|
||||||
|
logger.warning(f"Could not find folder ID for folder specification: {folder_name}")
|
||||||
|
|
||||||
|
|
||||||
|
search_result = {
|
||||||
|
"query": query,
|
||||||
|
"results": emails,
|
||||||
|
"count": len(emails),
|
||||||
|
"folder": folder,
|
||||||
|
"limit": limit,
|
||||||
|
"apiMetadata": {
|
||||||
|
"@odata.context": search_data.get("@odata.context"),
|
||||||
|
"@odata.count": search_data.get("@odata.count"),
|
||||||
|
"@odata.nextLink": search_data.get("@odata.nextLink")
|
||||||
|
},
|
||||||
|
"searchParams": params
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
except ImportError:
|
||||||
|
logger.error("requests module not available")
|
||||||
|
return ActionResult.isFailure(error="requests module not available")
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error searching emails via Microsoft Graph API: {str(e)}")
|
||||||
|
return ActionResult.isFailure(error=f"Failed to search emails: {str(e)}")
|
||||||
|
|
||||||
|
# Determine output format based on MIME type
|
||||||
|
mime_type_mapping = {
|
||||||
|
"application/json": ".json",
|
||||||
|
"text/plain": ".txt",
|
||||||
|
"text/csv": ".csv"
|
||||||
|
}
|
||||||
|
output_extension = mime_type_mapping.get(outputMimeType, ".json")
|
||||||
|
output_mime_type = outputMimeType
|
||||||
|
logger.info(f"Using output format: {output_extension} ({output_mime_type})")
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
result_data = {
|
||||||
|
"connectionReference": connectionReference,
|
||||||
|
"query": query,
|
||||||
|
"folder": folder,
|
||||||
|
"limit": limit,
|
||||||
|
"searchResults": search_result,
|
||||||
|
"connection": {
|
||||||
|
"id": connection["id"],
|
||||||
|
"authority": "microsoft",
|
||||||
|
"reference": connectionReference
|
||||||
|
},
|
||||||
|
"timestamp": self.services.utils.timestampGetUtc()
|
||||||
|
}
|
||||||
|
|
||||||
|
validationMetadata = {
|
||||||
|
"actionType": "outlook.searchEmails",
|
||||||
|
"connectionReference": connectionReference,
|
||||||
|
"query": query,
|
||||||
|
"folder": folder,
|
||||||
|
"limit": limit,
|
||||||
|
"resultCount": search_result.get("count", 0),
|
||||||
|
"outputMimeType": outputMimeType
|
||||||
|
}
|
||||||
|
|
||||||
|
return ActionResult(
|
||||||
|
success=True,
|
||||||
|
documents=[ActionDocument(
|
||||||
|
documentName=f"outlook_email_search_{self._format_timestamp_for_filename()}.json",
|
||||||
|
documentData=json.dumps(result_data, indent=2),
|
||||||
|
mimeType="application/json",
|
||||||
|
validationMetadata=validationMetadata
|
||||||
|
)]
|
||||||
|
)
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error searching emails: {str(e)}")
|
||||||
|
return ActionResult.isFailure(error=str(e))
|
||||||
|
|
||||||
|
|
@ -0,0 +1,312 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
"""
|
||||||
|
Send Draft Email action for Outlook operations.
|
||||||
|
Sends draft email(s) using draft email JSON document(s) from action outlook.composeAndDraftEmailWithContext.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import time
|
||||||
|
import json
|
||||||
|
import requests
|
||||||
|
from typing import Dict, Any
|
||||||
|
from modules.workflows.methods.methodBase import action
|
||||||
|
from modules.datamodels.datamodelChat import ActionResult, ActionDocument
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
@action
|
||||||
|
async def sendDraftEmail(self, parameters: Dict[str, Any]) -> ActionResult:
|
||||||
|
"""
|
||||||
|
GENERAL:
|
||||||
|
- Purpose: Send draft email(s) using draft email JSON document(s) from action outlook.composeAndDraftEmailWithContext.
|
||||||
|
- Input requirements: connectionReference (required); documentList with draft email JSON documents (required).
|
||||||
|
- Output format: JSON confirmation with sent mail metadata for all emails.
|
||||||
|
|
||||||
|
Parameters:
|
||||||
|
- connectionReference (str, required): Microsoft connection label.
|
||||||
|
- documentList (list, required): Document reference(s) to draft emails in JSON format (outputs from outlook.composeAndDraftEmailWithContext function).
|
||||||
|
"""
|
||||||
|
operationId = None
|
||||||
|
try:
|
||||||
|
# Init progress logger
|
||||||
|
workflowId = self.services.workflow.id if self.services.workflow else f"no-workflow-{int(time.time())}"
|
||||||
|
operationId = f"outlook_send_{workflowId}_{int(time.time())}"
|
||||||
|
|
||||||
|
# Start progress tracking
|
||||||
|
parentOperationId = parameters.get('parentOperationId')
|
||||||
|
self.services.chat.progressLogStart(
|
||||||
|
operationId,
|
||||||
|
"Send Draft Email",
|
||||||
|
"Outlook Email Sending",
|
||||||
|
f"Processing {len(parameters.get('documentList', []))} draft(s)",
|
||||||
|
parentOperationId=parentOperationId
|
||||||
|
)
|
||||||
|
|
||||||
|
connectionReference = parameters.get("connectionReference")
|
||||||
|
documentList = parameters.get("documentList", [])
|
||||||
|
|
||||||
|
if not connectionReference:
|
||||||
|
if operationId:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
return ActionResult.isFailure(error="Connection reference is required")
|
||||||
|
|
||||||
|
if not documentList:
|
||||||
|
if operationId:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
return ActionResult.isFailure(error="documentList is required and cannot be empty")
|
||||||
|
|
||||||
|
# Convert single value to list if needed
|
||||||
|
if isinstance(documentList, str):
|
||||||
|
documentList = [documentList]
|
||||||
|
|
||||||
|
# Get Microsoft connection
|
||||||
|
self.services.chat.progressLogUpdate(operationId, 0.2, "Getting Microsoft connection")
|
||||||
|
connection = self.connection.getMicrosoftConnection(connectionReference)
|
||||||
|
if not connection:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
return ActionResult.isFailure(error="No valid Microsoft connection found for the provided connection reference")
|
||||||
|
|
||||||
|
# Check permissions
|
||||||
|
self.services.chat.progressLogUpdate(operationId, 0.3, "Checking permissions")
|
||||||
|
permissions_ok = await self.connection.checkPermissions(connection)
|
||||||
|
if not permissions_ok:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
return ActionResult.isFailure(error="Connection lacks necessary permissions for Outlook operations")
|
||||||
|
|
||||||
|
# Read draft email JSON documents from documentList
|
||||||
|
self.services.chat.progressLogUpdate(operationId, 0.4, "Reading draft email documents")
|
||||||
|
draftEmails = []
|
||||||
|
for docRef in documentList:
|
||||||
|
try:
|
||||||
|
# Get documents from document reference
|
||||||
|
from modules.datamodels.datamodelDocref import DocumentReferenceList
|
||||||
|
chatDocuments = self.services.chat.getChatDocumentsFromDocumentList(DocumentReferenceList.from_string_list([docRef]))
|
||||||
|
if not chatDocuments:
|
||||||
|
logger.warning(f"No documents found for reference: {docRef}")
|
||||||
|
continue
|
||||||
|
|
||||||
|
# Process each document in the reference
|
||||||
|
for doc in chatDocuments:
|
||||||
|
try:
|
||||||
|
# Read file data
|
||||||
|
fileId = getattr(doc, 'fileId', None)
|
||||||
|
if not fileId:
|
||||||
|
logger.warning(f"Document {doc.fileName} has no fileId")
|
||||||
|
continue
|
||||||
|
|
||||||
|
fileData = self.services.chat.getFileData(fileId)
|
||||||
|
if not fileData:
|
||||||
|
logger.warning(f"No file data found for document: {doc.fileName}")
|
||||||
|
continue
|
||||||
|
|
||||||
|
# Parse JSON content
|
||||||
|
if isinstance(fileData, bytes):
|
||||||
|
jsonContent = fileData.decode('utf-8')
|
||||||
|
else:
|
||||||
|
jsonContent = str(fileData)
|
||||||
|
|
||||||
|
# Parse JSON - handle both direct JSON and JSON wrapped in documentData
|
||||||
|
try:
|
||||||
|
draftEmailData = json.loads(jsonContent)
|
||||||
|
|
||||||
|
# If the JSON contains a 'documentData' field, extract it
|
||||||
|
if isinstance(draftEmailData, dict) and 'documentData' in draftEmailData:
|
||||||
|
documentDataStr = draftEmailData['documentData']
|
||||||
|
if isinstance(documentDataStr, str):
|
||||||
|
draftEmailData = json.loads(documentDataStr)
|
||||||
|
|
||||||
|
# Validate draft email structure
|
||||||
|
if not isinstance(draftEmailData, dict):
|
||||||
|
logger.warning(f"Document {doc.fileName} does not contain a valid draft email JSON object")
|
||||||
|
continue
|
||||||
|
|
||||||
|
draftId = draftEmailData.get("draftId")
|
||||||
|
if not draftId:
|
||||||
|
logger.warning(f"Document {doc.fileName} does not contain 'draftId' field")
|
||||||
|
continue
|
||||||
|
|
||||||
|
draftEmails.append({
|
||||||
|
"draftEmailJson": draftEmailData,
|
||||||
|
"draftId": draftId,
|
||||||
|
"sourceDocument": doc.fileName,
|
||||||
|
"sourceReference": docRef
|
||||||
|
})
|
||||||
|
|
||||||
|
except json.JSONDecodeError as e:
|
||||||
|
logger.error(f"Failed to parse JSON from document {doc.fileName}: {str(e)}")
|
||||||
|
continue
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error processing document {doc.fileName}: {str(e)}")
|
||||||
|
continue
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error reading documents from reference {docRef}: {str(e)}")
|
||||||
|
continue
|
||||||
|
|
||||||
|
if not draftEmails:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
return ActionResult.isFailure(error="No valid draft email JSON documents found in documentList")
|
||||||
|
|
||||||
|
self.services.chat.progressLogUpdate(operationId, 0.6, f"Found {len(draftEmails)} draft email(s) to send")
|
||||||
|
|
||||||
|
# Send all draft emails
|
||||||
|
graph_url = "https://graph.microsoft.com/v1.0"
|
||||||
|
headers = {
|
||||||
|
"Authorization": f"Bearer {connection['accessToken']}",
|
||||||
|
"Content-Type": "application/json"
|
||||||
|
}
|
||||||
|
|
||||||
|
sentResults = []
|
||||||
|
failedResults = []
|
||||||
|
|
||||||
|
self.services.chat.progressLogUpdate(operationId, 0.7, "Sending emails")
|
||||||
|
for idx, draftEmail in enumerate(draftEmails):
|
||||||
|
draftEmailJson = draftEmail["draftEmailJson"]
|
||||||
|
draftId = draftEmail["draftId"]
|
||||||
|
sourceDocument = draftEmail["sourceDocument"]
|
||||||
|
|
||||||
|
try:
|
||||||
|
send_url = f"{graph_url}/me/messages/{draftId}/send"
|
||||||
|
sendResponse = requests.post(send_url, headers=headers)
|
||||||
|
|
||||||
|
# Extract email details from draft JSON for confirmation
|
||||||
|
subject = draftEmailJson.get("subject", "Unknown")
|
||||||
|
recipients = draftEmailJson.get("recipients", [])
|
||||||
|
cc = draftEmailJson.get("cc", [])
|
||||||
|
bcc = draftEmailJson.get("bcc", [])
|
||||||
|
attachmentsCount = draftEmailJson.get("attachments", 0)
|
||||||
|
|
||||||
|
if sendResponse.status_code in [200, 202, 204]:
|
||||||
|
sentResults.append({
|
||||||
|
"status": "sent",
|
||||||
|
"message": "Email sent successfully",
|
||||||
|
"draftId": draftId,
|
||||||
|
"subject": subject,
|
||||||
|
"recipients": recipients,
|
||||||
|
"cc": cc,
|
||||||
|
"bcc": bcc,
|
||||||
|
"attachments": attachmentsCount,
|
||||||
|
"sentTimestamp": self.services.utils.timestampGetUtc(),
|
||||||
|
"sourceDocument": sourceDocument
|
||||||
|
})
|
||||||
|
logger.info(f"Email sent successfully. Draft ID: {draftId}, Subject: {subject}")
|
||||||
|
self.services.chat.progressLogUpdate(operationId, 0.7 + (idx + 1) * 0.2 / len(draftEmails), f"Sent {idx + 1}/{len(draftEmails)}: {subject}")
|
||||||
|
else:
|
||||||
|
errorResult = {
|
||||||
|
"status": "error",
|
||||||
|
"message": "Failed to send draft email",
|
||||||
|
"draftId": draftId,
|
||||||
|
"subject": subject,
|
||||||
|
"recipients": recipients,
|
||||||
|
"sendError": {
|
||||||
|
"statusCode": sendResponse.status_code,
|
||||||
|
"response": sendResponse.text
|
||||||
|
},
|
||||||
|
"sentTimestamp": self.services.utils.timestampGetUtc(),
|
||||||
|
"sourceDocument": sourceDocument
|
||||||
|
}
|
||||||
|
failedResults.append(errorResult)
|
||||||
|
logger.error(f"Failed to send email. Draft ID: {draftId}, Status: {sendResponse.status_code}, Response: {sendResponse.text}")
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
errorResult = {
|
||||||
|
"status": "error",
|
||||||
|
"message": f"Exception while sending draft email: {str(e)}",
|
||||||
|
"draftId": draftId,
|
||||||
|
"subject": draftEmailJson.get("subject", "Unknown"),
|
||||||
|
"recipients": draftEmailJson.get("recipients", []),
|
||||||
|
"exception": str(e),
|
||||||
|
"sentTimestamp": self.services.utils.timestampGetUtc(),
|
||||||
|
"sourceDocument": sourceDocument
|
||||||
|
}
|
||||||
|
failedResults.append(errorResult)
|
||||||
|
logger.error(f"Error sending draft email {draftId}: {str(e)}")
|
||||||
|
|
||||||
|
# Build result summary
|
||||||
|
totalEmails = len(draftEmails)
|
||||||
|
successfulEmails = len(sentResults)
|
||||||
|
failedEmails = len(failedResults)
|
||||||
|
|
||||||
|
resultData = {
|
||||||
|
"totalEmails": totalEmails,
|
||||||
|
"successfulEmails": successfulEmails,
|
||||||
|
"failedEmails": failedEmails,
|
||||||
|
"sentResults": sentResults,
|
||||||
|
"failedResults": failedResults,
|
||||||
|
"timestamp": self.services.utils.timestampGetUtc()
|
||||||
|
}
|
||||||
|
|
||||||
|
# Determine overall success status
|
||||||
|
self.services.chat.progressLogUpdate(operationId, 0.9, f"Sent {successfulEmails}/{totalEmails} email(s)")
|
||||||
|
if successfulEmails == 0:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
validationMetadata = {
|
||||||
|
"actionType": "outlook.sendDraftEmail",
|
||||||
|
"connectionReference": connectionReference,
|
||||||
|
"totalEmails": totalEmails,
|
||||||
|
"successfulEmails": successfulEmails,
|
||||||
|
"failedEmails": failedEmails,
|
||||||
|
"status": "all_failed"
|
||||||
|
}
|
||||||
|
return ActionResult.isFailure(
|
||||||
|
error=f"Failed to send all {totalEmails} email(s)",
|
||||||
|
documents=[ActionDocument(
|
||||||
|
documentName=f"sent_mail_confirmation_{self._format_timestamp_for_filename()}.json",
|
||||||
|
documentData=json.dumps(resultData, indent=2),
|
||||||
|
mimeType="application/json",
|
||||||
|
validationMetadata=validationMetadata
|
||||||
|
)]
|
||||||
|
)
|
||||||
|
elif failedEmails > 0:
|
||||||
|
# Partial success
|
||||||
|
logger.warning(f"Sent {successfulEmails} out of {totalEmails} emails. {failedEmails} failed.")
|
||||||
|
validationMetadata = {
|
||||||
|
"actionType": "outlook.sendDraftEmail",
|
||||||
|
"connectionReference": connectionReference,
|
||||||
|
"totalEmails": totalEmails,
|
||||||
|
"successfulEmails": successfulEmails,
|
||||||
|
"failedEmails": failedEmails,
|
||||||
|
"status": "partial_success"
|
||||||
|
}
|
||||||
|
self.services.chat.progressLogFinish(operationId, True)
|
||||||
|
return ActionResult(
|
||||||
|
success=True,
|
||||||
|
documents=[ActionDocument(
|
||||||
|
documentName=f"sent_mail_confirmation_{self._format_timestamp_for_filename()}.json",
|
||||||
|
documentData=json.dumps(resultData, indent=2),
|
||||||
|
mimeType="application/json",
|
||||||
|
validationMetadata=validationMetadata
|
||||||
|
)]
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
# All successful
|
||||||
|
logger.info(f"Successfully sent all {totalEmails} email(s)")
|
||||||
|
validationMetadata = {
|
||||||
|
"actionType": "outlook.sendDraftEmail",
|
||||||
|
"connectionReference": connectionReference,
|
||||||
|
"totalEmails": totalEmails,
|
||||||
|
"successfulEmails": successfulEmails,
|
||||||
|
"failedEmails": failedEmails,
|
||||||
|
"status": "all_successful"
|
||||||
|
}
|
||||||
|
self.services.chat.progressLogFinish(operationId, True)
|
||||||
|
return ActionResult(
|
||||||
|
success=True,
|
||||||
|
documents=[ActionDocument(
|
||||||
|
documentName=f"sent_mail_confirmation_{self._format_timestamp_for_filename()}.json",
|
||||||
|
documentData=json.dumps(resultData, indent=2),
|
||||||
|
mimeType="application/json",
|
||||||
|
validationMetadata=validationMetadata
|
||||||
|
)]
|
||||||
|
)
|
||||||
|
|
||||||
|
except ImportError:
|
||||||
|
logger.error("requests module not available")
|
||||||
|
return ActionResult.isFailure(error="requests module not available")
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error in sendDraftEmail: {str(e)}")
|
||||||
|
return ActionResult.isFailure(error=str(e))
|
||||||
|
|
||||||
|
|
@ -0,0 +1,5 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
"""Helper modules for Outlook method operations."""
|
||||||
|
|
||||||
|
|
@ -0,0 +1,95 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
"""
|
||||||
|
Connection helper for Outlook operations.
|
||||||
|
Handles Microsoft connection management and permission checking.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import requests
|
||||||
|
from typing import Dict, Any, Optional
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
class ConnectionHelper:
|
||||||
|
"""Helper for Microsoft connection management in Outlook operations"""
|
||||||
|
|
||||||
|
def __init__(self, methodInstance):
|
||||||
|
"""
|
||||||
|
Initialize connection helper.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
methodInstance: Instance of MethodOutlook (for access to services)
|
||||||
|
"""
|
||||||
|
self.method = methodInstance
|
||||||
|
self.services = methodInstance.services
|
||||||
|
|
||||||
|
def getMicrosoftConnection(self, connectionReference: str) -> Optional[Dict[str, Any]]:
|
||||||
|
"""
|
||||||
|
Helper function to get Microsoft connection details.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
logger.debug(f"Getting Microsoft connection for reference: {connectionReference}")
|
||||||
|
|
||||||
|
# Get the connection from the service
|
||||||
|
userConnection = self.services.chat.getUserConnectionFromConnectionReference(connectionReference)
|
||||||
|
if not userConnection:
|
||||||
|
logger.error(f"Connection not found: {connectionReference}")
|
||||||
|
return None
|
||||||
|
|
||||||
|
logger.debug(f"Found connection: {userConnection.id}, status: {userConnection.status.value}, authority: {userConnection.authority.value}")
|
||||||
|
|
||||||
|
# Get a fresh token for this connection
|
||||||
|
token = self.services.chat.getFreshConnectionToken(userConnection.id)
|
||||||
|
if not token:
|
||||||
|
logger.error(f"Fresh token not found for connection: {userConnection.id}")
|
||||||
|
logger.debug(f"Connection details: {userConnection}")
|
||||||
|
return None
|
||||||
|
|
||||||
|
logger.debug(f"Fresh token retrieved for connection {userConnection.id}")
|
||||||
|
|
||||||
|
# Check if connection is active
|
||||||
|
if userConnection.status.value != "active":
|
||||||
|
logger.error(f"Connection is not active: {userConnection.id}, status: {userConnection.status.value}")
|
||||||
|
return None
|
||||||
|
|
||||||
|
return {
|
||||||
|
"id": userConnection.id,
|
||||||
|
"accessToken": token.tokenAccess,
|
||||||
|
"refreshToken": token.tokenRefresh,
|
||||||
|
"scopes": ["Mail.ReadWrite", "Mail.Send", "Mail.ReadWrite.Shared", "User.Read"] # Valid Microsoft Graph API scopes
|
||||||
|
}
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error getting Microsoft connection: {str(e)}")
|
||||||
|
return None
|
||||||
|
|
||||||
|
async def checkPermissions(self, connection: Dict[str, Any]) -> bool:
|
||||||
|
"""
|
||||||
|
Check if the current connection has the necessary permissions for Outlook operations.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
graph_url = "https://graph.microsoft.com/v1.0"
|
||||||
|
headers = {
|
||||||
|
"Authorization": f"Bearer {connection['accessToken']}",
|
||||||
|
"Content-Type": "application/json"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Test permissions by trying to access the user's mail folder
|
||||||
|
test_url = f"{graph_url}/me/mailFolders"
|
||||||
|
response = requests.get(test_url, headers=headers)
|
||||||
|
|
||||||
|
if response.status_code == 200:
|
||||||
|
return True
|
||||||
|
elif response.status_code == 403:
|
||||||
|
logger.error("Permission denied - connection lacks necessary mail permissions")
|
||||||
|
logger.error("Required scopes: Mail.ReadWrite, Mail.Send, Mail.ReadWrite.Shared")
|
||||||
|
return False
|
||||||
|
else:
|
||||||
|
logger.warning(f"Permission check returned status {response.status_code}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error checking permissions: {str(e)}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
@ -0,0 +1,184 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
"""
|
||||||
|
Email Processing helper for Outlook operations.
|
||||||
|
Handles email search query sanitization, search parameter building, and filter construction.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import re
|
||||||
|
from typing import Dict, Any
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
class EmailProcessingHelper:
|
||||||
|
"""Helper for email search and processing operations"""
|
||||||
|
|
||||||
|
def __init__(self, methodInstance):
|
||||||
|
"""
|
||||||
|
Initialize email processing helper.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
methodInstance: Instance of MethodOutlook (for access to services)
|
||||||
|
"""
|
||||||
|
self.method = methodInstance
|
||||||
|
self.services = methodInstance.services
|
||||||
|
|
||||||
|
def sanitizeSearchQuery(self, query: str) -> str:
|
||||||
|
"""
|
||||||
|
Sanitize and validate search query for Microsoft Graph API
|
||||||
|
|
||||||
|
Microsoft Graph API has specific requirements for search queries:
|
||||||
|
- Escape special characters properly
|
||||||
|
- Handle search operators correctly
|
||||||
|
- Ensure query format is valid
|
||||||
|
"""
|
||||||
|
if not query:
|
||||||
|
return ""
|
||||||
|
|
||||||
|
# Clean the query
|
||||||
|
clean_query = query.strip()
|
||||||
|
|
||||||
|
# Handle folder specifications first
|
||||||
|
if clean_query.lower().startswith('folder:'):
|
||||||
|
folder_name = clean_query[7:].strip()
|
||||||
|
if folder_name:
|
||||||
|
# Return the folder specification as-is
|
||||||
|
return clean_query
|
||||||
|
|
||||||
|
# Remove any double quotes that might cause issues
|
||||||
|
clean_query = clean_query.replace('"', '')
|
||||||
|
|
||||||
|
# Handle common search operators
|
||||||
|
# Recognize Graph operators including both singular and plural forms for hasAttachments
|
||||||
|
lowered = clean_query.lower()
|
||||||
|
if any(op in lowered for op in ['from:', 'to:', 'subject:', 'received:', 'hasattachment:', 'hasattachments:']):
|
||||||
|
# This is an advanced search query, return as-is
|
||||||
|
return clean_query
|
||||||
|
|
||||||
|
# For basic text search, ensure it's safe for contains() filter
|
||||||
|
# Remove any characters that might break the OData filter syntax
|
||||||
|
# Remove or escape characters that could break OData filter syntax
|
||||||
|
safe_query = re.sub(r'[\\\'"]', '', clean_query)
|
||||||
|
|
||||||
|
return safe_query
|
||||||
|
|
||||||
|
def buildSearchParameters(self, query: str, folder: str, limit: int) -> Dict[str, Any]:
|
||||||
|
"""
|
||||||
|
Build search parameters for Microsoft Graph API
|
||||||
|
|
||||||
|
This method handles the complexity of building search parameters
|
||||||
|
while avoiding conflicts between $search and $filter parameters.
|
||||||
|
"""
|
||||||
|
params = {
|
||||||
|
"$top": limit
|
||||||
|
}
|
||||||
|
|
||||||
|
if not query or not query.strip():
|
||||||
|
# No query specified, just get emails from folder
|
||||||
|
if folder and folder.lower() != "all":
|
||||||
|
# Use folder name directly for well-known folders, or get folder ID
|
||||||
|
if folder.lower() in ["inbox", "drafts", "sentitems", "deleteditems"]:
|
||||||
|
params["$filter"] = f"parentFolderId eq '{folder}'"
|
||||||
|
else:
|
||||||
|
# For custom folders, we need to get the folder ID first
|
||||||
|
# This will be handled by the calling method
|
||||||
|
params["$filter"] = f"parentFolderId eq '{folder}'"
|
||||||
|
# Add orderby for basic queries
|
||||||
|
params["$orderby"] = "receivedDateTime desc"
|
||||||
|
return params
|
||||||
|
|
||||||
|
clean_query = self.sanitizeSearchQuery(query)
|
||||||
|
|
||||||
|
# Check if this is a folder specification (e.g., "folder:Drafts", "folder:Inbox")
|
||||||
|
if clean_query.lower().startswith('folder:'):
|
||||||
|
folder_name = clean_query[7:].strip() # Remove "folder:" prefix
|
||||||
|
if folder_name:
|
||||||
|
# This is a folder specification, not a text search
|
||||||
|
# Just filter by folder and return
|
||||||
|
params["$filter"] = f"parentFolderId eq '{folder_name}'"
|
||||||
|
params["$orderby"] = "receivedDateTime desc"
|
||||||
|
return params
|
||||||
|
|
||||||
|
# Check if this is a complex search query with multiple operators
|
||||||
|
# Recognize Graph operators including both singular and plural forms for hasAttachments
|
||||||
|
lowered = clean_query.lower()
|
||||||
|
if any(op in lowered for op in ['from:', 'to:', 'subject:', 'received:', 'hasattachment:', 'hasattachments:']):
|
||||||
|
# This is an advanced search query, use $search
|
||||||
|
# Microsoft Graph API supports complex search syntax
|
||||||
|
params["$search"] = f'"{clean_query}"'
|
||||||
|
|
||||||
|
# Note: When using $search, we cannot combine it with $orderby or $filter for folder
|
||||||
|
# We'll need to filter results after the API call
|
||||||
|
# Folder filtering will be done after the API call
|
||||||
|
else:
|
||||||
|
# Use $filter for basic text search, but keep it simple to avoid "InefficientFilter" error
|
||||||
|
# Microsoft Graph API has limitations on complex filters
|
||||||
|
if len(clean_query) > 50:
|
||||||
|
# If query is too long, truncate it to avoid complex filter issues
|
||||||
|
clean_query = clean_query[:50]
|
||||||
|
|
||||||
|
|
||||||
|
# Use only subject search to keep filter simple
|
||||||
|
# Handle wildcard queries specially
|
||||||
|
if clean_query == "*" or clean_query == "":
|
||||||
|
# For wildcard or empty query, don't use contains filter
|
||||||
|
# Just use folder filter if specified
|
||||||
|
if folder and folder.lower() != "all":
|
||||||
|
params["$filter"] = f"parentFolderId eq '{folder}'"
|
||||||
|
else:
|
||||||
|
# No filter needed for wildcard search across all folders
|
||||||
|
pass
|
||||||
|
else:
|
||||||
|
params["$filter"] = f"contains(subject,'{clean_query}')"
|
||||||
|
|
||||||
|
# Add folder filter if specified
|
||||||
|
if folder and folder.lower() != "all":
|
||||||
|
params["$filter"] = f"{params['$filter']} and parentFolderId eq '{folder}'"
|
||||||
|
|
||||||
|
# Add orderby for basic queries
|
||||||
|
params["$orderby"] = "receivedDateTime desc"
|
||||||
|
|
||||||
|
|
||||||
|
return params
|
||||||
|
|
||||||
|
def buildGraphFilter(self, filter_text: str) -> Dict[str, str]:
|
||||||
|
"""
|
||||||
|
Build proper Microsoft Graph API filter parameters based on filter text
|
||||||
|
|
||||||
|
Args:
|
||||||
|
filter_text (str): The filter text to process
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Dict[str, str]: Dictionary with either $filter or $search parameter
|
||||||
|
"""
|
||||||
|
if not filter_text:
|
||||||
|
return {}
|
||||||
|
|
||||||
|
filter_text = filter_text.strip()
|
||||||
|
|
||||||
|
# Handle folder specifications (e.g., "folder:Drafts", "folder:Inbox")
|
||||||
|
if filter_text.lower().startswith('folder:'):
|
||||||
|
folder_name = filter_text[7:].strip() # Remove "folder:" prefix
|
||||||
|
if folder_name:
|
||||||
|
# This is a folder specification, return empty to let the main method handle it
|
||||||
|
return {}
|
||||||
|
|
||||||
|
# Handle search queries (from:, to:, subject:, etc.) - check this FIRST
|
||||||
|
# Support both singular and plural forms for hasAttachments
|
||||||
|
lt = filter_text.lower()
|
||||||
|
if any(lt.startswith(prefix) for prefix in ['from:', 'to:', 'subject:', 'received:', 'hasattachment:', 'hasattachments:']):
|
||||||
|
return {"$search": f'"{filter_text}"'}
|
||||||
|
|
||||||
|
# Handle email address filters (only if it's NOT a search query)
|
||||||
|
if '@' in filter_text and '.' in filter_text and ' ' not in filter_text and not filter_text.startswith('from:'):
|
||||||
|
return {"$filter": f"from/fromAddress/address eq '{filter_text}'"}
|
||||||
|
|
||||||
|
# Handle OData filter conditions (contains 'eq', 'ne', 'gt', 'lt', etc.)
|
||||||
|
if any(op in filter_text.lower() for op in [' eq ', ' ne ', ' gt ', ' lt ', ' ge ', ' le ', ' and ', ' or ']):
|
||||||
|
return {"$filter": filter_text}
|
||||||
|
|
||||||
|
# Handle text content - search in subject
|
||||||
|
return {"$filter": f"contains(subject,'{filter_text}')"}
|
||||||
|
|
||||||
|
|
@ -0,0 +1,110 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
"""
|
||||||
|
Folder Management helper for Outlook operations.
|
||||||
|
Handles folder ID resolution and folder name lookups.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import requests
|
||||||
|
from typing import Dict, Any, Optional
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
class FolderManagementHelper:
|
||||||
|
"""Helper for folder management operations"""
|
||||||
|
|
||||||
|
def __init__(self, methodInstance):
|
||||||
|
"""
|
||||||
|
Initialize folder management helper.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
methodInstance: Instance of MethodOutlook (for access to services)
|
||||||
|
"""
|
||||||
|
self.method = methodInstance
|
||||||
|
self.services = methodInstance.services
|
||||||
|
|
||||||
|
def getFolderId(self, folder_name: str, connection: Dict[str, Any]) -> Optional[str]:
|
||||||
|
"""
|
||||||
|
Get the folder ID for a given folder name
|
||||||
|
|
||||||
|
This is needed for proper filtering when using advanced search queries
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
graph_url = "https://graph.microsoft.com/v1.0"
|
||||||
|
headers = {
|
||||||
|
"Authorization": f"Bearer {connection['accessToken']}",
|
||||||
|
"Content-Type": "application/json"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Get mail folders
|
||||||
|
api_url = f"{graph_url}/me/mailFolders"
|
||||||
|
response = requests.get(api_url, headers=headers)
|
||||||
|
|
||||||
|
if response.status_code == 200:
|
||||||
|
folders_data = response.json()
|
||||||
|
all_folders = folders_data.get("value", [])
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
# Try exact match first
|
||||||
|
for folder in all_folders:
|
||||||
|
if folder.get("displayName", "").lower() == folder_name.lower():
|
||||||
|
|
||||||
|
return folder.get("id")
|
||||||
|
|
||||||
|
# Try common variations for Drafts folder
|
||||||
|
if folder_name.lower() == "drafts":
|
||||||
|
draft_variations = ["drafts", "draft", "entwürfe", "entwurf", "brouillons", "brouillon"]
|
||||||
|
for folder in all_folders:
|
||||||
|
folder_display_name = folder.get("displayName", "").lower()
|
||||||
|
if any(variation in folder_display_name for variation in draft_variations):
|
||||||
|
|
||||||
|
return folder.get("id")
|
||||||
|
|
||||||
|
# Try common variations for other folders
|
||||||
|
if folder_name.lower() == "sent items":
|
||||||
|
sent_variations = ["sent items", "sent", "gesendete elemente", "éléments envoyés"]
|
||||||
|
for folder in all_folders:
|
||||||
|
folder_display_name = folder.get("displayName", "").lower()
|
||||||
|
if any(variation in folder_display_name for variation in sent_variations):
|
||||||
|
|
||||||
|
return folder.get("id")
|
||||||
|
|
||||||
|
logger.warning(f"Folder '{folder_name}' not found. Available folders: {[f.get('displayName', 'Unknown') for f in all_folders]}")
|
||||||
|
return None
|
||||||
|
else:
|
||||||
|
logger.warning(f"Could not retrieve folders: {response.status_code}")
|
||||||
|
return None
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Error getting folder ID for '{folder_name}': {str(e)}")
|
||||||
|
return None
|
||||||
|
|
||||||
|
def getFolderNameById(self, folder_id: str, connection: Dict[str, Any]) -> str:
|
||||||
|
"""
|
||||||
|
Get the folder display name for a given folder ID
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
graph_url = "https://graph.microsoft.com/v1.0"
|
||||||
|
headers = {
|
||||||
|
"Authorization": f"Bearer {connection['accessToken']}",
|
||||||
|
"Content-Type": "application/json"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Get folder by ID
|
||||||
|
api_url = f"{graph_url}/me/mailFolders/{folder_id}"
|
||||||
|
response = requests.get(api_url, headers=headers)
|
||||||
|
|
||||||
|
if response.status_code == 200:
|
||||||
|
folder_data = response.json()
|
||||||
|
return folder_data.get("displayName", folder_id)
|
||||||
|
else:
|
||||||
|
logger.warning(f"Could not retrieve folder name for ID {folder_id}: {response.status_code}")
|
||||||
|
return folder_id
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Error getting folder name for ID '{folder_id}': {str(e)}")
|
||||||
|
return folder_id
|
||||||
|
|
||||||
237
modules/workflows/methods/methodOutlook/methodOutlook.py
Normal file
237
modules/workflows/methods/methodOutlook/methodOutlook.py
Normal file
|
|
@ -0,0 +1,237 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
import logging
|
||||||
|
from datetime import datetime, UTC
|
||||||
|
from modules.workflows.methods.methodBase import MethodBase
|
||||||
|
from modules.datamodels.datamodelWorkflowActions import WorkflowActionDefinition, WorkflowActionParameter
|
||||||
|
from modules.shared.frontendTypes import FrontendType
|
||||||
|
|
||||||
|
# Import helpers
|
||||||
|
from .helpers.connection import ConnectionHelper
|
||||||
|
from .helpers.emailProcessing import EmailProcessingHelper
|
||||||
|
from .helpers.folderManagement import FolderManagementHelper
|
||||||
|
|
||||||
|
# Import actions
|
||||||
|
from .actions.readEmails import readEmails
|
||||||
|
from .actions.searchEmails import searchEmails
|
||||||
|
from .actions.composeAndDraftEmailWithContext import composeAndDraftEmailWithContext
|
||||||
|
from .actions.sendDraftEmail import sendDraftEmail
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
class MethodOutlook(MethodBase):
|
||||||
|
"""Outlook method implementation for email operations"""
|
||||||
|
|
||||||
|
def __init__(self, services):
|
||||||
|
"""Initialize the Outlook method"""
|
||||||
|
super().__init__(services)
|
||||||
|
self.name = "outlook"
|
||||||
|
self.description = "Handle Microsoft Outlook email operations"
|
||||||
|
|
||||||
|
# Initialize helper modules
|
||||||
|
self.connection = ConnectionHelper(self)
|
||||||
|
self.emailProcessing = EmailProcessingHelper(self)
|
||||||
|
self.folderManagement = FolderManagementHelper(self)
|
||||||
|
|
||||||
|
# RBAC-Integration: Action-Definitionen mit actionId
|
||||||
|
self._actions = {
|
||||||
|
"readEmails": WorkflowActionDefinition(
|
||||||
|
actionId="outlook.readEmails",
|
||||||
|
description="Read emails and metadata from a mailbox folder",
|
||||||
|
parameters={
|
||||||
|
"connectionReference": WorkflowActionParameter(
|
||||||
|
name="connectionReference",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.USER_CONNECTION,
|
||||||
|
required=True,
|
||||||
|
description="Microsoft connection label"
|
||||||
|
),
|
||||||
|
"folder": WorkflowActionParameter(
|
||||||
|
name="folder",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.SELECT,
|
||||||
|
frontendOptions="outlook.folder",
|
||||||
|
required=False,
|
||||||
|
default="Inbox",
|
||||||
|
description="Folder to read from"
|
||||||
|
),
|
||||||
|
"limit": WorkflowActionParameter(
|
||||||
|
name="limit",
|
||||||
|
type="int",
|
||||||
|
frontendType=FrontendType.NUMBER,
|
||||||
|
required=False,
|
||||||
|
default=1000,
|
||||||
|
description="Maximum items to return",
|
||||||
|
validation={"min": 1, "max": 10000}
|
||||||
|
),
|
||||||
|
"filter": WorkflowActionParameter(
|
||||||
|
name="filter",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.TEXT,
|
||||||
|
required=False,
|
||||||
|
description="Sender, query operators, or subject text"
|
||||||
|
),
|
||||||
|
"outputMimeType": WorkflowActionParameter(
|
||||||
|
name="outputMimeType",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.SELECT,
|
||||||
|
frontendOptions=["application/json", "text/plain", "text/csv"],
|
||||||
|
required=False,
|
||||||
|
default="application/json",
|
||||||
|
description="MIME type for output file"
|
||||||
|
)
|
||||||
|
},
|
||||||
|
execute=readEmails.__get__(self, self.__class__)
|
||||||
|
),
|
||||||
|
"searchEmails": WorkflowActionDefinition(
|
||||||
|
actionId="outlook.searchEmails",
|
||||||
|
description="Search emails by query and return matching items with metadata",
|
||||||
|
parameters={
|
||||||
|
"connectionReference": WorkflowActionParameter(
|
||||||
|
name="connectionReference",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.USER_CONNECTION,
|
||||||
|
required=True,
|
||||||
|
description="Microsoft connection label"
|
||||||
|
),
|
||||||
|
"query": WorkflowActionParameter(
|
||||||
|
name="query",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.TEXT,
|
||||||
|
required=True,
|
||||||
|
description="Search expression"
|
||||||
|
),
|
||||||
|
"folder": WorkflowActionParameter(
|
||||||
|
name="folder",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.SELECT,
|
||||||
|
frontendOptions="outlook.folder",
|
||||||
|
required=False,
|
||||||
|
default="All",
|
||||||
|
description="Folder scope or All"
|
||||||
|
),
|
||||||
|
"limit": WorkflowActionParameter(
|
||||||
|
name="limit",
|
||||||
|
type="int",
|
||||||
|
frontendType=FrontendType.NUMBER,
|
||||||
|
required=False,
|
||||||
|
default=1000,
|
||||||
|
description="Maximum items to return",
|
||||||
|
validation={"min": 1, "max": 10000}
|
||||||
|
),
|
||||||
|
"outputMimeType": WorkflowActionParameter(
|
||||||
|
name="outputMimeType",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.SELECT,
|
||||||
|
frontendOptions=["application/json", "text/plain", "text/csv"],
|
||||||
|
required=False,
|
||||||
|
default="application/json",
|
||||||
|
description="MIME type for output file"
|
||||||
|
)
|
||||||
|
},
|
||||||
|
execute=searchEmails.__get__(self, self.__class__)
|
||||||
|
),
|
||||||
|
"composeAndDraftEmailWithContext": WorkflowActionDefinition(
|
||||||
|
actionId="outlook.composeAndDraftEmailWithContext",
|
||||||
|
description="Compose email content using AI from context and optional documents, then create a draft",
|
||||||
|
parameters={
|
||||||
|
"connectionReference": WorkflowActionParameter(
|
||||||
|
name="connectionReference",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.USER_CONNECTION,
|
||||||
|
required=True,
|
||||||
|
description="Microsoft connection label"
|
||||||
|
),
|
||||||
|
"to": WorkflowActionParameter(
|
||||||
|
name="to",
|
||||||
|
type="List[str]",
|
||||||
|
frontendType=FrontendType.MULTISELECT,
|
||||||
|
required=True,
|
||||||
|
description="Recipient email addresses"
|
||||||
|
),
|
||||||
|
"context": WorkflowActionParameter(
|
||||||
|
name="context",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.TEXTAREA,
|
||||||
|
required=True,
|
||||||
|
description="Detailed context for composing the email"
|
||||||
|
),
|
||||||
|
"documentList": WorkflowActionParameter(
|
||||||
|
name="documentList",
|
||||||
|
type="List[str]",
|
||||||
|
frontendType=FrontendType.DOCUMENT_REFERENCE,
|
||||||
|
required=False,
|
||||||
|
description="Document references for context/attachments"
|
||||||
|
),
|
||||||
|
"cc": WorkflowActionParameter(
|
||||||
|
name="cc",
|
||||||
|
type="List[str]",
|
||||||
|
frontendType=FrontendType.MULTISELECT,
|
||||||
|
required=False,
|
||||||
|
description="CC recipients"
|
||||||
|
),
|
||||||
|
"bcc": WorkflowActionParameter(
|
||||||
|
name="bcc",
|
||||||
|
type="List[str]",
|
||||||
|
frontendType=FrontendType.MULTISELECT,
|
||||||
|
required=False,
|
||||||
|
description="BCC recipients"
|
||||||
|
),
|
||||||
|
"emailStyle": WorkflowActionParameter(
|
||||||
|
name="emailStyle",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.SELECT,
|
||||||
|
frontendOptions=["formal", "casual", "business"],
|
||||||
|
required=False,
|
||||||
|
default="business",
|
||||||
|
description="Email style: formal, casual, or business"
|
||||||
|
),
|
||||||
|
"maxLength": WorkflowActionParameter(
|
||||||
|
name="maxLength",
|
||||||
|
type="int",
|
||||||
|
frontendType=FrontendType.NUMBER,
|
||||||
|
required=False,
|
||||||
|
default=1000,
|
||||||
|
description="Maximum length for generated content",
|
||||||
|
validation={"min": 100, "max": 10000}
|
||||||
|
)
|
||||||
|
},
|
||||||
|
execute=composeAndDraftEmailWithContext.__get__(self, self.__class__)
|
||||||
|
),
|
||||||
|
"sendDraftEmail": WorkflowActionDefinition(
|
||||||
|
actionId="outlook.sendDraftEmail",
|
||||||
|
description="Send draft email(s) using draft email JSON document(s) from action outlook.composeAndDraftEmailWithContext",
|
||||||
|
parameters={
|
||||||
|
"connectionReference": WorkflowActionParameter(
|
||||||
|
name="connectionReference",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.USER_CONNECTION,
|
||||||
|
required=True,
|
||||||
|
description="Microsoft connection label"
|
||||||
|
),
|
||||||
|
"documentList": WorkflowActionParameter(
|
||||||
|
name="documentList",
|
||||||
|
type="List[str]",
|
||||||
|
frontendType=FrontendType.DOCUMENT_REFERENCE,
|
||||||
|
required=True,
|
||||||
|
description="Document reference(s) to draft emails in JSON format (outputs from outlook.composeAndDraftEmailWithContext function)"
|
||||||
|
)
|
||||||
|
},
|
||||||
|
execute=sendDraftEmail.__get__(self, self.__class__)
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
# Validate actions after definition
|
||||||
|
self._validateActions()
|
||||||
|
|
||||||
|
# Register actions as methods (optional, für direkten Zugriff)
|
||||||
|
self.readEmails = readEmails.__get__(self, self.__class__)
|
||||||
|
self.searchEmails = searchEmails.__get__(self, self.__class__)
|
||||||
|
self.composeAndDraftEmailWithContext = composeAndDraftEmailWithContext.__get__(self, self.__class__)
|
||||||
|
self.sendDraftEmail = sendDraftEmail.__get__(self, self.__class__)
|
||||||
|
|
||||||
|
def _format_timestamp_for_filename(self) -> str:
|
||||||
|
"""Format current timestamp as YYYYMMDD-hhmmss for filenames."""
|
||||||
|
return datetime.now(UTC).strftime("%Y%m%d-%H%M%S")
|
||||||
|
|
||||||
7
modules/workflows/methods/methodSharepoint/__init__.py
Normal file
7
modules/workflows/methods/methodSharepoint/__init__.py
Normal file
|
|
@ -0,0 +1,7 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
from .methodSharepoint import MethodSharepoint
|
||||||
|
|
||||||
|
__all__ = ['MethodSharepoint']
|
||||||
|
|
||||||
|
|
@ -0,0 +1,28 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
"""Action modules for SharePoint operations."""
|
||||||
|
|
||||||
|
# Export all actions
|
||||||
|
from .findDocumentPath import findDocumentPath
|
||||||
|
from .readDocuments import readDocuments
|
||||||
|
from .uploadDocument import uploadDocument
|
||||||
|
from .listDocuments import listDocuments
|
||||||
|
from .analyzeFolderUsage import analyzeFolderUsage
|
||||||
|
from .findSiteByUrl import findSiteByUrl
|
||||||
|
from .downloadFileByPath import downloadFileByPath
|
||||||
|
from .copyFile import copyFile
|
||||||
|
from .uploadFile import uploadFile
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
'findDocumentPath',
|
||||||
|
'readDocuments',
|
||||||
|
'uploadDocument',
|
||||||
|
'listDocuments',
|
||||||
|
'analyzeFolderUsage',
|
||||||
|
'findSiteByUrl',
|
||||||
|
'downloadFileByPath',
|
||||||
|
'copyFile',
|
||||||
|
'uploadFile',
|
||||||
|
]
|
||||||
|
|
||||||
|
|
@ -0,0 +1,337 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
"""
|
||||||
|
Analyze Folder Usage action for SharePoint operations.
|
||||||
|
Analyzes usage intensity of folders and files in SharePoint.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import time
|
||||||
|
import json
|
||||||
|
from datetime import datetime, timezone, timedelta
|
||||||
|
from typing import Dict, Any
|
||||||
|
from modules.workflows.methods.methodBase import action
|
||||||
|
from modules.datamodels.datamodelChat import ActionResult, ActionDocument
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
@action
|
||||||
|
async def analyzeFolderUsage(self, parameters: Dict[str, Any]) -> ActionResult:
|
||||||
|
"""
|
||||||
|
GENERAL:
|
||||||
|
- Purpose: Analyze usage intensity of folders and files in SharePoint.
|
||||||
|
- Input requirements: connectionReference (required); documentList (required); optional startDateTime, endDateTime, interval.
|
||||||
|
- Output format: JSON with usage analytics grouped by time intervals.
|
||||||
|
|
||||||
|
Parameters:
|
||||||
|
- connectionReference (str, required): Microsoft connection label.
|
||||||
|
- documentList (list, required): Document list reference(s) containing findDocumentPath result.
|
||||||
|
- startDateTime (str, optional): Start date/time in ISO format (e.g., "2025-11-01T00:00:00Z"). Default: 30 days ago.
|
||||||
|
- endDateTime (str, optional): End date/time in ISO format (e.g., "2025-11-30T23:59:59Z"). Default: current time.
|
||||||
|
- interval (str, optional): Time interval for grouping activities. Options: "day", "week", "month". Default: "day".
|
||||||
|
"""
|
||||||
|
operationId = None
|
||||||
|
try:
|
||||||
|
# Init progress logger
|
||||||
|
workflowId = self.services.workflow.id if self.services.workflow else f"no-workflow-{int(time.time())}"
|
||||||
|
operationId = f"sharepoint_usage_{workflowId}_{int(time.time())}"
|
||||||
|
|
||||||
|
# Start progress tracking
|
||||||
|
parentOperationId = parameters.get('parentOperationId')
|
||||||
|
self.services.chat.progressLogStart(
|
||||||
|
operationId,
|
||||||
|
"Analyze Folder Usage",
|
||||||
|
"SharePoint Analytics",
|
||||||
|
"Processing document list",
|
||||||
|
parentOperationId=parentOperationId
|
||||||
|
)
|
||||||
|
|
||||||
|
connectionReference = parameters.get("connectionReference")
|
||||||
|
documentList = parameters.get("documentList")
|
||||||
|
pathQuery = parameters.get("pathQuery")
|
||||||
|
if isinstance(documentList, str):
|
||||||
|
documentList = [documentList]
|
||||||
|
startDateTime = parameters.get("startDateTime")
|
||||||
|
endDateTime = parameters.get("endDateTime")
|
||||||
|
interval = parameters.get("interval", "day")
|
||||||
|
|
||||||
|
if not connectionReference:
|
||||||
|
if operationId:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
return ActionResult.isFailure(error="Connection reference is required")
|
||||||
|
|
||||||
|
# Require either documentList or pathQuery
|
||||||
|
if not documentList and (not pathQuery or pathQuery.strip() == "" or pathQuery.strip() == "*"):
|
||||||
|
if operationId:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
return ActionResult.isFailure(error="Either documentList or pathQuery is required")
|
||||||
|
|
||||||
|
# Resolve folder/item information from documentList or pathQuery
|
||||||
|
siteId = None
|
||||||
|
driveId = None
|
||||||
|
itemId = None
|
||||||
|
folderPath = None
|
||||||
|
folderName = None
|
||||||
|
foundDocuments = None
|
||||||
|
|
||||||
|
if documentList:
|
||||||
|
foundDocuments, sites, errorMsg = await self.documentParsing.parseDocumentListForFoundDocuments(documentList)
|
||||||
|
if errorMsg:
|
||||||
|
if operationId:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
return ActionResult.isFailure(error=errorMsg)
|
||||||
|
|
||||||
|
if not foundDocuments:
|
||||||
|
if operationId:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
return ActionResult.isFailure(error="No documents found in documentList")
|
||||||
|
|
||||||
|
# Get siteId from first document (all should be from same site)
|
||||||
|
firstItem = foundDocuments[0]
|
||||||
|
siteId = firstItem.get("siteId")
|
||||||
|
if not siteId:
|
||||||
|
if operationId:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
return ActionResult.isFailure(error="Site ID missing from documentList")
|
||||||
|
|
||||||
|
# Get drive ID (needed for analytics)
|
||||||
|
driveId = await self.services.sharepoint.getDriveId(siteId)
|
||||||
|
if not driveId:
|
||||||
|
if operationId:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
return ActionResult.isFailure(error="Could not determine drive ID for the site")
|
||||||
|
|
||||||
|
# If no items from documentList, try pathQuery fallback
|
||||||
|
if not foundDocuments and pathQuery and pathQuery.strip() != "" and pathQuery.strip() != "*":
|
||||||
|
sites, errorMsg = await self.siteDiscovery.resolveSitesFromPathQuery(pathQuery)
|
||||||
|
if errorMsg:
|
||||||
|
if operationId:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
return ActionResult.isFailure(error=errorMsg)
|
||||||
|
|
||||||
|
if sites:
|
||||||
|
siteId = sites[0].get("id")
|
||||||
|
# Parse pathQuery to find the folder/item
|
||||||
|
pathQueryParsed, fileQuery, searchType, searchOptions = self.pathProcessing.parseSearchQuery(pathQuery)
|
||||||
|
|
||||||
|
# Extract folder path from pathQuery
|
||||||
|
folderPath = '/'
|
||||||
|
if pathQueryParsed and pathQueryParsed.startswith('/sites/'):
|
||||||
|
parsedPath = self.siteDiscovery.extractSiteFromStandardPath(pathQueryParsed)
|
||||||
|
if parsedPath:
|
||||||
|
innerPath = parsedPath.get("innerPath", "")
|
||||||
|
folderPath = '/' + innerPath if innerPath else '/'
|
||||||
|
elif pathQueryParsed:
|
||||||
|
folderPath = pathQueryParsed
|
||||||
|
|
||||||
|
# Get drive ID
|
||||||
|
driveId = await self.services.sharepoint.getDriveId(siteId)
|
||||||
|
if not driveId:
|
||||||
|
if operationId:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
return ActionResult.isFailure(error="Could not determine drive ID for the site")
|
||||||
|
|
||||||
|
# Get folder/item by path
|
||||||
|
folderInfo = await self.services.sharepoint.getFolderByPath(siteId, folderPath.lstrip('/'))
|
||||||
|
if not folderInfo:
|
||||||
|
if operationId:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
return ActionResult.isFailure(error=f"Folder or file not found at path: {folderPath}")
|
||||||
|
|
||||||
|
# Add pathQuery item to foundDocuments for processing
|
||||||
|
foundDocuments = [{
|
||||||
|
"id": folderInfo.get("id"),
|
||||||
|
"name": folderInfo.get("name", ""),
|
||||||
|
"type": "folder" if folderInfo.get("folder") else "file",
|
||||||
|
"siteId": siteId,
|
||||||
|
"fullPath": folderPath,
|
||||||
|
"webUrl": folderInfo.get("webUrl", "")
|
||||||
|
}]
|
||||||
|
|
||||||
|
if not siteId or not driveId:
|
||||||
|
if operationId:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
return ActionResult.isFailure(error="Either documentList must contain findDocumentPath result with folder information, or pathQuery must be provided. Use findDocumentPath first to get folder path, or provide pathQuery directly.")
|
||||||
|
|
||||||
|
self.services.chat.progressLogUpdate(operationId, 0.2, "Getting Microsoft connection")
|
||||||
|
# Get Microsoft connection
|
||||||
|
connection = self.connection.getMicrosoftConnection(connectionReference)
|
||||||
|
if not connection:
|
||||||
|
if operationId:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
return ActionResult.isFailure(error="No valid Microsoft connection found for the provided connection reference")
|
||||||
|
|
||||||
|
# Set access token
|
||||||
|
if not self.services.sharepoint.setAccessTokenFromConnection(connection):
|
||||||
|
if operationId:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
return ActionResult.isFailure(error="Failed to set SharePoint access token")
|
||||||
|
|
||||||
|
# Process all items from documentList or pathQuery
|
||||||
|
# IMPORTANT: Only analyze FOLDERS, not files (action is "analyzeFolderUsage")
|
||||||
|
itemsToAnalyze = []
|
||||||
|
if foundDocuments:
|
||||||
|
for item in foundDocuments:
|
||||||
|
itemId = item.get("id")
|
||||||
|
itemType = item.get("type", "").lower()
|
||||||
|
|
||||||
|
# Only process folders, skip files and site-level items
|
||||||
|
if itemId and itemType == "folder":
|
||||||
|
itemsToAnalyze.append({
|
||||||
|
"id": itemId,
|
||||||
|
"name": item.get("name", ""),
|
||||||
|
"type": itemType,
|
||||||
|
"path": item.get("fullPath", ""),
|
||||||
|
"webUrl": item.get("webUrl", "")
|
||||||
|
})
|
||||||
|
|
||||||
|
if not itemsToAnalyze:
|
||||||
|
if operationId:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
return ActionResult.isFailure(error="No valid folders found in documentList to analyze. Note: This action only analyzes folders, not files.")
|
||||||
|
|
||||||
|
self.services.chat.progressLogUpdate(operationId, 0.4, f"Analyzing {len(itemsToAnalyze)} folder(s)")
|
||||||
|
|
||||||
|
# Analyze each item
|
||||||
|
allAnalytics = []
|
||||||
|
totalActivities = 0
|
||||||
|
uniqueUsers = set()
|
||||||
|
activityTypes = {}
|
||||||
|
|
||||||
|
# Compute actual date range values (getFolderUsageAnalytics will set defaults if None)
|
||||||
|
# We need to compute them here to store in output, since getFolderUsageAnalytics modifies them
|
||||||
|
actualStartDateTime = startDateTime
|
||||||
|
actualEndDateTime = endDateTime
|
||||||
|
if not actualEndDateTime:
|
||||||
|
actualEndDateTime = datetime.now(timezone.utc).isoformat().replace('+00:00', 'Z')
|
||||||
|
if not actualStartDateTime:
|
||||||
|
startDate = datetime.now(timezone.utc) - timedelta(days=30)
|
||||||
|
actualStartDateTime = startDate.isoformat().replace('+00:00', 'Z')
|
||||||
|
|
||||||
|
for idx, item in enumerate(itemsToAnalyze):
|
||||||
|
progress = 0.4 + (idx / len(itemsToAnalyze)) * 0.5
|
||||||
|
self.services.chat.progressLogUpdate(operationId, progress, f"Analyzing folder {item['name']} ({idx+1}/{len(itemsToAnalyze)})")
|
||||||
|
|
||||||
|
# Get usage analytics for this folder
|
||||||
|
analyticsResult = await self.services.sharepoint.getFolderUsageAnalytics(
|
||||||
|
siteId=siteId,
|
||||||
|
driveId=driveId,
|
||||||
|
itemId=item["id"],
|
||||||
|
startDateTime=startDateTime,
|
||||||
|
endDateTime=endDateTime,
|
||||||
|
interval=interval
|
||||||
|
)
|
||||||
|
|
||||||
|
if "error" in analyticsResult:
|
||||||
|
logger.warning(f"Failed to get analytics for item {item['name']} ({item['id']}): {analyticsResult['error']}")
|
||||||
|
# Continue with other items even if one fails
|
||||||
|
itemAnalytics = {
|
||||||
|
"itemId": item["id"],
|
||||||
|
"itemName": item["name"],
|
||||||
|
"itemType": item["type"],
|
||||||
|
"itemPath": item["path"],
|
||||||
|
"error": analyticsResult.get("error", "Unknown error")
|
||||||
|
}
|
||||||
|
else:
|
||||||
|
# Process analytics for this item
|
||||||
|
itemActivities = 0
|
||||||
|
itemUsers = set()
|
||||||
|
itemActivityTypes = {}
|
||||||
|
|
||||||
|
if "value" in analyticsResult:
|
||||||
|
for intervalData in analyticsResult["value"]:
|
||||||
|
activities = intervalData.get("activities", [])
|
||||||
|
for activity in activities:
|
||||||
|
itemActivities += 1
|
||||||
|
totalActivities += 1
|
||||||
|
|
||||||
|
action = activity.get("action", {})
|
||||||
|
actionType = action.get("verb", "unknown")
|
||||||
|
itemActivityTypes[actionType] = itemActivityTypes.get(actionType, 0) + 1
|
||||||
|
activityTypes[actionType] = activityTypes.get(actionType, 0) + 1
|
||||||
|
|
||||||
|
actor = activity.get("actor", {})
|
||||||
|
userPrincipalName = actor.get("userPrincipalName", "")
|
||||||
|
if userPrincipalName:
|
||||||
|
itemUsers.add(userPrincipalName)
|
||||||
|
uniqueUsers.add(userPrincipalName)
|
||||||
|
|
||||||
|
itemAnalytics = {
|
||||||
|
"itemId": item["id"],
|
||||||
|
"itemName": item["name"],
|
||||||
|
"itemType": item["type"],
|
||||||
|
"itemPath": item["path"],
|
||||||
|
"webUrl": item["webUrl"],
|
||||||
|
"analytics": analyticsResult,
|
||||||
|
"summary": {
|
||||||
|
"totalActivities": itemActivities,
|
||||||
|
"uniqueUsers": len(itemUsers),
|
||||||
|
"activityTypes": itemActivityTypes
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
# Include note if analytics are not available
|
||||||
|
if "note" in analyticsResult:
|
||||||
|
itemAnalytics["note"] = analyticsResult["note"]
|
||||||
|
|
||||||
|
allAnalytics.append(itemAnalytics)
|
||||||
|
|
||||||
|
self.services.chat.progressLogUpdate(operationId, 0.9, "Processing analytics data")
|
||||||
|
|
||||||
|
# Process and format analytics data
|
||||||
|
resultData = {
|
||||||
|
"siteId": siteId,
|
||||||
|
"driveId": driveId,
|
||||||
|
"startDateTime": actualStartDateTime, # Store computed date range (not None)
|
||||||
|
"endDateTime": actualEndDateTime, # Store computed date range (not None)
|
||||||
|
"interval": interval,
|
||||||
|
"itemsAnalyzed": len(itemsToAnalyze),
|
||||||
|
"foldersAnalyzed": len([item for item in allAnalytics if item.get("itemType") == "folder"]),
|
||||||
|
"items": allAnalytics,
|
||||||
|
"summary": {
|
||||||
|
"totalActivities": totalActivities,
|
||||||
|
"uniqueUsers": len(uniqueUsers),
|
||||||
|
"activityTypes": activityTypes
|
||||||
|
},
|
||||||
|
"note": f"Analyzed {len(itemsToAnalyze)} folder(s) from {actualStartDateTime} to {actualEndDateTime}. " +
|
||||||
|
f"Found {totalActivities} total activities across {len(uniqueUsers)} unique user(s)." +
|
||||||
|
(f" Note: {len([item for item in allAnalytics if 'error' in item])} folder(s) had errors or no analytics data available." if any('error' in item for item in allAnalytics) else ""),
|
||||||
|
"timestamp": self.services.utils.timestampGetUtc()
|
||||||
|
}
|
||||||
|
|
||||||
|
self.services.chat.progressLogUpdate(operationId, 0.95, f"Found {totalActivities} total activities across {len(itemsToAnalyze)} folder(s)")
|
||||||
|
|
||||||
|
validationMetadata = {
|
||||||
|
"actionType": "sharepoint.analyzeFolderUsage",
|
||||||
|
"itemsAnalyzed": len(itemsToAnalyze),
|
||||||
|
"interval": interval,
|
||||||
|
"totalActivities": totalActivities,
|
||||||
|
"uniqueUsers": len(uniqueUsers)
|
||||||
|
}
|
||||||
|
|
||||||
|
self.services.chat.progressLogFinish(operationId, True)
|
||||||
|
return ActionResult(
|
||||||
|
success=True,
|
||||||
|
documents=[
|
||||||
|
ActionDocument(
|
||||||
|
documentName=self._generateMeaningfulFileName("sharepoint_usage_analysis", "json", None, "analyzeFolderUsage"),
|
||||||
|
documentData=json.dumps(resultData, indent=2),
|
||||||
|
mimeType="application/json",
|
||||||
|
validationMetadata=validationMetadata
|
||||||
|
)
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error analyzing folder usage: {str(e)}")
|
||||||
|
if operationId:
|
||||||
|
try:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
except:
|
||||||
|
pass
|
||||||
|
return ActionResult(
|
||||||
|
success=False,
|
||||||
|
error=str(e)
|
||||||
|
)
|
||||||
|
|
||||||
163
modules/workflows/methods/methodSharepoint/actions/copyFile.py
Normal file
163
modules/workflows/methods/methodSharepoint/actions/copyFile.py
Normal file
|
|
@ -0,0 +1,163 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
"""
|
||||||
|
Copy File action for SharePoint operations.
|
||||||
|
Copies file within SharePoint.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import json
|
||||||
|
from typing import Dict, Any
|
||||||
|
from modules.workflows.methods.methodBase import action
|
||||||
|
from modules.datamodels.datamodelChat import ActionResult, ActionDocument
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
@action
|
||||||
|
async def copyFile(self, parameters: Dict[str, Any]) -> ActionResult:
|
||||||
|
"""
|
||||||
|
Copy file within SharePoint.
|
||||||
|
|
||||||
|
Parameters:
|
||||||
|
- connectionReference (str, required): Microsoft connection label.
|
||||||
|
- siteId (str, required): SharePoint site ID (from findSiteByUrl result) or document reference containing site info
|
||||||
|
- sourceFolder (str, required): Source folder path relative to site root
|
||||||
|
- sourceFile (str, required): Source file name
|
||||||
|
- destFolder (str, required): Destination folder path relative to site root
|
||||||
|
- destFile (str, required): Destination file name
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
- ActionResult with ActionDocument containing copy result
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
connectionReference = parameters.get("connectionReference")
|
||||||
|
if not connectionReference:
|
||||||
|
return ActionResult.isFailure(error="connectionReference parameter is required")
|
||||||
|
|
||||||
|
siteIdParam = parameters.get("siteId")
|
||||||
|
if not siteIdParam:
|
||||||
|
return ActionResult.isFailure(error="siteId parameter is required")
|
||||||
|
|
||||||
|
sourceFolder = parameters.get("sourceFolder")
|
||||||
|
if not sourceFolder:
|
||||||
|
return ActionResult.isFailure(error="sourceFolder parameter is required")
|
||||||
|
|
||||||
|
sourceFile = parameters.get("sourceFile")
|
||||||
|
if not sourceFile:
|
||||||
|
return ActionResult.isFailure(error="sourceFile parameter is required")
|
||||||
|
|
||||||
|
destFolder = parameters.get("destFolder")
|
||||||
|
if not destFolder:
|
||||||
|
return ActionResult.isFailure(error="destFolder parameter is required")
|
||||||
|
|
||||||
|
destFile = parameters.get("destFile")
|
||||||
|
if not destFile:
|
||||||
|
return ActionResult.isFailure(error="destFile parameter is required")
|
||||||
|
|
||||||
|
# Extract siteId from document if it's a reference
|
||||||
|
siteId = None
|
||||||
|
if isinstance(siteIdParam, str):
|
||||||
|
from modules.datamodels.datamodelDocref import DocumentReferenceList
|
||||||
|
try:
|
||||||
|
docList = DocumentReferenceList.from_string_list([siteIdParam])
|
||||||
|
chatDocuments = self.services.chat.getChatDocumentsFromDocumentList(docList)
|
||||||
|
if chatDocuments and len(chatDocuments) > 0:
|
||||||
|
siteInfoJson = json.loads(chatDocuments[0].documentData)
|
||||||
|
siteId = siteInfoJson.get("id")
|
||||||
|
except:
|
||||||
|
pass
|
||||||
|
|
||||||
|
if not siteId:
|
||||||
|
siteId = siteIdParam
|
||||||
|
else:
|
||||||
|
siteId = siteIdParam
|
||||||
|
|
||||||
|
if not siteId:
|
||||||
|
return ActionResult.isFailure(error="Could not extract siteId from parameter")
|
||||||
|
|
||||||
|
# Get Microsoft connection
|
||||||
|
connection = self.connection.getMicrosoftConnection(connectionReference)
|
||||||
|
if not connection:
|
||||||
|
return ActionResult.isFailure(error="No valid Microsoft connection found for the provided connection reference")
|
||||||
|
|
||||||
|
# Copy file
|
||||||
|
await self.services.sharepoint.copyFileAsync(
|
||||||
|
siteId=siteId,
|
||||||
|
sourceFolder=sourceFolder,
|
||||||
|
sourceFile=sourceFile,
|
||||||
|
destFolder=destFolder,
|
||||||
|
destFile=destFile
|
||||||
|
)
|
||||||
|
|
||||||
|
logger.info(f"Copied file in SharePoint: {sourceFolder}/{sourceFile} -> {destFolder}/{destFile}")
|
||||||
|
|
||||||
|
# Generate filename
|
||||||
|
workflowContext = self.services.chat.getWorkflowContext() if hasattr(self.services, 'chat') else None
|
||||||
|
filename = self._generateMeaningfulFileName(
|
||||||
|
"file_copy_result",
|
||||||
|
"json",
|
||||||
|
workflowContext,
|
||||||
|
"copyFile"
|
||||||
|
)
|
||||||
|
|
||||||
|
result = {
|
||||||
|
"success": True,
|
||||||
|
"siteId": siteId,
|
||||||
|
"sourcePath": f"{sourceFolder}/{sourceFile}",
|
||||||
|
"destPath": f"{destFolder}/{destFile}"
|
||||||
|
}
|
||||||
|
|
||||||
|
validationMetadata = self._createValidationMetadata(
|
||||||
|
"copyFile",
|
||||||
|
siteId=siteId,
|
||||||
|
sourcePath=f"{sourceFolder}/{sourceFile}",
|
||||||
|
destPath=f"{destFolder}/{destFile}"
|
||||||
|
)
|
||||||
|
|
||||||
|
document = ActionDocument(
|
||||||
|
documentName=filename,
|
||||||
|
documentData=json.dumps(result, indent=2),
|
||||||
|
mimeType="application/json",
|
||||||
|
validationMetadata=validationMetadata
|
||||||
|
)
|
||||||
|
|
||||||
|
return ActionResult.isSuccess(documents=[document])
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
# Handle file not found gracefully
|
||||||
|
if "itemNotFound" in str(e) or "404" in str(e):
|
||||||
|
logger.warning(f"File not found for copy: {parameters.get('sourceFolder')}/{parameters.get('sourceFile')}")
|
||||||
|
# Return success with skipped status
|
||||||
|
workflowContext = self.services.chat.getWorkflowContext() if hasattr(self.services, 'chat') else None
|
||||||
|
filename = self._generateMeaningfulFileName(
|
||||||
|
"file_copy_result",
|
||||||
|
"json",
|
||||||
|
workflowContext,
|
||||||
|
"copyFile"
|
||||||
|
)
|
||||||
|
|
||||||
|
result = {
|
||||||
|
"success": True,
|
||||||
|
"skipped": True,
|
||||||
|
"reason": "File not found (may not exist yet)"
|
||||||
|
}
|
||||||
|
|
||||||
|
validationMetadata = self._createValidationMetadata(
|
||||||
|
"copyFile",
|
||||||
|
skipped=True
|
||||||
|
)
|
||||||
|
|
||||||
|
document = ActionDocument(
|
||||||
|
documentName=filename,
|
||||||
|
documentData=json.dumps(result, indent=2),
|
||||||
|
mimeType="application/json",
|
||||||
|
validationMetadata=validationMetadata
|
||||||
|
)
|
||||||
|
|
||||||
|
return ActionResult.isSuccess(documents=[document])
|
||||||
|
|
||||||
|
errorMsg = f"Error copying file in SharePoint: {str(e)}"
|
||||||
|
logger.error(errorMsg)
|
||||||
|
return ActionResult.isFailure(error=errorMsg)
|
||||||
|
|
||||||
|
|
@ -0,0 +1,117 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
"""
|
||||||
|
Download File By Path action for SharePoint operations.
|
||||||
|
Downloads file from SharePoint by exact file path.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import json
|
||||||
|
import base64
|
||||||
|
import os
|
||||||
|
from typing import Dict, Any
|
||||||
|
from modules.workflows.methods.methodBase import action
|
||||||
|
from modules.datamodels.datamodelChat import ActionResult, ActionDocument
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
@action
|
||||||
|
async def downloadFileByPath(self, parameters: Dict[str, Any]) -> ActionResult:
|
||||||
|
"""
|
||||||
|
Download file from SharePoint by exact file path.
|
||||||
|
|
||||||
|
Parameters:
|
||||||
|
- connectionReference (str, required): Microsoft connection label.
|
||||||
|
- siteId (str, required): SharePoint site ID (from findSiteByUrl result) or document reference containing site info
|
||||||
|
- filePath (str, required): Full file path relative to site root (e.g., "/General/50 Docs hosted by SELISE/file.xlsx")
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
- ActionResult with ActionDocument containing file content as base64-encoded bytes
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
connectionReference = parameters.get("connectionReference")
|
||||||
|
if not connectionReference:
|
||||||
|
return ActionResult.isFailure(error="connectionReference parameter is required")
|
||||||
|
|
||||||
|
siteIdParam = parameters.get("siteId")
|
||||||
|
if not siteIdParam:
|
||||||
|
return ActionResult.isFailure(error="siteId parameter is required")
|
||||||
|
|
||||||
|
filePath = parameters.get("filePath")
|
||||||
|
if not filePath:
|
||||||
|
return ActionResult.isFailure(error="filePath parameter is required")
|
||||||
|
|
||||||
|
# Extract siteId from document if it's a reference
|
||||||
|
siteId = None
|
||||||
|
if isinstance(siteIdParam, str):
|
||||||
|
# Try to parse from document reference
|
||||||
|
from modules.datamodels.datamodelDocref import DocumentReferenceList
|
||||||
|
try:
|
||||||
|
docList = DocumentReferenceList.from_string_list([siteIdParam])
|
||||||
|
chatDocuments = self.services.chat.getChatDocumentsFromDocumentList(docList)
|
||||||
|
if chatDocuments and len(chatDocuments) > 0:
|
||||||
|
siteInfoJson = json.loads(chatDocuments[0].documentData)
|
||||||
|
siteId = siteInfoJson.get("id")
|
||||||
|
except:
|
||||||
|
pass
|
||||||
|
|
||||||
|
if not siteId:
|
||||||
|
# Assume it's the site ID directly
|
||||||
|
siteId = siteIdParam
|
||||||
|
else:
|
||||||
|
siteId = siteIdParam
|
||||||
|
|
||||||
|
if not siteId:
|
||||||
|
return ActionResult.isFailure(error="Could not extract siteId from parameter")
|
||||||
|
|
||||||
|
# Get Microsoft connection
|
||||||
|
connection = self.connection.getMicrosoftConnection(connectionReference)
|
||||||
|
if not connection:
|
||||||
|
return ActionResult.isFailure(error="No valid Microsoft connection found for the provided connection reference")
|
||||||
|
|
||||||
|
# Download file
|
||||||
|
fileContent = await self.services.sharepoint.downloadFileByPath(
|
||||||
|
siteId=siteId,
|
||||||
|
filePath=filePath
|
||||||
|
)
|
||||||
|
|
||||||
|
if fileContent is None:
|
||||||
|
return ActionResult.isFailure(error=f"File not found or could not be downloaded: {filePath}")
|
||||||
|
|
||||||
|
logger.info(f"Downloaded file from SharePoint: {filePath} ({len(fileContent)} bytes)")
|
||||||
|
|
||||||
|
# Generate filename from filePath
|
||||||
|
fileName = os.path.basename(filePath) or "downloaded_file"
|
||||||
|
workflowContext = self.services.chat.getWorkflowContext() if hasattr(self.services, 'chat') else None
|
||||||
|
filename = self._generateMeaningfulFileName(
|
||||||
|
fileName.split('.')[0] if '.' in fileName else fileName,
|
||||||
|
fileName.split('.')[-1] if '.' in fileName else "bin",
|
||||||
|
workflowContext,
|
||||||
|
"downloadFileByPath"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Encode as base64
|
||||||
|
fileBase64 = base64.b64encode(fileContent).decode('utf-8')
|
||||||
|
|
||||||
|
validationMetadata = self._createValidationMetadata(
|
||||||
|
"downloadFileByPath",
|
||||||
|
siteId=siteId,
|
||||||
|
filePath=filePath,
|
||||||
|
fileSize=len(fileContent)
|
||||||
|
)
|
||||||
|
|
||||||
|
document = ActionDocument(
|
||||||
|
documentName=filename,
|
||||||
|
documentData=fileBase64,
|
||||||
|
mimeType="application/octet-stream",
|
||||||
|
validationMetadata=validationMetadata
|
||||||
|
)
|
||||||
|
|
||||||
|
return ActionResult.isSuccess(documents=[document])
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
errorMsg = f"Error downloading file from SharePoint: {str(e)}"
|
||||||
|
logger.error(errorMsg)
|
||||||
|
return ActionResult.isFailure(error=errorMsg)
|
||||||
|
|
||||||
|
|
@ -0,0 +1,497 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
"""
|
||||||
|
Find Document Path action for SharePoint operations.
|
||||||
|
Finds documents and folders by name/path across SharePoint sites.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import time
|
||||||
|
import json
|
||||||
|
import urllib.parse
|
||||||
|
from typing import Dict, Any
|
||||||
|
from modules.workflows.methods.methodBase import action
|
||||||
|
from modules.datamodels.datamodelChat import ActionResult, ActionDocument
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
@action
|
||||||
|
async def findDocumentPath(self, parameters: Dict[str, Any]) -> ActionResult:
|
||||||
|
"""
|
||||||
|
GENERAL:
|
||||||
|
- Purpose: Find documents and folders by name/path across sites.
|
||||||
|
- Input requirements: connectionReference (required); searchQuery (required); optional site, maxResults.
|
||||||
|
- Output format: JSON with found items and paths.
|
||||||
|
|
||||||
|
Parameters:
|
||||||
|
- connectionReference (str, required): Microsoft connection label.
|
||||||
|
- site (str, optional): Site hint.
|
||||||
|
- searchQuery (str, required): Search terms or path.
|
||||||
|
- maxResults (int, optional): Maximum items to return. Default: 1000.
|
||||||
|
"""
|
||||||
|
operationId = None
|
||||||
|
try:
|
||||||
|
# Init progress logger
|
||||||
|
workflowId = self.services.workflow.id if self.services.workflow else f"no-workflow-{int(time.time())}"
|
||||||
|
operationId = f"sharepoint_find_{workflowId}_{int(time.time())}"
|
||||||
|
|
||||||
|
# Start progress tracking
|
||||||
|
parentOperationId = parameters.get('parentOperationId')
|
||||||
|
self.services.chat.progressLogStart(
|
||||||
|
operationId,
|
||||||
|
"Find Document Path",
|
||||||
|
"SharePoint Search",
|
||||||
|
f"Query: {parameters.get('searchQuery', '*')}",
|
||||||
|
parentOperationId=parentOperationId
|
||||||
|
)
|
||||||
|
|
||||||
|
connectionReference = parameters.get("connectionReference")
|
||||||
|
site = parameters.get("site")
|
||||||
|
searchQuery = parameters.get("searchQuery", "*")
|
||||||
|
maxResults = parameters.get("maxResults", 1000)
|
||||||
|
|
||||||
|
if not connectionReference:
|
||||||
|
if operationId:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
return ActionResult.isFailure(error="Connection reference is required")
|
||||||
|
|
||||||
|
# Parse searchQuery to extract path, search terms, search type, and options
|
||||||
|
pathQuery, fileQuery, searchType, searchOptions = self.pathProcessing.parseSearchQuery(searchQuery)
|
||||||
|
logger.debug(f"Parsed searchQuery '{searchQuery}' -> pathQuery='{pathQuery}', fileQuery='{fileQuery}', searchType='{searchType}'")
|
||||||
|
|
||||||
|
self.services.chat.progressLogUpdate(operationId, 0.2, "Getting Microsoft connection")
|
||||||
|
connection = self.connection.getMicrosoftConnection(connectionReference)
|
||||||
|
if not connection:
|
||||||
|
if operationId:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
return ActionResult.isFailure(error="No valid Microsoft connection found for the provided connection reference")
|
||||||
|
|
||||||
|
# Extract site name from pathQuery if it contains Microsoft-standard path (/sites/SiteName/...)
|
||||||
|
siteFromPath = None
|
||||||
|
directSite = None
|
||||||
|
if pathQuery and pathQuery.startswith('/sites/'):
|
||||||
|
parsedPath = self.siteDiscovery.extractSiteFromStandardPath(pathQuery)
|
||||||
|
if parsedPath:
|
||||||
|
siteFromPath = parsedPath.get("siteName")
|
||||||
|
logger.info(f"Extracted site from Microsoft-standard pathQuery '{pathQuery}': '{siteFromPath}'")
|
||||||
|
|
||||||
|
# Try to get site directly by path (optimization - no need to load all 60 sites)
|
||||||
|
directSite = await self.siteDiscovery.getSiteByStandardPath(siteFromPath)
|
||||||
|
if directSite:
|
||||||
|
logger.info(f"Got site directly by standard path - no need to discover all sites")
|
||||||
|
sites = [directSite]
|
||||||
|
else:
|
||||||
|
logger.warning(f"Could not get site directly, falling back to site discovery")
|
||||||
|
directSite = None
|
||||||
|
else:
|
||||||
|
logger.warning(f"Failed to parse site from standard pathQuery '{pathQuery}'")
|
||||||
|
|
||||||
|
# If we didn't get the site directly, use discovery and filtering
|
||||||
|
if not directSite:
|
||||||
|
# Determine which site hint to use (priority: site parameter > site from pathQuery > site_hint from searchOptions)
|
||||||
|
siteHintToUse = site or siteFromPath or searchOptions.get("site_hint")
|
||||||
|
|
||||||
|
# Discover SharePoint sites - use targeted approach when site hint is available
|
||||||
|
self.services.chat.progressLogUpdate(operationId, 0.3, "Discovering SharePoint sites")
|
||||||
|
if siteHintToUse:
|
||||||
|
# When site hint is available, discover all sites first, then filter
|
||||||
|
allSites = await self.siteDiscovery.discoverSharePointSites()
|
||||||
|
if not allSites:
|
||||||
|
if operationId:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
return ActionResult.isFailure(error="No SharePoint sites found or accessible")
|
||||||
|
|
||||||
|
sites = self.siteDiscovery.filterSitesByHint(allSites, siteHintToUse)
|
||||||
|
logger.info(f"Filtered sites by site hint '{siteHintToUse}' -> {len(sites)} sites")
|
||||||
|
if not sites:
|
||||||
|
if operationId:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
return ActionResult.isFailure(error=f"No SharePoint sites found matching '{siteHintToUse}'")
|
||||||
|
else:
|
||||||
|
# No site hint - discover all sites
|
||||||
|
sites = await self.siteDiscovery.discoverSharePointSites()
|
||||||
|
if not sites:
|
||||||
|
if operationId:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
return ActionResult.isFailure(error="No SharePoint sites found or accessible")
|
||||||
|
|
||||||
|
# Resolve path query into search paths
|
||||||
|
searchPaths = self.pathProcessing.resolvePathQuery(pathQuery)
|
||||||
|
|
||||||
|
self.services.chat.progressLogUpdate(operationId, 0.5, f"Searching across {len(sites)} site(s)")
|
||||||
|
|
||||||
|
try:
|
||||||
|
# Search across all discovered sites
|
||||||
|
foundDocuments = []
|
||||||
|
allSitesSearched = []
|
||||||
|
|
||||||
|
# Handle different search approaches based on search type
|
||||||
|
if searchType == "folders" and fileQuery and fileQuery.strip() != "" and fileQuery.strip() != "*":
|
||||||
|
# Use unified search for folders - this is global and searches all sites
|
||||||
|
try:
|
||||||
|
|
||||||
|
# Use Microsoft Graph Search API syntax (simple term search only)
|
||||||
|
terms = [t for t in fileQuery.split() if t.strip()]
|
||||||
|
|
||||||
|
if len(terms) > 1:
|
||||||
|
# Multiple terms: search for ALL terms (AND) - more specific results
|
||||||
|
queryString = " AND ".join(terms)
|
||||||
|
else:
|
||||||
|
# Single term: search for the term
|
||||||
|
queryString = terms[0] if terms else fileQuery
|
||||||
|
logger.info(f"Using unified search for folders: {queryString}")
|
||||||
|
|
||||||
|
payload = {
|
||||||
|
"requests": [
|
||||||
|
{
|
||||||
|
"entityTypes": ["driveItem"],
|
||||||
|
"query": {"queryString": queryString},
|
||||||
|
"from": 0,
|
||||||
|
"size": 50
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
logger.info(f"Using unified search API for folders with queryString: {queryString}")
|
||||||
|
|
||||||
|
# Use global search endpoint (site-specific search not available)
|
||||||
|
unifiedResult = await self.apiClient.makeGraphApiCall(
|
||||||
|
"search/query",
|
||||||
|
method="POST",
|
||||||
|
data=json.dumps(payload).encode("utf-8")
|
||||||
|
)
|
||||||
|
|
||||||
|
if "error" in unifiedResult:
|
||||||
|
logger.warning(f"Unified search failed: {unifiedResult['error']}")
|
||||||
|
items = []
|
||||||
|
else:
|
||||||
|
# Flatten hits -> driveItem resources
|
||||||
|
items = []
|
||||||
|
for container in (unifiedResult.get("value", []) or []):
|
||||||
|
for hitsContainer in (container.get("hitsContainers", []) or []):
|
||||||
|
for hit in (hitsContainer.get("hits", []) or []):
|
||||||
|
resource = hit.get("resource")
|
||||||
|
if resource:
|
||||||
|
items.append(resource)
|
||||||
|
|
||||||
|
logger.info(f"Unified search returned {len(items)} items (pre-filter)")
|
||||||
|
|
||||||
|
# Apply our improved folder detection logic
|
||||||
|
folderItems = []
|
||||||
|
for item in items:
|
||||||
|
resource = item
|
||||||
|
|
||||||
|
# Use the same detection logic as our test
|
||||||
|
isFolder = self.services.sharepoint.detectFolderType(resource)
|
||||||
|
|
||||||
|
if isFolder:
|
||||||
|
folderItems.append(item)
|
||||||
|
|
||||||
|
items = folderItems
|
||||||
|
logger.info(f"Filtered to {len(items)} folders using improved detection logic")
|
||||||
|
|
||||||
|
# Process unified search results - extract site information from webUrl
|
||||||
|
for item in items:
|
||||||
|
itemName = item.get("name", "")
|
||||||
|
webUrl = item.get("webUrl", "")
|
||||||
|
|
||||||
|
# Extract site information from webUrl
|
||||||
|
siteName = "Unknown Site"
|
||||||
|
siteId = "unknown"
|
||||||
|
|
||||||
|
if webUrl and '/sites/' in webUrl:
|
||||||
|
try:
|
||||||
|
# Extract site name from URL: https://pcuster.sharepoint.com/sites/SiteName/...
|
||||||
|
urlParts = webUrl.split('/sites/')
|
||||||
|
if len(urlParts) > 1:
|
||||||
|
sitePath = urlParts[1].split('/')[0]
|
||||||
|
# Find matching site from discovered sites
|
||||||
|
# First try to match by site name (URL path)
|
||||||
|
for site in sites:
|
||||||
|
if site.get("name") == sitePath:
|
||||||
|
siteName = site.get("displayName", sitePath)
|
||||||
|
siteId = site.get("id", "unknown")
|
||||||
|
break
|
||||||
|
else:
|
||||||
|
# If no match by name, try to match by displayName
|
||||||
|
for site in sites:
|
||||||
|
if site.get("displayName") == sitePath:
|
||||||
|
siteName = site.get("displayName", sitePath)
|
||||||
|
siteId = site.get("id", "unknown")
|
||||||
|
break
|
||||||
|
else:
|
||||||
|
# If no exact match, use the site path as site name
|
||||||
|
siteName = sitePath
|
||||||
|
# Try to find a site with similar name
|
||||||
|
for site in sites:
|
||||||
|
if sitePath.lower() in site.get("name", "").lower() or sitePath.lower() in site.get("displayName", "").lower():
|
||||||
|
siteName = site.get("displayName", sitePath)
|
||||||
|
siteId = site.get("id", "unknown")
|
||||||
|
break
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Error extracting site info from URL {webUrl}: {e}")
|
||||||
|
|
||||||
|
# Use improved folder detection logic
|
||||||
|
isFolder = self.services.sharepoint.detectFolderType(item)
|
||||||
|
itemType = "folder" if isFolder else "file"
|
||||||
|
itemPath = item.get("parentReference", {}).get("path", "")
|
||||||
|
logger.debug(f"Processing {itemType}: '{itemName}' at path: '{itemPath}'")
|
||||||
|
|
||||||
|
# Simple filtering like test file - just check search type
|
||||||
|
if searchType == "files" and isFolder:
|
||||||
|
continue # Skip folders when searching for files
|
||||||
|
elif searchType == "folders" and not isFolder:
|
||||||
|
continue # Skip files when searching for folders
|
||||||
|
|
||||||
|
# Simple approach like test file - no complex filtering
|
||||||
|
logger.debug(f"Item '{itemName}' found - adding to results")
|
||||||
|
|
||||||
|
# Create result with full path information for proper action chaining
|
||||||
|
parentPath = item.get("parentReference", {}).get("path", "")
|
||||||
|
|
||||||
|
# Extract the full SharePoint path from webUrl or parentReference
|
||||||
|
fullPath = ""
|
||||||
|
if webUrl:
|
||||||
|
# Extract path from webUrl: https://pcuster.sharepoint.com/sites/SSSRESYNachfolge/Freigegebene%20Dokumente/General/Eskalation%20LogObject/Druckersteuerung
|
||||||
|
if '/sites/' in webUrl:
|
||||||
|
pathPart = webUrl.split('/sites/')[1]
|
||||||
|
# Decode URL encoding and convert to backslash format
|
||||||
|
decodedPath = urllib.parse.unquote(pathPart)
|
||||||
|
fullPath = "\\" + decodedPath.replace('/', '\\')
|
||||||
|
elif parentPath:
|
||||||
|
# Use parentReference path if available
|
||||||
|
fullPath = parentPath.replace('/', '\\')
|
||||||
|
|
||||||
|
docInfo = {
|
||||||
|
"id": item.get("id"),
|
||||||
|
"name": item.get("name"),
|
||||||
|
"type": "folder" if isFolder else "file",
|
||||||
|
"siteName": siteName,
|
||||||
|
"siteId": siteId,
|
||||||
|
"webUrl": webUrl,
|
||||||
|
"fullPath": fullPath,
|
||||||
|
"parentPath": parentPath
|
||||||
|
}
|
||||||
|
|
||||||
|
foundDocuments.append(docInfo)
|
||||||
|
|
||||||
|
logger.info(f"Found {len(foundDocuments)} documents from unified search")
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error performing unified folder search: {str(e)}")
|
||||||
|
# Fallback to site-by-site search
|
||||||
|
pass
|
||||||
|
|
||||||
|
# If no unified search was performed or it failed, fall back to site-by-site search
|
||||||
|
if not foundDocuments:
|
||||||
|
# Use simple approach like test file - no complex filtering
|
||||||
|
siteScopedSites = sites
|
||||||
|
|
||||||
|
for site in siteScopedSites:
|
||||||
|
siteId = site["id"]
|
||||||
|
siteName = site["displayName"]
|
||||||
|
siteUrl = site["webUrl"]
|
||||||
|
|
||||||
|
logger.info(f"Searching in site: {siteName} ({siteUrl})")
|
||||||
|
|
||||||
|
# Check if pathQuery contains a specific folder path (not just /sites/SiteName)
|
||||||
|
folderPath = None
|
||||||
|
if pathQuery and pathQuery.startswith('/sites/'):
|
||||||
|
parsedPath = self.siteDiscovery.extractSiteFromStandardPath(pathQuery)
|
||||||
|
if parsedPath:
|
||||||
|
innerPath = parsedPath.get("innerPath", "")
|
||||||
|
if innerPath and innerPath.strip():
|
||||||
|
# Remove leading slash if present
|
||||||
|
folderPath = innerPath.lstrip('/')
|
||||||
|
|
||||||
|
# Generic approach: Try to find the folder, if it fails, remove first segment
|
||||||
|
# This works for all languages because we test the actual API response
|
||||||
|
# In SharePoint Graph API, /drive/root already points to the default document library,
|
||||||
|
# so library names in paths should be removed
|
||||||
|
pathSegments = [s for s in folderPath.split('/') if s.strip()]
|
||||||
|
if len(pathSegments) > 1:
|
||||||
|
# Try with first segment removed (first segment is likely the document library)
|
||||||
|
testPath = '/'.join(pathSegments[1:])
|
||||||
|
# Quick test: try to get folder info (this is fast and doesn't require full search)
|
||||||
|
testEndpoint = f"sites/{siteId}/drive/root:/{urllib.parse.quote(testPath, safe='')}:"
|
||||||
|
testResult = await self.apiClient.makeGraphApiCall(testEndpoint)
|
||||||
|
if testResult and "error" not in testResult:
|
||||||
|
# Path without first segment works - first segment was likely the document library
|
||||||
|
folderPath = testPath
|
||||||
|
logger.info(f"Removed document library name '{pathSegments[0]}' from folder path (tested via API)")
|
||||||
|
else:
|
||||||
|
# Keep original path - first segment is not a document library
|
||||||
|
logger.info(f"Keeping original folder path '{folderPath}' (first segment is not a document library)")
|
||||||
|
elif len(pathSegments) == 1:
|
||||||
|
# Only one segment - likely the document library itself, use root
|
||||||
|
folderPath = None
|
||||||
|
logger.info(f"Only one segment '{pathSegments[0]}' found, likely document library - using root")
|
||||||
|
|
||||||
|
if folderPath:
|
||||||
|
logger.info(f"Extracted folder path from pathQuery: '{folderPath}'")
|
||||||
|
else:
|
||||||
|
logger.info(f"Folder path resolved to root (only document library in path)")
|
||||||
|
|
||||||
|
# Use Microsoft Graph API for this specific site
|
||||||
|
# Handle empty or wildcard queries
|
||||||
|
if not fileQuery or fileQuery.strip() == "" or fileQuery.strip() == "*":
|
||||||
|
# For wildcard/empty queries, list all items
|
||||||
|
if folderPath:
|
||||||
|
# List items in specific folder
|
||||||
|
encodedPath = urllib.parse.quote(folderPath, safe='')
|
||||||
|
endpoint = f"sites/{siteId}/drive/root:/{encodedPath}:/children"
|
||||||
|
logger.info(f"Listing items in folder: '{folderPath}'")
|
||||||
|
else:
|
||||||
|
# List all items in the drive root
|
||||||
|
endpoint = f"sites/{siteId}/drive/root/children"
|
||||||
|
|
||||||
|
# Make the API call to list items
|
||||||
|
listResult = await self.apiClient.makeGraphApiCall(endpoint)
|
||||||
|
if "error" in listResult:
|
||||||
|
logger.warning(f"List failed for site {siteName}: {listResult['error']}")
|
||||||
|
continue
|
||||||
|
# Process list results for this site
|
||||||
|
items = listResult.get("value", [])
|
||||||
|
logger.info(f"Retrieved {len(items)} items from site {siteName}")
|
||||||
|
else:
|
||||||
|
# For files, use regular search API
|
||||||
|
# Clean the query: remove path-like syntax and invalid KQL syntax
|
||||||
|
searchQueryCleaned = self.pathProcessing.cleanSearchQuery(fileQuery)
|
||||||
|
# URL-encode the query parameter
|
||||||
|
encodedQuery = urllib.parse.quote(searchQueryCleaned, safe='')
|
||||||
|
|
||||||
|
if folderPath:
|
||||||
|
# Search in specific folder
|
||||||
|
encodedPath = urllib.parse.quote(folderPath, safe='')
|
||||||
|
endpoint = f"sites/{siteId}/drive/root:/{encodedPath}:/search(q='{encodedQuery}')"
|
||||||
|
logger.info(f"Searching in folder '{folderPath}' with query: '{searchQueryCleaned}' (encoded: '{encodedQuery}')")
|
||||||
|
else:
|
||||||
|
# Search in drive root
|
||||||
|
endpoint = f"sites/{siteId}/drive/root/search(q='{encodedQuery}')"
|
||||||
|
logger.info(f"Using search API for files with query: '{searchQueryCleaned}' (encoded: '{encodedQuery}')")
|
||||||
|
|
||||||
|
# Make the search API call (files)
|
||||||
|
searchResult = await self.apiClient.makeGraphApiCall(endpoint)
|
||||||
|
if "error" in searchResult:
|
||||||
|
logger.warning(f"Search failed for site {siteName}: {searchResult['error']}")
|
||||||
|
continue
|
||||||
|
# Process search results for this site (files)
|
||||||
|
items = searchResult.get("value", [])
|
||||||
|
logger.info(f"Retrieved {len(items)} items from site {siteName}")
|
||||||
|
|
||||||
|
siteDocuments = []
|
||||||
|
|
||||||
|
for item in items:
|
||||||
|
itemName = item.get("name", "")
|
||||||
|
|
||||||
|
# Use improved folder detection logic
|
||||||
|
isFolder = self.services.sharepoint.detectFolderType(item)
|
||||||
|
|
||||||
|
itemType = "folder" if isFolder else "file"
|
||||||
|
itemPath = item.get("parentReference", {}).get("path", "")
|
||||||
|
logger.debug(f"Processing {itemType}: '{itemName}' at path: '{itemPath}'")
|
||||||
|
|
||||||
|
# Simple filtering like test file - just check search type
|
||||||
|
if searchType == "files" and isFolder:
|
||||||
|
continue # Skip folders when searching for files
|
||||||
|
elif searchType == "folders" and not isFolder:
|
||||||
|
continue # Skip files when searching for folders
|
||||||
|
|
||||||
|
# Simple approach like test file - no complex filtering
|
||||||
|
logger.debug(f"Item '{itemName}' found - adding to results")
|
||||||
|
|
||||||
|
# Create result with full path information for proper action chaining
|
||||||
|
webUrl = item.get("webUrl", "")
|
||||||
|
parentPath = item.get("parentReference", {}).get("path", "")
|
||||||
|
|
||||||
|
# Extract the full SharePoint path from webUrl or parentReference
|
||||||
|
fullPath = ""
|
||||||
|
if webUrl:
|
||||||
|
# Extract path from webUrl: https://pcuster.sharepoint.com/sites/SSSRESYNachfolge/Freigegebene%20Dokumente/General/Eskalation%20LogObject/Druckersteuerung
|
||||||
|
if '/sites/' in webUrl:
|
||||||
|
pathPart = webUrl.split('/sites/')[1]
|
||||||
|
# Decode URL encoding and convert to backslash format
|
||||||
|
decodedPath = urllib.parse.unquote(pathPart)
|
||||||
|
fullPath = "\\" + decodedPath.replace('/', '\\')
|
||||||
|
elif parentPath:
|
||||||
|
# Use parentReference path if available
|
||||||
|
fullPath = parentPath.replace('/', '\\')
|
||||||
|
|
||||||
|
docInfo = {
|
||||||
|
"id": item.get("id"),
|
||||||
|
"name": item.get("name"),
|
||||||
|
"type": "folder" if isFolder else "file",
|
||||||
|
"siteName": siteName,
|
||||||
|
"siteId": siteId,
|
||||||
|
"webUrl": webUrl,
|
||||||
|
"fullPath": fullPath,
|
||||||
|
"parentPath": parentPath
|
||||||
|
}
|
||||||
|
|
||||||
|
siteDocuments.append(docInfo)
|
||||||
|
|
||||||
|
foundDocuments.extend(siteDocuments)
|
||||||
|
allSitesSearched.append({
|
||||||
|
"siteName": siteName,
|
||||||
|
"siteUrl": siteUrl,
|
||||||
|
"siteId": siteId,
|
||||||
|
"documentsFound": len(siteDocuments)
|
||||||
|
})
|
||||||
|
|
||||||
|
logger.info(f"Found {len(siteDocuments)} documents in site {siteName}")
|
||||||
|
|
||||||
|
# Limit total results to maxResults
|
||||||
|
if len(foundDocuments) > maxResults:
|
||||||
|
foundDocuments = foundDocuments[:maxResults]
|
||||||
|
logger.info(f"Limited results to {maxResults} items")
|
||||||
|
|
||||||
|
self.services.chat.progressLogUpdate(operationId, 0.9, f"Found {len(foundDocuments)} document(s)")
|
||||||
|
|
||||||
|
resultData = {
|
||||||
|
"searchQuery": searchQuery,
|
||||||
|
"totalResults": len(foundDocuments),
|
||||||
|
"maxResults": maxResults,
|
||||||
|
"foundDocuments": foundDocuments,
|
||||||
|
"timestamp": self.services.utils.timestampGetUtc()
|
||||||
|
}
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error searching SharePoint: {str(e)}")
|
||||||
|
if operationId:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
return ActionResult.isFailure(error=str(e))
|
||||||
|
|
||||||
|
# Use default JSON format for output
|
||||||
|
outputExtension = ".json" # Default
|
||||||
|
outputMimeType = "application/json" # Default
|
||||||
|
|
||||||
|
validationMetadata = {
|
||||||
|
"actionType": "sharepoint.findDocumentPath",
|
||||||
|
"searchQuery": searchQuery,
|
||||||
|
"maxResults": maxResults,
|
||||||
|
"totalResults": len(foundDocuments),
|
||||||
|
"hasResults": len(foundDocuments) > 0
|
||||||
|
}
|
||||||
|
|
||||||
|
self.services.chat.progressLogFinish(operationId, True)
|
||||||
|
return ActionResult(
|
||||||
|
success=True,
|
||||||
|
documents=[
|
||||||
|
ActionDocument(
|
||||||
|
documentName=self._generateMeaningfulFileName("sharepoint_find_path", "json", None, "findDocumentPath"),
|
||||||
|
documentData=json.dumps(resultData, indent=2),
|
||||||
|
mimeType=outputMimeType,
|
||||||
|
validationMetadata=validationMetadata
|
||||||
|
)
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error finding document path: {str(e)}")
|
||||||
|
if operationId:
|
||||||
|
try:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
except:
|
||||||
|
pass
|
||||||
|
return ActionResult.isFailure(error=str(e))
|
||||||
|
|
||||||
|
|
@ -0,0 +1,88 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
"""
|
||||||
|
Find Site By URL action for SharePoint operations.
|
||||||
|
Finds SharePoint site by hostname and site path.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import json
|
||||||
|
from typing import Dict, Any
|
||||||
|
from modules.workflows.methods.methodBase import action
|
||||||
|
from modules.datamodels.datamodelChat import ActionResult, ActionDocument
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
@action
|
||||||
|
async def findSiteByUrl(self, parameters: Dict[str, Any]) -> ActionResult:
|
||||||
|
"""
|
||||||
|
Find SharePoint site by hostname and site path.
|
||||||
|
|
||||||
|
Parameters:
|
||||||
|
- connectionReference (str, required): Microsoft connection label.
|
||||||
|
- hostname (str, required): SharePoint hostname (e.g., "example.sharepoint.com")
|
||||||
|
- sitePath (str, required): Site path (e.g., "SteeringBPM" or "/sites/SteeringBPM")
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
- ActionResult with ActionDocument containing site information (id, displayName, name, webUrl)
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
connectionReference = parameters.get("connectionReference")
|
||||||
|
if not connectionReference:
|
||||||
|
return ActionResult.isFailure(error="connectionReference parameter is required")
|
||||||
|
|
||||||
|
hostname = parameters.get("hostname")
|
||||||
|
if not hostname:
|
||||||
|
return ActionResult.isFailure(error="hostname parameter is required")
|
||||||
|
|
||||||
|
sitePath = parameters.get("sitePath")
|
||||||
|
if not sitePath:
|
||||||
|
return ActionResult.isFailure(error="sitePath parameter is required")
|
||||||
|
|
||||||
|
# Get Microsoft connection
|
||||||
|
connection = self.connection.getMicrosoftConnection(connectionReference)
|
||||||
|
if not connection:
|
||||||
|
return ActionResult.isFailure(error="No valid Microsoft connection found for the provided connection reference")
|
||||||
|
|
||||||
|
# Find site by URL
|
||||||
|
siteInfo = await self.services.sharepoint.findSiteByUrl(
|
||||||
|
hostname=hostname,
|
||||||
|
sitePath=sitePath
|
||||||
|
)
|
||||||
|
|
||||||
|
if not siteInfo:
|
||||||
|
return ActionResult.isFailure(error=f"Site not found: {hostname}:/sites/{sitePath}")
|
||||||
|
|
||||||
|
logger.info(f"Found SharePoint site: {siteInfo.get('displayName')} (ID: {siteInfo.get('id')})")
|
||||||
|
|
||||||
|
# Generate filename
|
||||||
|
workflowContext = self.services.chat.getWorkflowContext() if hasattr(self.services, 'chat') else None
|
||||||
|
filename = self._generateMeaningfulFileName(
|
||||||
|
"sharepoint_site",
|
||||||
|
"json",
|
||||||
|
workflowContext,
|
||||||
|
"findSiteByUrl"
|
||||||
|
)
|
||||||
|
|
||||||
|
validationMetadata = self._createValidationMetadata(
|
||||||
|
"findSiteByUrl",
|
||||||
|
hostname=hostname,
|
||||||
|
sitePath=sitePath,
|
||||||
|
siteId=siteInfo.get("id")
|
||||||
|
)
|
||||||
|
|
||||||
|
document = ActionDocument(
|
||||||
|
documentName=filename,
|
||||||
|
documentData=json.dumps(siteInfo, indent=2),
|
||||||
|
mimeType="application/json",
|
||||||
|
validationMetadata=validationMetadata
|
||||||
|
)
|
||||||
|
|
||||||
|
return ActionResult.isSuccess(documents=[document])
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
errorMsg = f"Error finding SharePoint site: {str(e)}"
|
||||||
|
logger.error(errorMsg)
|
||||||
|
return ActionResult.isFailure(error=errorMsg)
|
||||||
|
|
||||||
|
|
@ -0,0 +1,345 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
"""
|
||||||
|
List Documents action for SharePoint operations.
|
||||||
|
Lists documents and folders in SharePoint paths across sites.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import time
|
||||||
|
import json
|
||||||
|
import urllib.parse
|
||||||
|
from typing import Dict, Any
|
||||||
|
from modules.workflows.methods.methodBase import action
|
||||||
|
from modules.datamodels.datamodelChat import ActionResult, ActionDocument
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
@action
|
||||||
|
async def listDocuments(self, parameters: Dict[str, Any]) -> ActionResult:
|
||||||
|
"""
|
||||||
|
GENERAL:
|
||||||
|
- Purpose: List documents and folders in SharePoint paths across sites.
|
||||||
|
- Input requirements: connectionReference (required); documentList (required); includeSubfolders (optional).
|
||||||
|
- Output format: JSON with folder items and metadata.
|
||||||
|
|
||||||
|
Parameters:
|
||||||
|
- connectionReference (str, required): Microsoft connection label.
|
||||||
|
- documentList (list, required): Document list reference(s) containing findDocumentPath result.
|
||||||
|
- includeSubfolders (bool, optional): Include one level of subfolders. Default: False.
|
||||||
|
"""
|
||||||
|
operationId = None
|
||||||
|
try:
|
||||||
|
# Init progress logger
|
||||||
|
workflowId = self.services.workflow.id if self.services.workflow else f"no-workflow-{int(time.time())}"
|
||||||
|
operationId = f"sharepoint_list_{workflowId}_{int(time.time())}"
|
||||||
|
|
||||||
|
# Start progress tracking
|
||||||
|
parentOperationId = parameters.get('parentOperationId')
|
||||||
|
self.services.chat.progressLogStart(
|
||||||
|
operationId,
|
||||||
|
"List Documents",
|
||||||
|
"SharePoint Listing",
|
||||||
|
"Processing document list",
|
||||||
|
parentOperationId=parentOperationId
|
||||||
|
)
|
||||||
|
|
||||||
|
connectionReference = parameters.get("connectionReference")
|
||||||
|
documentList = parameters.get("documentList")
|
||||||
|
pathQuery = parameters.get("pathQuery", "*")
|
||||||
|
if isinstance(documentList, str):
|
||||||
|
documentList = [documentList]
|
||||||
|
includeSubfolders = parameters.get("includeSubfolders", False) # Default to False for better UX
|
||||||
|
|
||||||
|
if not connectionReference:
|
||||||
|
if operationId:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
return ActionResult.isFailure(error="Connection reference is required")
|
||||||
|
|
||||||
|
# Require either documentList or pathQuery
|
||||||
|
if not documentList and (not pathQuery or pathQuery.strip() == "" or pathQuery.strip() == "*"):
|
||||||
|
if operationId:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
return ActionResult.isFailure(error="Either documentList or pathQuery is required")
|
||||||
|
|
||||||
|
# Parse documentList to extract folder path and site information
|
||||||
|
listQuery, sites, _, errorMsg = await self.documentParsing.parseDocumentListForFolder(documentList)
|
||||||
|
if errorMsg:
|
||||||
|
if operationId:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
return ActionResult.isFailure(error=errorMsg)
|
||||||
|
|
||||||
|
# If no folder path found from documentList, use pathQuery if provided
|
||||||
|
if not listQuery and pathQuery and pathQuery.strip() != "" and pathQuery.strip() != "*":
|
||||||
|
listQuery = pathQuery
|
||||||
|
logger.info(f"Using pathQuery for list query: {listQuery}")
|
||||||
|
# Resolve sites from pathQuery
|
||||||
|
sites, errorMsg = await self.siteDiscovery.resolveSitesFromPathQuery(pathQuery)
|
||||||
|
if errorMsg:
|
||||||
|
if operationId:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
return ActionResult.isFailure(error=errorMsg)
|
||||||
|
|
||||||
|
# Validate required parameters
|
||||||
|
if not listQuery:
|
||||||
|
if operationId:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
return ActionResult.isFailure(error="Either documentList must contain findDocumentPath result with folder information, or pathQuery must be provided. Use findDocumentPath first to get folder path, or provide pathQuery directly.")
|
||||||
|
|
||||||
|
if not sites:
|
||||||
|
if operationId:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
return ActionResult.isFailure(error="Site information missing. Cannot determine target site for list operation.")
|
||||||
|
|
||||||
|
# Get connection
|
||||||
|
self.services.chat.progressLogUpdate(operationId, 0.2, "Getting Microsoft connection")
|
||||||
|
connection = self.connection.getMicrosoftConnection(connectionReference)
|
||||||
|
if not connection:
|
||||||
|
if operationId:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
return ActionResult.isFailure(error="No valid Microsoft connection found for the provided connection reference")
|
||||||
|
|
||||||
|
logger.info(f"Starting SharePoint listDocuments for listQuery: {listQuery}")
|
||||||
|
logger.debug(f"Connection ID: {connection['id']}")
|
||||||
|
|
||||||
|
self.services.chat.progressLogUpdate(operationId, 0.3, "Processing folder path")
|
||||||
|
|
||||||
|
# Parse listQuery to extract path, search terms, search type, and options
|
||||||
|
pathQuery, fileQuery, searchType, searchOptions = self.pathProcessing.parseSearchQuery(listQuery)
|
||||||
|
|
||||||
|
# Check if listQuery is a folder ID (starts with 01PPXICCB...)
|
||||||
|
if listQuery.startswith('01PPXICCB') or listQuery.startswith('01'):
|
||||||
|
# Direct folder ID - use it directly
|
||||||
|
folderPaths = [listQuery]
|
||||||
|
logger.info(f"Using direct folder ID: {listQuery}")
|
||||||
|
else:
|
||||||
|
# Remove site prefix from pathQuery before resolving (it's only for site filtering)
|
||||||
|
pathQueryForResolve = pathQuery
|
||||||
|
# Microsoft-standard path: /sites/SiteName/Path -> /Path
|
||||||
|
if pathQuery.startswith('/sites/'):
|
||||||
|
parsedPath = self.siteDiscovery.extractSiteFromStandardPath(pathQuery)
|
||||||
|
if parsedPath:
|
||||||
|
innerPath = parsedPath.get("innerPath", "")
|
||||||
|
pathQueryForResolve = '/' + innerPath if innerPath else '/'
|
||||||
|
else:
|
||||||
|
pathQueryForResolve = '/'
|
||||||
|
|
||||||
|
# Remove first path segment if it looks like a document library name
|
||||||
|
# In SharePoint Graph API, /drive/root already points to the default document library,
|
||||||
|
# so library names in paths should be removed
|
||||||
|
# Generic approach: if path has multiple segments, store original for fallback
|
||||||
|
pathSegments = [s for s in pathQueryForResolve.split('/') if s.strip()]
|
||||||
|
if len(pathSegments) > 1:
|
||||||
|
# Path has multiple segments - first might be a library name
|
||||||
|
# Store original for potential fallback
|
||||||
|
originalPath = pathQueryForResolve
|
||||||
|
# Try without first segment (assuming it's a library name)
|
||||||
|
pathQueryForResolve = '/' + '/'.join(pathSegments[1:])
|
||||||
|
logger.info(f"Removed first path segment (potential library name), path changed from '{originalPath}' to '{pathQueryForResolve}'")
|
||||||
|
elif len(pathSegments) == 1:
|
||||||
|
# Only one segment - if it's a common library-like name, use root
|
||||||
|
firstSegmentLower = pathSegments[0].lower()
|
||||||
|
libraryIndicators = ['document', 'dokument', 'shared', 'freigegeben', 'library', 'bibliothek']
|
||||||
|
if any(indicator in firstSegmentLower for indicator in libraryIndicators):
|
||||||
|
pathQueryForResolve = '/'
|
||||||
|
logger.info(f"First segment '{pathSegments[0]}' appears to be a library name, using root")
|
||||||
|
|
||||||
|
# Resolve path query into folder paths
|
||||||
|
folderPaths = self.pathProcessing.resolvePathQuery(pathQueryForResolve)
|
||||||
|
logger.info(f"Resolved folder paths: {folderPaths}")
|
||||||
|
|
||||||
|
# Process each folder path across all sites
|
||||||
|
listResults = []
|
||||||
|
|
||||||
|
self.services.chat.progressLogUpdate(operationId, 0.5, f"Listing {len(folderPaths)} folder(s) across {len(sites)} site(s)")
|
||||||
|
|
||||||
|
for folderPath in folderPaths:
|
||||||
|
try:
|
||||||
|
folderResults = []
|
||||||
|
|
||||||
|
for site in sites:
|
||||||
|
siteId = site["id"]
|
||||||
|
siteName = site["displayName"]
|
||||||
|
siteUrl = site["webUrl"]
|
||||||
|
|
||||||
|
logger.info(f"Listing folder {folderPath} in site: {siteName}")
|
||||||
|
|
||||||
|
# Determine the endpoint based on folder path
|
||||||
|
if folderPath in ["/", ""] or folderPath == "*":
|
||||||
|
# Root folder
|
||||||
|
endpoint = f"sites/{siteId}/drive/root/children"
|
||||||
|
elif folderPath.startswith('01PPXICCB') or folderPath.startswith('01'):
|
||||||
|
# Direct folder ID
|
||||||
|
endpoint = f"sites/{siteId}/drive/items/{folderPath}/children"
|
||||||
|
else:
|
||||||
|
# Specific folder path - remove leading slash if present and URL encode
|
||||||
|
folderPathClean = folderPath.lstrip('/')
|
||||||
|
# URL encode the path for Graph API (spaces and special characters need encoding)
|
||||||
|
folderPathEncoded = urllib.parse.quote(folderPathClean, safe='/')
|
||||||
|
endpoint = f"sites/{siteId}/drive/root:/{folderPathEncoded}:/children"
|
||||||
|
|
||||||
|
# Make the API call to list folder contents
|
||||||
|
apiResult = await self.apiClient.makeGraphApiCall(endpoint)
|
||||||
|
|
||||||
|
if "error" in apiResult:
|
||||||
|
logger.warning(f"Failed to list folder {folderPath} in site {siteName}: {apiResult['error']}")
|
||||||
|
continue
|
||||||
|
|
||||||
|
# Process the results
|
||||||
|
items = apiResult.get("value", [])
|
||||||
|
processedItems = []
|
||||||
|
|
||||||
|
for item in items:
|
||||||
|
# Use improved folder detection logic
|
||||||
|
isFolder = self.services.sharepoint.detectFolderType(item)
|
||||||
|
|
||||||
|
itemInfo = {
|
||||||
|
"id": item.get("id"),
|
||||||
|
"name": item.get("name"),
|
||||||
|
"size": item.get("size", 0),
|
||||||
|
"createdDateTime": item.get("createdDateTime"),
|
||||||
|
"lastModifiedDateTime": item.get("lastModifiedDateTime"),
|
||||||
|
"webUrl": item.get("webUrl"),
|
||||||
|
"type": "folder" if isFolder else "file",
|
||||||
|
"siteName": siteName,
|
||||||
|
"siteUrl": siteUrl
|
||||||
|
}
|
||||||
|
|
||||||
|
# Add file-specific information
|
||||||
|
if "file" in item:
|
||||||
|
itemInfo.update({
|
||||||
|
"mimeType": item["file"].get("mimeType"),
|
||||||
|
"downloadUrl": item.get("@microsoft.graph.downloadUrl")
|
||||||
|
})
|
||||||
|
|
||||||
|
# Add folder-specific information
|
||||||
|
if "folder" in item:
|
||||||
|
itemInfo.update({
|
||||||
|
"childCount": item["folder"].get("childCount", 0)
|
||||||
|
})
|
||||||
|
|
||||||
|
processedItems.append(itemInfo)
|
||||||
|
|
||||||
|
# If include subfolders is enabled, get ONLY direct subfolder contents (1 level deep only)
|
||||||
|
if includeSubfolders:
|
||||||
|
folderItems = [item for item in processedItems if item['type'] == 'folder']
|
||||||
|
logger.info(f"Including subfolders - processing {len(folderItems)} folders")
|
||||||
|
subfolderCount = 0
|
||||||
|
maxSubfolders = 10 # Limit to prevent infinite loops
|
||||||
|
|
||||||
|
for item in processedItems[:]: # Use slice to avoid modifying list during iteration
|
||||||
|
if item["type"] == "folder" and subfolderCount < maxSubfolders:
|
||||||
|
subfolderCount += 1
|
||||||
|
subfolderPath = f"{folderPath.rstrip('/')}/{item['name']}"
|
||||||
|
subfolderEndpoint = f"sites/{siteId}/drive/items/{item['id']}/children"
|
||||||
|
|
||||||
|
logger.debug(f"Getting contents of subfolder: {item['name']}")
|
||||||
|
subfolderResult = await self.apiClient.makeGraphApiCall(subfolderEndpoint)
|
||||||
|
if "error" not in subfolderResult:
|
||||||
|
subfolderItems = subfolderResult.get("value", [])
|
||||||
|
logger.debug(f"Found {len(subfolderItems)} items in subfolder {item['name']}")
|
||||||
|
|
||||||
|
for subfolderItem in subfolderItems:
|
||||||
|
# Use improved folder detection logic for subfolder items
|
||||||
|
subfolderIsFolder = self.services.sharepoint.detectFolderType(subfolderItem)
|
||||||
|
|
||||||
|
# Only add files and direct subfolders, NO RECURSION
|
||||||
|
subfolderItemInfo = {
|
||||||
|
"id": subfolderItem.get("id"),
|
||||||
|
"name": subfolderItem.get("name"),
|
||||||
|
"size": subfolderItem.get("size", 0),
|
||||||
|
"createdDateTime": subfolderItem.get("createdDateTime"),
|
||||||
|
"lastModifiedDateTime": subfolderItem.get("lastModifiedDateTime"),
|
||||||
|
"webUrl": subfolderItem.get("webUrl"),
|
||||||
|
"type": "folder" if subfolderIsFolder else "file",
|
||||||
|
"parentPath": subfolderPath,
|
||||||
|
"siteName": siteName,
|
||||||
|
"siteUrl": siteUrl
|
||||||
|
}
|
||||||
|
|
||||||
|
if "file" in subfolderItem:
|
||||||
|
subfolderItemInfo.update({
|
||||||
|
"mimeType": subfolderItem["file"].get("mimeType"),
|
||||||
|
"downloadUrl": subfolderItem.get("@microsoft.graph.downloadUrl")
|
||||||
|
})
|
||||||
|
|
||||||
|
processedItems.append(subfolderItemInfo)
|
||||||
|
else:
|
||||||
|
logger.warning(f"Failed to get contents of subfolder {item['name']}: {subfolderResult.get('error')}")
|
||||||
|
elif subfolderCount >= maxSubfolders:
|
||||||
|
logger.warning(f"Reached maximum subfolder limit ({maxSubfolders}), skipping remaining folders")
|
||||||
|
break
|
||||||
|
|
||||||
|
logger.info(f"Processed {subfolderCount} subfolders, total items: {len(processedItems)}")
|
||||||
|
|
||||||
|
folderResults.append({
|
||||||
|
"siteName": siteName,
|
||||||
|
"siteUrl": siteUrl,
|
||||||
|
"itemCount": len(processedItems),
|
||||||
|
"items": processedItems
|
||||||
|
})
|
||||||
|
|
||||||
|
listResults.append({
|
||||||
|
"folderPath": folderPath,
|
||||||
|
"sitesProcessed": len(folderResults),
|
||||||
|
"siteResults": folderResults
|
||||||
|
})
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error listing folder {folderPath}: {str(e)}")
|
||||||
|
listResults.append({
|
||||||
|
"folderPath": folderPath,
|
||||||
|
"error": str(e),
|
||||||
|
"sitesProcessed": 0,
|
||||||
|
"siteResults": []
|
||||||
|
})
|
||||||
|
|
||||||
|
# Create result data
|
||||||
|
totalItems = sum(len(siteResult.get("items", [])) for result in listResults for siteResult in result.get("siteResults", []))
|
||||||
|
|
||||||
|
resultData = {
|
||||||
|
"listQuery": listQuery,
|
||||||
|
"pathQuery": pathQuery,
|
||||||
|
"totalItems": totalItems,
|
||||||
|
"foldersProcessed": len(listResults),
|
||||||
|
"listResults": listResults,
|
||||||
|
"includeSubfolders": includeSubfolders,
|
||||||
|
"timestamp": self.services.utils.timestampGetUtc()
|
||||||
|
}
|
||||||
|
|
||||||
|
self.services.chat.progressLogUpdate(operationId, 0.9, f"Found {totalItems} item(s) in {len(listResults)} folder(s)")
|
||||||
|
|
||||||
|
validationMetadata = {
|
||||||
|
"actionType": "sharepoint.listDocuments",
|
||||||
|
"listQuery": listQuery,
|
||||||
|
"totalItems": totalItems,
|
||||||
|
"foldersProcessed": len(listResults),
|
||||||
|
"includeSubfolders": includeSubfolders
|
||||||
|
}
|
||||||
|
|
||||||
|
self.services.chat.progressLogFinish(operationId, True)
|
||||||
|
return ActionResult(
|
||||||
|
success=True,
|
||||||
|
documents=[
|
||||||
|
ActionDocument(
|
||||||
|
documentName=self._generateMeaningfulFileName("sharepoint_list", "json", None, "listDocuments"),
|
||||||
|
documentData=json.dumps(resultData, indent=2),
|
||||||
|
mimeType="application/json",
|
||||||
|
validationMetadata=validationMetadata
|
||||||
|
)
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error listing SharePoint documents: {str(e)}")
|
||||||
|
if operationId:
|
||||||
|
try:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
except:
|
||||||
|
pass
|
||||||
|
return ActionResult(
|
||||||
|
success=False,
|
||||||
|
error=str(e)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
@ -0,0 +1,290 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
"""
|
||||||
|
Read Documents action for SharePoint operations.
|
||||||
|
Reads documents from SharePoint and extracts content/metadata.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import time
|
||||||
|
import json
|
||||||
|
import base64
|
||||||
|
from typing import Dict, Any
|
||||||
|
from modules.workflows.methods.methodBase import action
|
||||||
|
from modules.datamodels.datamodelChat import ActionResult, ActionDocument
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
@action
|
||||||
|
async def readDocuments(self, parameters: Dict[str, Any]) -> ActionResult:
|
||||||
|
"""
|
||||||
|
GENERAL:
|
||||||
|
- Purpose: Read documents from SharePoint and extract content/metadata.
|
||||||
|
- Input requirements: connectionReference (required); documentList or pathQuery (required); includeMetadata (optional).
|
||||||
|
- Output format: Standardized ActionDocument format (documentName, documentData, mimeType).
|
||||||
|
- Binary files (PDFs, etc.) are Base64-encoded in documentData.
|
||||||
|
- Text files are stored as plain text in documentData.
|
||||||
|
- Returns ActionResult with documents list for template processing.
|
||||||
|
|
||||||
|
Parameters:
|
||||||
|
- connectionReference (str, required): Microsoft connection label.
|
||||||
|
- documentList (list, optional): Document list reference(s) containing findDocumentPath result.
|
||||||
|
- pathQuery (str, optional): Direct path query if no documentList (e.g., /sites/SiteName/FolderPath).
|
||||||
|
- includeMetadata (bool, optional): Include metadata. Default: True.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
- ActionResult with documents: List[ActionDocument] where each ActionDocument contains:
|
||||||
|
- documentName: File name
|
||||||
|
- documentData: Base64-encoded content (binary files) or plain text (text files)
|
||||||
|
- mimeType: MIME type (e.g., application/pdf, text/plain)
|
||||||
|
"""
|
||||||
|
operationId = None
|
||||||
|
try:
|
||||||
|
# Init progress logger
|
||||||
|
workflowId = self.services.workflow.id if self.services.workflow else f"no-workflow-{int(time.time())}"
|
||||||
|
operationId = f"sharepoint_read_{workflowId}_{int(time.time())}"
|
||||||
|
|
||||||
|
# Start progress tracking
|
||||||
|
parentOperationId = parameters.get('parentOperationId')
|
||||||
|
self.services.chat.progressLogStart(
|
||||||
|
operationId,
|
||||||
|
"Read Documents",
|
||||||
|
"SharePoint Document Reading",
|
||||||
|
"Processing document list",
|
||||||
|
parentOperationId=parentOperationId
|
||||||
|
)
|
||||||
|
|
||||||
|
documentList = parameters.get("documentList")
|
||||||
|
pathQuery = parameters.get("pathQuery", "*")
|
||||||
|
connectionReference = parameters.get("connectionReference")
|
||||||
|
includeMetadata = parameters.get("includeMetadata", True)
|
||||||
|
|
||||||
|
# Validate connection reference
|
||||||
|
if not connectionReference:
|
||||||
|
if operationId:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
return ActionResult.isFailure(error="Connection reference is required")
|
||||||
|
|
||||||
|
# Require either documentList or pathQuery
|
||||||
|
if not documentList and (not pathQuery or pathQuery.strip() == "" or pathQuery.strip() == "*"):
|
||||||
|
if operationId:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
return ActionResult.isFailure(error="Either documentList or pathQuery is required")
|
||||||
|
|
||||||
|
# Get connection first
|
||||||
|
self.services.chat.progressLogUpdate(operationId, 0.2, "Getting Microsoft connection")
|
||||||
|
connection = self.connection.getMicrosoftConnection(connectionReference)
|
||||||
|
if not connection:
|
||||||
|
if operationId:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
return ActionResult.isFailure(error="No valid Microsoft connection found for the provided connection reference")
|
||||||
|
|
||||||
|
# Parse documentList to extract foundDocuments and site information
|
||||||
|
sharePointFileIds = None
|
||||||
|
sites = None
|
||||||
|
|
||||||
|
if documentList:
|
||||||
|
foundDocuments, sites, errorMsg = await self.documentParsing.parseDocumentListForFoundDocuments(documentList)
|
||||||
|
if errorMsg:
|
||||||
|
if operationId:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
return ActionResult.isFailure(error=errorMsg)
|
||||||
|
|
||||||
|
if foundDocuments:
|
||||||
|
# Extract SharePoint file IDs from foundDocuments
|
||||||
|
sharePointFileIds = [doc.get("id") for doc in foundDocuments if doc.get("type") == "file"]
|
||||||
|
if not sharePointFileIds:
|
||||||
|
if operationId:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
return ActionResult.isFailure(error="No files found in documentList from findDocumentPath result")
|
||||||
|
logger.info(f"Extracted {len(sharePointFileIds)} SharePoint file IDs from documentList")
|
||||||
|
|
||||||
|
# If we have SharePoint file IDs from documentList (findDocumentPath result), read them directly
|
||||||
|
if sharePointFileIds and sites:
|
||||||
|
# Read SharePoint files directly using their IDs
|
||||||
|
readResults = []
|
||||||
|
siteId = sites[0]['id']
|
||||||
|
|
||||||
|
self.services.chat.progressLogUpdate(operationId, 0.5, f"Reading {len(sharePointFileIds)} file(s) from SharePoint")
|
||||||
|
for idx, fileId in enumerate(sharePointFileIds):
|
||||||
|
try:
|
||||||
|
self.services.chat.progressLogUpdate(operationId, 0.5 + (idx * 0.3 / len(sharePointFileIds)), f"Reading file {idx + 1}/{len(sharePointFileIds)}")
|
||||||
|
# Get file info from SharePoint
|
||||||
|
endpoint = f"sites/{siteId}/drive/items/{fileId}"
|
||||||
|
fileInfo = await self.apiClient.makeGraphApiCall(endpoint)
|
||||||
|
|
||||||
|
if "error" in fileInfo:
|
||||||
|
logger.warning(f"Failed to get file info for {fileId}: {fileInfo['error']}")
|
||||||
|
continue
|
||||||
|
|
||||||
|
# Get file content using SharePoint service (handles binary data correctly)
|
||||||
|
fileName = fileInfo.get("name", f"file_{fileId}")
|
||||||
|
fileContent = await self.services.sharepoint.downloadFile(siteId, fileId)
|
||||||
|
|
||||||
|
# Create result document
|
||||||
|
resultItem = {
|
||||||
|
"fileId": fileId,
|
||||||
|
"fileName": fileName,
|
||||||
|
"sharepointFileId": fileId,
|
||||||
|
"siteName": sites[0]['displayName'],
|
||||||
|
"siteUrl": sites[0]['webUrl'],
|
||||||
|
"size": fileInfo.get("size", 0),
|
||||||
|
"createdDateTime": fileInfo.get("createdDateTime"),
|
||||||
|
"lastModifiedDateTime": fileInfo.get("lastModifiedDateTime"),
|
||||||
|
"webUrl": fileInfo.get("webUrl")
|
||||||
|
}
|
||||||
|
|
||||||
|
# Add content if available
|
||||||
|
if fileContent:
|
||||||
|
resultItem["content"] = fileContent
|
||||||
|
|
||||||
|
# Add metadata if requested
|
||||||
|
if includeMetadata:
|
||||||
|
resultItem["metadata"] = {
|
||||||
|
"mimeType": fileInfo.get("file", {}).get("mimeType"),
|
||||||
|
"downloadUrl": fileInfo.get("@microsoft.graph.downloadUrl"),
|
||||||
|
"createdBy": fileInfo.get("createdBy", {}),
|
||||||
|
"lastModifiedBy": fileInfo.get("lastModifiedBy", {}),
|
||||||
|
"parentReference": fileInfo.get("parentReference", {})
|
||||||
|
}
|
||||||
|
|
||||||
|
readResults.append(resultItem)
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error reading file {fileId}: {str(e)}")
|
||||||
|
continue
|
||||||
|
|
||||||
|
if not readResults:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
return ActionResult.isFailure(error="No files could be read from documentList")
|
||||||
|
|
||||||
|
# Convert read results to ActionDocument objects
|
||||||
|
# IMPORTANT: For binary files (PDFs), store Base64-encoded content directly in documentData
|
||||||
|
# The system will create FileData and ChatDocument automatically
|
||||||
|
self.services.chat.progressLogUpdate(operationId, 0.8, f"Processing {len(readResults)} document(s)")
|
||||||
|
|
||||||
|
actionDocuments = []
|
||||||
|
for resultItem in readResults:
|
||||||
|
fileContent = resultItem.get("content")
|
||||||
|
fileName = resultItem.get("fileName", f"file_{resultItem.get('fileId')}")
|
||||||
|
|
||||||
|
# Determine MIME type from metadata or file extension
|
||||||
|
mimeType = "application/octet-stream"
|
||||||
|
if resultItem.get("metadata", {}).get("mimeType"):
|
||||||
|
mimeType = resultItem["metadata"]["mimeType"]
|
||||||
|
elif fileName:
|
||||||
|
if fileName.endswith('.pdf'):
|
||||||
|
mimeType = "application/pdf"
|
||||||
|
elif fileName.endswith('.txt'):
|
||||||
|
mimeType = "text/plain"
|
||||||
|
elif fileName.endswith('.json'):
|
||||||
|
mimeType = "application/json"
|
||||||
|
|
||||||
|
# For binary files (PDFs, etc.), store Base64-encoded content directly
|
||||||
|
# The GenerationService will detect PDF mimeType and handle base64 decoding
|
||||||
|
if fileContent and isinstance(fileContent, bytes):
|
||||||
|
# Encode binary content as Base64 string
|
||||||
|
base64Content = base64.b64encode(fileContent).decode('utf-8')
|
||||||
|
validationMetadata = {
|
||||||
|
"actionType": "sharepoint.readDocuments",
|
||||||
|
"fileName": fileName,
|
||||||
|
"sharepointFileId": resultItem.get("sharepointFileId"),
|
||||||
|
"siteName": resultItem.get("siteName"),
|
||||||
|
"mimeType": mimeType,
|
||||||
|
"contentType": "binary",
|
||||||
|
"size": len(fileContent),
|
||||||
|
"includeMetadata": includeMetadata
|
||||||
|
}
|
||||||
|
actionDoc = ActionDocument(
|
||||||
|
documentName=fileName,
|
||||||
|
documentData=base64Content, # Base64 string for binary files
|
||||||
|
mimeType=mimeType,
|
||||||
|
validationMetadata=validationMetadata
|
||||||
|
)
|
||||||
|
actionDocuments.append(actionDoc)
|
||||||
|
logger.info(f"Stored binary file {fileName} ({len(fileContent)} bytes) as Base64 in ActionDocument")
|
||||||
|
elif fileContent:
|
||||||
|
# Text content - store directly in documentData
|
||||||
|
validationMetadata = {
|
||||||
|
"actionType": "sharepoint.readDocuments",
|
||||||
|
"fileName": fileName,
|
||||||
|
"sharepointFileId": resultItem.get("sharepointFileId"),
|
||||||
|
"siteName": resultItem.get("siteName"),
|
||||||
|
"mimeType": mimeType,
|
||||||
|
"contentType": "text",
|
||||||
|
"includeMetadata": includeMetadata
|
||||||
|
}
|
||||||
|
actionDoc = ActionDocument(
|
||||||
|
documentName=fileName,
|
||||||
|
documentData=fileContent if isinstance(fileContent, str) else str(fileContent),
|
||||||
|
mimeType=mimeType,
|
||||||
|
validationMetadata=validationMetadata
|
||||||
|
)
|
||||||
|
actionDocuments.append(actionDoc)
|
||||||
|
else:
|
||||||
|
# No content - store metadata only
|
||||||
|
docData = {
|
||||||
|
"fileName": fileName,
|
||||||
|
"sharepointFileId": resultItem.get("sharepointFileId"),
|
||||||
|
"siteName": resultItem.get("siteName"),
|
||||||
|
"siteUrl": resultItem.get("siteUrl"),
|
||||||
|
"size": resultItem.get("size"),
|
||||||
|
"createdDateTime": resultItem.get("createdDateTime"),
|
||||||
|
"lastModifiedDateTime": resultItem.get("lastModifiedDateTime"),
|
||||||
|
"webUrl": resultItem.get("webUrl")
|
||||||
|
}
|
||||||
|
if resultItem.get("metadata"):
|
||||||
|
docData["metadata"] = resultItem["metadata"]
|
||||||
|
|
||||||
|
validationMetadata = {
|
||||||
|
"actionType": "sharepoint.readDocuments",
|
||||||
|
"fileName": fileName,
|
||||||
|
"sharepointFileId": resultItem.get("sharepointFileId"),
|
||||||
|
"siteName": resultItem.get("siteName"),
|
||||||
|
"mimeType": mimeType,
|
||||||
|
"contentType": "metadata_only",
|
||||||
|
"includeMetadata": includeMetadata
|
||||||
|
}
|
||||||
|
actionDoc = ActionDocument(
|
||||||
|
documentName=fileName,
|
||||||
|
documentData=json.dumps(docData, indent=2),
|
||||||
|
mimeType=mimeType,
|
||||||
|
validationMetadata=validationMetadata
|
||||||
|
)
|
||||||
|
actionDocuments.append(actionDoc)
|
||||||
|
|
||||||
|
# Return success with action documents
|
||||||
|
self.services.chat.progressLogUpdate(operationId, 0.9, f"Read {len(actionDocuments)} document(s)")
|
||||||
|
self.services.chat.progressLogFinish(operationId, True)
|
||||||
|
return ActionResult.isSuccess(documents=actionDocuments)
|
||||||
|
|
||||||
|
# If no sites from documentList, try pathQuery fallback
|
||||||
|
if not sites and pathQuery and pathQuery.strip() != "" and pathQuery.strip() != "*":
|
||||||
|
sites, errorMsg = await self.siteDiscovery.resolveSitesFromPathQuery(pathQuery)
|
||||||
|
if errorMsg:
|
||||||
|
if operationId:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
return ActionResult.isFailure(error=errorMsg)
|
||||||
|
|
||||||
|
# If still no sites, return error
|
||||||
|
if not sites:
|
||||||
|
if operationId:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
return ActionResult.isFailure(error="Either documentList must contain findDocumentPath result with file information, or pathQuery must be provided. Use findDocumentPath first to get file paths, or provide pathQuery directly.")
|
||||||
|
|
||||||
|
# This should never be reached if logic above is correct
|
||||||
|
if operationId:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
return ActionResult.isFailure(error="Unexpected error: could not process documentList or pathQuery")
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error reading SharePoint documents: {str(e)}")
|
||||||
|
if operationId:
|
||||||
|
try:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
except:
|
||||||
|
pass # Don't fail on progress logging errors
|
||||||
|
return ActionResult(
|
||||||
|
success=False,
|
||||||
|
error=str(e)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
@ -0,0 +1,278 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
"""
|
||||||
|
Upload Document action for SharePoint operations.
|
||||||
|
Uploads documents to SharePoint.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import time
|
||||||
|
import json
|
||||||
|
import urllib.parse
|
||||||
|
from typing import Dict, Any
|
||||||
|
from modules.workflows.methods.methodBase import action
|
||||||
|
from modules.datamodels.datamodelChat import ActionResult, ActionDocument
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
@action
|
||||||
|
async def uploadDocument(self, parameters: Dict[str, Any]) -> ActionResult:
|
||||||
|
"""
|
||||||
|
GENERAL:
|
||||||
|
- Purpose: Upload documents to SharePoint. Only to choose this action with a connectionReference
|
||||||
|
- Input requirements: connectionReference (required); documentList (required); pathQuery (optional).
|
||||||
|
- Output format: JSON with upload status and file info.
|
||||||
|
|
||||||
|
Parameters:
|
||||||
|
- connectionReference (str, required): Microsoft connection label.
|
||||||
|
- documentList (list, required): Document reference(s) to upload. File names are taken from the documents.
|
||||||
|
- pathQuery (str, optional): Direct upload target path if documentList doesn't contain findDocumentPath result (e.g., /sites/SiteName/FolderPath).
|
||||||
|
"""
|
||||||
|
operationId = None
|
||||||
|
try:
|
||||||
|
# Init progress logger
|
||||||
|
workflowId = self.services.workflow.id if self.services.workflow else f"no-workflow-{int(time.time())}"
|
||||||
|
operationId = f"sharepoint_upload_{workflowId}_{int(time.time())}"
|
||||||
|
|
||||||
|
# Start progress tracking
|
||||||
|
parentOperationId = parameters.get('parentOperationId')
|
||||||
|
self.services.chat.progressLogStart(
|
||||||
|
operationId,
|
||||||
|
"Upload Document",
|
||||||
|
"SharePoint Upload",
|
||||||
|
"Processing document list",
|
||||||
|
parentOperationId=parentOperationId
|
||||||
|
)
|
||||||
|
|
||||||
|
connectionReference = parameters.get("connectionReference")
|
||||||
|
documentList = parameters.get("documentList")
|
||||||
|
pathQuery = parameters.get("pathQuery")
|
||||||
|
if isinstance(documentList, str):
|
||||||
|
documentList = [documentList]
|
||||||
|
|
||||||
|
if not connectionReference:
|
||||||
|
if operationId:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
return ActionResult.isFailure(error="Connection reference is required")
|
||||||
|
|
||||||
|
if not documentList:
|
||||||
|
if operationId:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
return ActionResult.isFailure(error="Document list is required")
|
||||||
|
|
||||||
|
# Parse documentList to extract folder path and site information
|
||||||
|
uploadPath, sites, filesToUpload, errorMsg = await self.documentParsing.parseDocumentListForFolder(documentList)
|
||||||
|
if errorMsg:
|
||||||
|
if operationId:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
return ActionResult.isFailure(error=errorMsg)
|
||||||
|
|
||||||
|
# If no folder path found from documentList, use pathQuery if provided
|
||||||
|
if not uploadPath and pathQuery and pathQuery.strip() != "" and pathQuery.strip() != "*":
|
||||||
|
uploadPath = pathQuery
|
||||||
|
logger.info(f"Using pathQuery for upload path: {uploadPath}")
|
||||||
|
# Resolve sites from pathQuery
|
||||||
|
sites, errorMsg = await self.siteDiscovery.resolveSitesFromPathQuery(pathQuery)
|
||||||
|
if errorMsg:
|
||||||
|
if operationId:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
return ActionResult.isFailure(error=errorMsg)
|
||||||
|
|
||||||
|
# Validate required parameters
|
||||||
|
if not uploadPath:
|
||||||
|
if operationId:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
return ActionResult.isFailure(error="Either documentList must contain findDocumentPath result with folder information, or pathQuery must be provided. Use findDocumentPath first to get upload folder, or provide pathQuery directly.")
|
||||||
|
|
||||||
|
if not sites:
|
||||||
|
if operationId:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
return ActionResult.isFailure(error="Site information missing. Cannot determine target site for upload.")
|
||||||
|
|
||||||
|
if not filesToUpload:
|
||||||
|
if operationId:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
return ActionResult.isFailure(error="No files to upload found in documentList.")
|
||||||
|
|
||||||
|
# Get connection
|
||||||
|
self.services.chat.progressLogUpdate(operationId, 0.3, "Getting Microsoft connection")
|
||||||
|
connection = self.connection.getMicrosoftConnection(connectionReference)
|
||||||
|
if not connection:
|
||||||
|
if operationId:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
return ActionResult.isFailure(error="No valid Microsoft connection found for the provided connection reference")
|
||||||
|
|
||||||
|
# Process upload paths
|
||||||
|
uploadPaths = []
|
||||||
|
if uploadPath.startswith('01PPXICCB') or uploadPath.startswith('01'):
|
||||||
|
# It's a folder ID - use it directly
|
||||||
|
uploadPaths = [uploadPath]
|
||||||
|
logger.info(f"Using folder ID directly for upload: {uploadPath}")
|
||||||
|
else:
|
||||||
|
# It's a path - resolve it normally
|
||||||
|
uploadPaths = self.pathProcessing.resolvePathQuery(uploadPath)
|
||||||
|
|
||||||
|
# Process each document upload
|
||||||
|
uploadResults = []
|
||||||
|
|
||||||
|
# Extract file names from documents
|
||||||
|
fileNames = [doc.fileName for doc in filesToUpload]
|
||||||
|
logger.info(f"Using file names from documentList: {fileNames}")
|
||||||
|
|
||||||
|
self.services.chat.progressLogUpdate(operationId, 0.5, f"Uploading {len(filesToUpload)} document(s)")
|
||||||
|
|
||||||
|
for i, (chatDocument, fileName) in enumerate(zip(filesToUpload, fileNames)):
|
||||||
|
try:
|
||||||
|
fileId = chatDocument.fileId
|
||||||
|
fileData = self.services.chat.getFileData(fileId)
|
||||||
|
|
||||||
|
if not fileData:
|
||||||
|
logger.warning(f"File data not found for fileId: {fileId}")
|
||||||
|
uploadResults.append({
|
||||||
|
"fileName": fileName,
|
||||||
|
"fileId": fileId,
|
||||||
|
"error": "File data not found",
|
||||||
|
"uploadStatus": "failed"
|
||||||
|
})
|
||||||
|
continue
|
||||||
|
|
||||||
|
# Upload to the first available site (or could be made configurable)
|
||||||
|
uploadSuccessful = False
|
||||||
|
|
||||||
|
for site in sites:
|
||||||
|
siteId = site["id"]
|
||||||
|
siteName = site["displayName"]
|
||||||
|
siteUrl = site["webUrl"]
|
||||||
|
|
||||||
|
# Use the first upload path or default to Documents
|
||||||
|
uploadPath = uploadPaths[0] if uploadPaths else "/Documents"
|
||||||
|
|
||||||
|
# Handle wildcard paths - replace with default Documents folder
|
||||||
|
if uploadPath == "*":
|
||||||
|
uploadPath = "/Documents"
|
||||||
|
logger.warning(f"Wildcard path '*' detected, using default '/Documents' folder for upload")
|
||||||
|
|
||||||
|
# Check if uploadPath is a folder ID or a regular path
|
||||||
|
if uploadPath.startswith('01PPXICCB') or uploadPath.startswith('01'):
|
||||||
|
# It's a folder ID - use the folder-specific upload endpoint
|
||||||
|
uploadEndpoint = f"sites/{siteId}/drive/items/{uploadPath}:/{fileName}:/content"
|
||||||
|
logger.info(f"Using folder ID upload endpoint: {uploadEndpoint}")
|
||||||
|
else:
|
||||||
|
# It's a regular path - use the root-based upload endpoint
|
||||||
|
uploadPath = uploadPath.rstrip('/') + '/' + fileName
|
||||||
|
uploadPathClean = uploadPath.lstrip('/')
|
||||||
|
uploadEndpoint = f"sites/{siteId}/drive/root:/{uploadPathClean}:/content"
|
||||||
|
logger.info(f"Using path-based upload endpoint: {uploadEndpoint}")
|
||||||
|
|
||||||
|
# Upload endpoint for small files (< 4MB)
|
||||||
|
if len(fileData) < 4 * 1024 * 1024: # 4MB
|
||||||
|
|
||||||
|
# Upload the file
|
||||||
|
uploadResult = await self.apiClient.makeGraphApiCall(
|
||||||
|
uploadEndpoint,
|
||||||
|
method="PUT",
|
||||||
|
data=fileData
|
||||||
|
)
|
||||||
|
|
||||||
|
if "error" not in uploadResult:
|
||||||
|
uploadResults.append({
|
||||||
|
"fileName": fileName,
|
||||||
|
"fileId": fileId,
|
||||||
|
"uploadStatus": "success",
|
||||||
|
"siteName": siteName,
|
||||||
|
"siteUrl": siteUrl,
|
||||||
|
"uploadPath": uploadPath,
|
||||||
|
"uploadEndpoint": uploadEndpoint,
|
||||||
|
"sharepointFileId": uploadResult.get("id"),
|
||||||
|
"webUrl": uploadResult.get("webUrl"),
|
||||||
|
"size": uploadResult.get("size"),
|
||||||
|
"createdDateTime": uploadResult.get("createdDateTime")
|
||||||
|
})
|
||||||
|
uploadSuccessful = True
|
||||||
|
break
|
||||||
|
else:
|
||||||
|
logger.warning(f"Upload failed to site {siteName}: {uploadResult['error']}")
|
||||||
|
else:
|
||||||
|
# For large files, we would need to implement resumable upload
|
||||||
|
logger.warning(f"File too large ({len(fileData)} bytes) for site {siteName}")
|
||||||
|
continue
|
||||||
|
|
||||||
|
if not uploadSuccessful:
|
||||||
|
uploadResults.append({
|
||||||
|
"fileName": fileName,
|
||||||
|
"fileId": fileId,
|
||||||
|
"error": f"File too large ({len(fileData)} bytes) or upload failed to all sites. Files larger than 4MB require resumable upload (not implemented).",
|
||||||
|
"uploadStatus": "failed"
|
||||||
|
})
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error uploading document {fileName}: {str(e)}")
|
||||||
|
uploadResults.append({
|
||||||
|
"fileName": fileName,
|
||||||
|
"fileId": fileId,
|
||||||
|
"error": str(e),
|
||||||
|
"uploadStatus": "failed"
|
||||||
|
})
|
||||||
|
|
||||||
|
# Update progress for each file
|
||||||
|
self.services.chat.progressLogUpdate(operationId, 0.5 + (i * 0.4 / len(filesToUpload)), f"Uploaded {i + 1}/{len(filesToUpload)} file(s)")
|
||||||
|
|
||||||
|
# Create result data
|
||||||
|
resultData = {
|
||||||
|
"connectionReference": connectionReference,
|
||||||
|
"uploadPath": uploadPath,
|
||||||
|
"documentList": documentList,
|
||||||
|
"fileNames": fileNames,
|
||||||
|
"sitesAvailable": len(sites),
|
||||||
|
"uploadResults": uploadResults,
|
||||||
|
"connection": {
|
||||||
|
"id": connection["id"],
|
||||||
|
"authority": "microsoft",
|
||||||
|
"reference": connectionReference
|
||||||
|
},
|
||||||
|
"timestamp": self.services.utils.timestampGetUtc()
|
||||||
|
}
|
||||||
|
|
||||||
|
# Use default JSON format for output
|
||||||
|
outputExtension = ".json" # Default
|
||||||
|
outputMimeType = "application/json" # Default
|
||||||
|
|
||||||
|
validationMetadata = {
|
||||||
|
"actionType": "sharepoint.uploadDocument",
|
||||||
|
"connectionReference": connectionReference,
|
||||||
|
"uploadPath": uploadPath,
|
||||||
|
"fileNames": fileNames,
|
||||||
|
"uploadCount": len(uploadResults),
|
||||||
|
"successfulUploads": len([r for r in uploadResults if r.get("uploadStatus") == "success"]),
|
||||||
|
"failedUploads": len([r for r in uploadResults if r.get("uploadStatus") == "failed"])
|
||||||
|
}
|
||||||
|
|
||||||
|
successfulUploads = len([r for r in uploadResults if r.get("uploadStatus") == "success"])
|
||||||
|
self.services.chat.progressLogUpdate(operationId, 0.9, f"Uploaded {successfulUploads}/{len(uploadResults)} file(s)")
|
||||||
|
self.services.chat.progressLogFinish(operationId, successfulUploads > 0)
|
||||||
|
|
||||||
|
return ActionResult(
|
||||||
|
success=True,
|
||||||
|
documents=[
|
||||||
|
ActionDocument(
|
||||||
|
documentName=self._generateMeaningfulFileName("sharepoint_upload", "json", None, "uploadDocument"),
|
||||||
|
documentData=json.dumps(resultData, indent=2),
|
||||||
|
mimeType=outputMimeType,
|
||||||
|
validationMetadata=validationMetadata
|
||||||
|
)
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error uploading to SharePoint: {str(e)}")
|
||||||
|
if operationId:
|
||||||
|
try:
|
||||||
|
self.services.chat.progressLogFinish(operationId, False)
|
||||||
|
except:
|
||||||
|
pass
|
||||||
|
return ActionResult(
|
||||||
|
success=False,
|
||||||
|
error=str(e)
|
||||||
|
)
|
||||||
|
|
||||||
145
modules/workflows/methods/methodSharepoint/actions/uploadFile.py
Normal file
145
modules/workflows/methods/methodSharepoint/actions/uploadFile.py
Normal file
|
|
@ -0,0 +1,145 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
"""
|
||||||
|
Upload File action for SharePoint operations.
|
||||||
|
Uploads raw file content (bytes) to SharePoint.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import json
|
||||||
|
import base64
|
||||||
|
from typing import Dict, Any
|
||||||
|
from modules.workflows.methods.methodBase import action
|
||||||
|
from modules.datamodels.datamodelChat import ActionResult, ActionDocument
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
@action
|
||||||
|
async def uploadFile(self, parameters: Dict[str, Any]) -> ActionResult:
|
||||||
|
"""
|
||||||
|
Upload raw file content (bytes) to SharePoint.
|
||||||
|
|
||||||
|
Parameters:
|
||||||
|
- connectionReference (str, required): Microsoft connection label.
|
||||||
|
- siteId (str, required): SharePoint site ID (from findSiteByUrl result) or document reference containing site info
|
||||||
|
- folderPath (str, required): Folder path relative to site root
|
||||||
|
- fileName (str, required): File name
|
||||||
|
- content (str, required): Document reference containing file content as base64-encoded bytes
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
- ActionResult with ActionDocument containing upload result
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
connectionReference = parameters.get("connectionReference")
|
||||||
|
if not connectionReference:
|
||||||
|
return ActionResult.isFailure(error="connectionReference parameter is required")
|
||||||
|
|
||||||
|
siteIdParam = parameters.get("siteId")
|
||||||
|
if not siteIdParam:
|
||||||
|
return ActionResult.isFailure(error="siteId parameter is required")
|
||||||
|
|
||||||
|
folderPath = parameters.get("folderPath")
|
||||||
|
if not folderPath:
|
||||||
|
return ActionResult.isFailure(error="folderPath parameter is required")
|
||||||
|
|
||||||
|
fileName = parameters.get("fileName")
|
||||||
|
if not fileName:
|
||||||
|
return ActionResult.isFailure(error="fileName parameter is required")
|
||||||
|
|
||||||
|
contentParam = parameters.get("content")
|
||||||
|
if not contentParam:
|
||||||
|
return ActionResult.isFailure(error="content parameter is required")
|
||||||
|
|
||||||
|
# Extract siteId from document if it's a reference
|
||||||
|
siteId = None
|
||||||
|
if isinstance(siteIdParam, str):
|
||||||
|
from modules.datamodels.datamodelDocref import DocumentReferenceList
|
||||||
|
try:
|
||||||
|
docList = DocumentReferenceList.from_string_list([siteIdParam])
|
||||||
|
chatDocuments = self.services.chat.getChatDocumentsFromDocumentList(docList)
|
||||||
|
if chatDocuments and len(chatDocuments) > 0:
|
||||||
|
siteInfoJson = json.loads(chatDocuments[0].documentData)
|
||||||
|
siteId = siteInfoJson.get("id")
|
||||||
|
except:
|
||||||
|
pass
|
||||||
|
|
||||||
|
if not siteId:
|
||||||
|
siteId = siteIdParam
|
||||||
|
else:
|
||||||
|
siteId = siteIdParam
|
||||||
|
|
||||||
|
if not siteId:
|
||||||
|
return ActionResult.isFailure(error="Could not extract siteId from parameter")
|
||||||
|
|
||||||
|
# Get file content from document
|
||||||
|
from modules.datamodels.datamodelDocref import DocumentReferenceList
|
||||||
|
docList = DocumentReferenceList.from_string_list([contentParam] if isinstance(contentParam, str) else contentParam)
|
||||||
|
chatDocuments = self.services.chat.getChatDocumentsFromDocumentList(docList)
|
||||||
|
if not chatDocuments or len(chatDocuments) == 0:
|
||||||
|
return ActionResult.isFailure(error="Could not get file content from document reference")
|
||||||
|
|
||||||
|
fileContentBase64 = chatDocuments[0].documentData
|
||||||
|
|
||||||
|
# Decode base64
|
||||||
|
try:
|
||||||
|
fileContent = base64.b64decode(fileContentBase64)
|
||||||
|
except Exception as e:
|
||||||
|
return ActionResult.isFailure(error=f"Could not decode base64 file content: {str(e)}")
|
||||||
|
|
||||||
|
# Get Microsoft connection
|
||||||
|
connection = self.connection.getMicrosoftConnection(connectionReference)
|
||||||
|
if not connection:
|
||||||
|
return ActionResult.isFailure(error="No valid Microsoft connection found for the provided connection reference")
|
||||||
|
|
||||||
|
# Upload file
|
||||||
|
uploadResult = await self.services.sharepoint.uploadFile(
|
||||||
|
siteId=siteId,
|
||||||
|
folderPath=folderPath,
|
||||||
|
fileName=fileName,
|
||||||
|
content=fileContent
|
||||||
|
)
|
||||||
|
|
||||||
|
if "error" in uploadResult:
|
||||||
|
return ActionResult.isFailure(error=f"Upload failed: {uploadResult['error']}")
|
||||||
|
|
||||||
|
logger.info(f"Uploaded file to SharePoint: {folderPath}/{fileName} ({len(fileContent)} bytes)")
|
||||||
|
|
||||||
|
# Generate filename
|
||||||
|
workflowContext = self.services.chat.getWorkflowContext() if hasattr(self.services, 'chat') else None
|
||||||
|
filename = self._generateMeaningfulFileName(
|
||||||
|
"file_upload_result",
|
||||||
|
"json",
|
||||||
|
workflowContext,
|
||||||
|
"uploadFile"
|
||||||
|
)
|
||||||
|
|
||||||
|
result = {
|
||||||
|
"success": True,
|
||||||
|
"siteId": siteId,
|
||||||
|
"filePath": f"{folderPath}/{fileName}",
|
||||||
|
"fileSize": len(fileContent),
|
||||||
|
"uploadResult": uploadResult
|
||||||
|
}
|
||||||
|
|
||||||
|
validationMetadata = self._createValidationMetadata(
|
||||||
|
"uploadFile",
|
||||||
|
siteId=siteId,
|
||||||
|
filePath=f"{folderPath}/{fileName}",
|
||||||
|
fileSize=len(fileContent)
|
||||||
|
)
|
||||||
|
|
||||||
|
document = ActionDocument(
|
||||||
|
documentName=filename,
|
||||||
|
documentData=json.dumps(result, indent=2),
|
||||||
|
mimeType="application/json",
|
||||||
|
validationMetadata=validationMetadata
|
||||||
|
)
|
||||||
|
|
||||||
|
return ActionResult.isSuccess(documents=[document])
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
errorMsg = f"Error uploading file to SharePoint: {str(e)}"
|
||||||
|
logger.error(errorMsg)
|
||||||
|
return ActionResult.isFailure(error=errorMsg)
|
||||||
|
|
||||||
|
|
@ -0,0 +1,5 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
"""Helper modules for SharePoint method operations."""
|
||||||
|
|
||||||
102
modules/workflows/methods/methodSharepoint/helpers/apiClient.py
Normal file
102
modules/workflows/methods/methodSharepoint/helpers/apiClient.py
Normal file
|
|
@ -0,0 +1,102 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
"""
|
||||||
|
API Client helper for SharePoint operations.
|
||||||
|
Handles Microsoft Graph API calls with timeout and error handling.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import aiohttp
|
||||||
|
import asyncio
|
||||||
|
from typing import Dict, Any
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
class ApiClientHelper:
|
||||||
|
"""Helper for Microsoft Graph API calls"""
|
||||||
|
|
||||||
|
def __init__(self, methodInstance):
|
||||||
|
"""
|
||||||
|
Initialize API client helper.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
methodInstance: Instance of MethodSharepoint (for access to services)
|
||||||
|
"""
|
||||||
|
self.method = methodInstance
|
||||||
|
self.services = methodInstance.services
|
||||||
|
|
||||||
|
async def makeGraphApiCall(self, endpoint: str, method: str = "GET", data: bytes = None) -> Dict[str, Any]:
|
||||||
|
"""
|
||||||
|
Make a Microsoft Graph API call with timeout and detailed logging.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
endpoint: API endpoint (without base URL)
|
||||||
|
method: HTTP method (GET, POST, PUT)
|
||||||
|
data: Optional request body data (bytes)
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Dict with API response or error information
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
if not hasattr(self.services, 'sharepoint') or not self.services.sharepoint._target.accessToken:
|
||||||
|
return {"error": "SharePoint service not configured with access token"}
|
||||||
|
|
||||||
|
headers = {
|
||||||
|
"Authorization": f"Bearer {self.services.sharepoint._target.accessToken}",
|
||||||
|
"Content-Type": "application/json" if data and method != "PUT" else "application/octet-stream" if data else "application/json"
|
||||||
|
}
|
||||||
|
|
||||||
|
url = f"https://graph.microsoft.com/v1.0/{endpoint}"
|
||||||
|
logger.info(f"Making Graph API call: {method} {url}")
|
||||||
|
|
||||||
|
# Set timeout to 30 seconds
|
||||||
|
timeout = aiohttp.ClientTimeout(total=30)
|
||||||
|
|
||||||
|
async with aiohttp.ClientSession(timeout=timeout) as session:
|
||||||
|
if method == "GET":
|
||||||
|
logger.debug(f"Starting GET request to {url}")
|
||||||
|
async with session.get(url, headers=headers) as response:
|
||||||
|
logger.info(f"Graph API response: {response.status}")
|
||||||
|
if response.status == 200:
|
||||||
|
result = await response.json()
|
||||||
|
logger.debug(f"Graph API success: {len(str(result))} characters response")
|
||||||
|
return result
|
||||||
|
else:
|
||||||
|
errorText = await response.text()
|
||||||
|
logger.error(f"Graph API call failed: {response.status} - {errorText}")
|
||||||
|
return {"error": f"API call failed: {response.status} - {errorText}"}
|
||||||
|
|
||||||
|
elif method == "PUT":
|
||||||
|
logger.debug(f"Starting PUT request to {url}")
|
||||||
|
async with session.put(url, headers=headers, data=data) as response:
|
||||||
|
logger.info(f"Graph API response: {response.status}")
|
||||||
|
if response.status in [200, 201]:
|
||||||
|
result = await response.json()
|
||||||
|
logger.debug(f"Graph API success: {len(str(result))} characters response")
|
||||||
|
return result
|
||||||
|
else:
|
||||||
|
errorText = await response.text()
|
||||||
|
logger.error(f"Graph API call failed: {response.status} - {errorText}")
|
||||||
|
return {"error": f"API call failed: {response.status} - {errorText}"}
|
||||||
|
|
||||||
|
elif method == "POST":
|
||||||
|
logger.debug(f"Starting POST request to {url}")
|
||||||
|
async with session.post(url, headers=headers, data=data) as response:
|
||||||
|
logger.info(f"Graph API response: {response.status}")
|
||||||
|
if response.status in [200, 201]:
|
||||||
|
result = await response.json()
|
||||||
|
logger.debug(f"Graph API success: {len(str(result))} characters response")
|
||||||
|
return result
|
||||||
|
else:
|
||||||
|
errorText = await response.text()
|
||||||
|
logger.error(f"Graph API call failed: {response.status} - {errorText}")
|
||||||
|
return {"error": f"API call failed: {response.status} - {errorText}"}
|
||||||
|
|
||||||
|
except asyncio.TimeoutError:
|
||||||
|
logger.error(f"Graph API call timed out after 30 seconds: {endpoint}")
|
||||||
|
return {"error": f"API call timed out after 30 seconds: {endpoint}"}
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error making Graph API call: {str(e)}")
|
||||||
|
return {"error": f"Error making Graph API call: {str(e)}"}
|
||||||
|
|
||||||
|
|
@ -0,0 +1,67 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
"""
|
||||||
|
Connection helper for SharePoint operations.
|
||||||
|
Handles Microsoft connection management and SharePoint service configuration.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
from typing import Dict, Any, Optional
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
class ConnectionHelper:
|
||||||
|
"""Helper for Microsoft connection management in SharePoint operations"""
|
||||||
|
|
||||||
|
def __init__(self, methodInstance):
|
||||||
|
"""
|
||||||
|
Initialize connection helper.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
methodInstance: Instance of MethodSharepoint (for access to services)
|
||||||
|
"""
|
||||||
|
self.method = methodInstance
|
||||||
|
self.services = methodInstance.services
|
||||||
|
|
||||||
|
def getMicrosoftConnection(self, connectionReference: str) -> Optional[Dict[str, Any]]:
|
||||||
|
"""
|
||||||
|
Get Microsoft connection from connection reference and configure SharePoint service.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
connectionReference: Connection reference string
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Dict with connection info or None if failed
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
userConnection = self.services.chat.getUserConnectionFromConnectionReference(connectionReference)
|
||||||
|
if not userConnection:
|
||||||
|
logger.warning(f"No user connection found for reference: {connectionReference}")
|
||||||
|
return None
|
||||||
|
|
||||||
|
if userConnection.authority.value != "msft":
|
||||||
|
logger.warning(f"Connection {userConnection.id} is not Microsoft (authority: {userConnection.authority.value})")
|
||||||
|
return None
|
||||||
|
|
||||||
|
# Check if connection is active or pending (pending means OAuth in progress)
|
||||||
|
if userConnection.status.value not in ["active", "pending"]:
|
||||||
|
logger.warning(f"Connection {userConnection.id} status is not active/pending: {userConnection.status.value}")
|
||||||
|
return None
|
||||||
|
|
||||||
|
# Configure SharePoint service with the UserConnection
|
||||||
|
if not self.services.sharepoint.setAccessTokenFromConnection(userConnection):
|
||||||
|
logger.warning(f"Failed to configure SharePoint service with connection {userConnection.id}")
|
||||||
|
return None
|
||||||
|
|
||||||
|
logger.info(f"Successfully configured SharePoint service with Microsoft connection: {userConnection.id}, status: {userConnection.status.value}, externalId: {userConnection.externalId}")
|
||||||
|
|
||||||
|
return {
|
||||||
|
"id": userConnection.id,
|
||||||
|
"userConnection": userConnection,
|
||||||
|
"scopes": ["Sites.ReadWrite.All", "Files.ReadWrite.All", "User.Read"] # SharePoint scopes
|
||||||
|
}
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error getting Microsoft connection: {str(e)}")
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
@ -0,0 +1,252 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
"""
|
||||||
|
Document Parsing helper for SharePoint operations.
|
||||||
|
Handles parsing of document lists and extracting found documents and site information.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import json
|
||||||
|
from typing import Dict, Any, List, Optional
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
class DocumentParsingHelper:
|
||||||
|
"""Helper for parsing document lists and extracting document information"""
|
||||||
|
|
||||||
|
def __init__(self, methodInstance):
|
||||||
|
"""
|
||||||
|
Initialize document parsing helper.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
methodInstance: Instance of MethodSharepoint (for access to services)
|
||||||
|
"""
|
||||||
|
self.method = methodInstance
|
||||||
|
self.services = methodInstance.services
|
||||||
|
|
||||||
|
async def parseDocumentListForFoundDocuments(self, documentList: Any) -> tuple[Optional[List[Dict[str, Any]]], Optional[List[Dict[str, Any]]], Optional[str]]:
|
||||||
|
"""
|
||||||
|
Parse documentList to extract foundDocuments and site information.
|
||||||
|
|
||||||
|
Parameters:
|
||||||
|
documentList: Document list (can be list, DocumentReferenceList, or string)
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
tuple: (foundDocuments, sites, errorMessage)
|
||||||
|
- foundDocuments: List of found documents from findDocumentPath result
|
||||||
|
- sites: List of site dictionaries with id, displayName, webUrl
|
||||||
|
- errorMessage: Error message if parsing failed, None otherwise
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
if isinstance(documentList, str):
|
||||||
|
documentList = [documentList]
|
||||||
|
|
||||||
|
# Resolve documentList to get actual documents
|
||||||
|
from modules.datamodels.datamodelDocref import DocumentReferenceList
|
||||||
|
if isinstance(documentList, DocumentReferenceList):
|
||||||
|
docRefList = documentList
|
||||||
|
elif isinstance(documentList, list):
|
||||||
|
docRefList = DocumentReferenceList.from_string_list(documentList)
|
||||||
|
else:
|
||||||
|
docRefList = DocumentReferenceList(references=[])
|
||||||
|
|
||||||
|
chatDocuments = self.services.chat.getChatDocumentsFromDocumentList(docRefList)
|
||||||
|
if not chatDocuments:
|
||||||
|
return None, None, "No documents found for the provided document list"
|
||||||
|
|
||||||
|
firstDocument = chatDocuments[0]
|
||||||
|
fileData = self.services.chat.getFileData(firstDocument.fileId)
|
||||||
|
if not fileData:
|
||||||
|
return None, None, None # No fileData, but not an error (might be regular file)
|
||||||
|
|
||||||
|
try:
|
||||||
|
resultData = json.loads(fileData)
|
||||||
|
foundDocuments = resultData.get("foundDocuments", [])
|
||||||
|
|
||||||
|
# If no foundDocuments, check if it's a listDocuments result (has listResults)
|
||||||
|
if not foundDocuments and "listResults" in resultData:
|
||||||
|
logger.info(f"documentList contains listResults from listDocuments, converting to foundDocuments format")
|
||||||
|
listResults = resultData.get("listResults", [])
|
||||||
|
foundDocuments = []
|
||||||
|
siteIdFromList = None
|
||||||
|
siteNameFromList = None
|
||||||
|
|
||||||
|
for listResult in listResults:
|
||||||
|
siteResults = listResult.get("siteResults", [])
|
||||||
|
for siteResult in siteResults:
|
||||||
|
items = siteResult.get("items", [])
|
||||||
|
# Extract site info from first item if available
|
||||||
|
if items and not siteIdFromList:
|
||||||
|
siteNameFromList = items[0].get("siteName")
|
||||||
|
|
||||||
|
for item in items:
|
||||||
|
# Convert listDocuments item format to foundDocuments format
|
||||||
|
if item.get("type") == "file":
|
||||||
|
foundDoc = {
|
||||||
|
"id": item.get("id"),
|
||||||
|
"name": item.get("name"),
|
||||||
|
"type": "file",
|
||||||
|
"siteName": item.get("siteName"),
|
||||||
|
"siteId": None, # Will be determined from site discovery
|
||||||
|
"webUrl": item.get("webUrl"),
|
||||||
|
"fullPath": item.get("webUrl", ""),
|
||||||
|
"parentPath": item.get("parentPath", "")
|
||||||
|
}
|
||||||
|
foundDocuments.append(foundDoc)
|
||||||
|
|
||||||
|
# Discover sites to get siteId if we have siteName
|
||||||
|
if foundDocuments and siteNameFromList and not siteIdFromList:
|
||||||
|
logger.info(f"Discovering sites to find siteId for '{siteNameFromList}'")
|
||||||
|
allSites = await self.method.siteDiscovery.discoverSharePointSites()
|
||||||
|
matchingSites = self.method.siteDiscovery.filterSitesByHint(allSites, siteNameFromList)
|
||||||
|
if matchingSites:
|
||||||
|
siteIdFromList = matchingSites[0].get("id")
|
||||||
|
# Update all foundDocuments with siteId
|
||||||
|
for doc in foundDocuments:
|
||||||
|
doc["siteId"] = siteIdFromList
|
||||||
|
logger.info(f"Found siteId '{siteIdFromList}' for site '{siteNameFromList}'")
|
||||||
|
|
||||||
|
logger.info(f"Converted {len(foundDocuments)} files from listResults format")
|
||||||
|
|
||||||
|
if not foundDocuments:
|
||||||
|
return None, None, None # No foundDocuments, but not an error
|
||||||
|
|
||||||
|
# Extract site information from foundDocuments
|
||||||
|
firstDoc = foundDocuments[0]
|
||||||
|
siteName = firstDoc.get("siteName")
|
||||||
|
siteId = firstDoc.get("siteId")
|
||||||
|
|
||||||
|
# If siteId is missing (from listDocuments conversion), discover sites to find it
|
||||||
|
if siteName and not siteId:
|
||||||
|
logger.info(f"Site ID missing, discovering sites to find siteId for '{siteName}'")
|
||||||
|
allSites = await self.method.siteDiscovery.discoverSharePointSites()
|
||||||
|
matchingSites = self.method.siteDiscovery.filterSitesByHint(allSites, siteName)
|
||||||
|
if matchingSites:
|
||||||
|
siteId = matchingSites[0].get("id")
|
||||||
|
logger.info(f"Found siteId '{siteId}' for site '{siteName}'")
|
||||||
|
|
||||||
|
sites = None
|
||||||
|
if siteName and siteId:
|
||||||
|
sites = [{
|
||||||
|
"id": siteId,
|
||||||
|
"displayName": siteName,
|
||||||
|
"webUrl": firstDoc.get("webUrl", "")
|
||||||
|
}]
|
||||||
|
logger.info(f"Using specific site from documentList: {siteName} (ID: {siteId})")
|
||||||
|
elif siteName:
|
||||||
|
# Try to get site by name
|
||||||
|
allSites = await self.method.siteDiscovery.discoverSharePointSites()
|
||||||
|
matchingSites = self.method.siteDiscovery.filterSitesByHint(allSites, siteName)
|
||||||
|
if matchingSites:
|
||||||
|
sites = [{
|
||||||
|
"id": matchingSites[0].get("id"),
|
||||||
|
"displayName": siteName,
|
||||||
|
"webUrl": matchingSites[0].get("webUrl", "")
|
||||||
|
}]
|
||||||
|
logger.info(f"Found site by name: {siteName} (ID: {sites[0]['id']})")
|
||||||
|
else:
|
||||||
|
return None, None, f"Site '{siteName}' not found. Cannot determine target site."
|
||||||
|
else:
|
||||||
|
return None, None, "Site information missing from documentList. Cannot determine target site."
|
||||||
|
|
||||||
|
return foundDocuments, sites, None
|
||||||
|
|
||||||
|
except json.JSONDecodeError as e:
|
||||||
|
return None, None, f"Invalid JSON in documentList: {str(e)}"
|
||||||
|
except Exception as e:
|
||||||
|
return None, None, f"Error processing documentList: {str(e)}"
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error parsing documentList: {str(e)}")
|
||||||
|
return None, None, f"Error parsing documentList: {str(e)}"
|
||||||
|
|
||||||
|
async def parseDocumentListForFolder(self, documentList: Any) -> tuple[Optional[str], Optional[List[Dict[str, Any]]], Optional[List], Optional[str]]:
|
||||||
|
"""
|
||||||
|
Parse documentList to extract folder path, site information, and files to upload.
|
||||||
|
|
||||||
|
Parameters:
|
||||||
|
documentList: Document list (can be list, DocumentReferenceList, or string)
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
tuple: (folderPath, sites, filesToUpload, errorMessage)
|
||||||
|
- folderPath: Folder path from findDocumentPath result (or None)
|
||||||
|
- sites: List of site dictionaries with id, displayName, webUrl
|
||||||
|
- filesToUpload: List of ChatDocument objects to upload (or None)
|
||||||
|
- errorMessage: Error message if parsing failed, None otherwise
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
if isinstance(documentList, str):
|
||||||
|
documentList = [documentList]
|
||||||
|
|
||||||
|
# Resolve documentList to get actual documents
|
||||||
|
from modules.datamodels.datamodelDocref import DocumentReferenceList
|
||||||
|
if isinstance(documentList, DocumentReferenceList):
|
||||||
|
docRefList = documentList
|
||||||
|
elif isinstance(documentList, list):
|
||||||
|
docRefList = DocumentReferenceList.from_string_list(documentList)
|
||||||
|
else:
|
||||||
|
docRefList = DocumentReferenceList(references=[])
|
||||||
|
|
||||||
|
chatDocuments = self.services.chat.getChatDocumentsFromDocumentList(docRefList)
|
||||||
|
if not chatDocuments:
|
||||||
|
return None, None, None, "No documents found for the provided document list"
|
||||||
|
|
||||||
|
# Check if first document is a findDocumentPath result (has foundDocuments)
|
||||||
|
firstDocument = chatDocuments[0]
|
||||||
|
fileData = self.services.chat.getFileData(firstDocument.fileId)
|
||||||
|
|
||||||
|
folderPath = None
|
||||||
|
sites = None
|
||||||
|
filesToUpload = None
|
||||||
|
|
||||||
|
if fileData:
|
||||||
|
try:
|
||||||
|
resultData = json.loads(fileData)
|
||||||
|
foundDocuments = resultData.get("foundDocuments", [])
|
||||||
|
|
||||||
|
if foundDocuments:
|
||||||
|
# Extract folder path from first found document
|
||||||
|
firstDoc = foundDocuments[0]
|
||||||
|
parentPath = firstDoc.get("parentPath", "")
|
||||||
|
if parentPath:
|
||||||
|
folderPath = parentPath
|
||||||
|
|
||||||
|
# Extract site information
|
||||||
|
siteName = firstDoc.get("siteName")
|
||||||
|
siteId = firstDoc.get("siteId")
|
||||||
|
|
||||||
|
if siteName and siteId:
|
||||||
|
sites = [{
|
||||||
|
"id": siteId,
|
||||||
|
"displayName": siteName,
|
||||||
|
"webUrl": firstDoc.get("webUrl", "")
|
||||||
|
}]
|
||||||
|
elif siteName:
|
||||||
|
# Discover sites to find siteId
|
||||||
|
allSites = await self.method.siteDiscovery.discoverSharePointSites()
|
||||||
|
matchingSites = self.method.siteDiscovery.filterSitesByHint(allSites, siteName)
|
||||||
|
if matchingSites:
|
||||||
|
sites = [{
|
||||||
|
"id": matchingSites[0].get("id"),
|
||||||
|
"displayName": siteName,
|
||||||
|
"webUrl": matchingSites[0].get("webUrl", "")
|
||||||
|
}]
|
||||||
|
|
||||||
|
# For uploadDocument: filesToUpload are the chatDocuments themselves
|
||||||
|
# (they contain the files to upload)
|
||||||
|
filesToUpload = chatDocuments
|
||||||
|
|
||||||
|
except json.JSONDecodeError:
|
||||||
|
# Not a findDocumentPath result - treat as regular files to upload
|
||||||
|
filesToUpload = chatDocuments
|
||||||
|
else:
|
||||||
|
# No fileData - treat as regular files to upload
|
||||||
|
filesToUpload = chatDocuments
|
||||||
|
|
||||||
|
return folderPath, sites, filesToUpload, None
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error parsing documentList for folder: {str(e)}")
|
||||||
|
return None, None, None, f"Error parsing documentList for folder: {str(e)}"
|
||||||
|
|
||||||
|
|
@ -0,0 +1,338 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
"""
|
||||||
|
Path Processing helper for SharePoint operations.
|
||||||
|
Handles search query parsing, path resolution, and query cleaning.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import re
|
||||||
|
from typing import List, Optional, Dict, Any
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
class PathProcessingHelper:
|
||||||
|
"""Helper for path and query processing"""
|
||||||
|
|
||||||
|
def __init__(self, methodInstance):
|
||||||
|
"""
|
||||||
|
Initialize path processing helper.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
methodInstance: Instance of MethodSharepoint (for access to services)
|
||||||
|
"""
|
||||||
|
self.method = methodInstance
|
||||||
|
self.services = methodInstance.services
|
||||||
|
|
||||||
|
def parseSearchQuery(self, searchQuery: str) -> tuple[str, str, str, dict]:
|
||||||
|
"""
|
||||||
|
Parse searchQuery to extract path, search terms, search type, and search options.
|
||||||
|
|
||||||
|
CRITICAL: NEVER convert words to paths! Words stay as search terms.
|
||||||
|
- "root document lesson" → fileQuery="root document lesson" (NOT "/root/document/lesson")
|
||||||
|
- "root, gose" → fileQuery="root, gose" (NOT "/root/gose")
|
||||||
|
- "druckersteuerung eskalation logobject" → fileQuery="druckersteuerung eskalation logobject"
|
||||||
|
|
||||||
|
Parameters:
|
||||||
|
searchQuery (str): Enhanced search query with options:
|
||||||
|
- "budget" -> pathQuery="*", fileQuery="budget", searchType="all", options={}
|
||||||
|
- "root document lesson" -> pathQuery="*", fileQuery="root document lesson", searchType="all", options={}
|
||||||
|
- "root, gose" -> pathQuery="*", fileQuery="root, gose", searchType="all", options={}
|
||||||
|
- "/Documents:budget" -> pathQuery="/Documents", fileQuery="budget", searchType="all", options={}
|
||||||
|
- "files:budget" -> pathQuery="*", fileQuery="budget", searchType="files", options={}
|
||||||
|
- "folders:DELTA" -> pathQuery="*", fileQuery="DELTA", searchType="folders", options={}
|
||||||
|
- "exact:\"Operations 2025\"" -> exact phrase matching
|
||||||
|
- "regex:^Operations.*2025$" -> regex pattern matching
|
||||||
|
- "case:DELTA" -> case-sensitive search
|
||||||
|
- "and:DELTA AND 2025 Mars AND Group" -> all AND terms must be present
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
tuple[str, str, str, dict]: (pathQuery, fileQuery, searchType, searchOptions)
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
if not searchQuery or not searchQuery.strip() or searchQuery.strip() == "*":
|
||||||
|
return "*", "*", "all", {}
|
||||||
|
|
||||||
|
searchQuery = searchQuery.strip()
|
||||||
|
searchOptions = {}
|
||||||
|
|
||||||
|
# CRITICAL: Do NOT convert space-separated or comma-separated words to paths!
|
||||||
|
# "root document lesson" should stay as "root document lesson", NOT "/root/document/lesson"
|
||||||
|
# "root, gose" should stay as "root, gose", NOT "/root/gose"
|
||||||
|
|
||||||
|
# Check for search type specification (files:, folders:, all:) FIRST
|
||||||
|
searchType = "all" # Default
|
||||||
|
if searchQuery.startswith(("files:", "folders:", "all:")):
|
||||||
|
typeParts = searchQuery.split(':', 1)
|
||||||
|
searchType = typeParts[0].strip()
|
||||||
|
searchQuery = typeParts[1].strip()
|
||||||
|
|
||||||
|
# Extract optional site hint tokens: support "site=Name" or leading "site:Name"
|
||||||
|
def _extractSiteHint(q: str) -> tuple[str, Optional[str]]:
|
||||||
|
try:
|
||||||
|
qStrip = q.strip()
|
||||||
|
# Leading form: site:KM LayerFinance ...
|
||||||
|
if qStrip.lower().startswith("site:"):
|
||||||
|
after = qStrip[5:].lstrip()
|
||||||
|
# site name until next space or end
|
||||||
|
if ' ' in after:
|
||||||
|
siteName, rest = after.split(' ', 1)
|
||||||
|
else:
|
||||||
|
siteName, rest = after, ''
|
||||||
|
return rest.strip(), siteName.strip()
|
||||||
|
# Inline key=value form anywhere
|
||||||
|
m = re.search(r"\bsite=([^;\s]+)", qStrip, flags=re.IGNORECASE)
|
||||||
|
if m:
|
||||||
|
siteName = m.group(1).strip()
|
||||||
|
# remove the token from query
|
||||||
|
qNew = re.sub(r"\bsite=[^;\s]+;?", "", qStrip, flags=re.IGNORECASE).strip()
|
||||||
|
return qNew, siteName
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
return q, None
|
||||||
|
|
||||||
|
searchQuery, extractedSite = _extractSiteHint(searchQuery)
|
||||||
|
if extractedSite:
|
||||||
|
searchOptions["site_hint"] = extractedSite
|
||||||
|
logger.info(f"Extracted site hint: '{extractedSite}'")
|
||||||
|
|
||||||
|
# Extract name="..." if present (for quoted multi-word names)
|
||||||
|
nameMatch = re.search(r"name=\"([^\"]+)\"", searchQuery)
|
||||||
|
if nameMatch:
|
||||||
|
searchQuery = nameMatch.group(1)
|
||||||
|
logger.info(f"Extracted name from quotes: '{searchQuery}'")
|
||||||
|
|
||||||
|
# Check for search mode specification (exact:, regex:, case:, and:)
|
||||||
|
if searchQuery.startswith(("exact:", "regex:", "case:", "and:")):
|
||||||
|
modeParts = searchQuery.split(':', 1)
|
||||||
|
mode = modeParts[0].strip()
|
||||||
|
searchQuery = modeParts[1].strip()
|
||||||
|
|
||||||
|
if mode == "exact":
|
||||||
|
searchOptions["exact_match"] = True
|
||||||
|
# Remove quotes if present
|
||||||
|
if searchQuery.startswith('"') and searchQuery.endswith('"'):
|
||||||
|
searchQuery = searchQuery[1:-1]
|
||||||
|
elif mode == "regex":
|
||||||
|
searchOptions["regex_match"] = True
|
||||||
|
elif mode == "case":
|
||||||
|
searchOptions["case_sensitive"] = True
|
||||||
|
elif mode == "and":
|
||||||
|
searchOptions["and_terms"] = True
|
||||||
|
|
||||||
|
# Check if it contains path:search format
|
||||||
|
# Microsoft-standard paths: /sites/SiteName/Path:files:.pdf
|
||||||
|
if ':' in searchQuery:
|
||||||
|
# For Microsoft-standard paths (/sites/...), find the colon that separates path from search
|
||||||
|
if searchQuery.startswith('/sites/'):
|
||||||
|
# Find the colon that separates path from search (after the full path)
|
||||||
|
# Look for pattern: /sites/SiteName/Path/...:files:.pdf
|
||||||
|
# We need to find the colon that's followed by search type or file extension
|
||||||
|
colonPositions = []
|
||||||
|
for i, char in enumerate(searchQuery):
|
||||||
|
if char == ':':
|
||||||
|
colonPositions.append(i)
|
||||||
|
|
||||||
|
# If we have colons, find the one that's followed by search type or file extension
|
||||||
|
splitPos = None
|
||||||
|
if colonPositions:
|
||||||
|
for pos in colonPositions:
|
||||||
|
afterColon = searchQuery[pos+1:pos+10].strip().lower()
|
||||||
|
# Check if this colon is followed by search type or looks like a file extension
|
||||||
|
if afterColon.startswith(('files:', 'folders:', 'all:', '.')) or afterColon == '':
|
||||||
|
splitPos = pos
|
||||||
|
break
|
||||||
|
|
||||||
|
# If no clear split found, use the last colon
|
||||||
|
if splitPos is None and colonPositions:
|
||||||
|
splitPos = colonPositions[-1]
|
||||||
|
|
||||||
|
if splitPos:
|
||||||
|
pathPart = searchQuery[:splitPos].strip()
|
||||||
|
searchPart = searchQuery[splitPos+1:].strip()
|
||||||
|
else:
|
||||||
|
# Fallback: split on first colon
|
||||||
|
parts = searchQuery.split(':', 1)
|
||||||
|
pathPart = parts[0].strip()
|
||||||
|
searchPart = parts[1].strip()
|
||||||
|
else:
|
||||||
|
# Regular path:search format - split on first colon
|
||||||
|
parts = searchQuery.split(':', 1)
|
||||||
|
pathPart = parts[0].strip()
|
||||||
|
searchPart = parts[1].strip()
|
||||||
|
|
||||||
|
# Check if searchPart starts with search type (files:, folders:, all:)
|
||||||
|
if searchPart.startswith(("files:", "folders:", "all:")):
|
||||||
|
typeParts = searchPart.split(':', 1)
|
||||||
|
searchType = typeParts[0].strip() # Update searchType
|
||||||
|
searchPart = typeParts[1].strip() if len(typeParts) > 1 else ""
|
||||||
|
|
||||||
|
# Handle path part
|
||||||
|
if not pathPart or pathPart == "*":
|
||||||
|
pathQuery = "*"
|
||||||
|
elif pathPart.startswith('/'):
|
||||||
|
pathQuery = pathPart
|
||||||
|
else:
|
||||||
|
pathQuery = f"/Documents/{pathPart}"
|
||||||
|
|
||||||
|
# Handle search part
|
||||||
|
if not searchPart or searchPart == "*":
|
||||||
|
fileQuery = "*"
|
||||||
|
else:
|
||||||
|
fileQuery = searchPart
|
||||||
|
|
||||||
|
return pathQuery, fileQuery, searchType, searchOptions
|
||||||
|
|
||||||
|
# No colon - check if it looks like a path
|
||||||
|
elif searchQuery.startswith('/'):
|
||||||
|
# It's a path only
|
||||||
|
return searchQuery, "*", searchType, searchOptions
|
||||||
|
|
||||||
|
else:
|
||||||
|
# It's a search term only - keep words as-is, do NOT convert to paths
|
||||||
|
# "root document lesson" stays as "root document lesson"
|
||||||
|
# "root, gose" stays as "root, gose"
|
||||||
|
return "*", searchQuery, searchType, searchOptions
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error parsing searchQuery '{searchQuery}': {str(e)}")
|
||||||
|
raise ValueError(f"Failed to parse searchQuery '{searchQuery}': {str(e)}")
|
||||||
|
|
||||||
|
def resolvePathQuery(self, pathQuery: str) -> List[str]:
|
||||||
|
"""
|
||||||
|
Resolve pathQuery into a list of search paths for SharePoint operations.
|
||||||
|
|
||||||
|
Parameters:
|
||||||
|
pathQuery (str): Query string that can contain:
|
||||||
|
- Direct paths (e.g., "/Documents/Project1")
|
||||||
|
- Wildcards (e.g., "/Documents/*")
|
||||||
|
- Multiple paths separated by semicolons (e.g., "/Docs; /Files")
|
||||||
|
- Single word relative paths (e.g., "Project1" -> resolved to default folder)
|
||||||
|
- Empty string or "*" for global search
|
||||||
|
- Space-separated words are treated as search terms, NOT folder paths
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
List[str]: List of resolved paths
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
if not pathQuery or not pathQuery.strip() or pathQuery.strip() == "*":
|
||||||
|
return ["*"] # Global search across all sites
|
||||||
|
|
||||||
|
# Split by semicolon to handle multiple paths
|
||||||
|
rawPaths = [path.strip() for path in pathQuery.split(';') if path.strip()]
|
||||||
|
resolvedPaths = []
|
||||||
|
|
||||||
|
for rawPath in rawPaths:
|
||||||
|
# Handle wildcards - return as-is
|
||||||
|
if '*' in rawPath:
|
||||||
|
resolvedPaths.append(rawPath)
|
||||||
|
# Handle absolute paths
|
||||||
|
elif rawPath.startswith('/'):
|
||||||
|
resolvedPaths.append(rawPath)
|
||||||
|
# Handle single word relative paths - prepend default folder
|
||||||
|
# BUT NOT space-separated words (those are search terms, not paths)
|
||||||
|
elif ' ' not in rawPath:
|
||||||
|
resolvedPaths.append(f"/Documents/{rawPath}")
|
||||||
|
else:
|
||||||
|
# Check if this looks like a path (has path separators) or search terms
|
||||||
|
if '\\' in rawPath or '/' in rawPath:
|
||||||
|
# This looks like a path with spaces in folder names - treat as valid path
|
||||||
|
resolvedPaths.append(rawPath)
|
||||||
|
logger.info(f"Path with spaces '{rawPath}' treated as valid folder path")
|
||||||
|
else:
|
||||||
|
# Space-separated words without path separators are search terms
|
||||||
|
# Return as "*" to search globally
|
||||||
|
logger.info(f"Space-separated words '{rawPath}' treated as search terms, not folder path")
|
||||||
|
resolvedPaths.append("*")
|
||||||
|
|
||||||
|
# Remove duplicates while preserving order
|
||||||
|
seen = set()
|
||||||
|
uniquePaths = []
|
||||||
|
for path in resolvedPaths:
|
||||||
|
if path not in seen:
|
||||||
|
seen.add(path)
|
||||||
|
uniquePaths.append(path)
|
||||||
|
|
||||||
|
logger.info(f"Resolved pathQuery '{pathQuery}' to {len(uniquePaths)} paths: {uniquePaths}")
|
||||||
|
return uniquePaths
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error resolving pathQuery '{pathQuery}': {str(e)}")
|
||||||
|
raise ValueError(f"Failed to resolve pathQuery '{pathQuery}': {str(e)}")
|
||||||
|
|
||||||
|
def cleanSearchQuery(self, query: str) -> str:
|
||||||
|
"""
|
||||||
|
Clean search query to make it compatible with Graph API KQL syntax.
|
||||||
|
Removes path-like syntax and invalid KQL constructs.
|
||||||
|
|
||||||
|
Parameters:
|
||||||
|
query (str): Raw search query that may contain paths and invalid syntax
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
str: Cleaned query suitable for Graph API search endpoint
|
||||||
|
"""
|
||||||
|
if not query or not query.strip():
|
||||||
|
return ""
|
||||||
|
|
||||||
|
query = query.strip()
|
||||||
|
|
||||||
|
# Handle patterns like: "Company Share/Freigegebene Dokumente/.../expenses:files:.pdf"
|
||||||
|
# Extract the search term and file extension
|
||||||
|
|
||||||
|
# First, extract file extension if present (format: :files:.pdf or just .pdf at the end)
|
||||||
|
fileExtension = ""
|
||||||
|
if ':files:' in query.lower() or ':folders:' in query.lower():
|
||||||
|
# Extract extension after the type filter
|
||||||
|
extMatch = re.search(r':(?:files|folders):(\.\w+)', query, re.IGNORECASE)
|
||||||
|
if extMatch:
|
||||||
|
fileExtension = extMatch.group(1)
|
||||||
|
# Remove the type filter part
|
||||||
|
query = re.sub(r':(?:files|folders):\.?\w*', '', query, flags=re.IGNORECASE)
|
||||||
|
elif query.endswith(('.pdf', '.doc', '.docx', '.xls', '.xlsx', '.txt', '.csv', '.ppt', '.pptx')):
|
||||||
|
# Extract extension from end
|
||||||
|
extMatch = re.search(r'(\.\w+)$', query)
|
||||||
|
if extMatch:
|
||||||
|
fileExtension = extMatch.group(1)
|
||||||
|
query = query[:-len(fileExtension)]
|
||||||
|
|
||||||
|
# Extract search term: get the last segment after the last slash (filename part)
|
||||||
|
queryNormalized = query.replace('\\', '/')
|
||||||
|
if '/' in queryNormalized:
|
||||||
|
# Extract the last segment (usually the filename/search term)
|
||||||
|
lastSegment = queryNormalized.split('/')[-1]
|
||||||
|
# Remove any remaining colons or type filters
|
||||||
|
if ':' in lastSegment:
|
||||||
|
lastSegment = lastSegment.split(':')[0]
|
||||||
|
searchTerm = lastSegment.strip()
|
||||||
|
else:
|
||||||
|
# No path separators, use the query as-is but remove type filters
|
||||||
|
if ':' in query:
|
||||||
|
searchTerm = query.split(':')[0].strip()
|
||||||
|
else:
|
||||||
|
searchTerm = query.strip()
|
||||||
|
|
||||||
|
# Remove any remaining type filters or invalid syntax
|
||||||
|
searchTerm = re.sub(r':(?:files|folders|all):?', '', searchTerm, flags=re.IGNORECASE)
|
||||||
|
searchTerm = searchTerm.strip()
|
||||||
|
|
||||||
|
# If we have a file extension, include it in the search term
|
||||||
|
# Note: Graph API search endpoint may not support filetype: syntax
|
||||||
|
# So we include the extension as part of the search term or filter results after
|
||||||
|
if fileExtension:
|
||||||
|
extWithoutDot = fileExtension.lstrip('.')
|
||||||
|
# Try simple approach: add extension as search term
|
||||||
|
# If this doesn't work, we'll filter results after search
|
||||||
|
if searchTerm:
|
||||||
|
# Include extension in search - Graph API will search in filename
|
||||||
|
searchTerm = f"{searchTerm} {extWithoutDot}"
|
||||||
|
else:
|
||||||
|
searchTerm = extWithoutDot
|
||||||
|
|
||||||
|
# Final cleanup: remove any remaining invalid characters for KQL
|
||||||
|
# Keep alphanumeric, spaces, hyphens, underscores, dots, and common search operators
|
||||||
|
searchTerm = re.sub(r'[^\w\s\-\.\*]', ' ', searchTerm)
|
||||||
|
searchTerm = ' '.join(searchTerm.split()) # Normalize whitespace
|
||||||
|
|
||||||
|
return searchTerm if searchTerm else "*"
|
||||||
|
|
||||||
|
|
@ -0,0 +1,173 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
"""
|
||||||
|
Site Discovery helper for SharePoint operations.
|
||||||
|
Handles SharePoint site discovery, filtering, and resolution.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import urllib.parse
|
||||||
|
from typing import Dict, Any, List, Optional
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
class SiteDiscoveryHelper:
|
||||||
|
"""Helper for SharePoint site discovery and resolution"""
|
||||||
|
|
||||||
|
def __init__(self, methodInstance):
|
||||||
|
"""
|
||||||
|
Initialize site discovery helper.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
methodInstance: Instance of MethodSharepoint (for access to services)
|
||||||
|
"""
|
||||||
|
self.method = methodInstance
|
||||||
|
self.services = methodInstance.services
|
||||||
|
|
||||||
|
async def discoverSharePointSites(self, limit: Optional[int] = None) -> List[Dict[str, Any]]:
|
||||||
|
"""
|
||||||
|
Discover SharePoint sites accessible to the user via Microsoft Graph API.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
limit: Optional limit on number of sites to return
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
List of site information dictionaries
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
# Query Microsoft Graph to get sites the user has access to
|
||||||
|
endpoint = "sites?search=*"
|
||||||
|
if limit:
|
||||||
|
endpoint += f"&$top={limit}"
|
||||||
|
|
||||||
|
result = await self.method.apiClient.makeGraphApiCall(endpoint)
|
||||||
|
|
||||||
|
if "error" in result:
|
||||||
|
logger.error(f"Error discovering SharePoint sites: {result['error']}")
|
||||||
|
return []
|
||||||
|
|
||||||
|
sites = result.get("value", [])
|
||||||
|
if limit:
|
||||||
|
sites = sites[:limit]
|
||||||
|
|
||||||
|
logger.info(f"Discovered {len(sites)} SharePoint sites" + (f" (limited to {limit})" if limit else ""))
|
||||||
|
|
||||||
|
# Process and return site information
|
||||||
|
processedSites = []
|
||||||
|
for site in sites:
|
||||||
|
siteInfo = {
|
||||||
|
"id": site.get("id"),
|
||||||
|
"displayName": site.get("displayName"),
|
||||||
|
"name": site.get("name"),
|
||||||
|
"webUrl": site.get("webUrl"),
|
||||||
|
"description": site.get("description"),
|
||||||
|
"createdDateTime": site.get("createdDateTime"),
|
||||||
|
"lastModifiedDateTime": site.get("lastModifiedDateTime")
|
||||||
|
}
|
||||||
|
processedSites.append(siteInfo)
|
||||||
|
logger.debug(f"Site: {siteInfo['displayName']} - {siteInfo['webUrl']}")
|
||||||
|
|
||||||
|
return processedSites
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error discovering SharePoint sites: {str(e)}")
|
||||||
|
return []
|
||||||
|
|
||||||
|
def extractHostnameFromWebUrl(self, webUrl: str) -> Optional[str]:
|
||||||
|
"""Extract hostname from SharePoint webUrl (e.g., https://pcuster.sharepoint.com)"""
|
||||||
|
try:
|
||||||
|
if not webUrl:
|
||||||
|
return None
|
||||||
|
parsed = urllib.parse.urlparse(webUrl)
|
||||||
|
return parsed.hostname
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error extracting hostname from webUrl '{webUrl}': {str(e)}")
|
||||||
|
return None
|
||||||
|
|
||||||
|
def extractSiteFromStandardPath(self, pathQuery: str) -> Optional[Dict[str, str]]:
|
||||||
|
"""
|
||||||
|
Extract site name from Microsoft-standard server-relative path.
|
||||||
|
Delegates to SharePoint service.
|
||||||
|
"""
|
||||||
|
return self.services.sharepoint.extractSiteFromStandardPath(pathQuery)
|
||||||
|
|
||||||
|
async def getSiteByStandardPath(self, sitePath: str) -> Optional[Dict[str, Any]]:
|
||||||
|
"""
|
||||||
|
Get SharePoint site directly by Microsoft-standard path.
|
||||||
|
Delegates to SharePoint service.
|
||||||
|
"""
|
||||||
|
return await self.services.sharepoint.getSiteByStandardPath(sitePath)
|
||||||
|
|
||||||
|
def filterSitesByHint(self, sites: List[Dict[str, Any]], siteHint: str) -> List[Dict[str, Any]]:
|
||||||
|
"""
|
||||||
|
Filter discovered sites by a human-entered site hint.
|
||||||
|
Delegates to SharePoint service.
|
||||||
|
"""
|
||||||
|
return self.services.sharepoint.filterSitesByHint(sites, siteHint)
|
||||||
|
|
||||||
|
async def getSiteId(self, hostname: str, sitePath: str) -> str:
|
||||||
|
"""
|
||||||
|
Get SharePoint site ID from hostname and site path.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
hostname: SharePoint hostname
|
||||||
|
sitePath: Site path
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Site ID string
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
endpoint = f"sites/{hostname}:/{sitePath}"
|
||||||
|
result = await self.method.apiClient.makeGraphApiCall(endpoint)
|
||||||
|
|
||||||
|
if "error" in result:
|
||||||
|
logger.error(f"Error getting site ID: {result['error']}")
|
||||||
|
return ""
|
||||||
|
|
||||||
|
return result.get("id", "")
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error getting site ID: {str(e)}")
|
||||||
|
return ""
|
||||||
|
|
||||||
|
async def resolveSitesFromPathQuery(self, pathQuery: str) -> tuple[List[Dict[str, Any]], Optional[str]]:
|
||||||
|
"""
|
||||||
|
Resolve sites from pathQuery using SharePoint service helper methods.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
pathQuery: Path query string
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Tuple of (sites list, error message)
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
# Validate pathQuery format
|
||||||
|
isValid, errorMsg = self.services.sharepoint.validatePathQuery(pathQuery)
|
||||||
|
if not isValid:
|
||||||
|
return [], errorMsg
|
||||||
|
|
||||||
|
# Resolve sites using service helper
|
||||||
|
sites = await self.services.sharepoint.resolveSitesFromPathQuery(pathQuery)
|
||||||
|
if not sites:
|
||||||
|
return [], "No SharePoint sites found or accessible"
|
||||||
|
|
||||||
|
return sites, None
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error resolving sites from pathQuery '{pathQuery}': {str(e)}")
|
||||||
|
return [], f"Error resolving sites from pathQuery: {str(e)}"
|
||||||
|
|
||||||
|
def parseSiteUrl(self, siteUrl: str) -> Dict[str, str]:
|
||||||
|
"""Parse SharePoint site URL to extract hostname and site path"""
|
||||||
|
try:
|
||||||
|
parsed = urllib.parse.urlparse(siteUrl)
|
||||||
|
hostname = parsed.hostname
|
||||||
|
path = parsed.path.strip('/')
|
||||||
|
|
||||||
|
return {
|
||||||
|
"hostname": hostname,
|
||||||
|
"sitePath": path
|
||||||
|
}
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error parsing site URL {siteUrl}: {str(e)}")
|
||||||
|
return {"hostname": "", "sitePath": ""}
|
||||||
|
|
||||||
387
modules/workflows/methods/methodSharepoint/methodSharepoint.py
Normal file
387
modules/workflows/methods/methodSharepoint/methodSharepoint.py
Normal file
|
|
@ -0,0 +1,387 @@
|
||||||
|
# Copyright (c) 2025 Patrick Motsch
|
||||||
|
# All rights reserved.
|
||||||
|
|
||||||
|
"""
|
||||||
|
SharePoint operations method module.
|
||||||
|
Handles SharePoint document operations using the SharePoint service.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
from modules.workflows.methods.methodBase import MethodBase
|
||||||
|
from modules.datamodels.datamodelWorkflowActions import WorkflowActionDefinition, WorkflowActionParameter
|
||||||
|
from modules.shared.frontendTypes import FrontendType
|
||||||
|
|
||||||
|
# Import helpers
|
||||||
|
from .helpers.connection import ConnectionHelper
|
||||||
|
from .helpers.siteDiscovery import SiteDiscoveryHelper
|
||||||
|
from .helpers.documentParsing import DocumentParsingHelper
|
||||||
|
from .helpers.pathProcessing import PathProcessingHelper
|
||||||
|
from .helpers.apiClient import ApiClientHelper
|
||||||
|
|
||||||
|
# Import actions
|
||||||
|
from .actions.findDocumentPath import findDocumentPath
|
||||||
|
from .actions.readDocuments import readDocuments
|
||||||
|
from .actions.uploadDocument import uploadDocument
|
||||||
|
from .actions.listDocuments import listDocuments
|
||||||
|
from .actions.analyzeFolderUsage import analyzeFolderUsage
|
||||||
|
from .actions.findSiteByUrl import findSiteByUrl
|
||||||
|
from .actions.downloadFileByPath import downloadFileByPath
|
||||||
|
from .actions.copyFile import copyFile
|
||||||
|
from .actions.uploadFile import uploadFile
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
class MethodSharepoint(MethodBase):
|
||||||
|
"""SharePoint operations methods."""
|
||||||
|
|
||||||
|
def __init__(self, services):
|
||||||
|
super().__init__(services)
|
||||||
|
self.name = "sharepoint"
|
||||||
|
self.description = "SharePoint operations methods"
|
||||||
|
|
||||||
|
# Initialize helper modules
|
||||||
|
self.connection = ConnectionHelper(self)
|
||||||
|
self.siteDiscovery = SiteDiscoveryHelper(self)
|
||||||
|
self.documentParsing = DocumentParsingHelper(self)
|
||||||
|
self.pathProcessing = PathProcessingHelper(self)
|
||||||
|
self.apiClient = ApiClientHelper(self)
|
||||||
|
|
||||||
|
# RBAC-Integration: Action-Definitionen mit actionId
|
||||||
|
self._actions = {
|
||||||
|
"findDocumentPath": WorkflowActionDefinition(
|
||||||
|
actionId="sharepoint.findDocumentPath",
|
||||||
|
description="Find documents and folders by name/path across sites",
|
||||||
|
parameters={
|
||||||
|
"connectionReference": WorkflowActionParameter(
|
||||||
|
name="connectionReference",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.USER_CONNECTION,
|
||||||
|
required=True,
|
||||||
|
description="Microsoft connection label"
|
||||||
|
),
|
||||||
|
"site": WorkflowActionParameter(
|
||||||
|
name="site",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.TEXT,
|
||||||
|
required=False,
|
||||||
|
description="Site hint"
|
||||||
|
),
|
||||||
|
"searchQuery": WorkflowActionParameter(
|
||||||
|
name="searchQuery",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.TEXT,
|
||||||
|
required=True,
|
||||||
|
description="Search terms or path"
|
||||||
|
),
|
||||||
|
"maxResults": WorkflowActionParameter(
|
||||||
|
name="maxResults",
|
||||||
|
type="int",
|
||||||
|
frontendType=FrontendType.NUMBER,
|
||||||
|
required=False,
|
||||||
|
default=1000,
|
||||||
|
description="Maximum items to return",
|
||||||
|
validation={"min": 1, "max": 10000}
|
||||||
|
)
|
||||||
|
},
|
||||||
|
execute=findDocumentPath.__get__(self, self.__class__)
|
||||||
|
),
|
||||||
|
"readDocuments": WorkflowActionDefinition(
|
||||||
|
actionId="sharepoint.readDocuments",
|
||||||
|
description="Read documents from SharePoint and extract content/metadata",
|
||||||
|
parameters={
|
||||||
|
"connectionReference": WorkflowActionParameter(
|
||||||
|
name="connectionReference",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.USER_CONNECTION,
|
||||||
|
required=True,
|
||||||
|
description="Microsoft connection label"
|
||||||
|
),
|
||||||
|
"documentList": WorkflowActionParameter(
|
||||||
|
name="documentList",
|
||||||
|
type="List[str]",
|
||||||
|
frontendType=FrontendType.DOCUMENT_REFERENCE,
|
||||||
|
required=False,
|
||||||
|
description="Document list reference(s) containing findDocumentPath result"
|
||||||
|
),
|
||||||
|
"pathQuery": WorkflowActionParameter(
|
||||||
|
name="pathQuery",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.TEXT,
|
||||||
|
required=False,
|
||||||
|
description="Direct path query if no documentList (e.g., /sites/SiteName/FolderPath)"
|
||||||
|
),
|
||||||
|
"includeMetadata": WorkflowActionParameter(
|
||||||
|
name="includeMetadata",
|
||||||
|
type="bool",
|
||||||
|
frontendType=FrontendType.CHECKBOX,
|
||||||
|
required=False,
|
||||||
|
default=True,
|
||||||
|
description="Include metadata"
|
||||||
|
)
|
||||||
|
},
|
||||||
|
execute=readDocuments.__get__(self, self.__class__)
|
||||||
|
),
|
||||||
|
"uploadDocument": WorkflowActionDefinition(
|
||||||
|
actionId="sharepoint.uploadDocument",
|
||||||
|
description="Upload documents to SharePoint",
|
||||||
|
parameters={
|
||||||
|
"connectionReference": WorkflowActionParameter(
|
||||||
|
name="connectionReference",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.USER_CONNECTION,
|
||||||
|
required=True,
|
||||||
|
description="Microsoft connection label"
|
||||||
|
),
|
||||||
|
"documentList": WorkflowActionParameter(
|
||||||
|
name="documentList",
|
||||||
|
type="List[str]",
|
||||||
|
frontendType=FrontendType.DOCUMENT_REFERENCE,
|
||||||
|
required=True,
|
||||||
|
description="Document reference(s) to upload. File names are taken from the documents"
|
||||||
|
),
|
||||||
|
"pathQuery": WorkflowActionParameter(
|
||||||
|
name="pathQuery",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.TEXT,
|
||||||
|
required=False,
|
||||||
|
description="Direct upload target path if documentList doesn't contain findDocumentPath result (e.g., /sites/SiteName/FolderPath)"
|
||||||
|
)
|
||||||
|
},
|
||||||
|
execute=uploadDocument.__get__(self, self.__class__)
|
||||||
|
),
|
||||||
|
"listDocuments": WorkflowActionDefinition(
|
||||||
|
actionId="sharepoint.listDocuments",
|
||||||
|
description="List documents and folders in SharePoint paths across sites",
|
||||||
|
parameters={
|
||||||
|
"connectionReference": WorkflowActionParameter(
|
||||||
|
name="connectionReference",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.USER_CONNECTION,
|
||||||
|
required=True,
|
||||||
|
description="Microsoft connection label"
|
||||||
|
),
|
||||||
|
"documentList": WorkflowActionParameter(
|
||||||
|
name="documentList",
|
||||||
|
type="List[str]",
|
||||||
|
frontendType=FrontendType.DOCUMENT_REFERENCE,
|
||||||
|
required=True,
|
||||||
|
description="Document list reference(s) containing findDocumentPath result"
|
||||||
|
),
|
||||||
|
"includeSubfolders": WorkflowActionParameter(
|
||||||
|
name="includeSubfolders",
|
||||||
|
type="bool",
|
||||||
|
frontendType=FrontendType.CHECKBOX,
|
||||||
|
required=False,
|
||||||
|
default=False,
|
||||||
|
description="Include one level of subfolders"
|
||||||
|
)
|
||||||
|
},
|
||||||
|
execute=listDocuments.__get__(self, self.__class__)
|
||||||
|
),
|
||||||
|
"analyzeFolderUsage": WorkflowActionDefinition(
|
||||||
|
actionId="sharepoint.analyzeFolderUsage",
|
||||||
|
description="Analyze usage intensity of folders and files in SharePoint",
|
||||||
|
parameters={
|
||||||
|
"connectionReference": WorkflowActionParameter(
|
||||||
|
name="connectionReference",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.USER_CONNECTION,
|
||||||
|
required=True,
|
||||||
|
description="Microsoft connection label"
|
||||||
|
),
|
||||||
|
"documentList": WorkflowActionParameter(
|
||||||
|
name="documentList",
|
||||||
|
type="List[str]",
|
||||||
|
frontendType=FrontendType.DOCUMENT_REFERENCE,
|
||||||
|
required=True,
|
||||||
|
description="Document list reference(s) containing findDocumentPath result"
|
||||||
|
),
|
||||||
|
"startDateTime": WorkflowActionParameter(
|
||||||
|
name="startDateTime",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.DATETIME,
|
||||||
|
required=False,
|
||||||
|
description="Start date/time in ISO format (e.g., 2025-11-01T00:00:00Z). Default: 30 days ago"
|
||||||
|
),
|
||||||
|
"endDateTime": WorkflowActionParameter(
|
||||||
|
name="endDateTime",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.DATETIME,
|
||||||
|
required=False,
|
||||||
|
description="End date/time in ISO format (e.g., 2025-11-30T23:59:59Z). Default: current time"
|
||||||
|
),
|
||||||
|
"interval": WorkflowActionParameter(
|
||||||
|
name="interval",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.SELECT,
|
||||||
|
frontendOptions=["day", "week", "month"],
|
||||||
|
required=False,
|
||||||
|
default="day",
|
||||||
|
description="Time interval for grouping activities"
|
||||||
|
)
|
||||||
|
},
|
||||||
|
execute=analyzeFolderUsage.__get__(self, self.__class__)
|
||||||
|
),
|
||||||
|
"findSiteByUrl": WorkflowActionDefinition(
|
||||||
|
actionId="sharepoint.findSiteByUrl",
|
||||||
|
description="Find SharePoint site by hostname and site path",
|
||||||
|
parameters={
|
||||||
|
"connectionReference": WorkflowActionParameter(
|
||||||
|
name="connectionReference",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.USER_CONNECTION,
|
||||||
|
required=True,
|
||||||
|
description="Microsoft connection label"
|
||||||
|
),
|
||||||
|
"hostname": WorkflowActionParameter(
|
||||||
|
name="hostname",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.TEXT,
|
||||||
|
required=True,
|
||||||
|
description="SharePoint hostname (e.g., example.sharepoint.com)"
|
||||||
|
),
|
||||||
|
"sitePath": WorkflowActionParameter(
|
||||||
|
name="sitePath",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.TEXT,
|
||||||
|
required=True,
|
||||||
|
description="Site path (e.g., SteeringBPM or /sites/SteeringBPM)"
|
||||||
|
)
|
||||||
|
},
|
||||||
|
execute=findSiteByUrl.__get__(self, self.__class__)
|
||||||
|
),
|
||||||
|
"downloadFileByPath": WorkflowActionDefinition(
|
||||||
|
actionId="sharepoint.downloadFileByPath",
|
||||||
|
description="Download file from SharePoint by exact file path",
|
||||||
|
parameters={
|
||||||
|
"connectionReference": WorkflowActionParameter(
|
||||||
|
name="connectionReference",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.USER_CONNECTION,
|
||||||
|
required=True,
|
||||||
|
description="Microsoft connection label"
|
||||||
|
),
|
||||||
|
"siteId": WorkflowActionParameter(
|
||||||
|
name="siteId",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.TEXT,
|
||||||
|
required=True,
|
||||||
|
description="SharePoint site ID (from findSiteByUrl result) or document reference containing site info"
|
||||||
|
),
|
||||||
|
"filePath": WorkflowActionParameter(
|
||||||
|
name="filePath",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.TEXT,
|
||||||
|
required=True,
|
||||||
|
description="Full file path relative to site root (e.g., /General/50 Docs hosted by SELISE/file.xlsx)"
|
||||||
|
)
|
||||||
|
},
|
||||||
|
execute=downloadFileByPath.__get__(self, self.__class__)
|
||||||
|
),
|
||||||
|
"copyFile": WorkflowActionDefinition(
|
||||||
|
actionId="sharepoint.copyFile",
|
||||||
|
description="Copy file within SharePoint",
|
||||||
|
parameters={
|
||||||
|
"connectionReference": WorkflowActionParameter(
|
||||||
|
name="connectionReference",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.USER_CONNECTION,
|
||||||
|
required=True,
|
||||||
|
description="Microsoft connection label"
|
||||||
|
),
|
||||||
|
"siteId": WorkflowActionParameter(
|
||||||
|
name="siteId",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.TEXT,
|
||||||
|
required=True,
|
||||||
|
description="SharePoint site ID (from findSiteByUrl result) or document reference containing site info"
|
||||||
|
),
|
||||||
|
"sourceFolder": WorkflowActionParameter(
|
||||||
|
name="sourceFolder",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.TEXT,
|
||||||
|
required=True,
|
||||||
|
description="Source folder path relative to site root"
|
||||||
|
),
|
||||||
|
"sourceFile": WorkflowActionParameter(
|
||||||
|
name="sourceFile",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.TEXT,
|
||||||
|
required=True,
|
||||||
|
description="Source file name"
|
||||||
|
),
|
||||||
|
"destFolder": WorkflowActionParameter(
|
||||||
|
name="destFolder",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.TEXT,
|
||||||
|
required=True,
|
||||||
|
description="Destination folder path relative to site root"
|
||||||
|
),
|
||||||
|
"destFile": WorkflowActionParameter(
|
||||||
|
name="destFile",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.TEXT,
|
||||||
|
required=True,
|
||||||
|
description="Destination file name"
|
||||||
|
)
|
||||||
|
},
|
||||||
|
execute=copyFile.__get__(self, self.__class__)
|
||||||
|
),
|
||||||
|
"uploadFile": WorkflowActionDefinition(
|
||||||
|
actionId="sharepoint.uploadFile",
|
||||||
|
description="Upload raw file content (bytes) to SharePoint",
|
||||||
|
parameters={
|
||||||
|
"connectionReference": WorkflowActionParameter(
|
||||||
|
name="connectionReference",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.USER_CONNECTION,
|
||||||
|
required=True,
|
||||||
|
description="Microsoft connection label"
|
||||||
|
),
|
||||||
|
"siteId": WorkflowActionParameter(
|
||||||
|
name="siteId",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.TEXT,
|
||||||
|
required=True,
|
||||||
|
description="SharePoint site ID (from findSiteByUrl result) or document reference containing site info"
|
||||||
|
),
|
||||||
|
"folderPath": WorkflowActionParameter(
|
||||||
|
name="folderPath",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.TEXT,
|
||||||
|
required=True,
|
||||||
|
description="Folder path relative to site root"
|
||||||
|
),
|
||||||
|
"fileName": WorkflowActionParameter(
|
||||||
|
name="fileName",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.TEXT,
|
||||||
|
required=True,
|
||||||
|
description="File name"
|
||||||
|
),
|
||||||
|
"content": WorkflowActionParameter(
|
||||||
|
name="content",
|
||||||
|
type="str",
|
||||||
|
frontendType=FrontendType.DOCUMENT_REFERENCE,
|
||||||
|
required=True,
|
||||||
|
description="Document reference containing file content as base64-encoded bytes"
|
||||||
|
)
|
||||||
|
},
|
||||||
|
execute=uploadFile.__get__(self, self.__class__)
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
# Validate actions after definition
|
||||||
|
self._validateActions()
|
||||||
|
|
||||||
|
# Register actions as methods (optional, für direkten Zugriff)
|
||||||
|
self.findDocumentPath = findDocumentPath.__get__(self, self.__class__)
|
||||||
|
self.readDocuments = readDocuments.__get__(self, self.__class__)
|
||||||
|
self.uploadDocument = uploadDocument.__get__(self, self.__class__)
|
||||||
|
self.listDocuments = listDocuments.__get__(self, self.__class__)
|
||||||
|
self.analyzeFolderUsage = analyzeFolderUsage.__get__(self, self.__class__)
|
||||||
|
self.findSiteByUrl = findSiteByUrl.__get__(self, self.__class__)
|
||||||
|
self.downloadFileByPath = downloadFileByPath.__get__(self, self.__class__)
|
||||||
|
self.copyFile = copyFile.__get__(self, self.__class__)
|
||||||
|
self.uploadFile = uploadFile.__get__(self, self.__class__)
|
||||||
|
|
||||||
|
|
@ -27,12 +27,16 @@ def discoverMethods(serviceCenter):
|
||||||
# Import the methods package
|
# Import the methods package
|
||||||
methodsPackage = importlib.import_module('modules.workflows.methods')
|
methodsPackage = importlib.import_module('modules.workflows.methods')
|
||||||
|
|
||||||
# Discover all modules in the package
|
# Discover all modules and packages in the methods package
|
||||||
for _, name, isPkg in pkgutil.iter_modules(methodsPackage.__path__):
|
for _, name, isPkg in pkgutil.iter_modules(methodsPackage.__path__):
|
||||||
if not isPkg and name.startswith('method'):
|
if name.startswith('method'):
|
||||||
try:
|
try:
|
||||||
# Import the module
|
if isPkg:
|
||||||
module = importlib.import_module(f'modules.workflows.methods.{name}')
|
# Package (folder) - import __init__.py which exports the Method class
|
||||||
|
module = importlib.import_module(f'modules.workflows.methods.{name}')
|
||||||
|
else:
|
||||||
|
# Module (file) - import directly
|
||||||
|
module = importlib.import_module(f'modules.workflows.methods.{name}')
|
||||||
|
|
||||||
# Find all classes in the module that inherit from MethodBase
|
# Find all classes in the module that inherit from MethodBase
|
||||||
for itemName, item in inspect.getmembers(module):
|
for itemName, item in inspect.getmembers(module):
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue