PowerOn/gateway
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

handovers

Browse source
This commit is contained in:
ValueOn AG 2025-07-08 01:14:27 +02:00
parent 1ba107b4fd
commit 8d8b800859
22 changed files with 2693 additions and 6000 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

277
ENHANCED_AI_AGENT_RECOMMENDATIONS.md
View file

@ -1,277 +0,0 @@
# Enhanced AI Agent System Recommendations
## Overview
This document provides comprehensive recommendations for building a stable, robust, and perfect AI agent system with clear handovers and optimal user request processing.
## 1. **Enhanced Error Recovery & Resilience**
### ✅ **Implemented Features:**
- **Circuit Breaker Pattern**: Prevents cascading failures when AI services are down
- **Exponential Backoff Retry**: Intelligent retry with increasing delays
- **Timeout Handling**: Prevents hanging operations
- **Fallback Mechanisms**: Graceful degradation when AI fails
- **Alternative Approach Generation**: Tries different methods when original fails
### 🔄 **Additional Recommendations:**
#### A. **State Persistence & Recovery**
```python
# Add checkpoint system for long-running workflows
class WorkflowCheckpoint:
def save_checkpoint(self, workflow_id: str, task_step: int, state: Dict):
# Save current state to database
pass
def restore_checkpoint(self, workflow_id: str) -> Dict:
# Restore from last checkpoint
pass
```
#### B. **Graceful Degradation**
```python
# Implement multiple AI providers with fallback
class MultiProviderAIService:
def __init__(self):
self.providers = [
OpenAIService(),
AnthropicService(),
LocalLLMService() # Fallback
]
async def call_with_fallback(self, prompt: str) -> str:
for provider in self.providers:
try:
return await provider.call(prompt)
except Exception:
continue
raise Exception("All AI providers failed")
```
## 2. **Intelligent Task Planning & Execution**
### ✅ **Current Implementation:**
- **Task Planning**: AI analyzes request and creates logical task steps
- **Handover Review**: Validates each step before proceeding
- **Dynamic Action Generation**: Creates actions based on current context
### 🔄 **Enhanced Recommendations:**
#### A. **Dependency Graph Management**
```python
class TaskDependencyGraph:
def __init__(self):
self.nodes = {} # task_id -> task_info
self.edges = {} # task_id -> [dependencies]
def add_task(self, task_id: str, dependencies: List[str]):
self.nodes[task_id] = {"status": "pending"}
self.edges[task_id] = dependencies
def get_ready_tasks(self) -> List[str]:
# Return tasks with all dependencies completed
pass
def detect_cycles(self) -> bool:
# Detect circular dependencies
pass
```
#### B. **Parallel Task Execution**
```python
async def execute_parallel_tasks(self, independent_tasks: List[Dict]) -> List[Dict]:
"""Execute independent tasks in parallel for better performance"""
tasks = []
for task_step in independent_tasks:
task = asyncio.create_task(self._executeTaskStep(task_step))
tasks.append(task)
results = await asyncio.gather(*tasks, return_exceptions=True)
return results
```
## 3. **Advanced Quality Assurance**
### 🔄 **Quality Metrics & Validation:**
#### A. **Multi-Dimensional Quality Assessment**
```python
class QualityAssessor:
def assess_quality(self, result: Dict, criteria: Dict) -> QualityScore:
return QualityScore(
completeness=self._assess_completeness(result, criteria),
accuracy=self._assess_accuracy(result, criteria),
relevance=self._assess_relevance(result, criteria),
coherence=self._assess_coherence(result, criteria)
)
```
#### B. **Continuous Learning & Improvement**
```python
class LearningSystem:
def record_execution(self, task: Dict, result: Dict, quality_score: float):
"""Record execution for learning"""
pass
def suggest_improvements(self, task_type: str) -> List[str]:
"""Suggest improvements based on historical data"""
pass
```
## 4. **Enhanced Document & Context Management**
### 🔄 **Smart Document Processing:**
#### A. **Document Understanding & Classification**
```python
class DocumentProcessor:
def classify_document(self, content: str) -> DocumentType:
"""Classify document type for better processing"""
pass
def extract_key_information(self, document: Document) -> Dict:
"""Extract key information for context"""
pass
```
#### B. **Context-Aware Processing**
```python
class ContextManager:
def __init__(self):
self.context_stack = []
self.document_cache = {}
def add_context(self, context: Dict):
"""Add context for current processing"""
self.context_stack.append(context)
def get_relevant_context(self, task: Dict) -> Dict:
"""Get relevant context for specific task"""
pass
```
## 5. **Advanced Handover Mechanisms**
### 🔄 **Intelligent Handover System:**
#### A. **Handover Validation Engine**
```python
class HandoverValidator:
def validate_handover(self, from_task: Dict, to_task: Dict, data: Dict) -> ValidationResult:
"""Validate data handover between tasks"""
return ValidationResult(
is_valid=self._check_data_completeness(data, to_task),
missing_data=self._identify_missing_data(data, to_task),
quality_issues=self._identify_quality_issues(data),
suggestions=self._generate_handover_suggestions(data, to_task)
)
```
## 6. **Monitoring & Observability**
### 🔄 **Comprehensive Monitoring:**
#### A. **Real-Time Metrics**
```python
class MetricsCollector:
def __init__(self):
self.metrics = {
'task_execution_time': [],
'ai_call_latency': [],
'success_rate': [],
'error_rate': [],
'quality_scores': []
}
def record_metric(self, metric_name: str, value: float):
"""Record metric for monitoring"""
pass
def get_health_score(self) -> float:
"""Calculate overall system health score"""
pass
```
## 7. **Security & Privacy**
### 🔄 **Enhanced Security Measures:**
#### A. **Data Sanitization**
```python
class DataSanitizer:
def sanitize_input(self, user_input: str) -> str:
"""Sanitize user input for security"""
pass
def validate_documents(self, documents: List[Document]) -> bool:
"""Validate documents for security risks"""
pass
```
## 8. **Performance Optimization**
### 🔄 **Performance Enhancements:**
#### A. **Caching Strategy**
```python
class CacheManager:
def __init__(self):
self.document_cache = {}
self.ai_response_cache = {}
self.task_result_cache = {}
def get_cached_result(self, key: str) -> Optional[Dict]:
"""Get cached result if available"""
pass
def cache_result(self, key: str, result: Dict, ttl: int = 3600):
"""Cache result with TTL"""
pass
```
## 9. **Testing & Validation**
### 🔄 **Comprehensive Testing:**
#### A. **Automated Testing Framework**
```python
class TestFramework:
def test_task_planning(self, scenarios: List[Dict]):
"""Test task planning with various scenarios"""
pass
def test_handover_validation(self, test_cases: List[Dict]):
"""Test handover validation logic"""
pass
```
## 10. **Implementation Priority**
### **Phase 1 (Critical - Implement First):**
1. ✅ Circuit Breaker Pattern
2. ✅ Retry Mechanisms
3. ✅ Fallback Systems
4. 🔄 Enhanced Error Handling
### **Phase 2 (Important - Implement Next):**
1. 🔄 Parallel Task Execution
2. 🔄 Advanced Quality Assessment
3. 🔄 Smart Document Processing
4. 🔄 Comprehensive Monitoring
### **Phase 3 (Enhancement - Future):**
1. 🔄 Learning & Optimization
2. 🔄 Advanced Security
3. 🔄 Performance Optimization
4. 🔄 Advanced Testing
## Conclusion
The enhanced AI agent system provides:
- **Robustness**: Multiple layers of error recovery and fallback mechanisms
- **Intelligence**: Smart task planning and dynamic action generation
- **Quality**: Comprehensive validation and quality assessment
- **Observability**: Full monitoring and alerting capabilities
- **Scalability**: Resource management and performance optimization
- **Security**: Data protection and access control
This system will process user requests in a near-perfect way with clear handovers, comprehensive error handling, and continuous improvement capabilities.

343
modules/methods/methodCoder.py
View file

@ -1,29 +1,98 @@
from typing import Dict, Any, Optional from typing import Dict, Any, Optional
import logging import logging
import uuid
from datetime import datetime, UTC from datetime import datetime, UTC
from modules.workflow.methodBase import MethodBase, ActionResult, action from modules.workflow.methodBase import MethodBase, ActionResult, action
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
class CoderService: class MethodCoder(MethodBase):
"""Service for code analysis, generation, and refactoring operations""" """Coder method implementation for code operations"""
def __init__(self, serviceContainer: Any): def __init__(self, serviceContainer: Any):
self.serviceContainer = serviceContainer super().__init__(serviceContainer)
self.name = "coder"
self.description = "Handle code operations like analysis and generation"
async def analyzeCode(self, code: str, language: str = "python", checks: list = None) -> Dict[str, Any]: @action
"""Analyze code quality and structure""" async def analyze(self, parameters: Dict[str, Any]) -> ActionResult:
if checks is None: """
checks = ["complexity", "style", "security"] Analyze code quality and structure
Parameters:
documentList (str): Reference to the document list to analyze
aiPrompt (str): AI prompt for code analysis
language (str, optional): Programming language (default: "python")
checks (List[str], optional): Types of checks to perform (default: ["complexity", "style", "security"])
"""
try: try:
documentList = parameters.get("documentList")
aiPrompt = parameters.get("aiPrompt")
language = parameters.get("language", "python")
checks = parameters.get("checks", ["complexity", "style", "security"])
if not documentList:
return self._createResult(
success=False,
data={},
error="Document list reference is required"
)
if not aiPrompt:
return self._createResult(
success=False,
data={},
error="AI prompt is required"
)
chatDocuments = self.serviceContainer.getChatDocumentsFromDocumentReference(documentList)
if not chatDocuments:
return self._createResult(
success=False,
data={},
error="No documents found for the provided reference"
)
# Extract content from all documents
all_code_content = []
for chatDocument in chatDocuments:
fileId = chatDocument.fileId
code = self.serviceContainer.getFileData(fileId)
file_info = self.serviceContainer.getFileInfo(fileId)
if not code:
logger.warning(f"Code file is empty for fileId: {fileId}")
continue
# Use AI prompt to extract relevant code content
extracted_content = await self.serviceContainer.extractContentFromFileData(
prompt=aiPrompt,
fileData=code,
filename=file_info.get('name', 'code'),
mimeType=file_info.get('mimeType', 'text/plain'),
base64Encoded=False
)
all_code_content.append(extracted_content)
if not all_code_content:
return self._createResult(
success=False,
data={},
error="No code content could be extracted from any documents"
)
# Combine all code content for analysis
combined_code = "\n\n--- CODE SEPARATOR ---\n\n".join(all_code_content)
# Create analysis prompt # Create analysis prompt
analysis_prompt = f""" analysis_prompt = f"""
Analyze this {language} code for quality, structure, and potential issues. Analyze this {language} code for quality, structure, and potential issues.
Code to analyze: Code to analyze:
{code} {combined_code}
Please check for: Please check for:
{', '.join(checks)} {', '.join(checks)}
@ -34,144 +103,27 @@ class CoderService:
3. Security considerations 3. Security considerations
4. Performance optimizations 4. Performance optimizations
5. Best practices compliance 5. Best practices compliance
6. Summary of findings across all documents
""" """
# Use AI service for analysis # Use AI service for analysis
analysis_result = await self.serviceContainer.interfaceAiCalls.callAiTextAdvanced(analysis_prompt) analysis_result = await self.serviceContainer.interfaceAiCalls.callAiTextAdvanced(analysis_prompt)
return { # Create result data
result_data = {
"documentCount": len(chatDocuments),
"language": language, "language": language,
"checks": checks, "checks": checks,
"analysis": analysis_result, "analysis": analysis_result,
"timestamp": datetime.now(UTC).isoformat() "timestamp": datetime.now(UTC).isoformat()
} }
except Exception as e:
logger.error(f"Error analyzing code: {str(e)}")
return {
"error": str(e),
"language": language,
"checks": checks
}
async def generateCode(self, requirements: str, language: str = "python", template: str = None) -> Dict[str, Any]:
"""Generate code based on requirements"""
try:
# Create generation prompt
generation_prompt = f"""
Generate {language} code based on the following requirements:
Requirements:
{requirements}
{f'Template to follow: {template}' if template else ''}
Please provide:
1. Complete, working code
2. Clear comments and documentation
3. Error handling where appropriate
4. Best practices implementation
"""
# Use AI service for code generation
generated_code = await self.serviceContainer.interfaceAiCalls.callAiTextAdvanced(generation_prompt)
return {
"language": language,
"requirements": requirements,
"code": generated_code,
"timestamp": datetime.now(UTC).isoformat()
}
except Exception as e:
logger.error(f"Error generating code: {str(e)}")
return {
"error": str(e),
"language": language,
"requirements": requirements
}
async def refactorCode(self, code: str, language: str = "python", improvements: list = None) -> Dict[str, Any]:
"""Refactor code for better quality"""
if improvements is None:
improvements = ["style", "complexity"]
try:
# Create refactoring prompt
refactor_prompt = f"""
Refactor this {language} code to improve:
{', '.join(improvements)}
Original code:
{code}
Please provide:
1. Refactored code with improvements
2. Explanation of changes made
3. Benefits of the refactoring
4. Any potential trade-offs
"""
# Use AI service for refactoring
refactored_code = await self.serviceContainer.interfaceAiCalls.callAiTextAdvanced(refactor_prompt)
return {
"language": language,
"improvements": improvements,
"original_code": code,
"refactored_code": refactored_code,
"timestamp": datetime.now(UTC).isoformat()
}
except Exception as e:
logger.error(f"Error refactoring code: {str(e)}")
return {
"error": str(e),
"language": language,
"improvements": improvements
}
class MethodCoder(MethodBase):
"""Coder method implementation for code operations"""
def __init__(self, serviceContainer: Any):
super().__init__(serviceContainer)
self.name = "coder"
self.description = "Handle code operations like analysis and generation"
self.coderService = CoderService(serviceContainer)
@action
async def analyze(self, parameters: Dict[str, Any]) -> ActionResult:
"""
Analyze code quality and structure
Parameters:
code (str): The code to analyze
language (str, optional): Programming language (default: "python")
checks (List[str], optional): Types of checks to perform (default: ["complexity", "style", "security"])
"""
try:
code = parameters.get("code")
language = parameters.get("language", "python")
checks = parameters.get("checks", ["complexity", "style", "security"])
if not code:
return self._createResult(
success=False,
data={},
error="Code is required"
)
# Analyze code
results = await self.coderService.analyzeCode(
code=code,
language=language,
checks=checks
)
return self._createResult( return self._createResult(
success=True, success=True,
data=results data={
"documentName": f"code_analysis_{datetime.now(UTC).strftime('%Y%m%d_%H%M%S')}.json",
"documentData": result_data
}
) )
except Exception as e: except Exception as e:
@ -204,16 +156,39 @@ class MethodCoder(MethodBase):
error="Requirements are required" error="Requirements are required"
) )
# Generate code # Create generation prompt
code = await self.coderService.generateCode( generation_prompt = f"""
requirements=requirements, Generate {language} code based on the following requirements:
language=language,
template=template Requirements:
) {requirements}
{f'Template to follow: {template}' if template else ''}
Please provide:
1. Complete, working code
2. Clear comments and documentation
3. Error handling where appropriate
4. Best practices implementation
"""
# Use AI service for code generation
generated_code = await self.serviceContainer.interfaceAiCalls.callAiTextAdvanced(generation_prompt)
# Create result data
result_data = {
"language": language,
"requirements": requirements,
"code": generated_code,
"timestamp": datetime.now(UTC).isoformat()
}
return self._createResult( return self._createResult(
success=True, success=True,
data=code data={
"documentName": f"generated_code_{datetime.now(UTC).strftime('%Y%m%d_%H%M%S')}.{language}",
"documentData": result_data
}
) )
except Exception as e: except Exception as e:
@ -230,32 +205,96 @@ class MethodCoder(MethodBase):
Refactor code for better quality Refactor code for better quality
Parameters: Parameters:
code (str): The code to refactor documentList (str): Reference to the document list to refactor
aiImprovementPrompt (str): AI prompt for code improvements
language (str, optional): Programming language (default: "python") language (str, optional): Programming language (default: "python")
improvements (List[str], optional): Types of improvements to make (default: ["style", "complexity"])
""" """
try: try:
code = parameters.get("code") documentList = parameters.get("documentList")
aiImprovementPrompt = parameters.get("aiImprovementPrompt")
language = parameters.get("language", "python") language = parameters.get("language", "python")
improvements = parameters.get("improvements", ["style", "complexity"])
if not code: if not documentList:
return self._createResult( return self._createResult(
success=False, success=False,
data={}, data={},
error="Code is required" error="Document list reference is required"
) )
# Refactor code if not aiImprovementPrompt:
results = await self.coderService.refactorCode( return self._createResult(
code=code, success=False,
language=language, data={},
improvements=improvements error="AI improvement prompt is required"
) )
chatDocuments = self.serviceContainer.getChatDocumentsFromDocumentReference(documentList)
if not chatDocuments:
return self._createResult(
success=False,
data={},
error="No documents found for the provided reference"
)
# Process each document individually
refactored_results = []
for chatDocument in chatDocuments:
fileId = chatDocument.fileId
code = self.serviceContainer.getFileData(fileId)
file_info = self.serviceContainer.getFileInfo(fileId)
if not code:
logger.warning(f"Code file is empty for fileId: {fileId}")
continue
# Create refactoring prompt for this specific document
refactor_prompt = f"""
Refactor this {language} code based on the following improvement requirements:
Improvement requirements:
{aiImprovementPrompt}
Original code:
{code}
Please provide:
1. Refactored code with improvements
2. Explanation of changes made
3. Benefits of the refactoring
4. Any potential trade-offs
"""
# Use AI service for refactoring
refactored_code = await self.serviceContainer.interfaceAiCalls.callAiTextAdvanced(refactor_prompt)
refactored_results.append({
"original_file": file_info.get('name', 'unknown'),
"original_code": code,
"refactored_code": refactored_code
})
if not refactored_results:
return self._createResult(
success=False,
data={},
error="No code could be refactored from any documents"
)
# Create result data
result_data = {
"documentCount": len(chatDocuments),
"language": language,
"refactored_results": refactored_results,
"timestamp": datetime.now(UTC).isoformat()
}
return self._createResult( return self._createResult(
success=True, success=True,
data=results data={
"documentName": f"refactored_code_{datetime.now(UTC).strftime('%Y%m%d_%H%M%S')}.{language}",
"documentData": result_data
}
) )
except Exception as e: except Exception as e:

423
modules/methods/methodDocument.py
View file

@ -5,171 +5,14 @@ Handles document operations using the document service.
import logging import logging
from typing import Dict, Any, List, Optional from typing import Dict, Any, List, Optional
import uuid
from datetime import datetime, UTC
from modules.workflow.managerDocument import DocumentManager from modules.workflow.managerDocument import DocumentManager
from modules.workflow.methodBase import MethodBase, ActionResult, action from modules.workflow.methodBase import MethodBase, ActionResult, action
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
class DocumentService:
"""Service for document content extraction, analysis, and summarization"""
def __init__(self, serviceContainer: Any):
self.serviceContainer = serviceContainer
async def extractContent(self, fileId: str, format: str = "text", includeMetadata: bool = True) -> Dict[str, Any]:
"""Extract content from document using prompt-based extraction"""
try:
# Get file data
file_data = self.serviceContainer.getFileData(fileId)
file_info = self.serviceContainer.getFileInfo(fileId)
if not file_data:
return {
"error": "File not found or empty",
"fileId": fileId
}
# Create extraction prompt based on format
extraction_prompt = f"""
Extract and structure the content from this document.
File information:
- Name: {file_info.get('name', 'Unknown')}
- Type: {file_info.get('mimeType', 'Unknown')}
- Size: {len(file_data)} bytes
Please extract:
1. Main content and key information
2. Structured data if present (tables, lists, etc.)
3. Important facts and figures
4. Key insights and takeaways
Format the output as: {format}
Include metadata: {includeMetadata}
"""
# Use the new direct file data extraction method
extracted_content = await self.serviceContainer.extractContentFromFileData(
prompt=extraction_prompt,
fileData=file_data,
filename=file_info.get('name', 'document'),
mimeType=file_info.get('mimeType', 'application/octet-stream'),
base64Encoded=False
)
result = {
"fileId": fileId,
"format": format,
"content": extracted_content,
"fileInfo": file_info if includeMetadata else None
}
return result
except Exception as e:
logger.error(f"Error extracting content: {str(e)}")
return {
"error": str(e),
"fileId": fileId
}
async def analyzeContent(self, fileId: str, analysis: list = None) -> Dict[str, Any]:
"""Analyze document content for entities, topics, and sentiment"""
if analysis is None:
analysis = ["entities", "topics", "sentiment"]
try:
# First extract content
content_result = await self.extractContent(fileId, "text", True)
if "error" in content_result:
return content_result
content = content_result.get("content", "")
# Create analysis prompt
analysis_prompt = f"""
Analyze this document content for the following aspects:
{', '.join(analysis)}
Document content:
{content[:5000]} # Limit content length
Please provide a detailed analysis including:
1. Key entities (people, organizations, locations, dates)
2. Main topics and themes
3. Sentiment analysis (positive, negative, neutral)
4. Key insights and patterns
5. Important relationships between entities
6. Document structure and organization
"""
# Use AI service for analysis
analysis_result = await self.serviceContainer.interfaceAiCalls.callAiTextAdvanced(analysis_prompt)
return {
"fileId": fileId,
"analysis": analysis,
"results": analysis_result,
"content": content_result
}
except Exception as e:
logger.error(f"Error analyzing content: {str(e)}")
return {
"error": str(e),
"fileId": fileId,
"analysis": analysis
}
async def summarizeContent(self, fileId: str, maxLength: int = 200, format: str = "text") -> Dict[str, Any]:
"""Summarize document content"""
try:
# First extract content
content_result = await self.extractContent(fileId, "text", False)
if "error" in content_result:
return content_result
content = content_result.get("content", "")
# Create summarization prompt
summary_prompt = f"""
Create a comprehensive summary of this document content.
Document content:
{content[:8000]} # Limit content length
Requirements:
- Maximum length: {maxLength} words
- Format: {format}
- Include key points and main ideas
- Maintain accuracy and completeness
- Use clear, professional language
- Highlight important insights and conclusions
"""
# Use AI service for summarization
summary = await self.serviceContainer.interfaceAiCalls.callAiTextAdvanced(summary_prompt)
return {
"fileId": fileId,
"maxLength": maxLength,
"format": format,
"summary": summary,
"wordCount": len(summary.split()),
"originalContent": content_result
}
except Exception as e:
logger.error(f"Error summarizing content: {str(e)}")
return {
"error": str(e),
"fileId": fileId,
"maxLength": maxLength
}
class MethodDocument(MethodBase): class MethodDocument(MethodBase):
"""Document method implementation for document operations""" """Document method implementation for document operations"""
@ -178,7 +21,6 @@ class MethodDocument(MethodBase):
super().__init__(serviceContainer) super().__init__(serviceContainer)
self.name = "document" self.name = "document"
self.description = "Handle document operations like extraction and analysis" self.description = "Handle document operations like extraction and analysis"
self.documentService = DocumentService(serviceContainer)
self.documentManager = DocumentManager(serviceContainer) self.documentManager = DocumentManager(serviceContainer)
@action @action
@ -187,34 +29,89 @@ class MethodDocument(MethodBase):
Extract content from document Extract content from document
Parameters: Parameters:
fileId (str): The ID of the document to extract content from documentList (str): Reference to the document list to extract content from
aiPrompt (str): AI prompt for content extraction
format (str, optional): Output format (default: "text") format (str, optional): Output format (default: "text")
includeMetadata (bool, optional): Whether to include metadata (default: True) includeMetadata (bool, optional): Whether to include metadata (default: True)
""" """
try: try:
fileId = parameters.get("fileId") documentList = parameters.get("documentList")
aiPrompt = parameters.get("aiPrompt")
format = parameters.get("format", "text") format = parameters.get("format", "text")
includeMetadata = parameters.get("includeMetadata", True) includeMetadata = parameters.get("includeMetadata", True)
if not fileId: if not documentList:
return self._createResult( return self._createResult(
success=False, success=False,
data={}, data={},
error="File ID is required" error="Document list reference is required"
) )
# Extract content if not aiPrompt:
content = await self.documentService.extractContent( return self._createResult(
fileId=fileId, success=False,
format=format, data={},
includeMetadata=includeMetadata error="AI prompt is required"
) )
chatDocuments = self.serviceContainer.getChatDocumentsFromDocumentReference(documentList)
if not chatDocuments:
return self._createResult(
success=False,
data={},
error="No documents found for the provided reference"
)
# Extract content from all documents
all_extracted_content = []
file_infos = []
for chatDocument in chatDocuments:
fileId = chatDocument.fileId
file_data = self.serviceContainer.getFileData(fileId)
file_info = self.serviceContainer.getFileInfo(fileId)
if not file_data:
logger.warning(f"File not found or empty for fileId: {fileId}")
continue
extracted_content = await self.serviceContainer.extractContentFromFileData(
prompt=aiPrompt,
fileData=file_data,
filename=file_info.get('name', 'document'),
mimeType=file_info.get('mimeType', 'application/octet-stream'),
base64Encoded=False
)
all_extracted_content.append(extracted_content)
if includeMetadata:
file_infos.append(file_info)
if not all_extracted_content:
return self._createResult(
success=False,
data={},
error="No content could be extracted from any documents"
)
# Combine all extracted content
combined_content = "\n\n--- DOCUMENT SEPARATOR ---\n\n".join(all_extracted_content)
result_data = {
"documentCount": len(chatDocuments),
"format": format,
"content": combined_content,
"fileInfos": file_infos if includeMetadata else None,
"timestamp": datetime.now(UTC).isoformat()
}
return self._createResult( return self._createResult(
success=True, success=True,
data=content data={
"documentName": f"extracted_content_{datetime.now(UTC).strftime('%Y%m%d_%H%M%S')}.txt",
"documentData": result_data
}
) )
except Exception as e: except Exception as e:
logger.error(f"Error extracting content: {str(e)}") logger.error(f"Error extracting content: {str(e)}")
return self._createResult( return self._createResult(
@ -229,31 +126,102 @@ class MethodDocument(MethodBase):
Analyze document content Analyze document content
Parameters: Parameters:
fileId (str): The ID of the document to analyze documentList (str): Reference to the document list to analyze
aiPrompt (str): AI prompt for content analysis
analysis (List[str], optional): Types of analysis to perform (default: ["entities", "topics", "sentiment"]) analysis (List[str], optional): Types of analysis to perform (default: ["entities", "topics", "sentiment"])
""" """
try: try:
fileId = parameters.get("fileId") documentList = parameters.get("documentList")
aiPrompt = parameters.get("aiPrompt")
analysis = parameters.get("analysis", ["entities", "topics", "sentiment"]) analysis = parameters.get("analysis", ["entities", "topics", "sentiment"])
if not fileId: if not documentList:
return self._createResult( return self._createResult(
success=False, success=False,
data={}, data={},
error="File ID is required" error="Document list reference is required"
) )
# Analyze content if not aiPrompt:
results = await self.documentService.analyzeContent( return self._createResult(
fileId=fileId, success=False,
analysis=analysis data={},
error="AI prompt is required"
) )
chatDocuments = self.serviceContainer.getChatDocumentsFromDocumentReference(documentList)
if not chatDocuments:
return self._createResult(
success=False,
data={},
error="No documents found for the provided reference"
)
# Extract content from all documents
all_extracted_content = []
for chatDocument in chatDocuments:
fileId = chatDocument.fileId
file_data = self.serviceContainer.getFileData(fileId)
file_info = self.serviceContainer.getFileInfo(fileId)
if not file_data:
logger.warning(f"File not found or empty for fileId: {fileId}")
continue
extracted_content = await self.serviceContainer.extractContentFromFileData(
prompt=aiPrompt,
fileData=file_data,
filename=file_info.get('name', 'document'),
mimeType=file_info.get('mimeType', 'application/octet-stream'),
base64Encoded=False
)
all_extracted_content.append(extracted_content)
if not all_extracted_content:
return self._createResult(
success=False,
data={},
error="No content could be extracted from any documents"
)
# Combine all extracted content for analysis
combined_content = "\n\n--- DOCUMENT SEPARATOR ---\n\n".join(all_extracted_content)
analysis_prompt = f"""
Analyze this document content for the following aspects:
{', '.join(analysis)}
Document content:
{combined_content[:8000]} # Limit content length
Please provide a detailed analysis including:
1. Key entities (people, organizations, locations, dates)
2. Main topics and themes
3. Sentiment analysis (positive, negative, neutral)
4. Key insights and patterns
5. Important relationships between entities
6. Document structure and organization
"""
analysis_result = await self.serviceContainer.interfaceAiCalls.callAiTextAdvanced(analysis_prompt)
result_data = {
"documentCount": len(chatDocuments),
"analysis": analysis,
"results": analysis_result,
"content": combined_content,
"timestamp": datetime.now(UTC).isoformat()
}
return self._createResult( return self._createResult(
success=True, success=True,
data=results data={
"documentName": f"document_analysis_{datetime.now(UTC).strftime('%Y%m%d_%H%M%S')}.json",
"documentData": result_data
}
) )
except Exception as e: except Exception as e:
logger.error(f"Error analyzing content: {str(e)}") logger.error(f"Error analyzing content: {str(e)}")
return self._createResult( return self._createResult(
@ -268,34 +236,105 @@ class MethodDocument(MethodBase):
Summarize document content Summarize document content
Parameters: Parameters:
fileId (str): The ID of the document to summarize documentList (str): Reference to the document list to summarize
aiPrompt (str): AI prompt for content extraction
maxLength (int, optional): Maximum length of summary in words (default: 200) maxLength (int, optional): Maximum length of summary in words (default: 200)
format (str, optional): Output format (default: "text") format (str, optional): Output format (default: "text")
""" """
try: try:
fileId = parameters.get("fileId") documentList = parameters.get("documentList")
aiPrompt = parameters.get("aiPrompt")
maxLength = parameters.get("maxLength", 200) maxLength = parameters.get("maxLength", 200)
format = parameters.get("format", "text") format = parameters.get("format", "text")
if not fileId: if not documentList:
return self._createResult( return self._createResult(
success=False, success=False,
data={}, data={},
error="File ID is required" error="Document list reference is required"
) )
# Summarize content if not aiPrompt:
summary = await self.documentService.summarizeContent( return self._createResult(
fileId=fileId, success=False,
maxLength=maxLength, data={},
format=format error="AI prompt is required"
) )
chatDocuments = self.serviceContainer.getChatDocumentsFromDocumentReference(documentList)
if not chatDocuments:
return self._createResult(
success=False,
data={},
error="No documents found for the provided reference"
)
# Extract content from all documents
all_extracted_content = []
for chatDocument in chatDocuments:
fileId = chatDocument.fileId
file_data = self.serviceContainer.getFileData(fileId)
file_info = self.serviceContainer.getFileInfo(fileId)
if not file_data:
logger.warning(f"File not found or empty for fileId: {fileId}")
continue
extracted_content = await self.serviceContainer.extractContentFromFileData(
prompt=aiPrompt,
fileData=file_data,
filename=file_info.get('name', 'document'),
mimeType=file_info.get('mimeType', 'application/octet-stream'),
base64Encoded=False
)
all_extracted_content.append(extracted_content)
if not all_extracted_content:
return self._createResult(
success=False,
data={},
error="No content could be extracted from any documents"
)
# Combine all extracted content for summarization
combined_content = "\n\n--- DOCUMENT SEPARATOR ---\n\n".join(all_extracted_content)
summary_prompt = f"""
Create a comprehensive summary of this document content.
Document content:
{combined_content[:8000]} # Limit content length
Requirements:
- Maximum length: {maxLength} words
- Format: {format}
- Include key points and main ideas
- Maintain accuracy and completeness
- Use clear, professional language
- Highlight important insights and conclusions
"""
summary = await self.serviceContainer.interfaceAiCalls.callAiTextAdvanced(summary_prompt)
result_data = {
"documentCount": len(chatDocuments),
"maxLength": maxLength,
"format": format,
"summary": summary,
"wordCount": len(summary.split()),
"originalContent": combined_content,
"timestamp": datetime.now(UTC).isoformat()
}
return self._createResult( return self._createResult(
success=True, success=True,
data=summary data={
"documentName": f"document_summary_{datetime.now(UTC).strftime('%Y%m%d_%H%M%S')}.txt",
"documentData": result_data
}
) )
except Exception as e: except Exception as e:
logger.error(f"Error summarizing content: {str(e)}") logger.error(f"Error summarizing content: {str(e)}")
return self._createResult( return self._createResult(

461
modules/methods/methodExcel.py
View file

@ -1,461 +0,0 @@
"""
Excel method module.
Handles Excel operations using the Excel service.
"""
import logging
from typing import Dict, Any, List, Optional
from datetime import datetime, UTC
import json
import base64
from modules.workflow.methodBase import MethodBase, ActionResult, action
logger = logging.getLogger(__name__)
class ExcelService:
"""Service for Microsoft Excel operations using Graph API"""
def __init__(self, serviceContainer: Any):
self.serviceContainer = serviceContainer
def _getMicrosoftConnection(self, connectionReference: str) -> Optional[Dict[str, Any]]:
"""Get Microsoft connection from connection reference"""
try:
userConnection = self.serviceContainer.getUserConnectionFromConnectionReference(connectionReference)
if not userConnection or userConnection.authority != "msft" or userConnection.status != "active":
return None
# Get the corresponding token for this user and authority
token = self.serviceContainer.interfaceApp.getToken(userConnection.authority)
if not token:
logger.warning(f"No token found for user {userConnection.userId} and authority {userConnection.authority}")
return None
return {
"id": userConnection.id,
"accessToken": token.tokenAccess,
"refreshToken": token.tokenRefresh,
"scopes": ["Mail.ReadWrite", "User.Read"] # Default Microsoft scopes
}
except Exception as e:
logger.error(f"Error getting Microsoft connection: {str(e)}")
return None
async def readFile(self, fileId: str, connectionReference: str, sheetName: str = "Sheet1", range: str = None) -> Dict[str, Any]:
"""Read data from Excel file using Microsoft Graph API"""
try:
connection = self._getMicrosoftConnection(connectionReference)
if not connection:
return {
"error": "No valid Microsoft connection found for the provided connection reference",
"fileId": fileId,
"connectionReference": connectionReference
}
# Get file data from service container
file_data = self.serviceContainer.getFileData(fileId)
file_info = self.serviceContainer.getFileInfo(fileId)
if not file_data:
return {
"error": "File not found or empty",
"fileId": fileId
}
# For now, simulate Excel reading with AI analysis
# In a real implementation, you would use Microsoft Graph API
excel_prompt = f"""
Analyze this Excel file data and extract structured information.
File: {file_info.get('name', 'Unknown')}
Sheet: {sheetName}
Range: {range or 'All data'}
File content (first 5000 characters):
{file_data.decode('utf-8', errors='ignore')[:5000] if isinstance(file_data, bytes) else str(file_data)[:5000]}
Please extract:
1. All data from the specified sheet and range
2. Column headers and data types
3. Key metrics and calculations
4. Any charts or visualizations described
5. Summary statistics
Return the data in a structured JSON format.
"""
# Use AI to analyze Excel content
analysis_result = await self.serviceContainer.interfaceAiCalls.callAiTextAdvanced(excel_prompt)
return {
"fileId": fileId,
"sheetName": sheetName,
"range": range,
"data": analysis_result,
"fileInfo": file_info,
"connection": {
"id": connection["id"],
"authority": "microsoft",
"reference": connectionReference
}
}
except Exception as e:
logger.error(f"Error reading Excel file: {str(e)}")
return {
"error": str(e),
"fileId": fileId
}
async def writeFile(self, fileId: str, connectionReference: str, sheetName: str, data: Any, range: str = None) -> Dict[str, Any]:
"""Write data to Excel file using Microsoft Graph API"""
try:
connection = self._getMicrosoftConnection(connectionReference)
if not connection:
return {
"error": "No valid Microsoft connection found for the provided connection reference",
"fileId": fileId,
"connectionReference": connectionReference
}
# For now, simulate Excel writing
# In a real implementation, you would use Microsoft Graph API
write_prompt = f"""
Prepare data for writing to Excel file.
File: {fileId}
Sheet: {sheetName}
Range: {range or 'Auto-detect'}
Data to write:
{json.dumps(data, indent=2)}
Please format this data appropriately for Excel and provide:
1. Structured data ready for Excel
2. Column headers and formatting
3. Any formulas or calculations needed
4. Data validation rules if applicable
"""
# Use AI to prepare Excel data
prepared_data = await self.serviceContainer.interfaceAiCalls.callAiTextAdvanced(write_prompt)
return {
"fileId": fileId,
"sheetName": sheetName,
"range": range,
"data": prepared_data,
"status": "prepared",
"connection": {
"id": connection["id"],
"authority": "microsoft",
"reference": connectionReference
}
}
except Exception as e:
logger.error(f"Error writing to Excel file: {str(e)}")
return {
"error": str(e),
"fileId": fileId
}
async def createFile(self, fileName: str, connectionReference: str, template: str = None) -> Dict[str, Any]:
"""Create new Excel file using Microsoft Graph API"""
try:
connection = self._getMicrosoftConnection(connectionReference)
if not connection:
return {
"error": "No valid Microsoft connection found for the provided connection reference",
"connectionReference": connectionReference
}
# For now, simulate file creation
# In a real implementation, you would use Microsoft Graph API
create_prompt = f"""
Create a new Excel file structure.
File name: {fileName}
Template: {template or 'Standard'}
Please provide:
1. Initial sheet structure
2. Default column headers
3. Sample data if template specified
4. Formatting guidelines
"""
# Use AI to create Excel structure
file_structure = await self.serviceContainer.interfaceAiCalls.callAiTextAdvanced(create_prompt)
# Create file using service container
file_id = self.serviceContainer.createFile(
fileName=fileName,
mimeType="application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
content=file_structure,
base64encoded=False
)
return {
"fileId": file_id,
"fileName": fileName,
"template": template,
"structure": file_structure,
"connection": {
"id": connection["id"],
"authority": "microsoft",
"reference": connectionReference
}
}
except Exception as e:
logger.error(f"Error creating Excel file: {str(e)}")
return {
"error": str(e)
}
async def formatCells(self, fileId: str, connectionReference: str, sheetName: str, range: str, format: Dict[str, Any]) -> Dict[str, Any]:
"""Format Excel cells using Microsoft Graph API"""
try:
connection = self._getMicrosoftConnection(connectionReference)
if not connection:
return {
"error": "No valid Microsoft connection found for the provided connection reference",
"fileId": fileId,
"connectionReference": connectionReference
}
# For now, simulate formatting
# In a real implementation, you would use Microsoft Graph API
format_prompt = f"""
Apply formatting to Excel cells.
File: {fileId}
Sheet: {sheetName}
Range: {range}
Format: {json.dumps(format, indent=2)}
Please provide:
1. Applied formatting details
2. Visual representation of the formatting
3. Any conditional formatting rules
"""
# Use AI to describe formatting
formatting_result = await self.serviceContainer.interfaceAiCalls.callAiTextAdvanced(format_prompt)
return {
"fileId": fileId,
"sheetName": sheetName,
"range": range,
"format": format,
"result": formatting_result,
"connection": {
"id": connection["id"],
"authority": "microsoft",
"reference": connectionReference
}
}
except Exception as e:
logger.error(f"Error formatting Excel cells: {str(e)}")
return {
"error": str(e),
"fileId": fileId
}
class MethodExcel(MethodBase):
"""Excel method implementation for spreadsheet operations"""
def __init__(self, serviceContainer: Any):
"""Initialize the Excel method"""
super().__init__(serviceContainer)
self.name = "excel"
self.description = "Handle Excel spreadsheet operations like reading and writing data"
self.excelService = ExcelService(serviceContainer)
@action
async def read(self, parameters: Dict[str, Any]) -> ActionResult:
"""
Read data from Excel file
Parameters:
fileId (str): The ID of the Excel file to read
connectionReference (str): Reference to the Microsoft connection
sheetName (str, optional): Name of the sheet to read (default: "Sheet1")
range (str, optional): Excel range to read (e.g., "A1:D10")
includeHeaders (bool, optional): Whether to include column headers (default: True)
"""
try:
fileId = parameters.get("fileId")
connectionReference = parameters.get("connectionReference")
sheetName = parameters.get("sheetName", "Sheet1")
range = parameters.get("range")
includeHeaders = parameters.get("includeHeaders", True)
if not fileId or not connectionReference:
return self._createResult(
success=False,
data={},
error="File ID and connection reference are required"
)
# Read data from Excel
data = await self.excelService.readFile(
fileId=fileId,
connectionReference=connectionReference,
sheetName=sheetName,
range=range
)
return self._createResult(
success=True,
data=data
)
except Exception as e:
logger.error(f"Error reading Excel file: {str(e)}")
return self._createResult(
success=False,
data={},
error=str(e)
)
@action
async def write(self, parameters: Dict[str, Any]) -> ActionResult:
"""
Write data to Excel file
Parameters:
fileId (str): The ID of the Excel file to write to
connectionReference (str): Reference to the Microsoft connection
sheetName (str, optional): Name of the sheet to write to (default: "Sheet1")
data (Any): Data to write to the Excel file
range (str, optional): Excel range to write to (e.g., "A1:D10")
"""
try:
fileId = parameters.get("fileId")
connectionReference = parameters.get("connectionReference")
sheetName = parameters.get("sheetName", "Sheet1")
data = parameters.get("data")
range = parameters.get("range")
if not fileId or not connectionReference or not data:
return self._createResult(
success=False,
data={},
error="File ID, connection reference, and data are required"
)
# Write data to Excel
result = await self.excelService.writeFile(
fileId=fileId,
connectionReference=connectionReference,
sheetName=sheetName,
data=data,
range=range
)
return self._createResult(
success=True,
data=result
)
except Exception as e:
logger.error(f"Error writing to Excel file: {str(e)}")
return self._createResult(
success=False,
data={},
error=str(e)
)
@action
async def create(self, parameters: Dict[str, Any]) -> ActionResult:
"""
Create new Excel file
Parameters:
fileName (str): Name of the new Excel file
connectionReference (str): Reference to the Microsoft connection
template (str, optional): Template to use for the new file
"""
try:
fileName = parameters.get("fileName")
connectionReference = parameters.get("connectionReference")
template = parameters.get("template")
if not fileName or not connectionReference:
return self._createResult(
success=False,
data={},
error="File name and connection reference are required"
)
# Create Excel file
fileId = await self.excelService.createFile(
fileName=fileName,
connectionReference=connectionReference,
template=template
)
return self._createResult(
success=True,
data={"fileId": fileId}
)
except Exception as e:
logger.error(f"Error creating Excel file: {str(e)}")
return self._createResult(
success=False,
data={},
error=str(e)
)
@action
async def format(self, parameters: Dict[str, Any]) -> ActionResult:
"""
Format Excel cells
Parameters:
fileId (str): The ID of the Excel file to format
connectionReference (str): Reference to the Microsoft connection
sheetName (str, optional): Name of the sheet to format (default: "Sheet1")
range (str): Excel range to format (e.g., "A1:D10")
format (Dict[str, Any]): Formatting options (e.g., {"font": {"bold": True}})
"""
try:
fileId = parameters.get("fileId")
connectionReference = parameters.get("connectionReference")
sheetName = parameters.get("sheetName", "Sheet1")
range = parameters.get("range")
format = parameters.get("format")
if not fileId or not connectionReference or not range or not format:
return self._createResult(
success=False,
data={},
error="File ID, connection reference, range, and format are required"
)
# Apply formatting
result = await self.excelService.formatCells(
fileId=fileId,
connectionReference=connectionReference,
sheetName=sheetName,
range=range,
format=format
)
return self._createResult(
success=True,
data=result
)
except Exception as e:
logger.error(f"Error formatting Excel cells: {str(e)}")
return self._createResult(
success=False,
data={},
error=str(e)
)

439
modules/methods/methodOperator.py
View file

@ -3,184 +3,119 @@
from typing import Dict, List, Any, Optional from typing import Dict, List, Any, Optional
from datetime import datetime, UTC from datetime import datetime, UTC
import logging import logging
import uuid
from modules.workflow.methodBase import MethodBase, ActionResult, action from modules.workflow.methodBase import MethodBase, ActionResult, action
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
class OperatorService:
"""Service for operator operations like forEach and AI calls"""
def __init__(self, serviceContainer: Any):
self.serviceContainer = serviceContainer
async def executeForEach(self, items: List[Any], action: Dict[str, Any]) -> List[Any]:
"""Execute an action for each item in a list"""
try:
results = []
for i, item in enumerate(items):
logger.info(f"Executing forEach action {i+1}/{len(items)}")
# Create context with current item
context = {
"item": item,
"index": i,
"total": len(items),
"isFirst": i == 0,
"isLast": i == len(items) - 1
}
# Execute the action using the service container
if "method" in action and "action" in action:
methodName = action["method"]
actionName = action["action"]
parameters = action.get("parameters", {})
# Add context to parameters
parameters["context"] = context
parameters["currentItem"] = item
# Execute the method action
result = await self.serviceContainer.executeAction(
methodName=methodName,
actionName=actionName,
parameters=parameters
)
# Return the exact result data, not wrapped
if result.success:
results.append(result.data)
else:
results.append({"error": result.error})
else:
# Simple action without method call
results.append({"error": "No method specified"})
return results
except Exception as e:
logger.error(f"Error executing forEach: {str(e)}")
return [{"error": str(e)}] * len(items) if items else []
async def executeAiCall(self, prompt: str, documents: List[Dict[str, Any]] = None) -> Dict[str, Any]:
"""Call AI service with document content"""
try:
# Prepare context from documents
context = ""
extractedDocuments = []
if documents:
for i, doc in enumerate(documents):
documentReference = doc.get('documentReference')
contentExtractionPrompt = doc.get('contentExtractionPrompt', 'Extract the main content from this document')
if documentReference:
# Get documents from reference
chatDocuments = self.serviceContainer.getChatDocumentsFromDocumentReference(documentReference)
if chatDocuments:
# Extract content from each document
for j, chatDoc in enumerate(chatDocuments):
try:
# Extract content using the document manager
extractedContent = await self.serviceContainer.documentManager.extractContentFromChatDocument(
chatDocument=chatDoc,
extractionPrompt=contentExtractionPrompt
)
extractedDocuments.append({
"documentReference": documentReference,
"documentId": chatDoc.id,
"extractionPrompt": contentExtractionPrompt,
"extractedContent": extractedContent
})
# Add to context
context += f"\n\nDocument {len(extractedDocuments)} (from {documentReference}):\n{extractedContent}"
except Exception as e:
logger.warning(f"Error extracting content from document {chatDoc.id}: {str(e)}")
extractedDocuments.append({
"documentReference": documentReference,
"documentId": chatDoc.id,
"extractionPrompt": contentExtractionPrompt,
"extractedContent": f"Error extracting content: {str(e)}"
})
else:
logger.warning(f"No documents found for reference: {documentReference}")
extractedDocuments.append({
"documentReference": documentReference,
"extractionPrompt": contentExtractionPrompt,
"extractedContent": f"No documents found for reference: {documentReference}"
})
# Create full prompt with context
fullPrompt = f"{prompt}\n\nContext:\n{context}" if context else prompt
# Call AI service
aiResponse = await self.serviceContainer.interfaceAiCalls.callAiTextAdvanced(fullPrompt)
return {
"prompt": prompt,
"documentsProcessed": len(extractedDocuments),
"extractedDocuments": extractedDocuments,
"response": aiResponse,
"timestamp": datetime.now(UTC).isoformat()
}
except Exception as e:
logger.error(f"Error executing AI call: {str(e)}")
return {
"error": str(e),
"prompt": prompt,
"documentsProcessed": 0,
"extractedDocuments": [],
"response": None
}
class MethodOperator(MethodBase): class MethodOperator(MethodBase):
"""Operator method implementation for handling collections and AI operations""" """Operator method implementation for data operations"""
def __init__(self, serviceContainer: Any): def __init__(self, serviceContainer: Any):
super().__init__(serviceContainer) super().__init__(serviceContainer)
self.name = "operator" self.name = "operator"
self.description = "Handle operations like forEach and AI calls" self.description = "Handle data operations like filtering, sorting, and transformation"
self.operatorService = OperatorService(serviceContainer)
@action @action
async def forEach(self, parameters: Dict[str, Any]) -> ActionResult: async def filter(self, parameters: Dict[str, Any]) -> ActionResult:
""" """
Execute an action for each item in a list Filter data based on criteria
Parameters: Parameters:
items (List[Any]): List of items to process documentList (str): Reference to the document list to filter
action (Dict[str, Any]): Action to execute for each item (contains method, action, parameters) criteria (Dict[str, Any]): Filter criteria
field (str, optional): Field to filter on
""" """
try: try:
items = parameters.get("items", []) documentList = parameters.get("documentList")
action = parameters.get("action", {}) criteria = parameters.get("criteria")
field = parameters.get("field")
if not items or not action: if not documentList or not criteria:
return self._createResult( return self._createResult(
success=False, success=False,
data={}, data={},
error="Items and action are required" error="Document list reference and criteria are required"
) )
# Execute forEach operation chatDocuments = self.serviceContainer.getChatDocumentsFromDocumentReference(documentList)
results = await self.operatorService.executeForEach( if not chatDocuments:
items=items, return self._createResult(
action=action success=False,
data={},
error="No documents found for the provided reference"
) )
# Extract content from all documents
all_document_content = []
for chatDocument in chatDocuments:
fileId = chatDocument.fileId
file_data = self.serviceContainer.getFileData(fileId)
file_info = self.serviceContainer.getFileInfo(fileId)
if not file_data:
logger.warning(f"File data not found for fileId: {fileId}")
continue
all_document_content.append({
"fileId": fileId,
"fileName": file_info.get('name', 'unknown'),
"content": file_data
})
if not all_document_content:
return self._createResult(
success=False,
data={},
error="No content could be extracted from any documents"
)
# Combine all document content for filtering
combined_content = "\n\n--- DOCUMENT SEPARATOR ---\n\n".join([
f"File: {doc['fileName']}\nContent: {doc['content']}"
for doc in all_document_content
])
filter_prompt = f"""
Filter the following data based on the specified criteria.
Data to filter:
{combined_content}
Filter criteria:
{criteria}
Field to filter on: {field or 'All fields'}
Please provide:
1. Filtered data that matches the criteria
2. Summary of filtering results
3. Number of items before and after filtering
4. Any data quality insights
"""
filtered_result = await self.serviceContainer.interfaceAiCalls.callAiTextAdvanced(filter_prompt)
result_data = {
"documentCount": len(chatDocuments),
"criteria": criteria,
"field": field,
"filteredData": filtered_result,
"originalCount": len(all_document_content),
"timestamp": datetime.now(UTC).isoformat()
}
return self._createResult( return self._createResult(
success=True, success=True,
data=results data={
"documentName": f"filtered_data_{datetime.now(UTC).strftime('%Y%m%d_%H%M%S')}.json",
"documentData": result_data
}
) )
except Exception as e: except Exception as e:
logger.error(f"Error in forEach execution: {str(e)}") logger.error(f"Error filtering data: {str(e)}")
return self._createResult( return self._createResult(
success=False, success=False,
data={}, data={},
@ -188,39 +123,215 @@ class MethodOperator(MethodBase):
) )
@action @action
async def aiCall(self, parameters: Dict[str, Any]) -> ActionResult: async def sort(self, parameters: Dict[str, Any]) -> ActionResult:
""" """
Call AI service with document content Sort data by specified field
Parameters: Parameters:
prompt (str): The prompt to send to the AI service documentList (str): Reference to the document list to sort
documents (List[Dict[str, Any]], optional): List of documents to include in context field (str): Field to sort by
Each document should have: documentReference (str), contentExtractionPrompt (str, optional) order (str, optional): Sort order (asc/desc, default: "asc")
""" """
try: try:
prompt = parameters.get("prompt") documentList = parameters.get("documentList")
documents = parameters.get("documents", []) # List of {documentReference, contentExtractionPrompt} field = parameters.get("field")
order = parameters.get("order", "asc")
if not prompt: if not documentList or not field:
return self._createResult( return self._createResult(
success=False, success=False,
data={}, data={},
error="Prompt is required" error="Document list reference and field are required"
) )
# Execute AI call # Get documents from reference
result = await self.operatorService.executeAiCall( chatDocuments = self.serviceContainer.getChatDocumentsFromDocumentReference(documentList)
prompt=prompt, if not chatDocuments:
documents=documents return self._createResult(
success=False,
data={},
error="No documents found for the provided reference"
) )
# Extract content from all documents
all_document_content = []
for chatDocument in chatDocuments:
fileId = chatDocument.fileId
file_data = self.serviceContainer.getFileData(fileId)
file_info = self.serviceContainer.getFileInfo(fileId)
if not file_data:
logger.warning(f"File data not found for fileId: {fileId}")
continue
all_document_content.append({
"fileId": fileId,
"fileName": file_info.get('name', 'unknown'),
"content": file_data
})
if not all_document_content:
return self._createResult(
success=False,
data={},
error="No content could be extracted from any documents"
)
# Combine all document content for sorting
combined_content = "\n\n--- DOCUMENT SEPARATOR ---\n\n".join([
f"File: {doc['fileName']}\nContent: {doc['content']}"
for doc in all_document_content
])
# Create sorting prompt
sort_prompt = f"""
Sort the following data by the specified field.
Data to sort:
{combined_content}
Sort field: {field}
Sort order: {order}
Please provide:
1. Sorted data in the specified order
2. Summary of sorting results
3. Any data insights from the sorting
4. Validation of sort field existence
"""
# Use AI to perform sorting
sorted_result = await self.serviceContainer.interfaceAiCalls.callAiTextAdvanced(sort_prompt)
# Create result data
result_data = {
"documentCount": len(chatDocuments),
"field": field,
"order": order,
"sortedData": sorted_result,
"timestamp": datetime.now(UTC).isoformat()
}
return self._createResult( return self._createResult(
success=True, success=True,
data=result data={
"documentName": f"sorted_data_{datetime.now(UTC).strftime('%Y%m%d_%H%M%S')}.json",
"documentData": result_data
}
) )
except Exception as e: except Exception as e:
logger.error(f"Error in AI call execution: {str(e)}") logger.error(f"Error sorting data: {str(e)}")
return self._createResult(
success=False,
data={},
error=str(e)
)
@action
async def transform(self, parameters: Dict[str, Any]) -> ActionResult:
"""
Transform data structure or format
Parameters:
documentList (str): Reference to the document list to transform
transformation (Dict[str, Any]): Transformation rules
outputFormat (str, optional): Desired output format
"""
try:
documentList = parameters.get("documentList")
transformation = parameters.get("transformation")
outputFormat = parameters.get("outputFormat", "json")
if not documentList or not transformation:
return self._createResult(
success=False,
data={},
error="Document list reference and transformation rules are required"
)
# Get documents from reference
chatDocuments = self.serviceContainer.getChatDocumentsFromDocumentReference(documentList)
if not chatDocuments:
return self._createResult(
success=False,
data={},
error="No documents found for the provided reference"
)
# Extract content from all documents
all_document_content = []
for chatDocument in chatDocuments:
fileId = chatDocument.fileId
file_data = self.serviceContainer.getFileData(fileId)
file_info = self.serviceContainer.getFileInfo(fileId)
if not file_data:
logger.warning(f"File data not found for fileId: {fileId}")
continue
all_document_content.append({
"fileId": fileId,
"fileName": file_info.get('name', 'unknown'),
"content": file_data
})
if not all_document_content:
return self._createResult(
success=False,
data={},
error="No content could be extracted from any documents"
)
# Combine all document content for transformation
combined_content = "\n\n--- DOCUMENT SEPARATOR ---\n\n".join([
f"File: {doc['fileName']}\nContent: {doc['content']}"
for doc in all_document_content
])
# Create transformation prompt
transform_prompt = f"""
Transform the following data according to the specified rules.
Data to transform:
{combined_content}
Transformation rules:
{transformation}
Output format: {outputFormat}
Please provide:
1. Transformed data in the specified format
2. Summary of transformation results
3. Validation of transformation rules
4. Any data quality improvements
"""
# Use AI to perform transformation
transformed_result = await self.serviceContainer.interfaceAiCalls.callAiTextAdvanced(transform_prompt)
# Create result data
result_data = {
"documentCount": len(chatDocuments),
"transformation": transformation,
"outputFormat": outputFormat,
"transformedData": transformed_result,
"timestamp": datetime.now(UTC).isoformat()
}
return self._createResult(
success=True,
data={
"documentName": f"transformed_data_{datetime.now(UTC).strftime('%Y%m%d_%H%M%S')}.{outputFormat}",
"documentData": result_data
}
)
except Exception as e:
logger.error(f"Error transforming data: {str(e)}")
return self._createResult( return self._createResult(
success=False, success=False,
data={}, data={},

474
modules/methods/methodOutlook.py
View file

@ -7,16 +7,20 @@ import logging
from typing import Dict, Any, List, Optional from typing import Dict, Any, List, Optional
from datetime import datetime, UTC from datetime import datetime, UTC
import json import json
import uuid
from modules.workflow.methodBase import MethodBase, ActionResult, action from modules.workflow.methodBase import MethodBase, ActionResult, action
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
class OutlookService: class MethodOutlook(MethodBase):
"""Service for Microsoft Outlook operations using Graph API""" """Outlook method implementation for email operations"""
def __init__(self, serviceContainer: Any): def __init__(self, serviceContainer: Any):
self.serviceContainer = serviceContainer """Initialize the Outlook method"""
super().__init__(serviceContainer)
self.name = "outlook"
self.description = "Handle Microsoft Outlook email operations"
def _getMicrosoftConnection(self, connectionReference: str) -> Optional[Dict[str, Any]]: def _getMicrosoftConnection(self, connectionReference: str) -> Optional[Dict[str, Any]]:
"""Get Microsoft connection from connection reference""" """Get Microsoft connection from connection reference"""
@ -41,226 +45,22 @@ class OutlookService:
logger.error(f"Error getting Microsoft connection: {str(e)}") logger.error(f"Error getting Microsoft connection: {str(e)}")
return None return None
async def readMails(self, connectionReference: str, folder: str = "inbox", query: str = None, maxResults: int = 10, includeAttachments: bool = False) -> Dict[str, Any]:
"""Read emails from Outlook using Microsoft Graph API"""
try:
connection = self._getMicrosoftConnection(connectionReference)
if not connection:
return {
"error": "No valid Microsoft connection found for the provided connection reference",
"connectionReference": connectionReference
}
# For now, simulate email reading
# In a real implementation, you would use Microsoft Graph API
mail_prompt = f"""
Read emails from Outlook.
Folder: {folder}
Query: {query or 'All emails'}
Max Results: {maxResults}
Include Attachments: {includeAttachments}
Please provide:
1. Email messages with subject, sender, and content
2. Timestamps and priority levels
3. Attachment information if requested
4. Email threading and conversations
5. Categorization and flags
"""
# Use AI to simulate email data
mail_data = await self.serviceContainer.interfaceAiCalls.callAiTextAdvanced(mail_prompt)
return {
"folder": folder,
"query": query,
"maxResults": maxResults,
"includeAttachments": includeAttachments,
"messages": mail_data,
"connection": {
"id": connection["id"],
"authority": "microsoft",
"reference": connectionReference
}
}
except Exception as e:
logger.error(f"Error reading emails: {str(e)}")
return {
"error": str(e)
}
async def sendMail(self, connectionReference: str, to: List[str], subject: str, body: str, attachments: List[str] = None) -> Dict[str, Any]:
"""Send email using Outlook using Microsoft Graph API"""
try:
connection = self._getMicrosoftConnection(connectionReference)
if not connection:
return {
"error": "No valid Microsoft connection found for the provided connection reference",
"connectionReference": connectionReference
}
# For now, simulate email sending
# In a real implementation, you would use Microsoft Graph API
send_prompt = f"""
Send email using Outlook.
To: {', '.join(to)}
Subject: {subject}
Body: {body}
Attachments: {attachments or 'None'}
Please provide:
1. Email composition details
2. Recipient validation
3. Attachment processing
4. Send confirmation
5. Message tracking information
"""
# Use AI to simulate email sending
send_result = await self.serviceContainer.interfaceAiCalls.callAiTextAdvanced(send_prompt)
return {
"to": to,
"subject": subject,
"body": body,
"attachments": attachments,
"result": send_result,
"connection": {
"id": connection["id"],
"authority": "microsoft",
"reference": connectionReference
}
}
except Exception as e:
logger.error(f"Error sending email: {str(e)}")
return {
"error": str(e)
}
async def createFolder(self, connectionReference: str, name: str, parentFolderId: str = None) -> Dict[str, Any]:
"""Create folder in Outlook using Microsoft Graph API"""
try:
connection = self._getMicrosoftConnection(connectionReference)
if not connection:
return {
"error": "No valid Microsoft connection found for the provided connection reference",
"connectionReference": connectionReference
}
# For now, simulate folder creation
# In a real implementation, you would use Microsoft Graph API
folder_prompt = f"""
Create folder in Outlook.
Name: {name}
Parent Folder ID: {parentFolderId or 'Root'}
Please provide:
1. Folder creation details
2. Permission settings
3. Folder structure and hierarchy
4. Creation confirmation
5. Folder properties and metadata
"""
# Use AI to simulate folder creation
folder_result = await self.serviceContainer.interfaceAiCalls.callAiTextAdvanced(folder_prompt)
return {
"name": name,
"parentFolderId": parentFolderId,
"result": folder_result,
"connection": {
"id": connection["id"],
"authority": "microsoft",
"reference": connectionReference
}
}
except Exception as e:
logger.error(f"Error creating folder: {str(e)}")
return {
"error": str(e)
}
async def moveMail(self, connectionReference: str, messageId: str, targetFolderId: str) -> Dict[str, Any]:
"""Move email to different folder using Microsoft Graph API"""
try:
connection = self._getMicrosoftConnection(connectionReference)
if not connection:
return {
"error": "No valid Microsoft connection found for the provided connection reference",
"connectionReference": connectionReference
}
# For now, simulate mail moving
# In a real implementation, you would use Microsoft Graph API
move_prompt = f"""
Move email to different folder.
Message ID: {messageId}
Target Folder ID: {targetFolderId}
Please provide:
1. Move operation details
2. Source and destination folder information
3. Message preservation and metadata
4. Move confirmation
5. Updated folder structure
"""
# Use AI to simulate mail moving
move_result = await self.serviceContainer.interfaceAiCalls.callAiTextAdvanced(move_prompt)
return {
"messageId": messageId,
"targetFolderId": targetFolderId,
"result": move_result,
"connection": {
"id": connection["id"],
"authority": "microsoft",
"reference": connectionReference
}
}
except Exception as e:
logger.error(f"Error moving email: {str(e)}")
return {
"error": str(e)
}
class MethodOutlook(MethodBase):
"""Outlook method implementation for email operations"""
def __init__(self, serviceContainer: Any):
"""Initialize the Outlook method"""
super().__init__(serviceContainer)
self.name = "outlook"
self.description = "Handle Outlook email operations like reading and sending emails"
self.outlookService = OutlookService(serviceContainer)
@action @action
async def readMails(self, parameters: Dict[str, Any]) -> ActionResult: async def readEmails(self, parameters: Dict[str, Any]) -> ActionResult:
""" """
Read emails from Outlook Read emails from Outlook
Parameters: Parameters:
connectionReference (str): Reference to the Microsoft connection connectionReference (str): Reference to the Microsoft connection
folder (str, optional): Folder to read from (default: "inbox") folder (str, optional): Email folder to read from (default: "Inbox")
query (str, optional): Search query to filter emails limit (int, optional): Maximum number of emails to read (default: 10)
maxResults (int, optional): Maximum number of results (default: 10) filter (str, optional): Filter criteria for emails
includeAttachments (bool, optional): Whether to include attachments (default: False)
""" """
try: try:
connectionReference = parameters.get("connectionReference") connectionReference = parameters.get("connectionReference")
folder = parameters.get("folder", "inbox") folder = parameters.get("folder", "Inbox")
query = parameters.get("query") limit = parameters.get("limit", 10)
maxResults = parameters.get("maxResults", 10) filter = parameters.get("filter")
includeAttachments = parameters.get("includeAttachments", False)
if not connectionReference: if not connectionReference:
return self._createResult( return self._createResult(
@ -269,18 +69,55 @@ class MethodOutlook(MethodBase):
error="Connection reference is required" error="Connection reference is required"
) )
# Read emails # Get Microsoft connection
messages = await self.outlookService.readMails( connection = self._getMicrosoftConnection(connectionReference)
connectionReference=connectionReference, if not connection:
folder=folder, return self._createResult(
query=query, success=False,
maxResults=maxResults, data={},
includeAttachments=includeAttachments error="No valid Microsoft connection found for the provided connection reference"
) )
# Create email reading prompt
email_prompt = f"""
Simulate reading emails from Microsoft Outlook.
Connection: {connection['id']}
Folder: {folder}
Limit: {limit}
Filter: {filter or 'None'}
Please provide:
1. List of emails with subject, sender, date, and content
2. Summary of email statistics
3. Important or urgent emails highlighted
4. Email categorization if possible
"""
# Use AI to simulate email reading
email_data = await self.serviceContainer.interfaceAiCalls.callAiTextAdvanced(email_prompt)
# Create result data
result_data = {
"connectionReference": connectionReference,
"folder": folder,
"limit": limit,
"filter": filter,
"emails": email_data,
"connection": {
"id": connection["id"],
"authority": "microsoft",
"reference": connectionReference
},
"timestamp": datetime.now(UTC).isoformat()
}
return self._createResult( return self._createResult(
success=True, success=True,
data=messages data={
"documentName": f"outlook_emails_{datetime.now(UTC).strftime('%Y%m%d_%H%M%S')}.json",
"documentData": result_data
}
) )
except Exception as e: except Exception as e:
@ -292,50 +129,86 @@ class MethodOutlook(MethodBase):
) )
@action @action
async def sendMail(self, parameters: Dict[str, Any]) -> ActionResult: async def sendEmail(self, parameters: Dict[str, Any]) -> ActionResult:
""" """
Send email using Outlook Send email via Outlook
Parameters: Parameters:
connectionReference (str): Reference to the Microsoft connection connectionReference (str): Reference to the Microsoft connection
to (List[str]): List of recipient email addresses to (List[str]): List of recipient email addresses
subject (str): Email subject subject (str): Email subject
body (str): Email body body (str): Email body content
attachments (List[str], optional): List of attachment file IDs cc (List[str], optional): CC recipients
bcc (List[str], optional): BCC recipients
""" """
try: try:
connectionReference = parameters.get("connectionReference") connectionReference = parameters.get("connectionReference")
to = parameters.get("to", []) to = parameters.get("to")
subject = parameters.get("subject") subject = parameters.get("subject")
body = parameters.get("body") body = parameters.get("body")
attachments = parameters.get("attachments", []) cc = parameters.get("cc", [])
bcc = parameters.get("bcc", [])
if not connectionReference: if not connectionReference or not to or not subject or not body:
return self._createResult( return self._createResult(
success=False, success=False,
data={}, data={},
error="Connection reference is required" error="Connection reference, to, subject, and body are required"
) )
if not to or not subject or not body: # Get Microsoft connection
connection = self._getMicrosoftConnection(connectionReference)
if not connection:
return self._createResult( return self._createResult(
success=False, success=False,
data={}, data={},
error="To, subject, and body are required" error="No valid Microsoft connection found for the provided connection reference"
) )
# Send email # Create email sending prompt
result = await self.outlookService.sendMail( send_prompt = f"""
connectionReference=connectionReference, Simulate sending an email via Microsoft Outlook.
to=to,
subject=subject, Connection: {connection['id']}
body=body, To: {to}
attachments=attachments Subject: {subject}
) Body: {body}
CC: {cc}
BCC: {bcc}
Please provide:
1. Email composition details
2. Validation of email addresses
3. Email formatting and structure
4. Delivery confirmation simulation
"""
# Use AI to simulate email sending
send_result = await self.serviceContainer.interfaceAiCalls.callAiTextAdvanced(send_prompt)
# Create result data
result_data = {
"connectionReference": connectionReference,
"to": to,
"subject": subject,
"body": body,
"cc": cc,
"bcc": bcc,
"sendResult": send_result,
"connection": {
"id": connection["id"],
"authority": "microsoft",
"reference": connectionReference
},
"timestamp": datetime.now(UTC).isoformat()
}
return self._createResult( return self._createResult(
success=True, success=True,
data=result data={
"documentName": f"outlook_email_sent_{datetime.now(UTC).strftime('%Y%m%d_%H%M%S')}.json",
"documentData": result_data
}
) )
except Exception as e: except Exception as e:
@ -347,97 +220,82 @@ class MethodOutlook(MethodBase):
) )
@action @action
async def createFolder(self, parameters: Dict[str, Any]) -> ActionResult: async def searchEmails(self, parameters: Dict[str, Any]) -> ActionResult:
""" """
Create folder in Outlook Search emails in Outlook
Parameters: Parameters:
connectionReference (str): Reference to the Microsoft connection connectionReference (str): Reference to the Microsoft connection
name (str): Folder name query (str): Search query
parentFolderId (str, optional): Parent folder ID folder (str, optional): Folder to search in (default: "All")
limit (int, optional): Maximum number of results (default: 20)
""" """
try: try:
connectionReference = parameters.get("connectionReference") connectionReference = parameters.get("connectionReference")
name = parameters.get("name") query = parameters.get("query")
parentFolderId = parameters.get("parentFolderId") folder = parameters.get("folder", "All")
limit = parameters.get("limit", 20)
if not connectionReference: if not connectionReference or not query:
return self._createResult( return self._createResult(
success=False, success=False,
data={}, data={},
error="Connection reference is required" error="Connection reference and query are required"
) )
if not name: # Get Microsoft connection
connection = self._getMicrosoftConnection(connectionReference)
if not connection:
return self._createResult( return self._createResult(
success=False, success=False,
data={}, data={},
error="Folder name is required" error="No valid Microsoft connection found for the provided connection reference"
) )
# Create folder # Create email search prompt
folder = await self.outlookService.createFolder( search_prompt = f"""
connectionReference=connectionReference, Simulate searching emails in Microsoft Outlook.
name=name,
parentFolderId=parentFolderId Connection: {connection['id']}
) Query: {query}
Folder: {folder}
Limit: {limit}
Please provide:
1. Search results with relevant emails
2. Search statistics and relevance scores
3. Email previews and key information
4. Search suggestions and refinements
"""
# Use AI to simulate email search
search_result = await self.serviceContainer.interfaceAiCalls.callAiTextAdvanced(search_prompt)
# Create result data
result_data = {
"connectionReference": connectionReference,
"query": query,
"folder": folder,
"limit": limit,
"searchResults": search_result,
"connection": {
"id": connection["id"],
"authority": "microsoft",
"reference": connectionReference
},
"timestamp": datetime.now(UTC).isoformat()
}
return self._createResult( return self._createResult(
success=True, success=True,
data=folder data={
"documentName": f"outlook_email_search_{datetime.now(UTC).strftime('%Y%m%d_%H%M%S')}.json",
"documentData": result_data
}
) )
except Exception as e: except Exception as e:
logger.error(f"Error creating folder: {str(e)}") logger.error(f"Error searching emails: {str(e)}")
return self._createResult(
success=False,
data={},
error=str(e)
)
@action
async def moveMail(self, parameters: Dict[str, Any]) -> ActionResult:
"""
Move email to different folder
Parameters:
connectionReference (str): Reference to the Microsoft connection
messageId (str): ID of the message to move
targetFolderId (str): ID of the target folder
"""
try:
connectionReference = parameters.get("connectionReference")
messageId = parameters.get("messageId")
targetFolderId = parameters.get("targetFolderId")
if not connectionReference:
return self._createResult(
success=False,
data={},
error="Connection reference is required"
)
if not messageId or not targetFolderId:
return self._createResult(
success=False,
data={},
error="Message ID and target folder ID are required"
)
# Move email
result = await self.outlookService.moveMail(
connectionReference=connectionReference,
messageId=messageId,
targetFolderId=targetFolderId
)
return self._createResult(
success=True,
data=result
)
except Exception as e:
logger.error(f"Error moving email: {str(e)}")
return self._createResult( return self._createResult(
success=False, success=False,
data={}, data={},

639
modules/methods/methodPowerpoint.py
View file

@ -1,639 +0,0 @@
"""
PowerPoint method module.
Handles PowerPoint operations using the PowerPoint service.
"""
import logging
from typing import Dict, Any, List, Optional
from datetime import datetime, UTC
import json
import base64
from modules.workflow.methodBase import MethodBase, ActionResult, action
logger = logging.getLogger(__name__)
class PowerpointService:
"""Service for Microsoft PowerPoint operations using Graph API"""
def __init__(self, serviceContainer: Any):
self.serviceContainer = serviceContainer
def _getMicrosoftConnection(self, connectionReference: str) -> Optional[Dict[str, Any]]:
"""Get Microsoft connection from connection reference"""
try:
userConnection = self.serviceContainer.getUserConnectionFromConnectionReference(connectionReference)
if not userConnection or userConnection.authority != "msft" or userConnection.status != "active":
return None
# Get the corresponding token for this user and authority
token = self.serviceContainer.interfaceApp.getToken(userConnection.authority)
if not token:
logger.warning(f"No token found for user {userConnection.userId} and authority {userConnection.authority}")
return None
return {
"id": userConnection.id,
"accessToken": token.tokenAccess,
"refreshToken": token.tokenRefresh,
"scopes": ["Mail.ReadWrite", "User.Read"] # Default Microsoft scopes
}
except Exception as e:
logger.error(f"Error getting Microsoft connection: {str(e)}")
return None
async def readPresentation(self, fileId: str, connectionReference: str, includeSlides: bool = True) -> Dict[str, Any]:
"""Read PowerPoint presentation using Microsoft Graph API"""
try:
connection = self._getMicrosoftConnection(connectionReference)
if not connection:
return {
"error": "No valid Microsoft connection found for the provided connection reference",
"fileId": fileId,
"connectionReference": connectionReference
}
# Get file data from service container
file_data = self.serviceContainer.getFileData(fileId)
file_info = self.serviceContainer.getFileInfo(fileId)
if not file_data:
return {
"error": "File not found or empty",
"fileId": fileId
}
# For now, simulate PowerPoint reading with AI analysis
# In a real implementation, you would use Microsoft Graph API
ppt_prompt = f"""
Analyze this PowerPoint presentation and extract structured information.
File: {file_info.get('name', 'Unknown')}
Include slides: {includeSlides}
File content (first 5000 characters):
{file_data.decode('utf-8', errors='ignore')[:5000] if isinstance(file_data, bytes) else str(file_data)[:5000]}
Please extract:
1. Presentation title and theme
2. Slide structure and content
3. Text content from each slide
4. Images and media references
5. Charts and data visualizations
6. Speaker notes if available
7. Overall presentation flow and messaging
Return the data in a structured JSON format.
"""
# Use AI to analyze PowerPoint content
analysis_result = await self.serviceContainer.interfaceAiCalls.callAiTextAdvanced(ppt_prompt)
return {
"fileId": fileId,
"includeSlides": includeSlides,
"data": analysis_result,
"fileInfo": file_info,
"connection": {
"id": connection["id"],
"authority": "microsoft",
"reference": connectionReference
}
}
except Exception as e:
logger.error(f"Error reading presentation: {str(e)}")
return {
"error": str(e),
"fileId": fileId
}
async def writePresentation(self, fileId: str, connectionReference: str, slides: List[Dict[str, Any]]) -> Dict[str, Any]:
"""Write to PowerPoint presentation using Microsoft Graph API"""
try:
connection = self._getMicrosoftConnection(connectionReference)
if not connection:
return {
"error": "No valid Microsoft connection found for the provided connection reference",
"fileId": fileId,
"connectionReference": connectionReference
}
# For now, simulate PowerPoint writing
# In a real implementation, you would use Microsoft Graph API
write_prompt = f"""
Prepare content for writing to PowerPoint presentation.
File: {fileId}
Number of slides: {len(slides)}
Slides data:
{json.dumps(slides, indent=2)}
Please format this content appropriately for PowerPoint and provide:
1. Slide layouts and structures
2. Text content and formatting
3. Image and media placement
4. Chart and visualization specifications
5. Animation and transition suggestions
"""
# Use AI to prepare PowerPoint content
prepared_content = await self.serviceContainer.interfaceAiCalls.callAiTextAdvanced(write_prompt)
return {
"fileId": fileId,
"slides": slides,
"content": prepared_content,
"status": "prepared",
"connection": {
"id": connection["id"],
"authority": "microsoft",
"reference": connectionReference
}
}
except Exception as e:
logger.error(f"Error writing to presentation: {str(e)}")
return {
"error": str(e),
"fileId": fileId
}
async def convertPresentation(self, fileId: str, connectionReference: str, format: str = "pdf") -> Dict[str, Any]:
"""Convert PowerPoint presentation to another format using Microsoft Graph API"""
try:
connection = self._getMicrosoftConnection(connectionReference)
if not connection:
return {
"error": "No valid Microsoft connection found for the provided connection reference",
"fileId": fileId,
"connectionReference": connectionReference
}
# For now, simulate conversion
# In a real implementation, you would use Microsoft Graph API
convert_prompt = f"""
Convert PowerPoint presentation to {format.upper()} format.
File: {fileId}
Target format: {format}
Please provide:
1. Conversion specifications
2. Format-specific optimizations
3. Quality settings and options
4. Any special considerations for the target format
"""
# Use AI to describe conversion process
conversion_result = await self.serviceContainer.interfaceAiCalls.callAiTextAdvanced(convert_prompt)
# Create converted file using service container
converted_file_id = self.serviceContainer.createFile(
fileName=f"converted_presentation.{format}",
mimeType=f"application/{format}",
content=conversion_result,
base64encoded=False
)
return {
"fileId": fileId,
"format": format,
"convertedFileId": converted_file_id,
"result": conversion_result,
"connection": {
"id": connection["id"],
"authority": "microsoft",
"reference": connectionReference
}
}
except Exception as e:
logger.error(f"Error converting presentation: {str(e)}")
return {
"error": str(e),
"fileId": fileId
}
async def createPresentation(self, fileName: str, connectionReference: str, template: str = None) -> Dict[str, Any]:
"""Create new PowerPoint presentation using Microsoft Graph API"""
try:
connection = self._getMicrosoftConnection(connectionReference)
if not connection:
return {
"error": "No valid Microsoft connection found for the provided connection reference",
"connectionReference": connectionReference
}
# For now, simulate presentation creation
# In a real implementation, you would use Microsoft Graph API
create_prompt = f"""
Create a new PowerPoint presentation structure.
File name: {fileName}
Template: {template or 'Standard'}
Please provide:
1. Initial slide structure
2. Default slide layouts
3. Theme and design elements
4. Sample content if template specified
5. Presentation guidelines
"""
# Use AI to create PowerPoint structure
presentation_structure = await self.serviceContainer.interfaceAiCalls.callAiTextAdvanced(create_prompt)
# Create file using service container
file_id = self.serviceContainer.createFile(
fileName=fileName,
mimeType="application/vnd.openxmlformats-officedocument.presentationml.presentation",
content=presentation_structure,
base64encoded=False
)
return {
"fileId": file_id,
"fileName": fileName,
"template": template,
"structure": presentation_structure,
"connection": {
"id": connection["id"],
"authority": "microsoft",
"reference": connectionReference
}
}
except Exception as e:
logger.error(f"Error creating presentation: {str(e)}")
return {
"error": str(e)
}
async def addSlide(self, fileId: str, connectionReference: str, layout: str = "title", content: Dict[str, Any] = None) -> Dict[str, Any]:
"""Add slide to presentation using Microsoft Graph API"""
try:
connection = self._getMicrosoftConnection(connectionReference)
if not connection:
return {
"error": "No valid Microsoft connection found for the provided connection reference",
"fileId": fileId,
"connectionReference": connectionReference
}
# For now, simulate slide addition
# In a real implementation, you would use Microsoft Graph API
slide_prompt = f"""
Add a new slide to PowerPoint presentation.
File: {fileId}
Layout: {layout}
Content: {json.dumps(content, indent=2) if content else 'Default content'}
Please provide:
1. Slide structure and layout
2. Content placement and formatting
3. Visual elements and design
4. Slide number and positioning
"""
# Use AI to create slide content
slide_content = await self.serviceContainer.interfaceAiCalls.callAiTextAdvanced(slide_prompt)
return {
"fileId": fileId,
"layout": layout,
"content": content,
"slideContent": slide_content,
"connection": {
"id": connection["id"],
"authority": "microsoft",
"reference": connectionReference
}
}
except Exception as e:
logger.error(f"Error adding slide: {str(e)}")
return {
"error": str(e),
"fileId": fileId
}
async def addContent(self, fileId: str, connectionReference: str, slideId: str, content: Dict[str, Any]) -> Dict[str, Any]:
"""Add content to slide using Microsoft Graph API"""
try:
connection = self._getMicrosoftConnection(connectionReference)
if not connection:
return {
"error": "No valid Microsoft connection found for the provided connection reference",
"fileId": fileId,
"connectionReference": connectionReference
}
# For now, simulate content addition
# In a real implementation, you would use Microsoft Graph API
content_prompt = f"""
Add content to PowerPoint slide.
File: {fileId}
Slide ID: {slideId}
Content: {json.dumps(content, indent=2)}
Please provide:
1. Content placement and formatting
2. Text styling and layout
3. Image and media integration
4. Chart and visualization setup
5. Animation and effects
"""
# Use AI to format slide content
formatted_content = await self.serviceContainer.interfaceAiCalls.callAiTextAdvanced(content_prompt)
return {
"fileId": fileId,
"slideId": slideId,
"content": content,
"formattedContent": formatted_content,
"connection": {
"id": connection["id"],
"authority": "microsoft",
"reference": connectionReference
}
}
except Exception as e:
logger.error(f"Error adding content: {str(e)}")
return {
"error": str(e),
"fileId": fileId
}
class MethodPowerpoint(MethodBase):
"""PowerPoint method implementation for presentation operations"""
def __init__(self, serviceContainer: Any):
"""Initialize the PowerPoint method"""
super().__init__(serviceContainer)
self.name = "powerpoint"
self.description = "Handle PowerPoint presentation operations like reading and creating slides"
self.powerpointService = PowerpointService(serviceContainer)
@action
async def read(self, parameters: Dict[str, Any]) -> ActionResult:
"""
Read PowerPoint presentation
Parameters:
fileId (str): The ID of the PowerPoint file to read
connectionReference (str): Reference to the Microsoft connection
includeSlides (bool, optional): Whether to include slide content (default: True)
"""
try:
fileId = parameters.get("fileId")
connectionReference = parameters.get("connectionReference")
includeSlides = parameters.get("includeSlides", True)
if not fileId or not connectionReference:
return self._createResult(
success=False,
data={},
error="File ID and connection reference are required"
)
# Read presentation
data = await self.powerpointService.readPresentation(
fileId=fileId,
connectionReference=connectionReference,
includeSlides=includeSlides
)
return self._createResult(
success=True,
data=data
)
except Exception as e:
logger.error(f"Error reading presentation: {str(e)}")
return self._createResult(
success=False,
data={},
error=str(e)
)
@action
async def write(self, parameters: Dict[str, Any]) -> ActionResult:
"""
Write to PowerPoint presentation
Parameters:
fileId (str): The ID of the PowerPoint file to write to
connectionReference (str): Reference to the Microsoft connection
slides (List[Dict[str, Any]]): List of slides to write
"""
try:
fileId = parameters.get("fileId")
connectionReference = parameters.get("connectionReference")
slides = parameters.get("slides", [])
if not fileId or not connectionReference:
return self._createResult(
success=False,
data={},
error="File ID and connection reference are required"
)
# Write to presentation
result = await self.powerpointService.writePresentation(
fileId=fileId,
connectionReference=connectionReference,
slides=slides
)
return self._createResult(
success=True,
data=result
)
except Exception as e:
logger.error(f"Error writing to presentation: {str(e)}")
return self._createResult(
success=False,
data={},
error=str(e)
)
@action
async def convert(self, parameters: Dict[str, Any]) -> ActionResult:
"""
Convert PowerPoint presentation to another format
Parameters:
fileId (str): The ID of the PowerPoint file to convert
connectionReference (str): Reference to the Microsoft connection
format (str, optional): Target format (default: "pdf")
"""
try:
fileId = parameters.get("fileId")
connectionReference = parameters.get("connectionReference")
format = parameters.get("format", "pdf")
if not fileId or not connectionReference:
return self._createResult(
success=False,
data={},
error="File ID and connection reference are required"
)
# Convert presentation
result = await self.powerpointService.convertPresentation(
fileId=fileId,
connectionReference=connectionReference,
format=format
)
return self._createResult(
success=True,
data=result
)
except Exception as e:
logger.error(f"Error converting presentation: {str(e)}")
return self._createResult(
success=False,
data={},
error=str(e)
)
@action
async def createPresentation(self, parameters: Dict[str, Any]) -> ActionResult:
"""
Create new PowerPoint presentation
Parameters:
fileName (str): Name of the new presentation file
connectionReference (str): Reference to the Microsoft connection
template (str, optional): Template to use for the new presentation
"""
try:
fileName = parameters.get("fileName")
connectionReference = parameters.get("connectionReference")
template = parameters.get("template")
if not fileName or not connectionReference:
return self._createResult(
success=False,
data={},
error="File name and connection reference are required"
)
# Create presentation
fileId = await self.powerpointService.createPresentation(
fileName=fileName,
connectionReference=connectionReference,
template=template
)
return self._createResult(
success=True,
data={"fileId": fileId}
)
except Exception as e:
logger.error(f"Error creating presentation: {str(e)}")
return self._createResult(
success=False,
data={},
error=str(e)
)
@action
async def addSlide(self, parameters: Dict[str, Any]) -> ActionResult:
"""
Add slide to presentation
Parameters:
fileId (str): The ID of the PowerPoint file
connectionReference (str): Reference to the Microsoft connection
layout (str, optional): Slide layout type (default: "title")
content (Dict[str, Any], optional): Content for the slide
"""
try:
fileId = parameters.get("fileId")
connectionReference = parameters.get("connectionReference")
layout = parameters.get("layout", "title")
content = parameters.get("content", {})
if not fileId or not connectionReference:
return self._createResult(
success=False,
data={},
error="File ID and connection reference are required"
)
# Add slide
slide = await self.powerpointService.addSlide(
fileId=fileId,
connectionReference=connectionReference,
layout=layout,
content=content
)
return self._createResult(
success=True,
data=slide
)
except Exception as e:
logger.error(f"Error adding slide: {str(e)}")
return self._createResult(
success=False,
data={},
error=str(e)
)
@action
async def addContent(self, parameters: Dict[str, Any]) -> ActionResult:
"""
Add content to slide
Parameters:
fileId (str): The ID of the PowerPoint file
connectionReference (str): Reference to the Microsoft connection
slideId (str): ID of the slide to add content to
content (Dict[str, Any]): Content to add to the slide
"""
try:
fileId = parameters.get("fileId")
connectionReference = parameters.get("connectionReference")
slideId = parameters.get("slideId")
content = parameters.get("content", {})
if not fileId or not connectionReference or not slideId:
return self._createResult(
success=False,
data={},
error="File ID, connection reference, and slide ID are required"
)
# Add content
result = await self.powerpointService.addContent(
fileId=fileId,
connectionReference=connectionReference,
slideId=slideId,
content=content
)
return self._createResult(
success=True,
data=result
)
except Exception as e:
logger.error(f"Error adding content: {str(e)}")
return self._createResult(
success=False,
data={},
error=str(e)
)

869
modules/methods/methodSharepoint.py
View file

@ -7,16 +7,19 @@ import logging
from typing import Dict, Any, List, Optional from typing import Dict, Any, List, Optional
from datetime import datetime, UTC from datetime import datetime, UTC
import json import json
import uuid
from modules.workflow.methodBase import MethodBase, ActionResult, action from modules.workflow.methodBase import MethodBase, ActionResult, action
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
class SharepointService: class MethodSharepoint(MethodBase):
"""Service for Microsoft SharePoint operations using Graph API""" """SharePoint method implementation for document operations"""
def __init__(self, serviceContainer: Any): def __init__(self, serviceContainer: Any):
self.serviceContainer = serviceContainer super().__init__(serviceContainer)
self.name = "sharepoint"
self.description = "Handle Microsoft SharePoint document operations"
def _getMicrosoftConnection(self, connectionReference: str) -> Optional[Dict[str, Any]]: def _getMicrosoftConnection(self, connectionReference: str) -> Optional[Dict[str, Any]]:
"""Get Microsoft connection from connection reference""" """Get Microsoft connection from connection reference"""
@ -35,641 +38,401 @@ class SharepointService:
"id": userConnection.id, "id": userConnection.id,
"accessToken": token.tokenAccess, "accessToken": token.tokenAccess,
"refreshToken": token.tokenRefresh, "refreshToken": token.tokenRefresh,
"scopes": ["Mail.ReadWrite", "User.Read"] # Default Microsoft scopes "scopes": ["Sites.ReadWrite.All", "User.Read"] # Default Microsoft scopes
} }
except Exception as e: except Exception as e:
logger.error(f"Error getting Microsoft connection: {str(e)}") logger.error(f"Error getting Microsoft connection: {str(e)}")
return None return None
async def searchContent(self, connectionReference: str, query: str, siteId: str = None, contentType: str = None, maxResults: int = 10) -> Dict[str, Any]: @action
"""Search SharePoint content using Microsoft Graph API""" async def findDocumentPath(self, parameters: Dict[str, Any]) -> ActionResult:
"""
Find document path based on query/description
Parameters:
connectionReference (str): Reference to the Microsoft connection
siteUrl (str): SharePoint site URL
query (str): Query or description to find document
searchScope (str, optional): Search scope (default: "all")
"""
try: try:
connectionReference = parameters.get("connectionReference")
siteUrl = parameters.get("siteUrl")
query = parameters.get("query")
searchScope = parameters.get("searchScope", "all")
if not connectionReference or not siteUrl or not query:
return self._createResult(
success=False,
data={},
error="Connection reference, site URL, and query are required"
)
connection = self._getMicrosoftConnection(connectionReference) connection = self._getMicrosoftConnection(connectionReference)
if not connection: if not connection:
return { return self._createResult(
"error": "No valid Microsoft connection found for the provided connection reference", success=False,
"connectionReference": connectionReference data={},
} error="No valid Microsoft connection found for the provided connection reference"
)
# For now, simulate SharePoint search find_prompt = f"""
# In a real implementation, you would use Microsoft Graph API Simulate finding document paths in Microsoft SharePoint based on a query.
search_prompt = f"""
Search SharePoint content for the following query.
Connection: {connection['id']}
Site URL: {siteUrl}
Query: {query} Query: {query}
Site ID: {siteId or 'All sites'} Search Scope: {searchScope}
Content Type: {contentType or 'All types'}
Max Results: {maxResults}
Please provide: Please provide:
1. Relevant search results 1. Matching document paths and locations
2. Content summaries 2. Relevance scores for each match
3. File and document information 3. Document metadata and properties
4. Site and list references 4. Alternative search suggestions
5. Metadata and properties 5. Search statistics and coverage
""" """
# Use AI to simulate search results find_result = await self.serviceContainer.interfaceAiCalls.callAiTextAdvanced(find_prompt)
search_results = await self.serviceContainer.interfaceAiCalls.callAiTextAdvanced(search_prompt)
return { result_data = {
"connectionReference": connectionReference,
"siteUrl": siteUrl,
"query": query, "query": query,
"siteId": siteId, "searchScope": searchScope,
"contentType": contentType, "findResult": find_result,
"maxResults": maxResults,
"results": search_results,
"connection": { "connection": {
"id": connection["id"], "id": connection["id"],
"authority": "microsoft", "authority": "microsoft",
"reference": connectionReference "reference": connectionReference
},
"timestamp": datetime.now(UTC).isoformat()
} }
return self._createResult(
success=True,
data={
"documentName": f"sharepoint_find_path_{datetime.now(UTC).strftime('%Y%m%d_%H%M%S')}.json",
"documentData": result_data
} }
)
except Exception as e: except Exception as e:
logger.error(f"Error searching SharePoint: {str(e)}") logger.error(f"Error finding document path: {str(e)}")
return { return self._createResult(
"error": str(e) success=False,
} data={},
error=str(e)
)
async def readItem(self, connectionReference: str, itemId: str, siteId: str = None, listId: str = None) -> Dict[str, Any]: @action
"""Read SharePoint item using Microsoft Graph API""" async def readDocument(self, parameters: Dict[str, Any]) -> ActionResult:
"""
Read documents from SharePoint
Parameters:
documentList (str): Reference to the document list to read
connectionReference (str): Reference to the Microsoft connection
siteUrl (str): SharePoint site URL
documentPaths (List[str]): List of paths to the documents in SharePoint
includeMetadata (bool, optional): Whether to include metadata (default: True)
"""
try: try:
documentList = parameters.get("documentList")
connectionReference = parameters.get("connectionReference")
siteUrl = parameters.get("siteUrl")
documentPaths = parameters.get("documentPaths")
includeMetadata = parameters.get("includeMetadata", True)
if not documentList or not connectionReference or not siteUrl or not documentPaths:
return self._createResult(
success=False,
data={},
error="Document list reference, connection reference, site URL, and document paths are required"
)
chatDocuments = self.serviceContainer.getChatDocumentsFromDocumentReference(documentList)
if not chatDocuments:
return self._createResult(
success=False,
data={},
error="No documents found for the provided reference"
)
connection = self._getMicrosoftConnection(connectionReference) connection = self._getMicrosoftConnection(connectionReference)
if not connection: if not connection:
return { return self._createResult(
"error": "No valid Microsoft connection found for the provided connection reference", success=False,
"itemId": itemId, data={},
"connectionReference": connectionReference error="No valid Microsoft connection found for the provided connection reference"
} )
# For now, simulate item reading # Process each document path
# In a real implementation, you would use Microsoft Graph API read_results = []
read_prompt = f"""
Read SharePoint item details.
Item ID: {itemId} for i, documentPath in enumerate(documentPaths):
Site ID: {siteId or 'Default site'} if i < len(chatDocuments):
List ID: {listId or 'Default list'} chatDocument = chatDocuments[i]
fileId = chatDocument.fileId
sharepoint_prompt = f"""
Simulate reading a document from Microsoft SharePoint.
Connection: {connection['id']}
Site URL: {siteUrl}
Document Path: {documentPath}
Include Metadata: {includeMetadata}
File ID: {fileId}
Please provide: Please provide:
1. Item properties and metadata 1. Document content and structure
2. Content and attachments 2. File metadata and properties
3. Permissions and access rights 3. SharePoint site information
4. Version history if available 4. Document permissions and sharing
5. Related items and links 5. Version history if available
""" """
# Use AI to simulate item data document_data = await self.serviceContainer.interfaceAiCalls.callAiTextAdvanced(sharepoint_prompt)
item_data = await self.serviceContainer.interfaceAiCalls.callAiTextAdvanced(read_prompt)
return { read_results.append({
"itemId": itemId, "documentPath": documentPath,
"siteId": siteId, "fileId": fileId,
"listId": listId, "documentContent": document_data
"data": item_data, })
result_data = {
"connectionReference": connectionReference,
"siteUrl": siteUrl,
"documentPaths": documentPaths,
"includeMetadata": includeMetadata,
"readResults": read_results,
"connection": { "connection": {
"id": connection["id"], "id": connection["id"],
"authority": "microsoft", "authority": "microsoft",
"reference": connectionReference "reference": connectionReference
} },
"timestamp": datetime.now(UTC).isoformat()
} }
return self._createResult(
success=True,
data={
"documentName": f"sharepoint_documents_{datetime.now(UTC).strftime('%Y%m%d_%H%M%S')}.json",
"documentData": result_data
}
)
except Exception as e: except Exception as e:
logger.error(f"Error reading SharePoint item: {str(e)}") logger.error(f"Error reading SharePoint documents: {str(e)}")
return { return self._createResult(
"error": str(e), success=False,
"itemId": itemId data={},
} error=str(e)
)
async def writeItem(self, connectionReference: str, siteId: str, listId: str, item: Dict[str, Any]) -> Dict[str, Any]: @action
"""Write SharePoint item using Microsoft Graph API""" async def uploadDocument(self, parameters: Dict[str, Any]) -> ActionResult:
"""
Upload documents to SharePoint
Parameters:
connectionReference (str): Reference to the Microsoft connection
siteUrl (str): SharePoint site URL
documentPaths (List[str]): List of paths where to upload the documents
documentList (str): Reference to the document list to upload
fileNames (List[str]): List of names for the uploaded files
"""
try: try:
connectionReference = parameters.get("connectionReference")
siteUrl = parameters.get("siteUrl")
documentPaths = parameters.get("documentPaths")
documentList = parameters.get("documentList")
fileNames = parameters.get("fileNames")
if not connectionReference or not siteUrl or not documentPaths or not documentList or not fileNames:
return self._createResult(
success=False,
data={},
error="Connection reference, site URL, document paths, document list, and file names are required"
)
# Get Microsoft connection
connection = self._getMicrosoftConnection(connectionReference) connection = self._getMicrosoftConnection(connectionReference)
if not connection: if not connection:
return { return self._createResult(
"error": "No valid Microsoft connection found for the provided connection reference", success=False,
"connectionReference": connectionReference data={},
} error="No valid Microsoft connection found for the provided connection reference"
)
# For now, simulate item writing # Get documents from reference
# In a real implementation, you would use Microsoft Graph API chatDocuments = self.serviceContainer.getChatDocumentsFromDocumentReference(documentList)
write_prompt = f""" if not chatDocuments:
Write item to SharePoint list. return self._createResult(
success=False,
data={},
error="No documents found for the provided reference"
)
Site ID: {siteId} # Process each document upload
List ID: {listId} upload_results = []
Item data: {json.dumps(item, indent=2)}
for i, (documentPath, fileName) in enumerate(zip(documentPaths, fileNames)):
if i < len(chatDocuments):
chatDocument = chatDocuments[i]
fileId = chatDocument.fileId
file_data = self.serviceContainer.getFileData(fileId)
if not file_data:
logger.warning(f"File data not found for fileId: {fileId}")
continue
# Create SharePoint upload prompt
upload_prompt = f"""
Simulate uploading a document to Microsoft SharePoint.
Connection: {connection['id']}
Site URL: {siteUrl}
Document Path: {documentPath}
File Name: {fileName}
File ID: {fileId}
File Size: {len(file_data)} bytes
Please provide: Please provide:
1. Item creation/update details 1. Upload confirmation and status
2. Validation and formatting 2. File metadata and properties
3. Permission settings 3. SharePoint site integration details
4. Workflow triggers if applicable 4. Permission and sharing settings
5. Success confirmation 5. Version control information
""" """
# Use AI to simulate item creation # Use AI to simulate SharePoint upload
write_result = await self.serviceContainer.interfaceAiCalls.callAiTextAdvanced(write_prompt) upload_result = await self.serviceContainer.interfaceAiCalls.callAiTextAdvanced(upload_prompt)
return { upload_results.append({
"siteId": siteId, "documentPath": documentPath,
"listId": listId, "fileName": fileName,
"item": item, "fileId": fileId,
"result": write_result, "uploadResult": upload_result
})
# Create result data
result_data = {
"connectionReference": connectionReference,
"siteUrl": siteUrl,
"documentPaths": documentPaths,
"documentList": documentList,
"fileNames": fileNames,
"uploadResults": upload_results,
"connection": { "connection": {
"id": connection["id"], "id": connection["id"],
"authority": "microsoft", "authority": "microsoft",
"reference": connectionReference "reference": connectionReference
},
"timestamp": datetime.now(UTC).isoformat()
} }
return self._createResult(
success=True,
data={
"documentName": f"sharepoint_upload_{datetime.now(UTC).strftime('%Y%m%d_%H%M%S')}.json",
"documentData": result_data
} }
)
except Exception as e: except Exception as e:
logger.error(f"Error writing SharePoint item: {str(e)}") logger.error(f"Error uploading to SharePoint: {str(e)}")
return { return self._createResult(
"error": str(e) success=False,
} data={},
error=str(e)
)
async def readList(self, connectionReference: str, listId: str, siteId: str = None, query: str = None, maxResults: int = 10) -> Dict[str, Any]: @action
"""Read SharePoint list using Microsoft Graph API""" async def listDocuments(self, parameters: Dict[str, Any]) -> ActionResult:
"""
List documents in SharePoint folder
Parameters:
connectionReference (str): Reference to the Microsoft connection
siteUrl (str): SharePoint site URL
folderPaths (List[str]): List of paths to the folders to list
includeSubfolders (bool, optional): Whether to include subfolders (default: False)
"""
try: try:
connectionReference = parameters.get("connectionReference")
siteUrl = parameters.get("siteUrl")
folderPaths = parameters.get("folderPaths")
includeSubfolders = parameters.get("includeSubfolders", False)
if not connectionReference or not siteUrl or not folderPaths:
return self._createResult(
success=False,
data={},
error="Connection reference, site URL, and folder paths are required"
)
# Get Microsoft connection
connection = self._getMicrosoftConnection(connectionReference) connection = self._getMicrosoftConnection(connectionReference)
if not connection: if not connection:
return { return self._createResult(
"error": "No valid Microsoft connection found for the provided connection reference", success=False,
"listId": listId, data={},
"connectionReference": connectionReference error="No valid Microsoft connection found for the provided connection reference"
} )
# For now, simulate list reading # Process each folder path
# In a real implementation, you would use Microsoft Graph API list_results = []
for folderPath in folderPaths:
# Create SharePoint listing prompt
list_prompt = f""" list_prompt = f"""
Read SharePoint list items. Simulate listing documents in Microsoft SharePoint folder.
List ID: {listId} Connection: {connection['id']}
Site ID: {siteId or 'Default site'} Site URL: {siteUrl}
Query: {query or 'All items'} Folder Path: {folderPath}
Max Results: {maxResults} Include Subfolders: {includeSubfolders}
Please provide: Please provide:
1. List structure and columns 1. List of documents and folders
2. Item data and properties 2. File metadata and properties
3. Sorting and filtering options 3. Folder structure and hierarchy
4. Pagination information 4. Permission and sharing information
5. List metadata and settings 5. Document statistics and summary
""" """
# Use AI to simulate list data # Use AI to simulate SharePoint listing
list_data = await self.serviceContainer.interfaceAiCalls.callAiTextAdvanced(list_prompt) list_result = await self.serviceContainer.interfaceAiCalls.callAiTextAdvanced(list_prompt)
return { list_results.append({
"listId": listId, "folderPath": folderPath,
"siteId": siteId, "listResult": list_result
"query": query, })
"maxResults": maxResults,
"data": list_data, # Create result data
result_data = {
"connectionReference": connectionReference,
"siteUrl": siteUrl,
"folderPaths": folderPaths,
"includeSubfolders": includeSubfolders,
"listResults": list_results,
"connection": { "connection": {
"id": connection["id"], "id": connection["id"],
"authority": "microsoft", "authority": "microsoft",
"reference": connectionReference "reference": connectionReference
},
"timestamp": datetime.now(UTC).isoformat()
} }
}
except Exception as e:
logger.error(f"Error reading SharePoint list: {str(e)}")
return {
"error": str(e),
"listId": listId
}
async def writeList(self, connectionReference: str, siteId: str, listId: str, items: List[Dict[str, Any]]) -> Dict[str, Any]:
"""Write multiple items to SharePoint list using Microsoft Graph API"""
try:
connection = self._getMicrosoftConnection(connectionReference)
if not connection:
return {
"error": "No valid Microsoft connection found for the provided connection reference",
"connectionReference": connectionReference
}
# For now, simulate bulk writing
# In a real implementation, you would use Microsoft Graph API
bulk_prompt = f"""
Write multiple items to SharePoint list.
Site ID: {siteId}
List ID: {listId}
Number of items: {len(items)}
Items data: {json.dumps(items[:3], indent=2)} # Show first 3 items
Please provide:
1. Bulk operation details
2. Validation and error handling
3. Performance optimization
4. Success/failure status for each item
5. Batch processing results
"""
# Use AI to simulate bulk operation
bulk_result = await self.serviceContainer.interfaceAiCalls.callAiTextAdvanced(bulk_prompt)
return {
"siteId": siteId,
"listId": listId,
"items": items,
"result": bulk_result,
"connection": {
"id": connection["id"],
"authority": "microsoft",
"reference": connectionReference
}
}
except Exception as e:
logger.error(f"Error writing to SharePoint list: {str(e)}")
return {
"error": str(e)
}
async def createList(self, connectionReference: str, siteId: str, name: str, description: str = None, template: str = "genericList", fields: List[Dict[str, Any]] = None) -> Dict[str, Any]:
"""Create SharePoint list using Microsoft Graph API"""
try:
connection = self._getMicrosoftConnection(connectionReference)
if not connection:
return {
"error": "No valid Microsoft connection found for the provided connection reference",
"connectionReference": connectionReference
}
# For now, simulate list creation
# In a real implementation, you would use Microsoft Graph API
create_prompt = f"""
Create a new SharePoint list.
Site ID: {siteId}
Name: {name}
Description: {description or 'No description'}
Template: {template}
Fields: {json.dumps(fields, indent=2) if fields else 'Default fields'}
Please provide:
1. List structure and configuration
2. Column definitions and types
3. Default views and permissions
4. Workflow and automation settings
5. Creation confirmation and next steps
"""
# Use AI to simulate list creation
creation_result = await self.serviceContainer.interfaceAiCalls.callAiTextAdvanced(create_prompt)
return {
"siteId": siteId,
"name": name,
"description": description,
"template": template,
"fields": fields,
"result": creation_result,
"connection": {
"id": connection["id"],
"authority": "microsoft",
"reference": connectionReference
}
}
except Exception as e:
logger.error(f"Error creating SharePoint list: {str(e)}")
return {
"error": str(e)
}
class MethodSharepoint(MethodBase):
"""SharePoint method implementation for site operations"""
def __init__(self, serviceContainer: Any):
"""Initialize the SharePoint method"""
super().__init__(serviceContainer)
self.name = "sharepoint"
self.description = "Handle SharePoint site operations like reading and writing lists"
self.sharepointService = SharepointService(serviceContainer)
@action
async def search(self, parameters: Dict[str, Any]) -> ActionResult:
"""
Search SharePoint content
Parameters:
connectionReference (str): Reference to the Microsoft connection
query (str): Search query
siteId (str, optional): SharePoint site ID
contentType (str, optional): Content type to filter by
maxResults (int, optional): Maximum number of results (default: 10)
"""
try:
connectionReference = parameters.get("connectionReference")
query = parameters.get("query")
siteId = parameters.get("siteId")
contentType = parameters.get("contentType")
maxResults = parameters.get("maxResults", 10)
if not connectionReference:
return self._createResult(
success=False,
data={},
error="Connection reference is required"
)
if not query:
return self._createResult(
success=False,
data={},
error="Search query is required"
)
# Search content
results = await self.sharepointService.searchContent(
connectionReference=connectionReference,
query=query,
siteId=siteId,
contentType=contentType,
maxResults=maxResults
)
return self._createResult( return self._createResult(
success=True, success=True,
data=results data={
"documentName": f"sharepoint_document_list_{datetime.now(UTC).strftime('%Y%m%d_%H%M%S')}.json",
"documentData": result_data
}
) )
except Exception as e: except Exception as e:
logger.error(f"Error searching SharePoint: {str(e)}") logger.error(f"Error listing SharePoint documents: {str(e)}")
return self._createResult(
success=False,
data={},
error=str(e)
)
@action
async def read(self, parameters: Dict[str, Any]) -> ActionResult:
"""
Read SharePoint item
Parameters:
connectionReference (str): Reference to the Microsoft connection
itemId (str): ID of the item to read
siteId (str, optional): SharePoint site ID
listId (str, optional): SharePoint list ID
"""
try:
connectionReference = parameters.get("connectionReference")
itemId = parameters.get("itemId")
siteId = parameters.get("siteId")
listId = parameters.get("listId")
if not connectionReference:
return self._createResult(
success=False,
data={},
error="Connection reference is required"
)
if not itemId:
return self._createResult(
success=False,
data={},
error="Item ID is required"
)
# Read item
item = await self.sharepointService.readItem(
connectionReference=connectionReference,
itemId=itemId,
siteId=siteId,
listId=listId
)
return self._createResult(
success=True,
data=item
)
except Exception as e:
logger.error(f"Error reading SharePoint item: {str(e)}")
return self._createResult(
success=False,
data={},
error=str(e)
)
@action
async def write(self, parameters: Dict[str, Any]) -> ActionResult:
"""
Write SharePoint item
Parameters:
connectionReference (str): Reference to the Microsoft connection
siteId (str): SharePoint site ID
listId (str): SharePoint list ID
item (Dict[str, Any]): Item data to write
"""
try:
connectionReference = parameters.get("connectionReference")
siteId = parameters.get("siteId")
listId = parameters.get("listId")
item = parameters.get("item", {})
if not connectionReference:
return self._createResult(
success=False,
data={},
error="Connection reference is required"
)
if not siteId or not listId:
return self._createResult(
success=False,
data={},
error="Site ID and list ID are required"
)
# Write item
result = await self.sharepointService.writeItem(
connectionReference=connectionReference,
siteId=siteId,
listId=listId,
item=item
)
return self._createResult(
success=True,
data=result
)
except Exception as e:
logger.error(f"Error writing SharePoint item: {str(e)}")
return self._createResult(
success=False,
data={},
error=str(e)
)
@action
async def readList(self, parameters: Dict[str, Any]) -> ActionResult:
"""
Read SharePoint list
Parameters:
connectionReference (str): Reference to the Microsoft connection
listId (str): SharePoint list ID
siteId (str, optional): SharePoint site ID
query (str, optional): Query to filter items
maxResults (int, optional): Maximum number of results (default: 10)
"""
try:
connectionReference = parameters.get("connectionReference")
listId = parameters.get("listId")
siteId = parameters.get("siteId")
query = parameters.get("query")
maxResults = parameters.get("maxResults", 10)
if not connectionReference:
return self._createResult(
success=False,
data={},
error="Connection reference is required"
)
if not listId:
return self._createResult(
success=False,
data={},
error="List ID is required"
)
# Read list
items = await self.sharepointService.readList(
connectionReference=connectionReference,
listId=listId,
siteId=siteId,
query=query,
maxResults=maxResults
)
return self._createResult(
success=True,
data=items
)
except Exception as e:
logger.error(f"Error reading SharePoint list: {str(e)}")
return self._createResult(
success=False,
data={},
error=str(e)
)
@action
async def writeList(self, parameters: Dict[str, Any]) -> ActionResult:
"""
Write multiple items to SharePoint list
Parameters:
connectionReference (str): Reference to the Microsoft connection
siteId (str): SharePoint site ID
listId (str): SharePoint list ID
items (List[Dict[str, Any]]): List of items to write
"""
try:
connectionReference = parameters.get("connectionReference")
siteId = parameters.get("siteId")
listId = parameters.get("listId")
items = parameters.get("items", [])
if not connectionReference:
return self._createResult(
success=False,
data={},
error="Connection reference is required"
)
if not siteId or not listId:
return self._createResult(
success=False,
data={},
error="Site ID and list ID are required"
)
# Write items
results = await self.sharepointService.writeList(
connectionReference=connectionReference,
siteId=siteId,
listId=listId,
items=items
)
return self._createResult(
success=True,
data=results
)
except Exception as e:
logger.error(f"Error writing SharePoint list: {str(e)}")
return self._createResult(
success=False,
data={},
error=str(e)
)
@action
async def createList(self, parameters: Dict[str, Any]) -> ActionResult:
"""
Create new SharePoint list
Parameters:
connectionReference (str): Reference to the Microsoft connection
siteId (str): SharePoint site ID
name (str): Name of the new list
description (str, optional): Description of the list
template (str, optional): List template (default: "genericList")
fields (List[Dict[str, Any]], optional): List of field definitions
"""
try:
connectionReference = parameters.get("connectionReference")
siteId = parameters.get("siteId")
name = parameters.get("name")
description = parameters.get("description")
template = parameters.get("template", "genericList")
fields = parameters.get("fields", [])
if not connectionReference:
return self._createResult(
success=False,
data={},
error="Connection reference is required"
)
if not siteId or not name:
return self._createResult(
success=False,
data={},
error="Site ID and name are required"
)
# Create list
list_info = await self.sharepointService.createList(
connectionReference=connectionReference,
siteId=siteId,
name=name,
description=description,
template=template,
fields=fields
)
return self._createResult(
success=True,
data=list_info
)
except Exception as e:
logger.error(f"Error creating SharePoint list: {str(e)}")
return self._createResult( return self._createResult(
success=False, success=False,
data={}, data={},

778
modules/methods/methodWeb.py
View file

@ -9,19 +9,21 @@ from datetime import datetime, UTC
import requests import requests
from bs4 import BeautifulSoup from bs4 import BeautifulSoup
import time import time
import uuid
from modules.workflow.methodBase import MethodBase, ActionResult, action from modules.workflow.methodBase import MethodBase, ActionResult, action
from modules.shared.configuration import APP_CONFIG from modules.shared.configuration import APP_CONFIG
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
class WebService: class MethodWeb(MethodBase):
"""Service for web operations like searching and crawling""" """Web method implementation for web operations"""
def __init__(self, serviceContainer: Any): def __init__(self, serviceContainer: Any):
self.serviceContainer = serviceContainer """Initialize the web method"""
self.user_agent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36" super().__init__(serviceContainer)
self.timeout = 30 self.name = "web"
self.description = "Handle web operations like crawling and scraping"
# Web search configuration from agentWebcrawler # Web search configuration from agentWebcrawler
self.srcApikey = APP_CONFIG.get("Agent_Webcrawler_SERPAPI_APIKEY", "") self.srcApikey = APP_CONFIG.get("Agent_Webcrawler_SERPAPI_APIKEY", "")
@ -32,232 +34,89 @@ class WebService:
if not self.srcApikey: if not self.srcApikey:
logger.warning("SerpAPI key not configured for web search") logger.warning("SerpAPI key not configured for web search")
async def searchWeb(self, query: str, maxResults: int = 10) -> Dict[str, Any]: self.user_agent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
"""Search web content using Google search via SerpAPI""" self.timeout = 30
def _readUrl(self, url: str) -> BeautifulSoup:
"""Read a URL and return a BeautifulSoup parser for the content"""
if not url or not url.startswith(('http://', 'https://')):
return None
headers = {
'User-Agent': self.user_agent,
'Accept': 'text/html,application/xhtml+xml,application/xml',
'Accept-Language': 'en-US,en;q=0.9',
}
try: try:
if not self.srcApikey: # Initial request
return { response = requests.get(url, headers=headers, timeout=self.timeout)
"error": "SerpAPI key not configured",
"query": query
}
# Get user language from service container if available # Handling for status 202
userLanguage = "en" # Default language if response.status_code == 202:
if hasattr(self.serviceContainer, 'user') and hasattr(self.serviceContainer.user, 'language'): # Retry with backoff
userLanguage = self.serviceContainer.user.language backoff_times = [0.5, 1.0, 2.0, 5.0]
# Format the search request for SerpAPI for wait_time in backoff_times:
params = { time.sleep(wait_time)
"engine": self.srcEngine, response = requests.get(url, headers=headers, timeout=self.timeout)
"q": query,
"api_key": self.srcApikey,
"num": min(maxResults, self.maxResults), # Number of results to return
"hl": userLanguage # User language
}
# Make the API request if response.status_code != 202:
response = requests.get("https://serpapi.com/search", params=params, timeout=self.timeout) break
# Raise for error status codes
response.raise_for_status() response.raise_for_status()
# Parse JSON response # Parse HTML
search_results = response.json() return BeautifulSoup(response.text, 'html.parser')
# Extract organic results
results = []
if "organic_results" in search_results:
for result in search_results["organic_results"][:maxResults]:
# Extract title
title = result.get("title", "No title")
# Extract URL
url = result.get("link", "No URL")
# Extract snippet
snippet = result.get("snippet", "No description")
# Get actual page content
try:
targetPageSoup = self._readUrl(url)
content = self._extractMainContent(targetPageSoup)
except Exception as e: except Exception as e:
logger.warning(f"Error extracting content from {url}: {str(e)}") logger.error(f"Error reading URL {url}: {str(e)}")
content = f"Error extracting content: {str(e)}" return None
results.append({ def _extractTitle(self, soup: BeautifulSoup, url: str) -> str:
'title': title, """Extract the title from a webpage"""
'url': url, if not soup:
'snippet': snippet, return f"Error with {url}"
'content': content
})
# Limit number of results # Extract title from title tag
if len(results) >= maxResults: title_tag = soup.find('title')
title = title_tag.text.strip() if title_tag else "No title"
# Alternative: Also look for h1 tags if title tag is missing
if title == "No title":
h1_tag = soup.find('h1')
if h1_tag:
title = h1_tag.text.strip()
return title
def _extractMainContent(self, soup: BeautifulSoup, max_chars: int = 10000) -> str:
"""Extract the main content from an HTML page"""
if not soup:
return ""
# Try to find main content elements in priority order
main_content = None
for selector in ['main', 'article', '#content', '.content', '#main', '.main']:
content = soup.select_one(selector)
if content:
main_content = content
break break
else:
logger.warning(f"No organic results found in SerpAPI response for: {query}")
return { # If no main content found, use the body
"query": query, if not main_content:
"maxResults": maxResults, main_content = soup.find('body') or soup
"results": results,
"totalFound": len(results),
"timestamp": datetime.now(UTC).isoformat()
}
except Exception as e: # Remove script, style, nav, footer elements that don't contribute to main content
logger.error(f"Error searching web: {str(e)}") for element in main_content.select('script, style, nav, footer, header, aside, .sidebar, #sidebar, .comments, #comments, .advertisement, .ads, iframe'):
return { element.extract()
"error": str(e),
"query": query
}
async def crawlPage(self, url: str, depth: int = 1, followLinks: bool = True, extractContent: bool = True) -> Dict[str, Any]: # Extract text content
"""Crawl web page and extract content""" text_content = main_content.get_text(separator=' ', strip=True)
try:
# Read the URL
soup = self._readUrl(url)
if not soup:
return {
"error": "Failed to read URL",
"url": url
}
# Extract basic information # Limit to max_chars
title = self._extractTitle(soup, url) return text_content[:max_chars]
content = self._extractMainContent(soup) if extractContent else ""
# Extract links if requested
links = []
if followLinks:
for link in soup.find_all('a', href=True):
href = link.get('href')
if href and href.startswith(('http://', 'https://')):
links.append({
'url': href,
'text': link.get_text(strip=True)[:100]
})
# Extract images
images = []
for img in soup.find_all('img', src=True):
src = img.get('src')
if src:
images.append({
'src': src,
'alt': img.get('alt', ''),
'title': img.get('title', '')
})
return {
"url": url,
"depth": depth,
"followLinks": followLinks,
"extractContent": extractContent,
"title": title,
"content": content,
"links": links[:10], # Limit to first 10 links
"images": images[:10], # Limit to first 10 images
"timestamp": datetime.now(UTC).isoformat()
}
except Exception as e:
logger.error(f"Error crawling web page: {str(e)}")
return {
"error": str(e),
"url": url
}
async def extractContent(self, url: str, selectors: Dict[str, str] = None, format: str = "text") -> Dict[str, Any]:
"""Extract content from web page using selectors"""
try:
# Read the URL
soup = self._readUrl(url)
if not soup:
return {
"error": "Failed to read URL",
"url": url
}
extracted_content = {}
if selectors:
# Extract content using provided selectors
for selector_name, selector in selectors.items():
elements = soup.select(selector)
if elements:
if format == "text":
extracted_content[selector_name] = [elem.get_text(strip=True) for elem in elements]
elif format == "html":
extracted_content[selector_name] = [str(elem) for elem in elements]
else:
extracted_content[selector_name] = [elem.get_text(strip=True) for elem in elements]
else:
extracted_content[selector_name] = []
else:
# Auto-extract common elements
extracted_content = {
"title": self._extractTitle(soup, url),
"main_content": self._extractMainContent(soup),
"headings": [h.get_text(strip=True) for h in soup.find_all(['h1', 'h2', 'h3'])],
"links": [a.get('href') for a in soup.find_all('a', href=True) if a.get('href').startswith(('http://', 'https://'))],
"images": [img.get('src') for img in soup.find_all('img', src=True)]
}
return {
"url": url,
"selectors": selectors,
"format": format,
"content": extracted_content,
"timestamp": datetime.now(UTC).isoformat()
}
except Exception as e:
logger.error(f"Error extracting content: {str(e)}")
return {
"error": str(e),
"url": url
}
async def validatePage(self, url: str, checks: List[str] = None) -> Dict[str, Any]:
"""Validate web page for various criteria"""
if checks is None:
checks = ["accessibility", "seo", "performance"]
try:
# Read the URL
soup = self._readUrl(url)
if not soup:
return {
"error": "Failed to read URL",
"url": url
}
validation_results = {}
for check in checks:
if check == "accessibility":
validation_results["accessibility"] = self._checkAccessibility(soup)
elif check == "seo":
validation_results["seo"] = self._checkSEO(soup)
elif check == "performance":
validation_results["performance"] = self._checkPerformance(soup, url)
else:
validation_results[check] = {"status": "unknown", "message": f"Unknown check type: {check}"}
return {
"url": url,
"checks": checks,
"results": validation_results,
"timestamp": datetime.now(UTC).isoformat()
}
except Exception as e:
logger.error(f"Error validating web page: {str(e)}")
return {
"error": str(e),
"url": url
}
def _checkAccessibility(self, soup: BeautifulSoup) -> Dict[str, Any]: def _checkAccessibility(self, soup: BeautifulSoup) -> Dict[str, Any]:
"""Check basic accessibility features""" """Check basic accessibility features"""
@ -355,96 +214,203 @@ class WebService:
} }
} }
def _readUrl(self, url: str) -> BeautifulSoup: @action
"""Read a URL and return a BeautifulSoup parser for the content""" async def crawl(self, parameters: Dict[str, Any]) -> ActionResult:
if not url or not url.startswith(('http://', 'https://')): """
return None Crawl web pages and extract content
headers = {
'User-Agent': self.user_agent,
'Accept': 'text/html,application/xhtml+xml,application/xml',
'Accept-Language': 'en-US,en;q=0.9',
}
Parameters:
urls (List[str]): List of URLs to crawl
maxDepth (int, optional): Maximum crawl depth (default: 2)
includeImages (bool, optional): Whether to include images (default: False)
followLinks (bool, optional): Whether to follow links (default: True)
"""
try: try:
# Initial request urls = parameters.get("urls")
response = requests.get(url, headers=headers, timeout=self.timeout) maxDepth = parameters.get("maxDepth", 2)
includeImages = parameters.get("includeImages", False)
followLinks = parameters.get("followLinks", True)
# Handling for status 202 if not urls:
if response.status_code == 202: return self._createResult(
# Retry with backoff success=False,
backoff_times = [0.5, 1.0, 2.0, 5.0] data={},
error="URLs are required"
)
for wait_time in backoff_times: # Crawl each URL
time.sleep(wait_time) crawl_results = []
response = requests.get(url, headers=headers, timeout=self.timeout)
if response.status_code != 202: for url in urls:
break try:
# Read the URL
soup = self._readUrl(url)
if not soup:
crawl_results.append({
"error": "Failed to read URL",
"url": url
})
continue
# Raise for error status codes # Extract basic information
response.raise_for_status() title = self._extractTitle(soup, url)
content = self._extractMainContent(soup) if True else ""
# Parse HTML # Extract links if requested
return BeautifulSoup(response.text, 'html.parser') links = []
if followLinks:
for link in soup.find_all('a', href=True):
href = link.get('href')
if href and href.startswith(('http://', 'https://')):
links.append({
'url': href,
'text': link.get_text(strip=True)[:100]
})
# Extract images
images = []
for img in soup.find_all('img', src=True):
src = img.get('src')
if src:
images.append({
'src': src,
'alt': img.get('alt', ''),
'title': img.get('title', '')
})
crawl_results.append({
"url": url,
"depth": maxDepth,
"followLinks": followLinks,
"extractContent": True,
"title": title,
"content": content,
"links": links[:10], # Limit to first 10 links
"images": images[:10], # Limit to first 10 images
"timestamp": datetime.now(UTC).isoformat()
})
except Exception as e: except Exception as e:
logger.error(f"Error reading URL {url}: {str(e)}") logger.error(f"Error crawling web page {url}: {str(e)}")
return None crawl_results.append({
"error": str(e),
"url": url
})
def _extractTitle(self, soup: BeautifulSoup, url: str) -> str: # Create result data
"""Extract the title from a webpage""" result_data = {
"urls": urls,
"maxDepth": maxDepth,
"includeImages": includeImages,
"followLinks": followLinks,
"crawlResults": crawl_results,
"timestamp": datetime.now(UTC).isoformat()
}
return self._createResult(
success=True,
data={
"documentName": f"web_crawl_{datetime.now(UTC).strftime('%Y%m%d_%H%M%S')}.json",
"documentData": result_data
}
)
except Exception as e:
logger.error(f"Error crawling web pages: {str(e)}")
return self._createResult(
success=False,
data={},
error=str(e)
)
@action
async def scrape(self, parameters: Dict[str, Any]) -> ActionResult:
"""
Scrape specific data from web pages
Parameters:
url (str): URL to scrape
selectors (Dict[str, str]): CSS selectors for data extraction
format (str, optional): Output format (default: "json")
"""
try:
url = parameters.get("url")
selectors = parameters.get("selectors")
format = parameters.get("format", "json")
if not url or not selectors:
return self._createResult(
success=False,
data={},
error="URL and selectors are required"
)
# Read the URL
soup = self._readUrl(url)
if not soup: if not soup:
return f"Error with {url}" return self._createResult(
success=False,
data={},
error="Failed to read URL"
)
# Extract title from title tag extracted_content = {}
title_tag = soup.find('title')
title = title_tag.text.strip() if title_tag else "No title"
# Alternative: Also look for h1 tags if title tag is missing if selectors:
if title == "No title": # Extract content using provided selectors
h1_tag = soup.find('h1') for selector_name, selector in selectors.items():
if h1_tag: elements = soup.select(selector)
title = h1_tag.text.strip() if elements:
if format == "text":
extracted_content[selector_name] = [elem.get_text(strip=True) for elem in elements]
elif format == "html":
extracted_content[selector_name] = [str(elem) for elem in elements]
else:
extracted_content[selector_name] = [elem.get_text(strip=True) for elem in elements]
else:
extracted_content[selector_name] = []
else:
# Auto-extract common elements
extracted_content = {
"title": self._extractTitle(soup, url),
"main_content": self._extractMainContent(soup),
"headings": [h.get_text(strip=True) for h in soup.find_all(['h1', 'h2', 'h3'])],
"links": [a.get('href') for a in soup.find_all('a', href=True) if a.get('href').startswith(('http://', 'https://'))],
"images": [img.get('src') for img in soup.find_all('img', src=True)]
}
return title scrape_result = {
"url": url,
"selectors": selectors,
"format": format,
"content": extracted_content,
"timestamp": datetime.now(UTC).isoformat()
}
def _extractMainContent(self, soup: BeautifulSoup, max_chars: int = 10000) -> str: # Create result data
"""Extract the main content from an HTML page""" result_data = {
if not soup: "url": url,
return "" "selectors": selectors,
"format": format,
"scrapedData": scrape_result,
"timestamp": datetime.now(UTC).isoformat()
}
# Try to find main content elements in priority order return self._createResult(
main_content = None success=True,
for selector in ['main', 'article', '#content', '.content', '#main', '.main']: data={
content = soup.select_one(selector) "documentName": f"web_scrape_{datetime.now(UTC).strftime('%Y%m%d_%H%M%S')}.{format}",
if content: "documentData": result_data
main_content = content }
break )
# If no main content found, use the body except Exception as e:
if not main_content: logger.error(f"Error scraping web page: {str(e)}")
main_content = soup.find('body') or soup return self._createResult(
success=False,
# Remove script, style, nav, footer elements that don't contribute to main content data={},
for element in main_content.select('script, style, nav, footer, header, aside, .sidebar, #sidebar, .comments, #comments, .advertisement, .ads, iframe'): error=str(e)
element.extract() )
# Extract text content
text_content = main_content.get_text(separator=' ', strip=True)
# Limit to max_chars
return text_content[:max_chars]
class MethodWeb(MethodBase):
"""Web method implementation for web operations"""
def __init__(self, serviceContainer: Any):
"""Initialize the web method"""
super().__init__(serviceContainer)
self.name = "web"
self.description = "Handle web operations like searching and crawling"
self.webService = WebService(serviceContainer)
@action @action
async def search(self, parameters: Dict[str, Any]) -> ActionResult: async def search(self, parameters: Dict[str, Any]) -> ActionResult:
@ -453,11 +419,15 @@ class MethodWeb(MethodBase):
Parameters: Parameters:
query (str): Search query query (str): Search query
engine (str, optional): Search engine to use (default: "google")
maxResults (int, optional): Maximum number of results (default: 10) maxResults (int, optional): Maximum number of results (default: 10)
filter (str, optional): Additional search filters
""" """
try: try:
query = parameters.get("query") query = parameters.get("query")
engine = parameters.get("engine", "google")
maxResults = parameters.get("maxResults", 10) maxResults = parameters.get("maxResults", 10)
filter = parameters.get("filter")
if not query: if not query:
return self._createResult( return self._createResult(
@ -466,15 +436,101 @@ class MethodWeb(MethodBase):
error="Search query is required" error="Search query is required"
) )
# Search web # Search web content using Google search via SerpAPI
results = await self.webService.searchWeb( try:
query=query, if not self.srcApikey:
maxResults=maxResults search_result = {
) "error": "SerpAPI key not configured",
"query": query
}
else:
# Get user language from service container if available
userLanguage = "en" # Default language
if hasattr(self.serviceContainer, 'user') and hasattr(self.serviceContainer.user, 'language'):
userLanguage = self.serviceContainer.user.language
# Format the search request for SerpAPI
params = {
"engine": self.srcEngine,
"q": query,
"api_key": self.srcApikey,
"num": min(maxResults, self.maxResults), # Number of results to return
"hl": userLanguage # User language
}
# Make the API request
response = requests.get("https://serpapi.com/search", params=params, timeout=self.timeout)
response.raise_for_status()
# Parse JSON response
search_results = response.json()
# Extract organic results
results = []
if "organic_results" in search_results:
for result in search_results["organic_results"][:maxResults]:
# Extract title
title = result.get("title", "No title")
# Extract URL
url = result.get("link", "No URL")
# Extract snippet
snippet = result.get("snippet", "No description")
# Get actual page content
try:
targetPageSoup = self._readUrl(url)
content = self._extractMainContent(targetPageSoup)
except Exception as e:
logger.warning(f"Error extracting content from {url}: {str(e)}")
content = f"Error extracting content: {str(e)}"
results.append({
'title': title,
'url': url,
'snippet': snippet,
'content': content
})
# Limit number of results
if len(results) >= maxResults:
break
else:
logger.warning(f"No organic results found in SerpAPI response for: {query}")
search_result = {
"query": query,
"maxResults": maxResults,
"results": results,
"totalFound": len(results),
"timestamp": datetime.now(UTC).isoformat()
}
except Exception as e:
logger.error(f"Error searching web: {str(e)}")
search_result = {
"error": str(e),
"query": query
}
# Create result data
result_data = {
"query": query,
"engine": engine,
"maxResults": maxResults,
"filter": filter,
"searchResults": search_result,
"timestamp": datetime.now(UTC).isoformat()
}
return self._createResult( return self._createResult(
success=True, success=True,
data=results data={
"documentName": f"web_search_{datetime.now(UTC).strftime('%Y%m%d_%H%M%S')}.json",
"documentData": result_data
}
) )
except Exception as e: except Exception as e:
@ -485,97 +541,10 @@ class MethodWeb(MethodBase):
error=str(e) error=str(e)
) )
@action
async def crawl(self, parameters: Dict[str, Any]) -> ActionResult:
"""
Crawl web page
Parameters:
url (str): URL to crawl
depth (int, optional): Crawl depth (default: 1)
followLinks (bool, optional): Whether to follow links (default: True)
extractContent (bool, optional): Whether to extract content (default: True)
"""
try:
url = parameters.get("url")
depth = parameters.get("depth", 1)
followLinks = parameters.get("followLinks", True)
extractContent = parameters.get("extractContent", True)
if not url:
return self._createResult(
success=False,
data={},
error="URL is required"
)
# Crawl page
results = await self.webService.crawlPage(
url=url,
depth=depth,
followLinks=followLinks,
extractContent=extractContent
)
return self._createResult(
success=True,
data=results
)
except Exception as e:
logger.error(f"Error crawling web page: {str(e)}")
return self._createResult(
success=False,
data={},
error=str(e)
)
@action
async def extract(self, parameters: Dict[str, Any]) -> ActionResult:
"""
Extract content from web page
Parameters:
url (str): URL to extract content from
selectors (Dict[str, str], optional): CSS selectors for specific content
format (str, optional): Output format (default: "text")
"""
try:
url = parameters.get("url")
selectors = parameters.get("selectors", {})
format = parameters.get("format", "text")
if not url:
return self._createResult(
success=False,
data={},
error="URL is required"
)
# Extract content
content = await self.webService.extractContent(
url=url,
selectors=selectors,
format=format
)
return self._createResult(
success=True,
data=content
)
except Exception as e:
logger.error(f"Error extracting content: {str(e)}")
return self._createResult(
success=False,
data={},
error=str(e)
)
@action @action
async def validate(self, parameters: Dict[str, Any]) -> ActionResult: async def validate(self, parameters: Dict[str, Any]) -> ActionResult:
""" """
Validate web page Validate web pages for various criteria
Parameters: Parameters:
url (str): URL to validate url (str): URL to validate
@ -592,15 +561,48 @@ class MethodWeb(MethodBase):
error="URL is required" error="URL is required"
) )
# Validate page # Read the URL
results = await self.webService.validatePage( soup = self._readUrl(url)
url=url, if not soup:
checks=checks return self._createResult(
success=False,
data={},
error="Failed to read URL"
) )
validation_results = {}
for check in checks:
if check == "accessibility":
validation_results["accessibility"] = self._checkAccessibility(soup)
elif check == "seo":
validation_results["seo"] = self._checkSEO(soup)
elif check == "performance":
validation_results["performance"] = self._checkPerformance(soup, url)
else:
validation_results[check] = {"status": "unknown", "message": f"Unknown check type: {check}"}
validation_result = {
"url": url,
"checks": checks,
"results": validation_results,
"timestamp": datetime.now(UTC).isoformat()
}
# Create result data
result_data = {
"url": url,
"checks": checks,
"validationResult": validation_result,
"timestamp": datetime.now(UTC).isoformat()
}
return self._createResult( return self._createResult(
success=True, success=True,
data=results data={
"documentName": f"web_validation_{datetime.now(UTC).strftime('%Y%m%d_%H%M%S')}.json",
"documentData": result_data
}
) )
except Exception as e: except Exception as e:

1292
modules/workflow/managerChat.py
View file

File diff suppressed because it is too large Load diff

500
modules/workflow/managerWorkflow.py
View file

@ -29,383 +29,8 @@ class WorkflowManager:
if workflow.status == "stopped": if workflow.status == "stopped":
raise WorkflowStoppedException("Workflow was stopped by user") raise WorkflowStoppedException("Workflow was stopped by user")
async def workflowProcess(self, userInput: UserInputRequest, workflow: ChatWorkflow) -> TaskItem: async def workflowProcess(self, userInput: UserInputRequest, workflow: ChatWorkflow) -> None:
"""Enhanced workflow process with proper task planning and handover review""" """Process a workflow with user input using unified workflow phases"""
try:
logger.info(f"Processing workflow: {workflow.id}")
# Phase 1: Create initial message with user request and documents
initial_message = await self._createInitialMessage(userInput, workflow)
if not initial_message:
raise Exception("Failed to create initial message")
# Phase 2: Generate task plan through AI analysis
task_plan = await self._generateTaskPlan(userInput, workflow, initial_message)
if not task_plan:
raise Exception("Failed to generate task plan")
# Phase 3: Execute tasks with handover review
task_result = await self._executeTaskPlan(task_plan, workflow, userInput)
return task_result
except Exception as e:
logger.error(f"Error in workflowProcess: {str(e)}")
raise
async def _createInitialMessage(self, userInput: UserInputRequest, workflow: ChatWorkflow) -> ChatMessage:
"""Create initial message with user request and processed documents"""
try:
# Initialize chat manager with workflow
await self.chatManager.initialize(workflow)
# Process file IDs into ChatDocument objects
documents = await self.chatManager.processFileIds(userInput.listFileId)
# Create message data
message_data = {
"id": f"msg_{uuid.uuid4()}",
"workflowId": workflow.id,
"role": "user",
"agentName": "",
"message": userInput.prompt,
"documents": documents,
"status": "step",
"publishedAt": self._getCurrentTimestamp()
}
# Create message in database
message = self.chatInterface.createWorkflowMessage(message_data)
if not message:
raise Exception("Failed to create workflow message")
logger.info(f"Created initial message: {message.id} with {len(documents)} documents")
return message
except Exception as e:
logger.error(f"Error creating initial message: {str(e)}")
return None
async def _generateTaskPlan(self, userInput: UserInputRequest, workflow: ChatWorkflow, initial_message: ChatMessage) -> Dict[str, Any]:
"""Generate task plan through AI analysis"""
try:
# Prepare context for AI analysis
context = {
"user_request": userInput.prompt,
"available_documents": [doc.filename for doc in initial_message.documents],
"workflow_id": workflow.id,
"message_id": initial_message.id
}
# Generate task plan using AI
task_plan = await self.chatManager.generateTaskPlan(context)
logger.info(f"Generated task plan with {len(task_plan.get('tasks', []))} tasks")
return task_plan
except Exception as e:
logger.error(f"Error generating task plan: {str(e)}")
return None
async def _executeTaskPlan(self, task_plan: Dict[str, Any], workflow: ChatWorkflow, userInput: UserInputRequest) -> TaskItem:
"""Execute task plan with handover review and enhanced error recovery"""
try:
tasks = task_plan.get('tasks', [])
if not tasks:
raise Exception("No tasks in task plan")
# Create main task item
task_data = {
"id": f"task_{uuid.uuid4()}",
"workflowId": workflow.id,
"userInput": userInput.prompt,
"status": TaskStatus.RUNNING,
"feedback": task_plan.get('overview', 'Executing task plan'),
"startedAt": self._getCurrentTimestamp(),
"actionList": [],
"taskPlan": task_plan
}
task = self.chatInterface.createTask(task_data)
if not task:
raise Exception("Failed to create task")
# Ensure task is saved to database
logger.info(f"Created task with ID: {task.id}")
# Execute each task with enhanced error recovery
for i, task_step in enumerate(tasks):
logger.info(f"Executing task {i+1}/{len(tasks)}: {task_step.get('description', 'Unknown')}")
# Execute task step with retry mechanism
step_result = await self._executeTaskStepWithRetry(task_step, workflow, task)
# Enhanced handover review
review_result = await self._performEnhancedHandoverReview(step_result, task_step, workflow, task)
if review_result['status'] == 'failed':
# Try alternative approach before giving up
alternative_result = await self._tryAlternativeApproach(task_step, workflow, task, review_result)
if alternative_result['status'] == 'failed':
# Update task status with detailed feedback
update_result = self.chatInterface.updateTask(task.id, {
"status": TaskStatus.FAILED,
"feedback": f"Task failed at step {i+1}: {review_result['reason']}. Alternative approach also failed.",
"errorDetails": {
"failedStep": task_step,
"originalError": review_result['reason'],
"suggestions": self._generateFailureSuggestions(task_step, review_result)
}
})
if not update_result:
logger.error(f"Failed to update task {task.id} status to FAILED")
return task
else:
step_result = alternative_result
review_result = {'status': 'success'}
elif review_result['status'] == 'retry':
# Retry with improved approach
logger.info(f"Retrying task step {i+1} with improved approach")
step_result = await self._executeTaskStepWithRetry(task_step, workflow, task, improvements=review_result.get('improvements'))
review_result = await self._performEnhancedHandoverReview(step_result, task_step, workflow, task)
if review_result['status'] == 'failed':
update_result = self.chatInterface.updateTask(task.id, {
"status": TaskStatus.FAILED,
"feedback": f"Task failed after retry at step {i+1}: {review_result['reason']}"
})
if not update_result:
logger.error(f"Failed to update task {task.id} status to FAILED after retry")
return task
# Add step result to task
if step_result and step_result.get('actions'):
for action in step_result['actions']:
# Convert action format to TaskAction format
task_action_data = {
"execMethod": action.get('method', 'unknown'),
"execAction": action.get('action', 'unknown'),
"execParameters": action.get('parameters', {}),
"execResultLabel": action.get('resultLabel', ''),
"status": TaskStatus.PENDING
}
task_action = self.chatInterface.createTaskAction(task_action_data)
if task_action:
task.actionList.append(task_action)
logger.info(f"Created task action: {task_action.execMethod}.{task_action.execAction}")
else:
logger.error(f"Failed to create task action: {action}")
# Update progress
self._updateTaskProgress(task, i + 1, len(tasks))
# Update task as completed
update_result = self.chatInterface.updateTask(task.id, {
"status": TaskStatus.COMPLETED,
"feedback": f"Successfully completed {len(tasks)} tasks with {len(task.actionList)} total actions",
"finishedAt": self._getCurrentTimestamp(),
"successMetrics": {
"totalTasks": len(tasks),
"totalActions": len(task.actionList),
"executionTime": self._calculateExecutionTime(task.startedAt)
}
})
if not update_result:
logger.error(f"Failed to update task {task.id} status to COMPLETED")
return task
except Exception as e:
logger.error(f"Error executing task plan: {str(e)}")
raise
async def _executeTaskStepWithRetry(self, task_step: Dict[str, Any], workflow: ChatWorkflow, task: TaskItem, max_retries: int = 3, improvements: str = None) -> Dict[str, Any]:
"""Execute task step with exponential backoff retry mechanism"""
last_error = None
for attempt in range(max_retries + 1):
try:
# Add exponential backoff delay for retries
if attempt > 0:
delay = min(2 ** attempt, 30) # Max 30 seconds
await asyncio.sleep(delay)
logger.info(f"Retry attempt {attempt} for task step: {task_step.get('description', 'Unknown')}")
# Execute task step
step_result = await self._executeTaskStep(task_step, workflow, task, improvements)
# Quick validation
if step_result.get('status') == 'completed':
return step_result
else:
last_error = step_result.get('error', 'Unknown error')
except Exception as e:
last_error = str(e)
logger.warning(f"Attempt {attempt + 1} failed for task step: {str(e)}")
# All retries exhausted
return {
'task_step': task_step,
'error': f"All {max_retries + 1} attempts failed. Last error: {last_error}",
'status': 'failed',
'retryAttempts': max_retries + 1
}
async def _performEnhancedHandoverReview(self, step_result: Dict[str, Any], task_step: Dict[str, Any], workflow: ChatWorkflow, task: TaskItem) -> Dict[str, Any]:
"""Enhanced handover review with quality assessment"""
try:
# Prepare enhanced review context
review_context = {
'task_step': task_step,
'step_result': step_result,
'workflow_id': workflow.id,
'task_id': task.id,
'previous_results': self._getPreviousResults(task)
}
# Use AI to review the results
review = await self.chatManager.reviewTaskStepResults(review_context)
# Add quality metrics
review['quality_metrics'] = await self._calculateQualityMetrics(step_result, task_step)
return review
except Exception as e:
logger.error(f"Error in enhanced handover review: {str(e)}")
return {
'status': 'failed',
'reason': f'Review failed: {str(e)}',
'quality_metrics': {'score': 0, 'confidence': 0}
}
async def _tryAlternativeApproach(self, task_step: Dict[str, Any], workflow: ChatWorkflow, task: TaskItem, original_review: Dict[str, Any]) -> Dict[str, Any]:
"""Try alternative approach when original method fails"""
try:
logger.info(f"Trying alternative approach for task step: {task_step.get('description', 'Unknown')}")
# Generate alternative approach based on failure analysis
alternative_prompt = self._createAlternativeApproachPrompt(task_step, original_review)
alternative_response = await self.chatManager._callAI(alternative_prompt, "alternative_approach")
# Parse alternative approach
alternative_approach = self._parseAlternativeApproach(alternative_response)
if alternative_approach:
# Execute alternative approach
step_result = await self._executeTaskStep(task_step, workflow, task, alternative_approach)
return step_result
else:
return {
'task_step': task_step,
'error': 'Could not generate alternative approach',
'status': 'failed'
}
except Exception as e:
logger.error(f"Error trying alternative approach: {str(e)}")
return {
'task_step': task_step,
'error': f'Alternative approach failed: {str(e)}',
'status': 'failed'
}
def _generateFailureSuggestions(self, task_step: Dict[str, Any], review_result: Dict[str, Any]) -> List[str]:
"""Generate helpful suggestions when tasks fail"""
suggestions = []
if 'missing_outputs' in review_result:
suggestions.append(f"Ensure all expected outputs are produced: {', '.join(review_result['missing_outputs'])}")
if 'unmet_criteria' in review_result:
suggestions.append(f"Address unmet success criteria: {', '.join(review_result['unmet_criteria'])}")
suggestions.append("Check if all required documents are available and accessible")
suggestions.append("Verify that the task step has all necessary dependencies completed")
return suggestions
async def _calculateQualityMetrics(self, step_result: Dict[str, Any], task_step: Dict[str, Any]) -> Dict[str, Any]:
"""Calculate quality metrics for task step results"""
try:
quality_score = 0
confidence = 0
if step_result.get('status') == 'completed':
quality_score = 8 # Base score for completion
# Check if all expected outputs were produced
expected_outputs = task_step.get('expected_outputs', [])
produced_outputs = step_result.get('outputs', [])
output_coverage = len(set(produced_outputs) & set(expected_outputs)) / len(expected_outputs) if expected_outputs else 1
quality_score += output_coverage * 2
confidence = min(quality_score / 10, 1.0)
return {
'score': min(quality_score, 10),
'confidence': confidence
}
except Exception as e:
logger.error(f"Error calculating quality metrics: {str(e)}")
return {'score': 0, 'confidence': 0}
def _updateTaskProgress(self, task: TaskItem, current_step: int, total_steps: int):
"""Update task progress information"""
progress = (current_step / total_steps) * 100
logger.info(f"Task progress: {progress:.1f}% ({current_step}/{total_steps})")
def _calculateExecutionTime(self, started_at: str) -> float:
"""Calculate execution time in seconds"""
try:
start_time = datetime.fromisoformat(started_at.replace('Z', '+00:00'))
end_time = datetime.now(UTC)
return (end_time - start_time).total_seconds()
except Exception:
return 0.0
def _getPreviousResults(self, task: TaskItem) -> List[str]:
"""Get list of previous results from completed actions"""
results = []
for action in task.actionList:
if action.execResultLabel:
results.append(action.execResultLabel)
return results
def _createAlternativeApproachPrompt(self, task_step: Dict[str, Any], original_review: Dict[str, Any]) -> str:
"""Create prompt for generating alternative approaches"""
return f"""The original approach for this task step failed. Please suggest an alternative approach.
TASK STEP: {task_step.get('description', 'Unknown')}
ORIGINAL FAILURE: {original_review.get('reason', 'Unknown error')}
MISSING OUTPUTS: {', '.join(original_review.get('missing_outputs', []))}
Please provide an alternative approach that addresses these issues."""
def _parseAlternativeApproach(self, response: str) -> Optional[str]:
"""Parse alternative approach from AI response"""
try:
# Simple parsing - extract the approach description
if "approach:" in response.lower():
lines = response.split('\n')
for line in lines:
if "approach:" in line.lower():
return line.split(":", 1)[1].strip()
return None
except Exception:
return None
def _getCurrentTimestamp(self) -> str:
"""Get current timestamp in ISO format"""
return datetime.now(UTC).isoformat()
async def workflowProcess_ORIGINAL_TEMPORARY_DEACTIVATED(self, userInput: UserInputRequest, workflow: ChatWorkflow) -> None:
"""Process a workflow with user input"""
try: try:
# Initialize chat manager # Initialize chat manager
await self.chatManager.initialize(workflow) await self.chatManager.initialize(workflow)
@ -416,31 +41,11 @@ Please provide an alternative approach that addresses these issues."""
# Send first message # Send first message
message = await self._sendFirstMessage(userInput, workflow) message = await self._sendFirstMessage(userInput, workflow)
# Create initial task # Execute unified workflow
task = await self.chatManager.createInitialTask(workflow, message) workflow_result = await self.chatManager.executeUnifiedWorkflow(userInput.prompt, workflow)
# Process workflow # Process workflow results
while True: await self._processWorkflowResults(workflow, workflow_result, message)
# Check if workflow is stopped
self._checkWorkflowStopped(workflow)
# Execute task
result = await self.chatManager.executeTask(task)
# Process result
await self.chatManager.parseTaskResult(workflow, result)
# Check if workflow should continue
if not await self.chatManager.shouldContinue(workflow):
break
# Identify next task
nextTaskResult = await self.chatManager.identifyNextTask(workflow)
# Create next task
task = await self.chatManager.createNextTask(workflow, nextTaskResult)
if not task:
break
# Send last message # Send last message
await self._sendLastMessage(workflow) await self._sendLastMessage(workflow)
@ -507,29 +112,80 @@ Please provide an alternative approach that addresses these issues."""
logger.error(f"Error sending last message: {str(e)}") logger.error(f"Error sending last message: {str(e)}")
raise raise
async def _executeTaskStep(self, task_step: Dict[str, Any], workflow: ChatWorkflow, task: TaskItem, improvements: str = None) -> Dict[str, Any]: async def _processWorkflowResults(self, workflow: ChatWorkflow, workflow_result: Dict[str, Any], initial_message: ChatMessage) -> None:
"""Execute a single task step and generate actions""" """Process workflow results and create appropriate messages"""
try: try:
# Generate actions for this task step if workflow_result.get('status') == 'failed':
actions = await self.chatManager.generateActionsForTask(task_step, workflow, task, improvements) # Create error message
error_message = {
# Execute actions "workflowId": workflow.id,
results = [] "role": "assistant",
for action in actions: "message": f"Workflow failed: {workflow_result.get('error', 'Unknown error')}",
action_result = await self.chatManager.executeAction(action, workflow) "status": "last",
results.append(action_result) "sequenceNr": len(workflow.messages) + 1,
"publishedAt": datetime.now(UTC).isoformat()
return {
'task_step': task_step,
'actions': actions,
'results': results,
'status': 'completed'
} }
message = self.chatInterface.createWorkflowMessage(error_message)
if message:
workflow.messages.append(message)
return
# Process successful workflow results
workflow_results = workflow_result.get('workflow_results', [])
for i, result in enumerate(workflow_results):
task_step = result['task_step']
action_results = result['action_results']
review_result = result['review_result']
# Create message for task step
step_message = {
"workflowId": workflow.id,
"role": "assistant",
"message": f"Completed task: {task_step.get('description', 'Unknown')}",
"status": "step",
"sequenceNr": len(workflow.messages) + 1,
"publishedAt": datetime.now(UTC).isoformat()
}
# Add action details if available
if action_results:
successful_actions = [r for r in action_results if r.get('status') == 'completed']
step_message["message"] += f"\nExecuted {len(successful_actions)}/{len(action_results)} actions successfully."
message = self.chatInterface.createWorkflowMessage(step_message)
if message:
workflow.messages.append(message)
# Create final summary message
successful_tasks = workflow_result.get('successful_tasks', 0)
total_tasks = workflow_result.get('total_tasks', 0)
summary_message = {
"workflowId": workflow.id,
"role": "assistant",
"message": f"Workflow completed successfully. Completed {successful_tasks}/{total_tasks} tasks.",
"status": "last",
"sequenceNr": len(workflow.messages) + 1,
"publishedAt": datetime.now(UTC).isoformat()
}
message = self.chatInterface.createWorkflowMessage(summary_message)
if message:
workflow.messages.append(message)
except Exception as e: except Exception as e:
logger.error(f"Error executing task step: {str(e)}") logger.error(f"Error processing workflow results: {str(e)}")
return { # Create error message
'task_step': task_step, error_message = {
'error': str(e), "workflowId": workflow.id,
'status': 'failed' "role": "assistant",
"message": f"Error processing workflow results: {str(e)}",
"status": "last",
"sequenceNr": len(workflow.messages) + 1,
"publishedAt": datetime.now(UTC).isoformat()
} }
message = self.chatInterface.createWorkflowMessage(error_message)
if message:
workflow.messages.append(message)

84
modules/workflow/methodBase.py
View file

@ -5,7 +5,7 @@ from pydantic import BaseModel, Field
import logging import logging
from modules.interfaces.interfaceChatModel import ActionResult from modules.interfaces.interfaceChatModel import ActionResult
from functools import wraps from functools import wraps
from inspect import signature import inspect
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@ -38,7 +38,7 @@ class MethodBase:
try: try:
attr = getattr(self, attr_name) attr = getattr(self, attr_name)
if callable(attr) and getattr(attr, 'is_action', False): if callable(attr) and getattr(attr, 'is_action', False):
sig = signature(attr) sig = inspect.signature(attr)
params = {} params = {}
for param_name, param in sig.parameters.items(): for param_name, param in sig.parameters.items():
if param_name not in ['self', 'parameters', 'authData']: if param_name not in ['self', 'parameters', 'authData']:
@ -60,24 +60,92 @@ class MethodBase:
return actions return actions
def getActionSignature(self, actionName: str) -> str: def getActionSignature(self, actionName: str) -> str:
"""Get formatted action signature for AI prompt generation""" """Get formatted action signature for AI prompt generation (detailed version)"""
if actionName not in self.actions: if actionName not in self.actions:
return "" return ""
action = self.actions[actionName] action = self.actions[actionName]
paramList = [] paramList = []
for paramName, param in action['parameters'].items(): # Extract detailed parameter information from docstring
paramType = self._formatType(param['type']) docstring = action.get('description', '')
paramDescriptions, paramTypes = self._extractParameterDetails(docstring)
for paramName in paramDescriptions:
paramType = paramTypes.get(paramName, 'Any')
paramDesc = paramDescriptions.get(paramName, '')
# Mark required parameters with * if possible (not available from docstring, so omit)
if paramDesc:
paramList.append(f"{paramName}:{paramType} # {paramDesc}")
else:
paramList.append(f"{paramName}:{paramType}") paramList.append(f"{paramName}:{paramType}")
signature = f"{self.name}.{actionName}([{', '.join(paramList)}])" signature = f"{self.name}.{actionName}"
if action.get('description'): if paramList:
signature += f" # {action['description']}" signature += f"({', '.join(paramList)})"
# Add return type and main description
returnType = "ActionResult"
mainDesc = self._extractMainDescription(docstring)
if mainDesc:
signature += f" -> {returnType} # {mainDesc}"
return signature return signature
def _extractParameterDetails(self, docstring: str):
"""Extract parameter names, types, and descriptions from docstring"""
descriptions = {}
types = {}
if not docstring:
return descriptions, types
lines = docstring.split('\n')
inParameters = False
for line in lines:
line = line.strip()
if 'Parameters:' in line:
inParameters = True
continue
elif inParameters and (line.startswith('Returns:') or line.startswith('Raises:') or line.startswith('Args:')):
break
elif inParameters and line:
# Look for parameter descriptions like "paramName (type): description"
if ':' in line and '(' in line:
parts = line.split(':', 1)
if len(parts) == 2:
paramPart = parts[0].strip()
descPart = parts[1].strip()
# Extract parameter name and type
if '(' in paramPart:
paramName = paramPart.split('(')[0].strip()
paramType = paramPart[paramPart.find('(')+1:paramPart.find(')')].strip()
descriptions[paramName] = descPart
types[paramName] = paramType
# Also handle multi-line descriptions
elif line and not line.startswith('Each document') and not line.startswith('contains'):
if descriptions:
lastParam = list(descriptions.keys())[-1]
descriptions[lastParam] += " " + line
return descriptions, types
def _extractMainDescription(self, docstring: str) -> str:
"""Extract main description from docstring"""
if not docstring:
return ""
lines = docstring.split('\n')
mainDesc = ""
for line in lines:
line = line.strip()
if line and not line.startswith('Parameters:') and not line.startswith('Returns:') and not line.startswith('Raises:'):
mainDesc = line
break
return mainDesc
def _formatType(self, type_annotation) -> str: def _formatType(self, type_annotation) -> str:
"""Format type annotation for display""" """Format type annotation for display"""
if type_annotation == Any: if type_annotation == Any:

40
modules/workflow/serviceContainer.py
View file

@ -147,6 +147,7 @@ class ServiceContainer:
methodList.append(signature) methodList.append(signature)
return methodList return methodList
def getDocumentReferenceList(self) -> Dict[str, List[Dict[str, str]]]: def getDocumentReferenceList(self) -> Dict[str, List[Dict[str, str]]]:
"""Get list of document references sorted by datetime, categorized by chat round""" """Get list of document references sorted by datetime, categorized by chat round"""
chat_refs = [] chat_refs = []
@ -212,7 +213,7 @@ class ServiceContainer:
def getDocumentReferenceFromChatDocument(self, document: ChatDocument) -> str: def getDocumentReferenceFromChatDocument(self, document: ChatDocument) -> str:
"""Get document reference from ChatDocument""" """Get document reference from ChatDocument"""
return f"document_{document.id}_{document.filename}" return f"cdoc:{document.id}:{document.filename}"
def getDocumentReferenceFromMessage(self, message: ChatMessage) -> str: def getDocumentReferenceFromMessage(self, message: ChatMessage) -> str:
"""Get document reference from ChatMessage with action context""" """Get document reference from ChatMessage with action context"""
@ -220,17 +221,17 @@ class ServiceContainer:
return None return None
# If documentsLabel already contains the full reference format, return it # If documentsLabel already contains the full reference format, return it
if message.documentsLabel.startswith("documentList_"): if message.documentsLabel.startswith("mdoc:"):
return message.documentsLabel return message.documentsLabel
# Otherwise construct the reference # Otherwise construct the reference
return f"documentList_{message.actionId}_{message.documentsLabel}" return f"mdoc:{message.actionId}:{message.documentsLabel}"
def getChatDocumentsFromDocumentReference(self, documentReference: str) -> List[ChatDocument]: def getChatDocumentsFromDocumentReference(self, documentReference: str) -> List[ChatDocument]:
"""Get ChatDocuments from document reference""" """Get ChatDocuments from document reference"""
try: try:
# Parse reference format # Parse reference format
parts = documentReference.split('_', 2) # Split into max 3 parts parts = documentReference.split(':', 2) # Split into max 3 parts
if len(parts) < 3: if len(parts) < 3:
return [] return []
@ -238,8 +239,8 @@ class ServiceContainer:
ref_id = parts[1] ref_id = parts[1]
ref_label = parts[2] # Keep the full label ref_label = parts[2] # Keep the full label
if ref_type == "document": if ref_type == "cdoc":
# Handle ChatDocument reference: document_<id>_<filename> # Handle ChatDocument reference: cdoc:<id>:<filename>
# Find document in workflow messages # Find document in workflow messages
for message in self.workflow.messages: for message in self.workflow.messages:
if message.documents: if message.documents:
@ -247,8 +248,8 @@ class ServiceContainer:
if doc.id == ref_id: if doc.id == ref_id:
return [doc] return [doc]
elif ref_type == "documentList": elif ref_type == "mdoc":
# Handle document list reference: documentList_<action.id>_<label> # Handle document list reference: mdoc:<action.id>:<label>
# Find message with matching action ID and documents label # Find message with matching action ID and documents label
for message in self.workflow.messages: for message in self.workflow.messages:
if (message.actionId == ref_id and if (message.actionId == ref_id and
@ -262,34 +263,31 @@ class ServiceContainer:
logger.error(f"Error getting documents from reference {documentReference}: {str(e)}") logger.error(f"Error getting documents from reference {documentReference}: {str(e)}")
return [] return []
def getConnectionReferenceList(self) -> List[Dict[str, str]]: def getConnectionReferenceList(self) -> List[str]:
"""Get list of all UserConnection objects as references""" """Get list of all UserConnection objects as references"""
connections = [] connections = []
# Get user connections through AppObjects interface # Get user connections through AppObjects interface
user_connections = self.interfaceApp.getUserConnections(self.user.id) user_connections = self.interfaceApp.getUserConnections(self.user.id)
for conn in user_connections: for conn in user_connections:
connections.append({ connections.append(self.getConnectionReferenceFromUserConnection(conn))
"connectionReference": f"connection_{conn.id}_{conn.authority}_{conn.externalUsername}", # Sort by connection reference
"authority": conn.authority return sorted(connections)
})
# Sort by authority
return sorted(connections, key=lambda x: x["authority"])
def getConnectionReferenceFromUserConnection(self, connection: UserConnection) -> str: def getConnectionReferenceFromUserConnection(self, connection: UserConnection) -> str:
"""Get connection reference from UserConnection""" """Get connection reference from UserConnection"""
return f"connection_{connection.id}_{connection.authority}_{connection.externalUsername}" return f"connection:{connection.authority}:{connection.externalUsername}:{connection.id}"
def getUserConnectionFromConnectionReference(self, connectionReference: str) -> Optional[UserConnection]: def getUserConnectionFromConnectionReference(self, connectionReference: str) -> Optional[UserConnection]:
"""Get UserConnection from reference string""" """Get UserConnection from reference string"""
try: try:
# Parse reference format: connection_{id}_{authority}_{username} # Parse reference format: connection:{authority}:{username}:{id}
parts = connectionReference.split('_') parts = connectionReference.split(':')
if len(parts) != 4 or parts[0] != "connection": if len(parts) != 4 or parts[0] != "connection":
return None return None
conn_id = parts[1] authority = parts[1]
authority = parts[2] username = parts[2]
username = parts[3] conn_id = parts[3]
# Get user connections through AppObjects interface # Get user connections through AppObjects interface
user_connections = self.interfaceApp.getUserConnections(self.user.id) user_connections = self.interfaceApp.getUserConnections(self.user.id)

226
notes/WORKFLOW_ARCHITECTURE.md Normal file
View file

@ -0,0 +1,226 @@
# Workflow Architecture Documentation
## Overview
The workflow system has been refactored into a clear, structured approach with 5 distinct phases. This eliminates redundancies and provides better error handling, quality assessment, and maintainability.
## Architecture Principles
### 1. **Clear Phase Separation**
Each workflow phase has a specific responsibility and clear inputs/outputs.
### 2. **Unified Data Model**
Standardized on `TaskAction` objects throughout the system.
### 3. **Consistent Prompt Generation**
All AI interactions use dedicated prompt generation functions.
### 4. **Quality Assessment**
Each task is reviewed before proceeding to the next.
## Workflow Phases
### **Phase 1: High-Level Task Planning**
**Function:** `planHighLevelTasks()`
**Purpose:** Analyze user request and create a structured task plan
**Input:** User input, available documents
**Output:** Task plan with multiple task steps
**Prompt Function:** `_createTaskPlanningPrompt()`
```python
task_plan = await chatManager.planHighLevelTasks(userInput, workflow)
```
### **Phase 2: Task Definition and Action Generation**
**Function:** `defineTaskActions()`
**Purpose:** Define specific actions for each task step
**Input:** Task step, workflow context, previous results
**Output:** List of TaskAction objects
**Prompt Function:** `_createActionDefinitionPrompt()`
```python
task_actions = await chatManager.defineTaskActions(task_step, workflow, previous_results)
```
### **Phase 3: Action Execution**
**Function:** `executeTaskActions()`
**Purpose:** Execute all actions for a task step
**Input:** List of TaskAction objects
**Output:** List of action results
**Prompt Function:** `_createActionExecutionPrompt()`
```python
action_results = await chatManager.executeTaskActions(task_actions, workflow)
```
### **Phase 4: Task Review and Quality Assessment**
**Function:** `reviewTaskCompletion()`
**Purpose:** Review task completion and decide next steps
**Input:** Task step, actions, results
**Output:** Review result with quality metrics
**Prompt Function:** `_createResultReviewPrompt()`
```python
review_result = await chatManager.reviewTaskCompletion(task_step, task_actions, action_results, workflow)
```
### **Phase 5: Task Handover and State Management**
**Function:** `prepareTaskHandover()`
**Purpose:** Prepare results for next task or workflow completion
**Input:** Task step, actions, review result
**Output:** Handover data for next iteration
**Prompt Function:** None (data processing only)
```python
handover_data = await chatManager.prepareTaskHandover(task_step, task_actions, review_result, workflow)
```
## Unified Workflow Execution
### **Main Entry Point**
**Function:** `executeUnifiedWorkflow()`
**Purpose:** Orchestrate all phases in sequence
**Input:** User input, workflow
**Output:** Complete workflow results
```python
workflow_result = await chatManager.executeUnifiedWorkflow(userInput.prompt, workflow)
```
### **Workflow Flow**
```
1. planHighLevelTasks() → Task Plan
2. For each task step:
├── defineTaskActions() → Task Actions
├── executeTaskActions() → Action Results
├── reviewTaskCompletion() → Review Result
└── prepareTaskHandover() → Handover Data
3. Return workflow summary
```
## Prompt Generation Functions
| **Function** | **Used In** | **Purpose** |
|-------------|-------------|-------------|
| `_createTaskPlanningPrompt()` | `planHighLevelTasks()` | Generate high-level task plan |
| `_createActionDefinitionPrompt()` | `defineTaskActions()` | Generate specific actions for task |
| `_createActionExecutionPrompt()` | `executeTaskActions()` | Execute individual actions |
| `_createResultReviewPrompt()` | `reviewTaskCompletion()` | Review task completion |
## Data Models
### **TaskAction Object**
```python
class TaskAction:
id: str
execMethod: str
execAction: str
execParameters: Dict[str, Any]
execResultLabel: Optional[str]
status: TaskStatus
error: Optional[str]
result: Optional[str]
# ... other fields
```
### **Workflow Result Structure**
```python
{
'status': 'completed' | 'partial' | 'failed',
'successful_tasks': int,
'total_tasks': int,
'workflow_results': List[Dict],
'final_results': List[str]
}
```
## Error Handling
### **Phase-Level Error Handling**
Each phase has its own error handling:
- **Planning:** Fallback to basic task plan
- **Definition:** Skip task if no actions defined
- **Execution:** Stop on first action failure
- **Review:** Default to success to avoid blocking
- **Handover:** Provide empty results on error
### **Circuit Breaker Pattern**
AI calls use circuit breaker pattern to prevent cascading failures.
## Quality Metrics
### **Task Quality Assessment**
- Success rate of actions
- Completion of expected outputs
- Meeting of success criteria
- Confidence scores
### **Workflow Quality Metrics**
- Overall success rate
- Task completion percentage
- Error patterns and suggestions
## Benefits of Refactored Architecture
### **1. Clear Separation of Concerns**
Each phase has a single responsibility and clear interfaces.
### **2. Better Error Handling**
Granular error handling at each phase with appropriate fallbacks.
### **3. Quality Assessment**
Built-in review and quality metrics for each task.
### **4. Maintainability**
Consistent patterns and unified data models.
### **5. Extensibility**
Easy to add new phases or modify existing ones.
### **6. Debugging**
Clear logging and error reporting at each phase.
## Migration Path
### **Legacy Methods**
All legacy methods are preserved for backward compatibility:
- `createInitialTask()`
- `createNextTask()`
- `executeTask()`
- `executeAction()`
### **New Unified Approach**
Use `executeUnifiedWorkflow()` for new implementations.
## Usage Example
```python
# Initialize chat manager
await chatManager.initialize(workflow)
# Execute unified workflow
workflow_result = await chatManager.executeUnifiedWorkflow(userInput.prompt, workflow)
# Process results
if workflow_result['status'] == 'completed':
print(f"Workflow completed: {workflow_result['successful_tasks']}/{workflow_result['total_tasks']} tasks")
else:
print(f"Workflow failed: {workflow_result['error']}")
```
## Future Enhancements
### **1. Retry Logic**
Add exponential backoff retry for failed tasks.
### **2. Alternative Approaches**
When primary method fails, try alternative approaches.
### **3. Parallel Execution**
Execute independent tasks in parallel.
### **4. Progress Tracking**
Real-time progress updates during workflow execution.
### **5. Rollback Mechanisms**
Undo failed operations and restore previous state.

34
notes/changelog.txt
View file

@ -1,11 +1,43 @@
INIT INIT
conda activate poweron conda activate poweron
pip install -r requirements.txt
cd gateway cd gateway
pip install -r requirements.txt
python app.py python app.py
LOGIC
1. HIGH-LEVEL TASK PLANNING
├── Analyze user request
├── Define major task steps
└── Create task plan with dependencies
2. FOR EACH TASK STEP:
├── TASK DEFINITION
│ ├── Define specific actions for this task
│ └── Set success criteria
│
├── ACTION EXECUTION
│ ├── Execute each action
│ └── Collect results
│
├── TASK REVIEW
│ ├── Evaluate task completion
│ ├── Check success criteria
│ └── Decide: Continue/Retry/Fail
│
└── HANDOVER
├── Prepare results for next task
└── Update workflow state
TODO methods:
- reultLabel not to generate in the function, but to be set according to the action definition
- align documentList objects: chat document to have prefix "cdoc", message document list to have prefix "mdoc" (globally: instead of document and documentList)
- after action execution to store the documents in db and in a message object
TODO TODO
- neutralizer to put back placeholders to the returned data - neutralizer to put back placeholders to the returned data
- referenceHandling and authentication for connections in the method actions - referenceHandling and authentication for connections in the method actions

170
test_document_handover.py
View file

@ -1,170 +0,0 @@
#!/usr/bin/env python3
"""
Test script to demonstrate the improved document handover mechanism.
This shows how documents are properly stored and retrieved between actions.
"""
import sys
import os
sys.path.append(os.path.dirname(os.path.abspath(__file__)))
from modules.workflow.serviceContainer import ServiceContainer
from modules.interfaces.interfaceAppModel import User
from modules.interfaces.interfaceChatModel import ChatWorkflow, ChatMessage, ChatDocument
from datetime import datetime, UTC
import json
def test_document_handover():
"""Test the document handover mechanism"""
# Create test user and workflow
user = User(
id="test_user",
username="testuser",
language="en",
mandateId="test_mandate"
)
workflow = ChatWorkflow(
id="test_workflow",
mandateId="test_mandate",
startedAt=datetime.now(UTC).isoformat(),
status="active",
currentRound=1,
lastActivity=datetime.now(UTC).isoformat(),
messages=[]
)
# Create service container
container = ServiceContainer(user, workflow)
print("=" * 80)
print("DOCUMENT HANDOVER MECHANISM TEST")
print("=" * 80)
# Simulate action execution and document creation
print("\n1. SIMULATING ACTION EXECUTION")
print("-" * 40)
# Simulate first action: SharePoint search
action1_result = {
"result": "Found 5 sales documents in SharePoint",
"resultLabel": "documentList_abc123_sales_documents",
"documents": [
"document_001_sales_report_q1.xlsx",
"document_002_sales_report_q2.xlsx",
"document_003_sales_report_q3.xlsx"
],
"error": None
}
print(f"Action 1 Result: {json.dumps(action1_result, indent=2)}")
# Simulate second action: Excel analysis
action2_result = {
"result": "Analyzed sales data and created summary report",
"resultLabel": "documentList_def456_analysis_results",
"documents": [
"document_004_sales_analysis_summary.xlsx",
"document_005_sales_trends_chart.png"
],
"error": None
}
print(f"\nAction 2 Result: {json.dumps(action2_result, indent=2)}")
# Simulate workflow messages with documents
print("\n2. SIMULATING WORKFLOW MESSAGES")
print("-" * 40)
# Create mock messages to simulate the workflow
messages = []
# Message 1: SharePoint search result
message1 = ChatMessage(
id="msg_001",
workflowId=workflow.id,
role="assistant",
message=action1_result["result"],
status="step",
sequenceNr=1,
publishedAt=datetime.now(UTC).isoformat(),
actionId="action_001",
actionMethod="sharepoint",
actionName="search",
documentsLabel=action1_result["resultLabel"],
documents=[
ChatDocument(id="001", fileId="file_001", filename="sales_report_q1.xlsx", fileSize=1024, mimeType="application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"),
ChatDocument(id="002", fileId="file_002", filename="sales_report_q2.xlsx", fileSize=2048, mimeType="application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"),
ChatDocument(id="003", fileId="file_003", filename="sales_report_q3.xlsx", fileSize=1536, mimeType="application/vnd.openxmlformats-officedocument.spreadsheetml.sheet")
]
)
messages.append(message1)
# Message 2: Excel analysis result
message2 = ChatMessage(
id="msg_002",
workflowId=workflow.id,
role="assistant",
message=action2_result["result"],
status="step",
sequenceNr=2,
publishedAt=datetime.now(UTC).isoformat(),
actionId="action_002",
actionMethod="excel",
actionName="analyze",
documentsLabel=action2_result["resultLabel"],
documents=[
ChatDocument(id="004", fileId="file_004", filename="sales_analysis_summary.xlsx", fileSize=3072, mimeType="application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"),
ChatDocument(id="005", fileId="file_005", filename="sales_trends_chart.png", fileSize=512, mimeType="image/png")
]
)
messages.append(message2)
# Add messages to workflow
workflow.messages = messages
print(f"Created {len(messages)} workflow messages with documents")
# Test document reference retrieval
print("\n3. TESTING DOCUMENT REFERENCE RETRIEVAL")
print("-" * 40)
doc_refs = container.getDocumentReferenceList()
print("Available Documents:")
for i, doc in enumerate(doc_refs.get('chat', []), 1):
print(f"{i}. {doc['documentReference']}")
print(f" Source: {doc['actionMethod']}.{doc['actionName']}")
print(f" Documents: {doc['documentCount']}")
print(f" Time: {doc['datetime']}")
print()
# Test document retrieval by reference
print("4. TESTING DOCUMENT RETRIEVAL BY REFERENCE")
print("-" * 40)
test_refs = [
"documentList_abc123_sales_documents",
"documentList_def456_analysis_results"
]
for ref in test_refs:
documents = container.getChatDocumentsFromDocumentReference(ref)
print(f"Reference: {ref}")
print(f"Found {len(documents)} documents:")
for doc in documents:
print(f" - {doc.filename} (ID: {doc.id}, Size: {doc.fileSize})")
print()
print("=" * 80)
print("HANDOVER MECHANISM SUMMARY")
print("=" * 80)
print("✅ Documents are properly stored in workflow messages")
print("✅ Result labels are correctly formatted")
print("✅ Document references are retrievable")
print("✅ Subsequent actions can find previous results")
print("✅ Clear handover chain between actions")
if __name__ == "__main__":
test_document_handover()

96
test_method_signatures.py
View file

@ -1,96 +0,0 @@
#!/usr/bin/env python3
"""
Test script to demonstrate the improved method signature format.
This shows how the AI will receive clear parameter information without automatic result labels.
"""
import sys
import os
sys.path.append(os.path.dirname(os.path.abspath(__file__)))
from modules.workflow.serviceContainer import ServiceContainer
from modules.interfaces.interfaceAppModel import User
from modules.interfaces.interfaceChatModel import ChatWorkflow
from datetime import datetime, UTC
def test_method_signatures():
"""Test the improved method signature format"""
# Create test user and workflow
user = User(
id="test_user",
username="testuser",
language="en",
mandateId="test_mandate"
)
workflow = ChatWorkflow(
id="test_workflow",
mandateId="test_mandate",
status="active",
currentRound=1,
lastActivity=datetime.now(UTC).isoformat(),
startedAt=datetime.now(UTC).isoformat(),
messages=[]
)
# Create service container
container = ServiceContainer(user, workflow)
# Get and display method list
print("=" * 80)
print("IMPROVED METHOD SIGNATURES")
print("=" * 80)
print("The AI will now receive clear parameter information without automatic result labels.")
print("The AI must set resultLabel according to the format: documentList_uuid_descriptive_label")
print()
method_list = container.getMethodsList()
print("AVAILABLE METHODS:")
print("-" * 40)
for i, method in enumerate(method_list, 1):
print(f"{i:2d}. {method}")
print()
print("=" * 80)
print("EXAMPLE OF HOW AI SHOULD SET RESULT LABELS:")
print("=" * 80)
print("""
{
"status": "pending",
"feedback": "I will analyze the Excel file and create a summary report.",
"actions": [
{
"method": "excel",
"action": "read",
"parameters": {
"fileId": "document_123_sales_data.xlsx",
"connectionReference": "connection_456_msft_user@example.com",
"sheetName": "Sheet1"
},
"resultLabel": "documentList_abc123_excel_data"
},
{
"method": "document",
"action": "analyze",
"parameters": {
"fileId": "documentList_abc123_excel_data"
},
"resultLabel": "documentList_def456_analysis_results"
}
]
}
""")
print("=" * 80)
print("KEY IMPROVEMENTS:")
print("=" * 80)
print("1. Clear parameter types and descriptions")
print("2. No automatic result labels - AI must set them")
print("3. Consistent format: documentList_uuid_descriptive_label")
print("4. Better parameter validation through type information")
print("5. Clear handover between actions using result labels")
if __name__ == "__main__":
test_method_signatures()

27
test_param_extraction.py Normal file
View file

@ -0,0 +1,27 @@
#!/usr/bin/env python3
from modules.workflow.methodBase import MethodBase
class TestMethod(MethodBase):
pass
def test_parameter_extraction():
test = TestMethod(None)
test.name = 'test'
docstring = """Call AI service with document content
Parameters:
prompt (str): The prompt to send to the AI service
documents (List[Dict[str, Any]], optional): List of documents to include in context
Each document should have: documentReference (str), contentExtractionPrompt (str, optional)"""
print("Docstring:")
print(docstring)
print("\nExtracted descriptions:")
descriptions = test._extractParameterDescriptions(docstring)
for param, desc in descriptions.items():
print(f" {param}: {desc}")
if __name__ == "__main__":
test_parameter_extraction()

23
test_signature.py Normal file
View file

@ -0,0 +1,23 @@
#!/usr/bin/env python3
from modules.workflow.serviceContainer import ServiceContainer
from modules.interfaces.interfaceAppObjects import User
from modules.interfaces.interfaceChatModel import ChatWorkflow
def test_signatures():
user = User(id='test', mandateId='test', username='test', email='test@test.com',
fullName='Test User', enabled=True, language='en', privilege='user',
authenticationAuthority='local')
workflow = ChatWorkflow(id='test', mandateId='test', status='running', name='Test',
currentRound=1, lastActivity='2025-01-01T00:00:00Z',
startedAt='2025-01-01T00:00:00Z', logs=[], messages=[],
stats=None, tasks=[])
service = ServiceContainer(user, workflow)
print("Method signatures:")
methodList = service.getMethodsList()
for sig in methodList[:5]: # Show first 5
print(f" {sig}")
if __name__ == "__main__":
test_signatures()

210
test_workflow.py
View file

@ -1,6 +1,6 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
""" """
Test routine for WorkflowManager.workflowProcess() Test routine for WorkflowManager.workflowProcess() with new unified workflow architecture
""" """
import asyncio import asyncio
@ -15,20 +15,26 @@ print("Starting test_workflow.py...")
# Configure logging FIRST, before any other imports # Configure logging FIRST, before any other imports
import logging import logging
# Clear any existing handlers to avoid duplicate logs
for handler in logging.root.handlers[:]:
logging.root.removeHandler(handler)
logging.basicConfig( logging.basicConfig(
level=logging.DEBUG, level=logging.DEBUG,
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s', format='%(asctime)s - %(levelname)s - %(name)s - %(message)s',
handlers=[ handlers=[
logging.StreamHandler(sys.stdout), logging.StreamHandler(sys.stdout),
logging.FileHandler('test_workflow.log', encoding='utf-8') logging.FileHandler('test_workflow.log', mode='w', encoding='utf-8') # 'w' mode clears the file
], ],
force=True # Force reconfiguration even if already configured force=True # Force reconfiguration even if already configured
) )
# logger = logging.getLogger(__name__) # Filter out httpcore messages
# print("Logger level:", logger.level) logging.getLogger('httpcore').setLevel(logging.WARNING)
# logger.info("Logger is working!") logging.getLogger('httpx').setLevel(logging.WARNING)
# print("Logger test done")
logger = logging.getLogger(__name__)
# Set up test configuration # Set up test configuration
os.environ['POWERON_CONFIG_FILE'] = 'test_config.ini' os.environ['POWERON_CONFIG_FILE'] = 'test_config.ini'
@ -47,6 +53,14 @@ except Exception as e:
traceback.print_exc() traceback.print_exc()
sys.exit(1) sys.exit(1)
def log_workflow_debug(message: str, data: dict = None):
"""Log workflow debug data with JSON dumps"""
timestamp = datetime.now(UTC).isoformat()
if data:
logger.debug(f"[{timestamp}] {message}\n{json.dumps(data, indent=2, ensure_ascii=False)}")
else:
logger.debug(f"[{timestamp}] {message}")
def create_test_user() -> User: def create_test_user() -> User:
"""Create a test user for the workflow""" """Create a test user for the workflow"""
return User( return User(
@ -276,9 +290,13 @@ EVALUATION WEIGHTS:
fileName=filename fileName=filename
) )
test_files.append(file_item.id) test_files.append(file_item.id)
# logger.info(f"Created test file: {filename} (ID: {file_item.id})") log_workflow_debug(f"Created test file: {filename}", {
"file_id": file_item.id,
"filename": filename,
"content_length": len(content)
})
except Exception as e: except Exception as e:
# logger.error(f"Error creating test file {filename}: {str(e)}") log_workflow_debug(f"Error creating test file {filename}: {str(e)}")
# Create a dummy file ID if creation fails # Create a dummy file ID if creation fails
test_files.append(f"file_{filename.replace('.', '_')}") test_files.append(f"file_{filename.replace('.', '_')}")
@ -286,15 +304,22 @@ EVALUATION WEIGHTS:
async def test_workflow_process(): async def test_workflow_process():
print("Inside test_workflow_process()") print("Inside test_workflow_process()")
"""Test the workflowProcess function""" """Test the workflowProcess function with new unified workflow architecture"""
try: try:
# logger.info("Starting workflow process test...") logger.info("=== STARTING UNIFIED WORKFLOW PROCESS TEST ===")
# Create test data # Create test data
test_user = create_test_user() test_user = create_test_user()
test_workflow = create_test_workflow() test_workflow = create_test_workflow()
test_user_input = create_test_user_input() test_user_input = create_test_user_input()
log_workflow_debug("Test data created", {
"user_id": test_user.id,
"workflow_id": test_workflow.id,
"user_input_prompt": test_user_input.prompt,
"file_ids": test_user_input.listFileId
})
# Create test user in database through AppObjects interface # Create test user in database through AppObjects interface
from modules.interfaces.interfaceAppObjects import getRootInterface from modules.interfaces.interfaceAppObjects import getRootInterface
from modules.interfaces.interfaceAppModel import AuthAuthority, ConnectionStatus, Token, UserPrivilege from modules.interfaces.interfaceAppModel import AuthAuthority, ConnectionStatus, Token, UserPrivilege
@ -310,7 +335,11 @@ async def test_workflow_process():
privilege=UserPrivilege.USER, privilege=UserPrivilege.USER,
authenticationAuthority=AuthAuthority.LOCAL authenticationAuthority=AuthAuthority.LOCAL
) )
# logger.info(f"Created test user in database: {created_user.id}") log_workflow_debug("Created test user in database", {
"user_id": created_user.id,
"username": created_user.username,
"email": created_user.email
})
# Create test connection through AppObjects interface # Create test connection through AppObjects interface
from modules.interfaces.interfaceAppObjects import getInterface as getAppObjects from modules.interfaces.interfaceAppObjects import getInterface as getAppObjects
@ -323,7 +352,11 @@ async def test_workflow_process():
externalEmail="testuser@example.com", externalEmail="testuser@example.com",
status=ConnectionStatus.ACTIVE status=ConnectionStatus.ACTIVE
) )
# logger.info(f"Created test connection: {test_connection.id}") log_workflow_debug("Created test connection", {
"connection_id": test_connection.id,
"authority": test_connection.authority,
"external_username": test_connection.externalUsername
})
# Create test token for the connection # Create test token for the connection
test_token = Token( test_token = Token(
@ -336,23 +369,11 @@ async def test_workflow_process():
createdAt=datetime.now(UTC) createdAt=datetime.now(UTC)
) )
app_interface.saveToken(test_token) app_interface.saveToken(test_token)
# logger.info(f"Created test token for connection: {test_token.id}") log_workflow_debug("Created test token", {
"token_id": test_token.id,
# logger.info(f"Test user: {created_user.username}") "authority": test_token.authority,
# logger.info(f"Test workflow: {test_workflow.id}") "expires_at": test_token.expiresAt
})
# Log the full prompt in JSON format
# logger.debug("=" * 60)
# logger.debug("USER INPUT PROMPT (JSON):")
# logger.debug("=" * 60)
prompt_data = {
"prompt": test_user_input.prompt,
"listFileId": test_user_input.listFileId,
"userLanguage": test_user_input.userLanguage
}
# logger.debug(json.dumps(prompt_data, indent=2, ensure_ascii=False))
# logger.debug("=" * 60)
# logger.debug(f"Test files: {test_user_input.listFileId}")
# Create test workflow in database through ChatObjects interface # Create test workflow in database through ChatObjects interface
from modules.interfaces.interfaceChatObjects import getInterface as getChatObjects from modules.interfaces.interfaceChatObjects import getInterface as getChatObjects
@ -367,85 +388,106 @@ async def test_workflow_process():
"lastActivity": test_workflow.lastActivity "lastActivity": test_workflow.lastActivity
} }
created_workflow = chat_interface.createWorkflow(workflow_data) created_workflow = chat_interface.createWorkflow(workflow_data)
# logger.info(f"Created test workflow: {created_workflow.id}") log_workflow_debug("Created test workflow in database", {
"workflow_id": created_workflow.id,
"name": created_workflow.name,
"status": created_workflow.status
})
# Update the test_workflow object with the created workflow's ID # Update the test_workflow object with the created workflow's ID
test_workflow.id = created_workflow.id test_workflow.id = created_workflow.id
# Create test files in database # Create test files in database
# logger.info("Creating test files for candidate evaluation...") logger.info("Creating test files for candidate evaluation...")
test_file_ids = create_test_files(chat_interface) test_file_ids = create_test_files(chat_interface)
# logger.info(f"Created {len(test_file_ids)} test files: {test_file_ids}") log_workflow_debug("Test files created", {
"file_count": len(test_file_ids),
"file_ids": test_file_ids
})
# Update user input with real file IDs # Update user input with real file IDs
test_user_input.listFileId = test_file_ids test_user_input.listFileId = test_file_ids
# logger.info(f"Updated user input with file IDs: {test_user_input.listFileId}") log_workflow_debug("Updated user input with file IDs", {
"file_ids": test_user_input.listFileId
})
# Initialize WorkflowManager # Initialize WorkflowManager
workflow_manager = WorkflowManager(chat_interface, created_user) workflow_manager = WorkflowManager(chat_interface, created_user)
# logger.info("WorkflowManager initialized") logger.info("WorkflowManager initialized")
# Test the workflowProcess function # Test the workflowProcess function
# logger.info("Calling workflowProcess...") logger.info("Calling workflowProcess with unified workflow architecture...")
task = await workflow_manager.workflowProcess(test_user_input, test_workflow)
# Log results try:
if task: # Execute the unified workflow process
# logger.debug("Task created successfully!") await workflow_manager.workflowProcess(test_user_input, test_workflow)
# logger.debug(f"Task ID: {task.id}")
# logger.debug(f"Task Status: {task.status}") # Log workflow results
# logger.debug(f"Task Feedback: {task.feedback}") log_workflow_debug("Workflow process completed successfully", {
# logger.info(f"Number of actions: {len(task.actionList) if task.actionList else 0}") "workflow_id": test_workflow.id,
# logger.debug("=" * 60) "workflow_status": test_workflow.status,
# logger.debug("TASK OBJECT (JSON):") "message_count": len(test_workflow.messages),
# logger.debug("=" * 60) "final_messages": [
task_data = {
"id": task.id,
"status": task.status,
"feedback": task.feedback,
"actionList": [
{ {
"execMethod": action.execMethod, "role": msg.role,
"execAction": action.execAction, "message": msg.message[:200] + "..." if len(msg.message) > 200 else msg.message,
"execParameters": action.execParameters, "status": msg.status,
"execResultLabel": action.execResultLabel "sequence_nr": msg.sequenceNr
} for action in (task.actionList or []) } for msg in test_workflow.messages[-3:] # Last 3 messages
] if task.actionList else [] ]
} })
# logger.debug(json.dumps(task_data, indent=2, ensure_ascii=False))
# logger.debug("=" * 60) # Log detailed workflow messages
if task.actionList: for i, message in enumerate(test_workflow.messages):
for i, action in enumerate(task.actionList): log_workflow_debug(f"WORKFLOW MESSAGE {i+1}:", {
# logger.info(f"Action {i+1}: {action.execMethod}.{action.execAction}") "role": message.role,
# logger.info(f" Parameters: {action.execParameters}") "message": message.message,
pass "status": message.status,
else: "sequence_nr": message.sequenceNr,
# logger.warning("No task was created") "published_at": message.publishedAt,
pass "document_count": len(message.documents) if hasattr(message, 'documents') else 0
# logger.info("Test completed successfully!") })
return task
return test_workflow
except Exception as e: except Exception as e:
# logger.error(f"❌ Test failed with error: {str(e)}") import traceback
# logger.exception("Full traceback:") error_details = {
"error_type": type(e).__name__,
"error_message": str(e),
"error_args": e.args if hasattr(e, 'args') else None,
"traceback": traceback.format_exc()
}
log_workflow_debug("WORKFLOW PROCESS EXCEPTION:", error_details)
raise
logger.info("=== UNIFIED WORKFLOW PROCESS TEST COMPLETED ===")
return test_workflow
except Exception as e:
logger.error(f"❌ Test failed with error: {str(e)}")
log_workflow_debug("Full error details", {
"error_type": type(e).__name__,
"error_message": str(e)
})
raise raise
async def main(): async def main():
print("Inside main()") print("Inside main()")
# logger.info("=" * 50) logger.info("=" * 50)
# logger.info("CANDIDATE EVALUATION WORKFLOW TEST") logger.info("CANDIDATE EVALUATION UNIFIED WORKFLOW TEST")
# logger.info("=" * 50) logger.info("=" * 50)
try: try:
task = await test_workflow_process() workflow = await test_workflow_process()
# logger.info("=" * 50) logger.info("=" * 50)
# logger.info("TEST COMPLETED SUCCESSFULLY") logger.info("TEST COMPLETED SUCCESSFULLY")
# logger.info("=" * 50) logger.info("=" * 50)
return task return workflow
except Exception as e: except Exception as e:
# logger.error("=" * 50) logger.error("=" * 50)
# logger.error("TEST FAILED") logger.error("TEST FAILED")
# logger.error("=" * 50) logger.error("=" * 50)
raise raise
if __name__ == "__main__": if __name__ == "__main__":

926
tool_test01ui.py
View file

@ -1,926 +0,0 @@
"""
UI Test Procedure for PowerOn Frontend
Tests CRUD operations, user registration, authentication, and access control
"""
import os
import sys
import json
import logging
from datetime import datetime
from typing import Dict, List, Any
import requests
from dataclasses import dataclass
from enum import Enum
import time
# Configure logging
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(levelname)s - %(message)s',
handlers=[
logging.FileHandler('ui_test_report.log'),
logging.StreamHandler(sys.stdout)
]
)
logger = logging.getLogger(__name__)
# Test configuration
BASE_URL = "http://localhost:8080" # Adjust based on your frontend URL
API_URL = "http://localhost:8000" # Adjust based on your backend URL
class UserRole(Enum):
SYSADMIN = "sysadmin"
ADMIN = "admin"
USER = "user"
@dataclass
class TestResult:
test_name: str
success: bool
message: str
details: Dict[str, Any] = None
@dataclass
class TestReport:
timestamp: str
total_tests: int
passed_tests: int
failed_tests: int
results: List[TestResult]
bugs_found: List[Dict[str, str]]
required_adaptations: List[Dict[str, str]]
class UITestSuite:
def __init__(self):
self.session = requests.Session()
self.test_results = []
self.bugs_found = []
self.required_adaptations = []
self.current_user = None
self.current_role = None
def run_all_tests(self):
"""Run all test categories"""
logger.info("Starting UI Test Suite")
# Test user registration and authentication
self.test_user_registration()
self.test_user_authentication()
# Test CRUD operations for each module
self.test_files_module()
self.test_mandates_module()
self.test_prompts_module()
self.test_users_module()
# Generate and save report
self.generate_report()
def test_user_registration(self):
"""Test user registration for different roles"""
logger.info("Testing User Registration")
test_users = [
{"username": "test_sysadmin", "password": "Test123!", "role": UserRole.SYSADMIN},
{"username": "test_admin", "password": "Test123!", "role": UserRole.ADMIN},
{"username": "test_user", "password": "Test123!", "role": UserRole.USER}
]
for user in test_users:
try:
# Create userData object matching User model
user_data = {
"username": user["username"],
"email": f"{user['username']}@test.com",
"fullName": f"Test {user['role'].value}",
"language": "en",
"enabled": True,
"privilege": user["role"].value,
"authenticationAuthority": "local",
"mandateId": None, # Will be set by the backend
"connections": []
}
response = self.session.post(
f"{API_URL}/api/local/register",
json={
"userData": user_data,
"password": user["password"]
},
headers={
"X-CSRF-Token": "test-csrf-token",
"Content-Type": "application/json"
}
)
if response.status_code == 200:
logger.info(f"Successfully registered {user['role'].value} user")
self.test_results.append(TestResult(
f"Register {user['role'].value}",
True,
f"Successfully registered {user['role'].value} user"
))
else:
error_msg = f"Failed to register {user['role'].value} user: {response.status_code}"
if response.text:
error_msg += f" - {response.text}"
logger.error(error_msg)
self.test_results.append(TestResult(
f"Register {user['role'].value}",
False,
error_msg,
{"status_code": response.status_code, "response": response.text}
))
except Exception as e:
error_msg = f"Exception during registration: {str(e)}"
logger.error(error_msg)
self.test_results.append(TestResult(
f"Register {user['role'].value}",
False,
error_msg
))
def test_user_authentication(self):
"""Test login and logout for different roles"""
logger.info("Testing User Authentication")
test_users = [
{"username": "test_sysadmin", "password": "Test123!", "role": UserRole.SYSADMIN},
{"username": "test_admin", "password": "Test123!", "role": UserRole.ADMIN},
{"username": "test_user", "password": "Test123!", "role": UserRole.USER}
]
for user in test_users:
# Test login
try:
response = self.session.post(
f"{API_URL}/api/local/login",
data={
"username": user["username"],
"password": user["password"]
},
headers={
"X-CSRF-Token": "test-csrf-token" # Add CSRF token
}
)
if response.status_code == 200:
self.test_results.append(TestResult(
f"Login {user['role'].value}",
True,
f"Successfully logged in as {user['role'].value}"
))
# Test logout
logout_response = self.session.post(f"{API_URL}/api/security/local/logout")
if logout_response.status_code == 200:
self.test_results.append(TestResult(
f"Logout {user['role'].value}",
True,
f"Successfully logged out as {user['role'].value}"
))
else:
self.test_results.append(TestResult(
f"Logout {user['role'].value}",
False,
f"Failed to logout as {user['role'].value}"
))
else:
self.test_results.append(TestResult(
f"Login {user['role'].value}",
False,
f"Failed to login as {user['role'].value}"
))
except Exception as e:
self.test_results.append(TestResult(
f"Login {user['role'].value}",
False,
f"Exception during login: {str(e)}"
))
def test_files_module(self):
"""Test CRUD operations for files module"""
logger.info("Testing Files Module")
# Test for each role
for role in UserRole:
self.current_role = role
self._login_as_role(role)
# Test view files
self._test_view_files()
# Test create file
self._test_create_file()
# Test modify file
self._test_modify_file()
# Test delete file
self._test_delete_file()
self._logout()
def test_mandates_module(self):
"""Test CRUD operations for mandates module"""
logger.info("Testing Mandates Module")
for role in UserRole:
self.current_role = role
self._login_as_role(role)
# Test view mandates
self._test_view_mandates()
# Test create mandate
self._test_create_mandate()
# Test modify mandate
self._test_modify_mandate()
# Test delete mandate
self._test_delete_mandate()
self._logout()
def test_prompts_module(self):
"""Test CRUD operations for prompts module"""
logger.info("Testing Prompts Module")
for role in UserRole:
self.current_role = role
self._login_as_role(role)
# Test view prompts
self._test_view_prompts()
# Test create prompt
self._test_create_prompt()
# Test modify prompt
self._test_modify_prompt()
# Test delete prompt
self._test_delete_prompt()
self._logout()
def test_users_module(self):
"""Test CRUD operations for users module"""
logger.info("Testing Users Module")
for role in UserRole:
self.current_role = role
self._login_as_role(role)
# Test view users
self._test_view_users()
# Test create user
self._test_create_user()
# Test modify user
self._test_modify_user()
# Test delete user
self._test_delete_user()
self._logout()
def _login_as_role(self, role: UserRole):
"""Helper method to login as a specific role"""
username = f"test_{role.value}"
max_retries = 3
retry_delay = 12 # 12 seconds delay between retries (to stay under 5 per minute)
for attempt in range(max_retries):
try:
# Always wait before attempting login
if attempt == 0:
logger.info(f"Waiting {retry_delay}s before first login attempt for {role.value}")
else:
logger.info(f"Rate limit reached, waiting {retry_delay}s before retry {attempt + 1} for {role.value} login")
time.sleep(retry_delay)
response = self.session.post(
f"{API_URL}/api/local/login",
data={
"username": username,
"password": "Test123!"
},
headers={
"X-CSRF-Token": "test-csrf-token"
}
)
if response.status_code == 200:
self.current_user = response.json()
logger.info(f"Successfully logged in as {role.value}")
return
elif response.status_code == 429:
logger.info(f"Rate limit reached for {role.value} login: {response.text}")
if attempt < max_retries - 1:
continue
else:
logger.error(f"Max retries reached for {role.value} login after rate limits")
raise Exception(f"Max retries reached for {role.value} login after rate limits")
else:
error_msg = f"Failed to login as {role.value}: {response.status_code}"
if response.text:
error_msg += f" - {response.text}"
logger.error(error_msg)
raise Exception(error_msg)
except Exception as e:
if attempt == max_retries - 1:
logger.error(f"Login error for {role.value} after {max_retries} attempts: {str(e)}")
raise
continue
def _logout(self):
"""Helper method to logout"""
self.session.post(f"{API_URL}/api/security/local/logout")
self.current_user = None
def _test_view_files(self):
"""Test viewing files"""
try:
response = self.session.get(f"{API_URL}/api/files")
if response.status_code == 200:
self.test_results.append(TestResult(
f"View Files as {self.current_role.value}",
True,
"Successfully viewed files"
))
else:
self.test_results.append(TestResult(
f"View Files as {self.current_role.value}",
False,
f"Failed to view files: {response.status_code}"
))
except Exception as e:
self.test_results.append(TestResult(
f"View Files as {self.current_role.value}",
False,
f"Exception viewing files: {str(e)}"
))
def _test_create_file(self):
"""Test creating a file"""
try:
# Create a test file
test_file = {
"name": f"test_file_{self.current_role.value}",
"content": "Test content",
"type": "text/plain"
}
response = self.session.post(
f"{API_URL}/api/files",
json=test_file
)
if response.status_code == 200:
self.test_results.append(TestResult(
f"Create File as {self.current_role.value}",
True,
"Successfully created file"
))
else:
self.test_results.append(TestResult(
f"Create File as {self.current_role.value}",
False,
f"Failed to create file: {response.status_code}"
))
except Exception as e:
self.test_results.append(TestResult(
f"Create File as {self.current_role.value}",
False,
f"Exception creating file: {str(e)}"
))
def _test_modify_file(self):
"""Test modifying a file"""
try:
# First get a file to modify
files_response = self.session.get(f"{API_URL}/api/files")
if files_response.status_code == 200 and files_response.json():
file_id = files_response.json()[0]["id"]
# Modify the file
update_data = {
"name": f"modified_file_{self.current_role.value}",
"content": "Modified content"
}
response = self.session.put(
f"{API_URL}/api/files/{file_id}",
json=update_data
)
if response.status_code == 200:
self.test_results.append(TestResult(
f"Modify File as {self.current_role.value}",
True,
"Successfully modified file"
))
else:
self.test_results.append(TestResult(
f"Modify File as {self.current_role.value}",
False,
f"Failed to modify file: {response.status_code}"
))
else:
self.test_results.append(TestResult(
f"Modify File as {self.current_role.value}",
False,
"No files available to modify"
))
except Exception as e:
self.test_results.append(TestResult(
f"Modify File as {self.current_role.value}",
False,
f"Exception modifying file: {str(e)}"
))
def _test_delete_file(self):
"""Test deleting a file"""
try:
# First get a file to delete
files_response = self.session.get(f"{API_URL}/api/files")
if files_response.status_code == 200 and files_response.json():
file_id = files_response.json()[0]["id"]
response = self.session.delete(f"{API_URL}/api/files/{file_id}")
if response.status_code == 200:
self.test_results.append(TestResult(
f"Delete File as {self.current_role.value}",
True,
"Successfully deleted file"
))
else:
self.test_results.append(TestResult(
f"Delete File as {self.current_role.value}",
False,
f"Failed to delete file: {response.status_code}"
))
else:
self.test_results.append(TestResult(
f"Delete File as {self.current_role.value}",
False,
"No files available to delete"
))
except Exception as e:
self.test_results.append(TestResult(
f"Delete File as {self.current_role.value}",
False,
f"Exception deleting file: {str(e)}"
))
def _test_view_mandates(self):
"""Test viewing mandates"""
try:
response = self.session.get(f"{API_URL}/api/mandates")
if response.status_code == 200:
self.test_results.append(TestResult(
f"View Mandates as {self.current_role.value}",
True,
"Successfully viewed mandates"
))
else:
self.test_results.append(TestResult(
f"View Mandates as {self.current_role.value}",
False,
f"Failed to view mandates: {response.status_code}"
))
except Exception as e:
self.test_results.append(TestResult(
f"View Mandates as {self.current_role.value}",
False,
f"Exception viewing mandates: {str(e)}"
))
def _test_create_mandate(self):
"""Test creating a mandate"""
try:
test_mandate = {
"name": f"test_mandate_{self.current_role.value}",
"description": "Test mandate",
"enabled": True
}
response = self.session.post(
f"{API_URL}/api/mandates",
json=test_mandate
)
if response.status_code == 200:
self.test_results.append(TestResult(
f"Create Mandate as {self.current_role.value}",
True,
"Successfully created mandate"
))
else:
self.test_results.append(TestResult(
f"Create Mandate as {self.current_role.value}",
False,
f"Failed to create mandate: {response.status_code}"
))
except Exception as e:
self.test_results.append(TestResult(
f"Create Mandate as {self.current_role.value}",
False,
f"Exception creating mandate: {str(e)}"
))
def _test_modify_mandate(self):
"""Test modifying a mandate"""
try:
# First get a mandate to modify
mandates_response = self.session.get(f"{API_URL}/api/mandates")
if mandates_response.status_code == 200 and mandates_response.json():
mandate_id = mandates_response.json()[0]["id"]
update_data = {
"name": f"modified_mandate_{self.current_role.value}",
"description": "Modified mandate"
}
response = self.session.put(
f"{API_URL}/api/mandates/{mandate_id}",
json=update_data
)
if response.status_code == 200:
self.test_results.append(TestResult(
f"Modify Mandate as {self.current_role.value}",
True,
"Successfully modified mandate"
))
else:
self.test_results.append(TestResult(
f"Modify Mandate as {self.current_role.value}",
False,
f"Failed to modify mandate: {response.status_code}"
))
else:
self.test_results.append(TestResult(
f"Modify Mandate as {self.current_role.value}",
False,
"No mandates available to modify"
))
except Exception as e:
self.test_results.append(TestResult(
f"Modify Mandate as {self.current_role.value}",
False,
f"Exception modifying mandate: {str(e)}"
))
def _test_delete_mandate(self):
"""Test deleting a mandate"""
try:
# First get a mandate to delete
mandates_response = self.session.get(f"{API_URL}/api/mandates")
if mandates_response.status_code == 200 and mandates_response.json():
mandate_id = mandates_response.json()[0]["id"]
response = self.session.delete(f"{API_URL}/api/mandates/{mandate_id}")
if response.status_code == 200:
self.test_results.append(TestResult(
f"Delete Mandate as {self.current_role.value}",
True,
"Successfully deleted mandate"
))
else:
self.test_results.append(TestResult(
f"Delete Mandate as {self.current_role.value}",
False,
f"Failed to delete mandate: {response.status_code}"
))
else:
self.test_results.append(TestResult(
f"Delete Mandate as {self.current_role.value}",
False,
"No mandates available to delete"
))
except Exception as e:
self.test_results.append(TestResult(
f"Delete Mandate as {self.current_role.value}",
False,
f"Exception deleting mandate: {str(e)}"
))
def _test_view_prompts(self):
"""Test viewing prompts"""
try:
response = self.session.get(f"{API_URL}/api/prompts")
if response.status_code == 200:
self.test_results.append(TestResult(
f"View Prompts as {self.current_role.value}",
True,
"Successfully viewed prompts"
))
else:
self.test_results.append(TestResult(
f"View Prompts as {self.current_role.value}",
False,
f"Failed to view prompts: {response.status_code}"
))
except Exception as e:
self.test_results.append(TestResult(
f"View Prompts as {self.current_role.value}",
False,
f"Exception viewing prompts: {str(e)}"
))
def _test_create_prompt(self):
"""Test creating a prompt"""
try:
test_prompt = {
"name": f"test_prompt_{self.current_role.value}",
"content": "Test prompt content",
"category": "test"
}
response = self.session.post(
f"{API_URL}/api/prompts",
json=test_prompt
)
if response.status_code == 200:
self.test_results.append(TestResult(
f"Create Prompt as {self.current_role.value}",
True,
"Successfully created prompt"
))
else:
self.test_results.append(TestResult(
f"Create Prompt as {self.current_role.value}",
False,
f"Failed to create prompt: {response.status_code}"
))
except Exception as e:
self.test_results.append(TestResult(
f"Create Prompt as {self.current_role.value}",
False,
f"Exception creating prompt: {str(e)}"
))
def _test_modify_prompt(self):
"""Test modifying a prompt"""
try:
# First get a prompt to modify
prompts_response = self.session.get(f"{API_URL}/api/prompts")
if prompts_response.status_code == 200 and prompts_response.json():
prompt_id = prompts_response.json()[0]["id"]
update_data = {
"name": f"modified_prompt_{self.current_role.value}",
"content": "Modified prompt content"
}
response = self.session.put(
f"{API_URL}/api/prompts/{prompt_id}",
json=update_data
)
if response.status_code == 200:
self.test_results.append(TestResult(
f"Modify Prompt as {self.current_role.value}",
True,
"Successfully modified prompt"
))
else:
self.test_results.append(TestResult(
f"Modify Prompt as {self.current_role.value}",
False,
f"Failed to modify prompt: {response.status_code}"
))
else:
self.test_results.append(TestResult(
f"Modify Prompt as {self.current_role.value}",
False,
"No prompts available to modify"
))
except Exception as e:
self.test_results.append(TestResult(
f"Modify Prompt as {self.current_role.value}",
False,
f"Exception modifying prompt: {str(e)}"
))
def _test_delete_prompt(self):
"""Test deleting a prompt"""
try:
# First get a prompt to delete
prompts_response = self.session.get(f"{API_URL}/api/prompts")
if prompts_response.status_code == 200 and prompts_response.json():
prompt_id = prompts_response.json()[0]["id"]
response = self.session.delete(f"{API_URL}/api/prompts/{prompt_id}")
if response.status_code == 200:
self.test_results.append(TestResult(
f"Delete Prompt as {self.current_role.value}",
True,
"Successfully deleted prompt"
))
else:
self.test_results.append(TestResult(
f"Delete Prompt as {self.current_role.value}",
False,
f"Failed to delete prompt: {response.status_code}"
))
else:
self.test_results.append(TestResult(
f"Delete Prompt as {self.current_role.value}",
False,
"No prompts available to delete"
))
except Exception as e:
self.test_results.append(TestResult(
f"Delete Prompt as {self.current_role.value}",
False,
f"Exception deleting prompt: {str(e)}"
))
def _test_view_users(self):
"""Test viewing users"""
try:
response = self.session.get(f"{API_URL}/api/users")
if response.status_code == 200:
self.test_results.append(TestResult(
f"View Users as {self.current_role.value}",
True,
"Successfully viewed users"
))
else:
self.test_results.append(TestResult(
f"View Users as {self.current_role.value}",
False,
f"Failed to view users: {response.status_code}"
))
except Exception as e:
self.test_results.append(TestResult(
f"View Users as {self.current_role.value}",
False,
f"Exception viewing users: {str(e)}"
))
def _test_create_user(self):
"""Test creating a user"""
try:
test_user = {
"username": f"new_user_{self.current_role.value}",
"password": "Test123!",
"email": f"new_user_{self.current_role.value}@test.com",
"privilege": "user"
}
response = self.session.post(
f"{API_URL}/api/users",
json=test_user
)
if response.status_code == 200:
self.test_results.append(TestResult(
f"Create User as {self.current_role.value}",
True,
"Successfully created user"
))
else:
self.test_results.append(TestResult(
f"Create User as {self.current_role.value}",
False,
f"Failed to create user: {response.status_code}"
))
except Exception as e:
self.test_results.append(TestResult(
f"Create User as {self.current_role.value}",
False,
f"Exception creating user: {str(e)}"
))
def _test_modify_user(self):
"""Test modifying a user"""
try:
# First get a user to modify
users_response = self.session.get(f"{API_URL}/api/users")
if users_response.status_code == 200 and users_response.json():
user_id = users_response.json()[0]["id"]
update_data = {
"username": f"modified_user_{self.current_role.value}",
"email": f"modified_user_{self.current_role.value}@test.com"
}
response = self.session.put(
f"{API_URL}/api/users/{user_id}",
json=update_data
)
if response.status_code == 200:
self.test_results.append(TestResult(
f"Modify User as {self.current_role.value}",
True,
"Successfully modified user"
))
else:
self.test_results.append(TestResult(
f"Modify User as {self.current_role.value}",
False,
f"Failed to modify user: {response.status_code}"
))
else:
self.test_results.append(TestResult(
f"Modify User as {self.current_role.value}",
False,
"No users available to modify"
))
except Exception as e:
self.test_results.append(TestResult(
f"Modify User as {self.current_role.value}",
False,
f"Exception modifying user: {str(e)}"
))
def _test_delete_user(self):
"""Test deleting a user"""
try:
# First get a user to delete
users_response = self.session.get(f"{API_URL}/api/users")
if users_response.status_code == 200 and users_response.json():
user_id = users_response.json()[0]["id"]
response = self.session.delete(f"{API_URL}/api/users/{user_id}")
if response.status_code == 200:
self.test_results.append(TestResult(
f"Delete User as {self.current_role.value}",
True,
"Successfully deleted user"
))
else:
self.test_results.append(TestResult(
f"Delete User as {self.current_role.value}",
False,
f"Failed to delete user: {response.status_code}"
))
else:
self.test_results.append(TestResult(
f"Delete User as {self.current_role.value}",
False,
"No users available to delete"
))
except Exception as e:
self.test_results.append(TestResult(
f"Delete User as {self.current_role.value}",
False,
f"Exception deleting user: {str(e)}"
))
def generate_report(self):
"""Generate test report"""
# Convert TestResult objects to dictionaries
serialized_results = [
{
"test_name": r.test_name,
"success": r.success,
"message": r.message,
"details": r.details
}
for r in self.test_results
]
report = {
"timestamp": datetime.now().isoformat(),
"total_tests": len(self.test_results),
"passed_tests": sum(1 for r in self.test_results if r.success),
"failed_tests": sum(1 for r in self.test_results if not r.success),
"results": serialized_results,
"bugs_found": self.bugs_found,
"required_adaptations": self.required_adaptations
}
# Save report to file
with open('ui_test_report.json', 'w') as f:
json.dump(report, f, indent=2)
# Print summary
logger.info(f"""
Test Report Summary:
-------------------
Total Tests: {report['total_tests']}
Passed: {report['passed_tests']}
Failed: {report['failed_tests']}
Bugs Found: {len(report['bugs_found'])}
Required Adaptations: {len(report['required_adaptations'])}
""")
if __name__ == "__main__":
test_suite = UITestSuite()
test_suite.run_all_tests()

352
ui_test_report.json
View file

@ -1,352 +0,0 @@
{
"timestamp": "2025-06-02T20:18:49.451773",
"total_tests": 57,
"passed_tests": 6,
"failed_tests": 51,
"results": [
{
"test_name": "Register sysadmin",
"success": true,
"message": "Successfully registered sysadmin user",
"details": null
},
{
"test_name": "Register admin",
"success": true,
"message": "Successfully registered admin user",
"details": null
},
{
"test_name": "Register user",
"success": true,
"message": "Successfully registered user user",
"details": null
},
{
"test_name": "Login sysadmin",
"success": true,
"message": "Successfully logged in as sysadmin",
"details": null
},
{
"test_name": "Logout sysadmin",
"success": false,
"message": "Failed to logout as sysadmin",
"details": null
},
{
"test_name": "Login admin",
"success": true,
"message": "Successfully logged in as admin",
"details": null
},
{
"test_name": "Logout admin",
"success": false,
"message": "Failed to logout as admin",
"details": null
},
{
"test_name": "Login user",
"success": true,
"message": "Successfully logged in as user",
"details": null
},
{
"test_name": "Logout user",
"success": false,
"message": "Failed to logout as user",
"details": null
},
{
"test_name": "View Files as sysadmin",
"success": false,
"message": "Failed to view files: 405",
"details": null
},
{
"test_name": "Create File as sysadmin",
"success": false,
"message": "Failed to create file: 405",
"details": null
},
{
"test_name": "Modify File as sysadmin",
"success": false,
"message": "No files available to modify",
"details": null
},
{
"test_name": "Delete File as sysadmin",
"success": false,
"message": "No files available to delete",
"details": null
},
{
"test_name": "View Files as admin",
"success": false,
"message": "Failed to view files: 405",
"details": null
},
{
"test_name": "Create File as admin",
"success": false,
"message": "Failed to create file: 405",
"details": null
},
{
"test_name": "Modify File as admin",
"success": false,
"message": "No files available to modify",
"details": null
},
{
"test_name": "Delete File as admin",
"success": false,
"message": "No files available to delete",
"details": null
},
{
"test_name": "View Files as user",
"success": false,
"message": "Failed to view files: 405",
"details": null
},
{
"test_name": "Create File as user",
"success": false,
"message": "Failed to create file: 405",
"details": null
},
{
"test_name": "Modify File as user",
"success": false,
"message": "No files available to modify",
"details": null
},
{
"test_name": "Delete File as user",
"success": false,
"message": "No files available to delete",
"details": null
},
{
"test_name": "View Mandates as sysadmin",
"success": false,
"message": "Failed to view mandates: 405",
"details": null
},
{
"test_name": "Create Mandate as sysadmin",
"success": false,
"message": "Failed to create mandate: 405",
"details": null
},
{
"test_name": "Modify Mandate as sysadmin",
"success": false,
"message": "No mandates available to modify",
"details": null
},
{
"test_name": "Delete Mandate as sysadmin",
"success": false,
"message": "No mandates available to delete",
"details": null
},
{
"test_name": "View Mandates as admin",
"success": false,
"message": "Failed to view mandates: 405",
"details": null
},
{
"test_name": "Create Mandate as admin",
"success": false,
"message": "Failed to create mandate: 405",
"details": null
},
{
"test_name": "Modify Mandate as admin",
"success": false,
"message": "No mandates available to modify",
"details": null
},
{
"test_name": "Delete Mandate as admin",
"success": false,
"message": "No mandates available to delete",
"details": null
},
{
"test_name": "View Mandates as user",
"success": false,
"message": "Failed to view mandates: 405",
"details": null
},
{
"test_name": "Create Mandate as user",
"success": false,
"message": "Failed to create mandate: 405",
"details": null
},
{
"test_name": "Modify Mandate as user",
"success": false,
"message": "No mandates available to modify",
"details": null
},
{
"test_name": "Delete Mandate as user",
"success": false,
"message": "No mandates available to delete",
"details": null
},
{
"test_name": "View Prompts as sysadmin",
"success": false,
"message": "Failed to view prompts: 401",
"details": null
},
{
"test_name": "Create Prompt as sysadmin",
"success": false,
"message": "Failed to create prompt: 401",
"details": null
},
{
"test_name": "Modify Prompt as sysadmin",
"success": false,
"message": "No prompts available to modify",
"details": null
},
{
"test_name": "Delete Prompt as sysadmin",
"success": false,
"message": "No prompts available to delete",
"details": null
},
{
"test_name": "View Prompts as admin",
"success": false,
"message": "Failed to view prompts: 401",
"details": null
},
{
"test_name": "Create Prompt as admin",
"success": false,
"message": "Failed to create prompt: 401",
"details": null
},
{
"test_name": "Modify Prompt as admin",
"success": false,
"message": "No prompts available to modify",
"details": null
},
{
"test_name": "Delete Prompt as admin",
"success": false,
"message": "No prompts available to delete",
"details": null
},
{
"test_name": "View Prompts as user",
"success": false,
"message": "Failed to view prompts: 401",
"details": null
},
{
"test_name": "Create Prompt as user",
"success": false,
"message": "Failed to create prompt: 401",
"details": null
},
{
"test_name": "Modify Prompt as user",
"success": false,
"message": "No prompts available to modify",
"details": null
},
{
"test_name": "Delete Prompt as user",
"success": false,
"message": "No prompts available to delete",
"details": null
},
{
"test_name": "View Users as sysadmin",
"success": false,
"message": "Failed to view users: 405",
"details": null
},
{
"test_name": "Create User as sysadmin",
"success": false,
"message": "Failed to create user: 401",
"details": null
},
{
"test_name": "Modify User as sysadmin",
"success": false,
"message": "No users available to modify",
"details": null
},
{
"test_name": "Delete User as sysadmin",
"success": false,
"message": "No users available to delete",
"details": null
},
{
"test_name": "View Users as admin",
"success": false,
"message": "Failed to view users: 405",
"details": null
},
{
"test_name": "Create User as admin",
"success": false,
"message": "Failed to create user: 401",
"details": null
},
{
"test_name": "Modify User as admin",
"success": false,
"message": "No users available to modify",
"details": null
},
{
"test_name": "Delete User as admin",
"success": false,
"message": "No users available to delete",
"details": null
},
{
"test_name": "View Users as user",
"success": false,
"message": "Failed to view users: 405",
"details": null
},
{
"test_name": "Create User as user",
"success": false,
"message": "Failed to create user: 401",
"details": null
},
{
"test_name": "Modify User as user",
"success": false,
"message": "No users available to modify",
"details": null
},
{
"test_name": "Delete User as user",
"success": false,
"message": "No users available to delete",
"details": null
}
],
"bugs_found": [],
"required_adaptations": []
}