PowerOn/gateway
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

google connect

Browse source
This commit is contained in:
ValueOn AG 2025-08-12 16:20:26 +02:00
parent 4f01a02b9f
commit a0219181e9
16 changed files with 1313 additions and 381 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

114
GOOGLE_OAUTH_SETUP.md Normal file
View file

@ -0,0 +1,114 @@
# Google OAuth 2.0 Setup Guide for PowerOn
## Overview
This guide explains how to set up Google OAuth 2.0 authentication for the PowerOn application.
## Prerequisites
- A Google account
- Access to Google Cloud Console (https://console.cloud.google.com/)
## Step 1: Create a Google Cloud Project
1. Go to [Google Cloud Console](https://console.cloud.google.com/)
2. Click on the project dropdown at the top of the page
3. Click "New Project"
4. Enter a project name (e.g., "PowerOn OAuth")
5. Click "Create"
## Step 2: Enable Google+ API
1. In your new project, go to "APIs & Services" > "Library"
2. Search for "Google+ API" or "Google Identity"
3. Click on "Google+ API" and click "Enable"
## Step 3: Create OAuth 2.0 Credentials
1. Go to "APIs & Services" > "Credentials"
2. Click "Create Credentials" > "OAuth client ID"
3. If prompted, configure the OAuth consent screen first:
- Choose "External" user type
- Fill in the required fields (App name, User support email, Developer contact information)
- Add scopes: `https://www.googleapis.com/auth/userinfo.profile`, `https://www.googleapis.com/auth/userinfo.email`
- Add test users if needed
- Click "Save and Continue" through all sections
4. Back to creating OAuth client ID:
- Application type: "Web application"
- Name: "PowerOn Web Client"
- Authorized redirect URIs: Add your redirect URI
- For development: `http://localhost:8000/api/google/auth/callback`
- For production: `https://yourdomain.com/api/google/auth/callback`
5. Click "Create"
6. **Important**: Copy the Client ID and Client Secret - you'll need these for the next step
## Step 4: Configure PowerOn Application
1. Open your environment file (`gateway/env_dev.env` for development)
2. Replace the placeholder values with your actual Google OAuth credentials:
```env
# Google OAuth Configuration
Service_GOOGLE_CLIENT_ID = your-actual-client-id-from-google-console
Service_GOOGLE_CLIENT_SECRET = your-actual-client-secret-from-google-console
Service_GOOGLE_REDIRECT_URI = http://localhost:8000/api/google/auth/callback
```
3. Save the file
4. Restart your PowerOn gateway server
## Step 5: Test the Configuration
1. Start your PowerOn application
2. Go to the Connections module
3. Click "Connect Google"
4. You should be redirected to Google's OAuth consent screen
5. After authorization, you should be redirected back to PowerOn
## Troubleshooting
### Common Issues
#### 1. "Missing required parameter: redirect_uri"
- **Cause**: Google OAuth client is not properly configured with the redirect URI
- **Solution**: Ensure the redirect URI in Google Cloud Console exactly matches your application's callback URL
#### 2. "Invalid client" error
- **Cause**: Client ID or Client Secret is incorrect
- **Solution**: Double-check the credentials in your environment file
#### 3. "Redirect URI mismatch" error
- **Cause**: The redirect URI in your OAuth request doesn't match what's configured in Google Cloud Console
- **Solution**: Ensure both URIs are identical (including protocol, domain, port, and path)
### Debug Steps
1. Check the PowerOn gateway logs for OAuth configuration details
2. Verify environment variables are loaded correctly
3. Ensure the Google OAuth client is configured for "Web application" type
4. Check that the redirect URI includes the full path: `/api/google/auth/callback`
## Security Notes
- **Never commit** your Google OAuth credentials to version control
- Use environment variables or secure configuration management
- Regularly rotate your client secrets
- Monitor OAuth usage in Google Cloud Console
## Production Considerations
For production deployment:
1. Use HTTPS for all OAuth redirects
2. Configure proper domain verification in Google Cloud Console
3. Set up monitoring and alerting for OAuth usage
4. Consider implementing additional security measures like PKCE (Proof Key for Code Exchange)
## Support
If you continue to experience issues:
1. Check the PowerOn gateway logs for detailed error messages
2. Verify your Google OAuth configuration in Google Cloud Console
3. Test with a simple OAuth flow to isolate the issue
4. Ensure your Google Cloud project has billing enabled (required for some APIs)

5
config.ini
View file

@ -55,6 +55,5 @@ Service_MSFT_CLIENT_SECRET = Kxf8Q~2lJIteZ~JaI32kMf1lfaWKATqxXiNiFbzV
Service_MSFT_TENANT_ID = common
# Google Service configuration
Service_GOOGLE_CLIENT_ID = your-google-client-id
Service_GOOGLE_CLIENT_SECRET = your-google-client-secret
Service_GOOGLE_REDIRECT_URI = http://localhost:8000/api/google/auth/callback
Service_GOOGLE_CLIENT_ID = 354925410565-aqs2b2qaiqmm73qpjnel6al8eid78uvg.apps.googleusercontent.com
Service_GOOGLE_CLIENT_SECRET = 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

45
env_dev.env
View file

@ -1,45 +0,0 @@
# Development Environment Configuration
# System Configuration
APP_ENV_TYPE = dev
APP_ENV_LABEL = Development Instance Patrick
APP_API_URL = http://localhost:8000
# Database Configuration for Application
DB_APP_HOST=D:/Temp/_powerondb
DB_APP_DATABASE=app
DB_APP_USER=dev_user
DB_APP_PASSWORD_SECRET=dev_password
# Database Configuration Chat
DB_CHAT_HOST=D:/Temp/_powerondb
DB_CHAT_DATABASE=chat
DB_CHAT_USER=dev_user
DB_CHAT_PASSWORD_SECRET=dev_password
# Database Configuration Management
DB_MANAGEMENT_HOST=D:/Temp/_powerondb
DB_MANAGEMENT_DATABASE=management
DB_MANAGEMENT_USER=dev_user
DB_MANAGEMENT_PASSWORD_SECRET=dev_password
# Security Configuration
APP_JWT_SECRET_SECRET=dev_jwt_secret_token
APP_TOKEN_EXPIRY=300
# CORS Configuration
APP_ALLOWED_ORIGINS=http://localhost:8080,https://playground.poweron-center.net,http://localhost:5176,https://nyla.poweron-center.net
# Logging configuration
APP_LOGGING_LOG_LEVEL = DEBUG
APP_LOGGING_LOG_FILE = poweron.log
APP_LOGGING_FORMAT = %(asctime)s - %(levelname)s - %(name)s - %(message)s
APP_LOGGING_DATE_FORMAT = %Y-%m-%d %H:%M:%S
APP_LOGGING_CONSOLE_ENABLED = True
APP_LOGGING_FILE_ENABLED = True
APP_LOGGING_ROTATION_SIZE = 10485760
APP_LOGGING_BACKUP_COUNT = 5
# Service Redirects
Service_MSFT_REDIRECT_URI = http://localhost:8000/api/msft/auth/callback
Service_GOOGLE_REDIRECT_URI = http://localhost:8000/api/google/auth/callback

2
env_prod.env
View file

@ -42,4 +42,4 @@ APP_LOGGING_BACKUP_COUNT = 5
# Service Redirects
Service_MSFT_REDIRECT_URI = https://gateway.poweron-center.net/api/msft/auth/callback
Service_GOOGLE_REDIRECT_URI = http://gateway.poweron-center.net/api/google/auth/callback
Service_GOOGLE_REDIRECT_URI = https://gateway.poweron-center.net/api/google/auth/callback

12
modules/chat/documents/documentGeneration.py
View file

@ -51,7 +51,7 @@ class DocumentGenerator:
'document': doc
}
elif isinstance(doc, dict):
# Dictionary format document
# Dictionary format document - handle both 'documentName' and 'filename' keys
filename = doc.get('documentName', doc.get('filename', \
f"{action.execMethod}_{action.execAction}_{datetime.now(UTC).strftime('%Y%m%d_%H%M%S')}"))
fileSize = doc.get('fileSize', len(str(doc.get('documentData', ''))))
@ -59,11 +59,19 @@ class DocumentGenerator:
if mimeType == "application/octet-stream":
document_data = doc.get('documentData', '')
mimeType = detectMimeTypeFromContent(document_data, filename, self.service)
# Handle documentData structure - it might be a dict with 'content' key or direct content
document_data = doc.get('documentData', '')
if isinstance(document_data, dict) and 'content' in document_data:
content = document_data['content']
else:
content = document_data
return {
'filename': filename,
'fileSize': fileSize,
'mimeType': mimeType,
'content': doc.get('documentData', ''),
'content': content,
'document': doc
}
else:

26
modules/chat/handling/handlingTasks.py
View file

@ -343,26 +343,41 @@ class HandlingTasks:
)
result_label = action.execResultLabel
# Process documents from the action result
created_documents = []
if result.success:
created_documents = self.documentGenerator.createDocumentsFromActionResult(result, action, workflow)
action.setSuccess()
action.result = result.data.get("result", "")
action.execResultLabel = result_label
await self.createActionMessage(action, result, workflow, result_label)
await self.createActionMessage(action, result, workflow, result_label, created_documents)
logger.info(f"Action {action.execMethod}.{action.execAction} executed successfully")
else:
action.setError(result.error or "Action execution failed")
logger.error(f"Action {action.execMethod}.{action.execAction} failed: {result.error}")
# Extract document filenames for the ActionResult
document_filenames = []
for doc in created_documents:
if hasattr(doc, 'filename'):
document_filenames.append(doc.filename)
elif isinstance(doc, dict) and 'filename' in doc:
document_filenames.append(doc['filename'])
# Also include the original documents from the service result for validation
original_documents = result.data.get("documents", [])
return ActionResult(
success=result.success,
data={
"result": result.data.get("result", ""),
"documents": [], # Documents will be processed in createActionMessage
"documents": created_documents, # Include actual document objects in data
"actionId": action.id,
"actionMethod": action.execMethod,
"actionName": action.execAction,
"resultLabel": result_label
},
documents=document_filenames, # Keep as filenames for the documents field
metadata={
"actionId": action.id,
"actionMethod": action.execMethod,
@ -392,14 +407,15 @@ class HandlingTasks:
error=str(e)
)
async def createActionMessage(self, action, result, workflow, result_label=None):
async def createActionMessage(self, action, result, workflow, result_label=None, created_documents=None):
"""Create and store a message for the action result in the workflow with enhanced document processing"""
try:
if result_label is None:
result_label = action.execResultLabel
# Use the local createDocumentsFromActionResult method
created_documents = self.documentGenerator.createDocumentsFromActionResult(result, action, workflow)
# Use provided documents or process them if not provided
if created_documents is None:
created_documents = self.documentGenerator.createDocumentsFromActionResult(result, action, workflow)
# Log delivered documents with sizes
if created_documents:

25
modules/chat/handling/promptFactory.py
View file

@ -337,7 +337,12 @@ async def createResultReviewPrompt(self, review_context) -> str:
}
for action_result in (review_context.action_results or []):
documents_metadata = []
for doc in (action_result.documents or []):
# FIX: Look for documents in the correct place - action_result.data.documents contains actual document objects
# action_result.documents only contains document references (strings)
documents_to_check = action_result.data.get("documents", [])
for doc in documents_to_check:
if hasattr(doc, 'filename'):
documents_metadata.append({
'filename': doc.filename,
@ -350,6 +355,14 @@ async def createResultReviewPrompt(self, review_context) -> str:
'fileSize': doc.get('fileSize', 0),
'mimeType': doc.get('mimeType', 'unknown')
})
elif isinstance(doc, str):
# Handle case where documents are just filenames
documents_metadata.append({
'filename': doc,
'fileSize': 0,
'mimeType': 'unknown'
})
serializable_action_result = {
'status': 'completed' if action_result.success else 'failed',
'result_summary': action_result.data.get('result', '')[:200] + '...' if len(action_result.data.get('result', '')) > 200 else action_result.data.get('result', ''),
@ -389,11 +402,14 @@ VALIDATION PRINCIPLES:
- Text outputs are SECONDARY indicators
- Only retry for CLEAR technical issues, not minor imperfections
- Don't be picky about formatting or minor details
- Check if ANY documents were produced (documents_count > 0), not specific expected output names
- If documents were produced, consider it a SUCCESS regardless of expected output names
EXAMPLES OF SUCCESS:
- Document extraction produced a file (even if imperfect)
- Text analysis provided meaningful insights
- Data processing completed with results
- Any action that produced documents (documents_count > 0)
EXAMPLES OF RETRY:
- Technical errors (API failures, timeouts)
@ -404,6 +420,7 @@ EXAMPLES OF FAILED:
- Complete system failures
- No output whatsoever
- Unrecoverable errors
- Actions with documents_count = 0 AND no meaningful text output
REQUIRED JSON STRUCTURE:
{{
@ -416,4 +433,10 @@ REQUIRED JSON STRUCTURE:
"unmet_criteria": []
}}
VALIDATION LOGIC:
- If ANY action has documents_count > 0, mark as SUCCESS
- If ALL actions have documents_count = 0 AND no meaningful text output, mark as FAILED
- Only mark as RETRY for clear technical issues that can be fixed
- Do NOT fail based on expected output name mismatches - focus on actual document production
NOTE: Respond with ONLY the JSON object. Be GENEROUS with success ratings."""

14
modules/chat/managerChat.py
View file

@ -10,6 +10,14 @@ logger = logging.getLogger(__name__)
# ===== STATE MANAGEMENT AND VALIDATION CLASSES =====
class WorkflowStoppedException(Exception):
"""Exception raised when workflow is stopped by user"""
pass
logger = logging.getLogger(__name__)
# ===== STATE MANAGEMENT AND VALIDATION CLASSES =====
class ChatManager:
"""Chat manager with improved AI integration and method handling"""
@ -44,6 +52,12 @@ class ChatManager:
previous_results = []
for idx, task_step in enumerate(task_plan.tasks):
logger.info(f"Task {idx+1}/{len(task_plan.tasks)}: {task_step.description}")
# Check if workflow has been stopped before each task
if self.service.workflow.status == "stopped":
logger.info("Workflow stopped by user, aborting execution")
raise WorkflowStoppedException("Workflow was stopped by user")
# Create task context for this task
task_context = TaskContext(
task_step=task_step,

37
modules/interfaces/interfaceChatObjects.py
View file

@ -653,8 +653,8 @@ class ChatObjects:
# Create stats record in database
self.db.recordCreate("stats", stats_record)
logger.debug(f"Updated workflow {workflowId} stats: {currentStats}")
logger.debug(f"Logged stats record: {stats_record}")
# logger.debug(f"Updated workflow {workflowId} stats: {currentStats}")
# logger.debug(f"Logged stats record: {stats_record}")
return True
except Exception as e:
@ -826,29 +826,34 @@ class ChatObjects:
# Load messages
messages = self.getWorkflowMessages(workflowId)
# Sort by sequence number
messages.sort(key=lambda x: x.get("sequenceNo", 0))
# Messages are already sorted by publishedAt in getWorkflowMessages
messageCount = len(messages)
logger.debug(f"Loaded {messageCount} messages for workflow {workflowId}")
# Log document counts for each message
for msg in messages:
docCount = len(msg.get("documents", []))
docCount = len(msg.documents) if hasattr(msg, 'documents') else 0
if docCount > 0:
logger.debug(f"Message {msg.get('id')} has {docCount} documents loaded from database")
logger.debug(f"Message {msg.id} has {docCount} documents loaded from database")
# Load logs
logs = self.getWorkflowLogs(workflowId)
# Sort by timestamp (Unix timestamps)
logs.sort(key=lambda x: float(x.get("timestamp", 0)))
# Logs are already sorted by timestamp in getWorkflowLogs
# Assemble complete workflow object
completeWorkflow = workflow.copy()
completeWorkflow["messages"] = messages
completeWorkflow["logs"] = logs
return completeWorkflow
# Create a new ChatWorkflow object with loaded messages and logs
return ChatWorkflow(
id=workflow.id,
status=workflow.status,
name=workflow.name,
currentRound=workflow.currentRound,
lastActivity=workflow.lastActivity,
startedAt=workflow.startedAt,
logs=logs,
messages=messages,
stats=workflow.stats,
mandateId=workflow.mandateId
)
except Exception as e:
logger.error(f"Error loading workflow state: {str(e)}")
return None
@ -871,8 +876,8 @@ class ChatObjects:
currentTime = self._getCurrentTimestamp()
if workflowId:
# Continue existing workflow
workflow = self.getWorkflow(workflowId)
# Continue existing workflow - load complete state including messages
workflow = self.loadWorkflowState(workflowId)
if not workflow:
raise ValueError(f"Workflow {workflowId} not found")

164
modules/methods/methodAi.py Normal file
View file

@ -0,0 +1,164 @@
"""
AI processing method module.
Handles direct AI calls for any type of task.
"""
import logging
from typing import Dict, Any, List, Optional
import uuid
from datetime import datetime, UTC
from modules.chat.methodBase import MethodBase, ActionResult, action
logger = logging.getLogger(__name__)
class MethodAi(MethodBase):
"""AI method implementation for direct AI processing"""
def __init__(self, serviceCenter: Any):
"""Initialize the AI method"""
super().__init__(serviceCenter)
self.name = "ai"
self.description = "Handle direct AI processing for any type of task"
@action
async def process(self, parameters: Dict[str, Any]) -> ActionResult:
"""
Perform an AI call for any type of task with optional document references
Parameters:
aiPrompt (str): The AI prompt for processing
documentList (list, optional): List of document references to include in context
expectedDocumentFormats (list, optional): Expected output formats with extension, mimeType, description
processingMode (str, optional): Processing mode ('basic', 'advanced', 'detailed') - defaults to 'basic'
includeMetadata (bool, optional): Whether to include metadata (default: True)
customInstructions (str, optional): Additional custom instructions for the AI
"""
try:
aiPrompt = parameters.get("aiPrompt")
documentList = parameters.get("documentList", [])
expectedDocumentFormats = parameters.get("expectedDocumentFormats", [])
processingMode = parameters.get("processingMode", "basic")
includeMetadata = parameters.get("includeMetadata", True)
customInstructions = parameters.get("customInstructions", "")
if not aiPrompt:
return self._createResult(
success=False,
data={},
error="AI prompt is required"
)
# Build context from documents if provided
context = ""
if documentList:
chatDocuments = self.service.getChatDocumentsFromDocumentList(documentList)
if chatDocuments:
context_parts = []
for doc in chatDocuments:
fileId = doc.fileId
file_data = self.service.getFileData(fileId)
file_info = self.service.getFileInfo(fileId)
if file_data:
try:
# Try to decode as text for context
content = file_data.decode('utf-8')
metadata_info = ""
if file_info and includeMetadata:
metadata_info = f" (Size: {file_info.get('fileSize', 'unknown')}, Type: {file_info.get('mimeType', 'unknown')})"
# Adjust context length based on processing mode
max_length = 5000 if processingMode == "detailed" else 3000 if processingMode == "advanced" else 2000
context_parts.append(f"Document: {doc.filename}{metadata_info}\nContent:\n{content[:max_length]}...")
except UnicodeDecodeError:
context_parts.append(f"Document: {doc.filename} [Binary content]")
if context_parts:
context = "\n\n".join(context_parts)
logger.info(f"Included {len(chatDocuments)} documents in AI context")
# Determine output format
output_extension = ".txt" # Default
output_mime_type = "text/plain" # Default
if expectedDocumentFormats and len(expectedDocumentFormats) > 0:
expected_format = expectedDocumentFormats[0]
output_extension = expected_format.get("extension", ".txt")
output_mime_type = expected_format.get("mimeType", "text/plain")
logger.info(f"Using expected format: {output_extension} ({output_mime_type})")
# Build enhanced prompt
enhanced_prompt = aiPrompt
# Add processing mode instructions if specified (generic, not analysis-specific)
if processingMode == "detailed":
enhanced_prompt += "\n\nPlease provide a detailed response with comprehensive information."
elif processingMode == "advanced":
enhanced_prompt += "\n\nPlease provide an advanced response with deep insights."
# Add custom instructions if provided
if customInstructions:
enhanced_prompt += f"\n\nAdditional Instructions: {customInstructions}"
# Add format-specific instructions only if non-text format is requested
if output_extension != ".txt":
if output_extension == ".csv":
enhanced_prompt += f"\n\nCRITICAL: Deliver the result as pure CSV data without any markdown formatting, code blocks, or additional text. Output only the CSV content with proper headers and data rows."
elif output_extension == ".json":
enhanced_prompt += f"\n\nCRITICAL: Deliver the result as pure JSON data without any markdown formatting, code blocks, or additional text. Output only the JSON content."
elif output_extension == ".xml":
enhanced_prompt += f"\n\nCRITICAL: Deliver the result as pure XML data without any markdown formatting, code blocks, or additional text. Output only the XML content."
else:
enhanced_prompt += f"\n\nCRITICAL: Deliver the result as pure {output_extension.upper()} data without any markdown formatting, code blocks, or additional text."
# Call appropriate AI service based on processing mode
logger.info(f"Executing AI call with mode: {processingMode}, prompt length: {len(enhanced_prompt)}")
if context:
logger.info(f"Including context from {len(documentList)} documents")
if processingMode in ["advanced", "detailed"]:
result = await self.service.callAiTextAdvanced(enhanced_prompt, context)
else:
result = await self.service.callAiTextBasic(enhanced_prompt, context)
# Create result document
timestamp = datetime.now(UTC).strftime('%Y%m%d_%H%M%S')
filename = f"ai_{processingMode}_{timestamp}{output_extension}"
# Create document through service (but don't add to workflow - let calling layer handle that)
document = self.service.createDocument(
fileName=filename,
mimeType=output_mime_type,
content=result,
base64encoded=False
)
return self._createResult(
success=True,
data={
"result": result,
"filename": filename,
"documentId": document.id if hasattr(document, 'id') else None,
"processedDocuments": len(documentList) if documentList else 0,
"processingMode": processingMode,
"document": document # Include the created document in the result data
},
metadata={
"method": "ai.process",
"promptLength": len(aiPrompt),
"contextLength": len(context),
"outputFormat": output_extension,
"includeMetadata": includeMetadata,
"processingMode": processingMode,
"hasCustomInstructions": bool(customInstructions)
}
)
except Exception as e:
logger.error(f"Error in ai.process: {str(e)}")
return self._createResult(
success=False,
data={},
error=f"AI processing failed: {str(e)}"
)

644
modules/methods/methodSharepoint.py
View file

@ -8,6 +8,9 @@ from typing import Dict, Any, List, Optional
from datetime import datetime, UTC
import json
import uuid
import aiohttp
import asyncio
from urllib.parse import urlparse
from modules.chat.methodBase import MethodBase, ActionResult, action
@ -25,7 +28,7 @@ class MethodSharepoint(MethodBase):
"""Get Microsoft connection from connection reference"""
try:
userConnection = self.service.getUserConnectionFromConnectionReference(connectionReference)
if not userConnection or userConnection.authority != "msft" or userConnection.status != "active":
if not userConnection or userConnection.authority.value != "msft" or userConnection.status.value != "active":
return None
# Get the corresponding token for this user and authority
@ -38,12 +41,103 @@ class MethodSharepoint(MethodBase):
"id": userConnection.id,
"accessToken": token.tokenAccess,
"refreshToken": token.tokenRefresh,
"scopes": ["Sites.ReadWrite.All", "User.Read"] # Default Microsoft scopes
"scopes": ["Sites.ReadWrite.All", "Files.ReadWrite.All", "User.Read"] # SharePoint scopes
}
except Exception as e:
logger.error(f"Error getting Microsoft connection: {str(e)}")
return None
def _parseSiteUrl(self, siteUrl: str) -> Dict[str, str]:
"""Parse SharePoint site URL to extract hostname and site path"""
try:
parsed = urlparse(siteUrl)
hostname = parsed.hostname
path = parsed.path.strip('/')
return {
"hostname": hostname,
"sitePath": path
}
except Exception as e:
logger.error(f"Error parsing site URL {siteUrl}: {str(e)}")
return {"hostname": "", "sitePath": ""}
async def _makeGraphApiCall(self, access_token: str, endpoint: str, method: str = "GET", data: bytes = None) -> Dict[str, Any]:
"""Make a Microsoft Graph API call with timeout and detailed logging"""
try:
headers = {
"Authorization": f"Bearer {access_token}",
"Content-Type": "application/json" if data and method != "PUT" else "application/octet-stream" if data else "application/json"
}
url = f"https://graph.microsoft.com/v1.0/{endpoint}"
logger.info(f"Making Graph API call: {method} {url}")
# Set timeout to 30 seconds
timeout = aiohttp.ClientTimeout(total=30)
async with aiohttp.ClientSession(timeout=timeout) as session:
if method == "GET":
logger.debug(f"Starting GET request to {url}")
async with session.get(url, headers=headers) as response:
logger.info(f"Graph API response: {response.status}")
if response.status == 200:
result = await response.json()
logger.debug(f"Graph API success: {len(str(result))} characters response")
return result
else:
error_text = await response.text()
logger.error(f"Graph API call failed: {response.status} - {error_text}")
return {"error": f"API call failed: {response.status} - {error_text}"}
elif method == "PUT":
logger.debug(f"Starting PUT request to {url}")
async with session.put(url, headers=headers, data=data) as response:
logger.info(f"Graph API response: {response.status}")
if response.status in [200, 201]:
result = await response.json()
logger.debug(f"Graph API success: {len(str(result))} characters response")
return result
else:
error_text = await response.text()
logger.error(f"Graph API call failed: {response.status} - {error_text}")
return {"error": f"API call failed: {response.status} - {error_text}"}
elif method == "POST":
logger.debug(f"Starting POST request to {url}")
async with session.post(url, headers=headers, data=data) as response:
logger.info(f"Graph API response: {response.status}")
if response.status in [200, 201]:
result = await response.json()
logger.debug(f"Graph API success: {len(str(result))} characters response")
return result
else:
error_text = await response.text()
logger.error(f"Graph API call failed: {response.status} - {error_text}")
return {"error": f"API call failed: {response.status} - {error_text}"}
except asyncio.TimeoutError:
logger.error(f"Graph API call timed out after 30 seconds: {endpoint}")
return {"error": f"API call timed out after 30 seconds: {endpoint}"}
except Exception as e:
logger.error(f"Error making Graph API call: {str(e)}")
return {"error": f"Error making Graph API call: {str(e)}"}
async def _getSiteId(self, access_token: str, hostname: str, site_path: str) -> str:
"""Get SharePoint site ID from hostname and site path"""
try:
endpoint = f"sites/{hostname}:/{site_path}"
result = await self._makeGraphApiCall(access_token, endpoint)
if "error" in result:
logger.error(f"Error getting site ID: {result['error']}")
return ""
return result.get("id", "")
except Exception as e:
logger.error(f"Error getting site ID: {str(e)}")
return ""
@action
async def findDocumentPath(self, parameters: Dict[str, Any]) -> ActionResult:
"""
@ -78,37 +172,98 @@ class MethodSharepoint(MethodBase):
error="No valid Microsoft connection found for the provided connection reference"
)
find_prompt = f"""
Simulate finding document paths in Microsoft SharePoint based on a query.
# Parse site URL to get hostname and site path
site_info = self._parseSiteUrl(siteUrl)
if not site_info["hostname"] or not site_info["sitePath"]:
return self._createResult(
success=False,
data={},
error=f"Invalid SharePoint site URL: {siteUrl}"
)
Connection: {connection['id']}
Site URL: {siteUrl}
Query: {query}
Search Scope: {searchScope}
# Get site ID
site_id = await self._getSiteId(connection["accessToken"], site_info["hostname"], site_info["sitePath"])
if not site_id:
return self._createResult(
success=False,
data={},
error="Failed to get SharePoint site ID"
)
Please provide:
1. Matching document paths and locations
2. Relevance scores for each match
3. Document metadata and properties
4. Alternative search suggestions
5. Search statistics and coverage
"""
find_result = await self.service.interfaceAiCalls.callAiTextAdvanced(find_prompt)
result_data = {
"connectionReference": connectionReference,
"siteUrl": siteUrl,
"query": query,
"searchScope": searchScope,
"findResult": find_result,
"connection": {
"id": connection["id"],
"authority": "microsoft",
"reference": connectionReference
},
"timestamp": datetime.now(UTC).isoformat()
}
try:
# Use Microsoft Graph search API
search_query = query.replace("'", "''") # Escape single quotes for OData
endpoint = f"sites/{site_id}/drive/root/search(q='{search_query}')"
# Make the search API call
search_result = await self._makeGraphApiCall(connection["accessToken"], endpoint)
if "error" in search_result:
return self._createResult(
success=False,
data={},
error=f"Search failed: {search_result['error']}"
)
# Process search results
items = search_result.get("value", [])
found_documents = []
for item in items:
# Filter by search scope if specified
if searchScope == "documents" and "folder" in item:
continue
elif searchScope == "pages" and "file" in item and not item["file"].get("mimeType", "").startswith("text/html"):
continue
doc_info = {
"id": item.get("id"),
"name": item.get("name"),
"path": item.get("parentReference", {}).get("path", "") + "/" + item.get("name", ""),
"size": item.get("size", 0),
"createdDateTime": item.get("createdDateTime"),
"lastModifiedDateTime": item.get("lastModifiedDateTime"),
"webUrl": item.get("webUrl"),
"type": "folder" if "folder" in item else "file"
}
# Add file-specific information
if "file" in item:
doc_info.update({
"mimeType": item["file"].get("mimeType"),
"downloadUrl": item.get("@microsoft.graph.downloadUrl")
})
# Add folder-specific information
if "folder" in item:
doc_info.update({
"childCount": item["folder"].get("childCount", 0)
})
found_documents.append(doc_info)
result_data = {
"connectionReference": connectionReference,
"siteUrl": siteUrl,
"query": query,
"searchScope": searchScope,
"totalResults": len(found_documents),
"foundDocuments": found_documents,
"connection": {
"id": connection["id"],
"authority": "microsoft",
"reference": connectionReference
},
"timestamp": datetime.now(UTC).isoformat()
}
except Exception as e:
logger.error(f"Error searching SharePoint: {str(e)}")
return self._createResult(
success=False,
data={},
error=str(e)
)
# Determine output format based on expected formats
output_extension = ".json" # Default
@ -172,8 +327,23 @@ class MethodSharepoint(MethodBase):
error="Document list reference, connection reference, site URL, and document paths are required"
)
# Get documents from reference
# Get documents from reference - ensure documentList is a list, not a string
if isinstance(documentList, str):
documentList = [documentList] # Convert string to list
chatDocuments = self.service.getChatDocumentsFromDocumentList(documentList)
# For testing: if no chat documents found, create mock documents based on document paths
if not chatDocuments and documentPaths:
logger.info("No chat documents found, creating mock documents for testing based on document paths")
chatDocuments = []
for i, path in enumerate(documentPaths):
mock_doc = type('MockChatDocument', (), {
'fileId': f'mock_file_id_{i}',
'filename': path.split('/')[-1] if '/' in path else path
})()
chatDocuments.append(mock_doc)
logger.info(f"Created {len(chatDocuments)} mock documents for testing")
if not chatDocuments:
return self._createResult(
success=False,
@ -189,37 +359,112 @@ class MethodSharepoint(MethodBase):
error="No valid Microsoft connection found for the provided connection reference"
)
# Parse site URL to get hostname and site path
site_info = self._parseSiteUrl(siteUrl)
if not site_info["hostname"] or not site_info["sitePath"]:
return self._createResult(
success=False,
data={},
error=f"Invalid SharePoint site URL: {siteUrl}"
)
# Get site ID
site_id = await self._getSiteId(connection["accessToken"], site_info["hostname"], site_info["sitePath"])
if not site_id:
return self._createResult(
success=False,
data={},
error="Failed to get SharePoint site ID"
)
# Process each document path
read_results = []
for i, documentPath in enumerate(documentPaths):
if i < len(chatDocuments):
chatDocument = chatDocuments[i]
fileId = chatDocument.fileId
try:
# Check if documentPath is actually a file ID (starts with 016GRP6V)
if documentPath.startswith('016GRP6V'):
# Use file ID directly
file_endpoint = f"sites/{site_id}/drive/items/{documentPath}"
logger.info(f"Reading file by ID: {documentPath}")
else:
# First, find the file by its path
path_clean = documentPath.lstrip('/')
file_endpoint = f"sites/{site_id}/drive/root:/{path_clean}"
logger.info(f"Reading file by path: {path_clean}")
sharepoint_prompt = f"""
Simulate reading a document from Microsoft SharePoint.
# Get file metadata
file_info_result = await self._makeGraphApiCall(connection["accessToken"], file_endpoint)
Connection: {connection['id']}
Site URL: {siteUrl}
Document Path: {documentPath}
Include Metadata: {includeMetadata}
File ID: {fileId}
if "error" in file_info_result:
read_results.append({
"documentPath": documentPath,
"error": f"File not found: {file_info_result['error']}",
"content": None
})
continue
Please provide:
1. Document content and structure
2. File metadata and properties
3. SharePoint site information
4. Document permissions and sharing
5. Version history if available
"""
file_id = file_info_result.get("id")
if not file_id:
read_results.append({
"documentPath": documentPath,
"error": "Could not get file ID",
"content": None
})
continue
document_data = await self.service.interfaceAiCalls.callAiTextAdvanced(sharepoint_prompt)
# Build result with metadata
result_item = {
"documentPath": documentPath,
"fileId": file_id,
"fileName": file_info_result.get("name"),
"size": file_info_result.get("size", 0),
"createdDateTime": file_info_result.get("createdDateTime"),
"lastModifiedDateTime": file_info_result.get("lastModifiedDateTime"),
"webUrl": file_info_result.get("webUrl")
}
# Add metadata if requested
if includeMetadata:
result_item["metadata"] = {
"mimeType": file_info_result.get("file", {}).get("mimeType"),
"downloadUrl": file_info_result.get("@microsoft.graph.downloadUrl"),
"createdBy": file_info_result.get("createdBy", {}),
"lastModifiedBy": file_info_result.get("lastModifiedBy", {}),
"parentReference": file_info_result.get("parentReference", {})
}
# Get file content if it's a readable format
mime_type = file_info_result.get("file", {}).get("mimeType", "")
if mime_type.startswith("text/") or mime_type in [
"application/json", "application/xml", "application/javascript"
]:
# Download the file content
content_endpoint = f"sites/{site_id}/drive/items/{file_id}/content"
# For content download, we need to handle binary data
try:
async with aiohttp.ClientSession() as session:
headers = {"Authorization": f"Bearer {connection['accessToken']}"}
async with session.get(f"https://graph.microsoft.com/v1.0/{content_endpoint}", headers=headers) as response:
if response.status == 200:
content = await response.text()
result_item["content"] = content
else:
result_item["content"] = f"Could not download content: HTTP {response.status}"
except Exception as e:
result_item["content"] = f"Error downloading content: {str(e)}"
else:
result_item["content"] = f"Binary file type ({mime_type}) - content not retrieved"
read_results.append(result_item)
except Exception as e:
logger.error(f"Error reading document {documentPath}: {str(e)}")
read_results.append({
"documentPath": documentPath,
"fileId": fileId,
"documentContent": document_data
"error": str(e),
"content": None
})
result_data = {
@ -306,7 +551,9 @@ class MethodSharepoint(MethodBase):
error="No valid Microsoft connection found for the provided connection reference"
)
# Get documents from reference
# Get documents from reference - ensure documentList is a list, not a string
if isinstance(documentList, str):
documentList = [documentList] # Convert string to list
chatDocuments = self.service.getChatDocumentsFromDocumentList(documentList)
if not chatDocuments:
return self._createResult(
@ -315,46 +562,107 @@ class MethodSharepoint(MethodBase):
error="No documents found for the provided reference"
)
# Parse site URL to get hostname and site path
site_info = self._parseSiteUrl(siteUrl)
if not site_info["hostname"] or not site_info["sitePath"]:
return self._createResult(
success=False,
data={},
error=f"Invalid SharePoint site URL: {siteUrl}"
)
# Get site ID
site_id = await self._getSiteId(connection["accessToken"], site_info["hostname"], site_info["sitePath"])
if not site_id:
return self._createResult(
success=False,
data={},
error="Failed to get SharePoint site ID"
)
# Process each document upload
upload_results = []
for i, (documentPath, fileName) in enumerate(zip(documentPaths, fileNames)):
if i < len(chatDocuments):
chatDocument = chatDocuments[i]
fileId = chatDocument.fileId
file_data = self.service.getFileData(fileId)
if not file_data:
logger.warning(f"File data not found for fileId: {fileId}")
continue
# Create SharePoint upload prompt
upload_prompt = f"""
Simulate uploading a document to Microsoft SharePoint.
Connection: {connection['id']}
Site URL: {siteUrl}
Document Path: {documentPath}
File Name: {fileName}
File ID: {fileId}
File Size: {len(file_data)} bytes
Please provide:
1. Upload confirmation and status
2. File metadata and properties
3. SharePoint site integration details
4. Permission and sharing settings
5. Version control information
"""
# Use AI to simulate SharePoint upload
upload_result = await self.service.interfaceAiCalls.callAiTextAdvanced(upload_prompt)
try:
if i < len(chatDocuments):
chatDocument = chatDocuments[i]
fileId = chatDocument.fileId
file_data = self.service.getFileData(fileId)
if not file_data:
logger.warning(f"File data not found for fileId: {fileId}")
upload_results.append({
"documentPath": documentPath,
"fileName": fileName,
"fileId": fileId,
"error": "File data not found",
"uploadStatus": "failed"
})
continue
# Prepare upload path
upload_path = documentPath.rstrip('/') + '/' + fileName
upload_path_clean = upload_path.lstrip('/')
# Upload endpoint for small files (< 4MB)
if len(file_data) < 4 * 1024 * 1024: # 4MB
upload_endpoint = f"sites/{site_id}/drive/root:/{upload_path_clean}:/content"
# Upload the file
upload_result = await self._makeGraphApiCall(
connection["accessToken"],
upload_endpoint,
method="PUT",
data=file_data
)
if "error" in upload_result:
upload_results.append({
"documentPath": documentPath,
"fileName": fileName,
"fileId": fileId,
"error": upload_result["error"],
"uploadStatus": "failed"
})
else:
upload_results.append({
"documentPath": documentPath,
"fileName": fileName,
"fileId": fileId,
"uploadStatus": "success",
"sharepointFileId": upload_result.get("id"),
"webUrl": upload_result.get("webUrl"),
"size": upload_result.get("size"),
"createdDateTime": upload_result.get("createdDateTime")
})
else:
# For large files, we would need to implement resumable upload
# For now, return an error for large files
upload_results.append({
"documentPath": documentPath,
"fileName": fileName,
"fileId": fileId,
"error": f"File too large ({len(file_data)} bytes). Files larger than 4MB require resumable upload (not implemented).",
"uploadStatus": "failed"
})
else:
upload_results.append({
"documentPath": documentPath,
"fileName": fileName,
"fileId": None,
"error": "No corresponding chat document found",
"uploadStatus": "failed"
})
except Exception as e:
logger.error(f"Error uploading document {fileName}: {str(e)}")
upload_results.append({
"documentPath": documentPath,
"fileName": fileName,
"fileId": fileId,
"uploadResult": upload_result
"fileId": fileId if i < len(chatDocuments) else None,
"error": str(e),
"uploadStatus": "failed"
})
# Create result data
@ -423,7 +731,7 @@ class MethodSharepoint(MethodBase):
connectionReference = parameters.get("connectionReference")
siteUrl = parameters.get("siteUrl")
folderPaths = parameters.get("folderPaths")
includeSubfolders = parameters.get("includeSubfolders", False)
includeSubfolders = parameters.get("includeSubfolders", False) # Default to False for better UX
expectedDocumentFormats = parameters.get("expectedDocumentFormats", [])
if not connectionReference or not siteUrl or not folderPaths:
@ -442,34 +750,148 @@ class MethodSharepoint(MethodBase):
error="No valid Microsoft connection found for the provided connection reference"
)
logger.info(f"Starting SharePoint listDocuments for site: {siteUrl}")
logger.debug(f"Connection ID: {connection['id']}")
logger.debug(f"Folder paths: {folderPaths}")
# Parse site URL to get hostname and site path
site_info = self._parseSiteUrl(siteUrl)
logger.info(f"Parsed site info - hostname: {site_info['hostname']}, sitePath: {site_info['sitePath']}")
if not site_info["hostname"] or not site_info["sitePath"]:
logger.error(f"Failed to parse site URL: {siteUrl}")
return self._createResult(
success=False,
data={},
error=f"Invalid SharePoint site URL: {siteUrl}"
)
# Get site ID
logger.info(f"Getting site ID for hostname: {site_info['hostname']}, path: {site_info['sitePath']}")
site_id = await self._getSiteId(connection["accessToken"], site_info["hostname"], site_info["sitePath"])
logger.info(f"Site ID result: {site_id}")
if not site_id:
return self._createResult(
success=False,
data={},
error="Failed to get SharePoint site ID"
)
# Process each folder path
list_results = []
for folderPath in folderPaths:
# Create SharePoint listing prompt
list_prompt = f"""
Simulate listing documents in Microsoft SharePoint folder.
Connection: {connection['id']}
Site URL: {siteUrl}
Folder Path: {folderPath}
Include Subfolders: {includeSubfolders}
Please provide:
1. List of documents and folders
2. File metadata and properties
3. Folder structure and hierarchy
4. Permission and sharing information
5. Document statistics and summary
"""
# Use AI to simulate SharePoint listing
list_result = await self.service.interfaceAiCalls.callAiTextAdvanced(list_prompt)
list_results.append({
"folderPath": folderPath,
"listResult": list_result
})
try:
# Determine the endpoint based on folder path
if folderPath in ["/", ""]:
# Root folder
endpoint = f"sites/{site_id}/drive/root/children"
else:
# Specific folder - remove leading slash if present
folder_path_clean = folderPath.lstrip('/')
endpoint = f"sites/{site_id}/drive/root:/{folder_path_clean}:/children"
# Make the API call to list folder contents
api_result = await self._makeGraphApiCall(connection["accessToken"], endpoint)
if "error" in api_result:
list_results.append({
"folderPath": folderPath,
"error": api_result["error"],
"items": []
})
continue
# Process the results
items = api_result.get("value", [])
processed_items = []
for item in items:
item_info = {
"id": item.get("id"),
"name": item.get("name"),
"size": item.get("size", 0),
"createdDateTime": item.get("createdDateTime"),
"lastModifiedDateTime": item.get("lastModifiedDateTime"),
"webUrl": item.get("webUrl"),
"type": "folder" if "folder" in item else "file"
}
# Add file-specific information
if "file" in item:
item_info.update({
"mimeType": item["file"].get("mimeType"),
"downloadUrl": item.get("@microsoft.graph.downloadUrl")
})
# Add folder-specific information
if "folder" in item:
item_info.update({
"childCount": item["folder"].get("childCount", 0)
})
processed_items.append(item_info)
# If include subfolders is enabled, get ONLY direct subfolder contents (1 level deep only)
if includeSubfolders:
logger.info(f"Including subfolders - processing {len([item for item in processed_items if item['type'] == 'folder'])} folders")
subfolder_count = 0
max_subfolders = 10 # Limit to prevent infinite loops
for item in processed_items[:]: # Use slice to avoid modifying list during iteration
if item["type"] == "folder" and subfolder_count < max_subfolders:
subfolder_count += 1
subfolder_path = f"{folderPath.rstrip('/')}/{item['name']}"
subfolder_endpoint = f"sites/{site_id}/drive/items/{item['id']}/children"
logger.debug(f"Getting contents of subfolder: {item['name']}")
subfolder_result = await self._makeGraphApiCall(connection["accessToken"], subfolder_endpoint)
if "error" not in subfolder_result:
subfolder_items = subfolder_result.get("value", [])
logger.debug(f"Found {len(subfolder_items)} items in subfolder {item['name']}")
for subfolder_item in subfolder_items:
# Only add files and direct subfolders, NO RECURSION
subfolder_item_info = {
"id": subfolder_item.get("id"),
"name": subfolder_item.get("name"),
"size": subfolder_item.get("size", 0),
"createdDateTime": subfolder_item.get("createdDateTime"),
"lastModifiedDateTime": subfolder_item.get("lastModifiedDateTime"),
"webUrl": subfolder_item.get("webUrl"),
"type": "folder" if "folder" in subfolder_item else "file",
"parentPath": subfolder_path
}
if "file" in subfolder_item:
subfolder_item_info.update({
"mimeType": subfolder_item["file"].get("mimeType"),
"downloadUrl": subfolder_item.get("@microsoft.graph.downloadUrl")
})
processed_items.append(subfolder_item_info)
else:
logger.warning(f"Failed to get contents of subfolder {item['name']}: {subfolder_result.get('error')}")
elif subfolder_count >= max_subfolders:
logger.warning(f"Reached maximum subfolder limit ({max_subfolders}), skipping remaining folders")
break
logger.info(f"Processed {subfolder_count} subfolders, total items: {len(processed_items)}")
list_results.append({
"folderPath": folderPath,
"itemCount": len(processed_items),
"items": processed_items
})
except Exception as e:
logger.error(f"Error listing folder {folderPath}: {str(e)}")
list_results.append({
"folderPath": folderPath,
"error": str(e),
"items": []
})
# Create result data
result_data = {

287
modules/methods/methodWeb.py
View file

@ -474,46 +474,107 @@ class MethodWeb(MethodBase):
return approaches
@action
def search(self, parameters: Dict[str, Any]) -> ActionResult:
async def search(self, parameters: Dict[str, Any]) -> ActionResult:
"""
Perform a web search and output a .txt file with a plain list of URLs (one per line).
Parameters:
query (str): Search query to perform
maxResults (int, optional): Maximum number of results (default: 10)
filter (str, optional): Filter criteria for search results
expectedDocumentFormats (list, optional): Expected document formats with extension, mimeType, description
"""
query = parameters.get("query")
max_results = parameters.get("maxResults", 10)
filter_param = parameters.get("filter")
if not query:
return ActionResult.failure("Search query is required")
if not self.srcApikey:
return ActionResult.failure("SerpAPI key not configured")
userLanguage = "en"
if hasattr(self.service, 'user') and hasattr(self.service.user, 'language'):
userLanguage = self.service.user.language
params = {
"engine": self.srcEngine,
"q": query,
"api_key": self.srcApikey,
"num": min(max_results, self.maxResults),
"hl": userLanguage
}
if filter_param:
params["filter"] = filter_param
try:
query = parameters.get("query")
max_results = parameters.get("maxResults", 10)
filter_param = parameters.get("filter")
expectedDocumentFormats = parameters.get("expectedDocumentFormats", [])
if not query:
return self._createResult(
success=False,
data={},
error="Search query is required"
)
if not self.srcApikey:
return self._createResult(
success=False,
data={},
error="SerpAPI key not configured"
)
userLanguage = "en"
if hasattr(self.service, 'user') and hasattr(self.service.user, 'language'):
userLanguage = self.service.user.language
params = {
"engine": self.srcEngine,
"q": query,
"api_key": self.srcApikey,
"num": min(max_results, self.maxResults),
"hl": userLanguage
}
if filter_param:
params["filter"] = filter_param
response = requests.get("https://serpapi.com/search", params=params, timeout=self.timeout)
response.raise_for_status()
search_results = response.json()
results = []
if "organic_results" in search_results:
results = search_results["organic_results"][:max_results]
# Assume 'results' is a list of dicts with 'url' keys
urls = [item['url'] for item in results if 'url' in item and isinstance(item['url'], str)]
url_list_str = "\n".join(urls)
filename = f"web_search_{datetime.now(UTC).strftime('%Y%m%d_%H%M%S')}.txt"
with open(filename, "w", encoding="utf-8") as f:
f.write(url_list_str)
return ActionResult.success(documents=[filename], resultLabel=parameters.get("resultLabel"))
# Determine output format based on expected formats
output_extension = ".txt" # Default
output_mime_type = "text/plain" # Default
if expectedDocumentFormats and len(expectedDocumentFormats) > 0:
# Use the first expected format
expected_format = expectedDocumentFormats[0]
output_extension = expected_format.get("extension", ".txt")
output_mime_type = expected_format.get("mimeType", "text/plain")
logger.info(f"Using expected format: {output_extension} ({output_mime_type})")
else:
logger.info("No expected format specified, using default .txt format")
# Create result data
result_data = {
"query": query,
"maxResults": max_results,
"filter": filter_param,
"totalResults": len(urls),
"urls": urls,
"urlList": url_list_str,
"timestamp": datetime.now(UTC).isoformat()
}
return self._createResult(
success=True,
data={
"documents": [
{
"documentName": f"web_search_{datetime.now(UTC).strftime('%Y%m%d_%H%M%S')}{output_extension}",
"documentData": result_data,
"mimeType": output_mime_type
}
]
}
)
except Exception as e:
logger.error(f"Error searching web: {str(e)}")
return ActionResult.failure(error=str(e))
return self._createResult(
success=False,
data={},
error=str(e)
)
def _selenium_extract_content(self, url: str) -> Optional[str]:
"""Use Selenium to fetch and extract main content from a JS-heavy page."""
@ -540,70 +601,126 @@ class MethodWeb(MethodBase):
return None
@action
def crawl(self, parameters: Dict[str, Any]) -> ActionResult:
async def crawl(self, parameters: Dict[str, Any]) -> ActionResult:
"""
Crawl a list of URLs provided in a document (.txt) with URLs separated by newline, comma, or semicolon.
Parameters:
document (str): Document containing URL list
expectedDocumentFormats (list, optional): Expected document formats with extension, mimeType, description
"""
document = parameters.get("document")
if not document:
return ActionResult.failure("No document with URL list provided.")
# Read the document content
with open(document, "r", encoding="utf-8") as f:
content = f.read()
# Split URLs by newline, comma, or semicolon
import re
urls = re.split(r'[\n,;]+', content)
urls = [u.strip() for u in urls if u.strip()]
if not urls:
return ActionResult.failure("No valid URLs provided in the document.")
crawl_results = []
for url in urls:
try:
logger.info(f"Crawling URL: {url}")
# Try Selenium first
content = self._selenium_extract_content(url)
if not content:
# Fallback to requests/BeautifulSoup
soup = self._readUrl(url)
content = self._extractMainContent(soup)
title = self._extractTitle(BeautifulSoup(content, 'html.parser'), url) if content else "No title"
meta_info = {"url": url, "title": title}
content_length = len(content) if content else 0
crawl_results.append({
"url": url,
"title": title,
"content": content,
"content_length": content_length,
"meta_info": meta_info,
"timestamp": datetime.now(UTC).isoformat()
})
logger.info(f"Successfully crawled {url} - extracted {content_length} characters")
except Exception as e:
logger.error(f"Error crawling web page {url}: {str(e)}")
crawl_results.append({
"error": str(e),
"url": url,
"suggestions": [
"Check if the URL is accessible",
"Try with a different user agent",
"Verify the site doesn't block automated access"
try:
document = parameters.get("document")
expectedDocumentFormats = parameters.get("expectedDocumentFormats", [])
if not document:
return self._createResult(
success=False,
data={},
error="No document with URL list provided."
)
# Read the document content
with open(document, "r", encoding="utf-8") as f:
content = f.read()
# Split URLs by newline, comma, or semicolon
import re
urls = re.split(r'[\n,;]+', content)
urls = [u.strip() for u in urls if u.strip()]
if not urls:
return self._createResult(
success=False,
data={},
error="No valid URLs provided in the document."
)
crawl_results = []
for url in urls:
try:
logger.info(f"Crawling URL: {url}")
# Try Selenium first
content = self._selenium_extract_content(url)
if not content:
# Fallback to requests/BeautifulSoup
soup = self._readUrl(url)
content = self._extractMainContent(soup)
title = self._extractTitle(BeautifulSoup(content, 'html.parser'), url) if content else "No title"
meta_info = {"url": url, "title": title}
content_length = len(content) if content else 0
crawl_results.append({
"url": url,
"title": title,
"content": content,
"content_length": content_length,
"meta_info": meta_info,
"timestamp": datetime.now(UTC).isoformat()
})
logger.info(f"Successfully crawled {url} - extracted {content_length} characters")
except Exception as e:
logger.error(f"Error crawling web page {url}: {str(e)}")
crawl_results.append({
"error": str(e),
"url": url,
"suggestions": [
"Check if the URL is accessible",
"Try with a different user agent",
"Verify the site doesn't block automated access"
]
})
# Determine output format based on expected formats
output_extension = ".json" # Default
output_mime_type = "application/json" # Default
if expectedDocumentFormats and len(expectedDocumentFormats) > 0:
# Use the first expected format
expected_format = expectedDocumentFormats[0]
output_extension = expected_format.get("extension", ".json")
output_mime_type = expected_format.get("mimeType", "application/json")
logger.info(f"Using expected format: {output_extension} ({output_mime_type})")
else:
logger.info("No expected format specified, using default .json format")
result_data = {
"urls": urls,
"maxDepth": 1, # Simplified crawl
"includeImages": False,
"followLinks": True,
"crawlResults": crawl_results,
"summary": {
"total_urls": len(urls),
"successful_crawls": len([r for r in crawl_results if "error" not in r]),
"failed_crawls": len([r for r in crawl_results if "error" in r]),
"total_content_chars": sum([r.get("content_length", 0) for r in crawl_results if "content_length" in r])
},
"timestamp": datetime.now(UTC).isoformat()
}
return self._createResult(
success=True,
data={
"documents": [
{
"documentName": f"web_crawl_{datetime.now(UTC).strftime('%Y%m%d_%H%M%S')}{output_extension}",
"documentData": result_data,
"mimeType": output_mime_type
}
]
})
result_data = {
"urls": urls,
"maxDepth": 1, # Simplified crawl
"includeImages": False,
"followLinks": True,
"crawlResults": crawl_results,
"summary": {
"total_urls": len(urls),
"successful_crawls": len([r for r in crawl_results if "error" not in r]),
"failed_crawls": len([r for r in crawl_results if "error" in r]),
"total_content_chars": sum([r.get("content_length", 0) for r in crawl_results if "content_length" in r])
},
"timestamp": datetime.now(UTC).isoformat()
}
return ActionResult.success(result=result_data, resultLabel=parameters.get("resultLabel"))
}
)
except Exception as e:
logger.error(f"Error crawling web pages: {str(e)}")
return self._createResult(
success=False,
data={},
error=str(e)
)
@action
async def scrape(self, parameters: Dict[str, Any]) -> ActionResult:

143
modules/routes/routeDataConnections.py
View file

@ -61,7 +61,7 @@ async def create_connection(
connection_data: Dict[str, Any] = Body(...),
currentUser: User = Depends(getCurrentUser)
) -> UserConnection:
"""Create a new connection for the current user or update existing one"""
"""Create a new connection for the current user"""
try:
interface = getInterface(currentUser)
@ -86,57 +86,28 @@ async def create_connection(
detail="User not found"
)
# Check for existing connection of the same authority
existing_connection = None
connections = interface.getUserConnections(currentUser.id)
for conn in connections:
if conn.authority == authority:
existing_connection = conn
break
# Always create a new connection with PENDING status
connection = interface.addUserConnection(
userId=currentUser.id,
authority=authority,
externalId="", # Will be set after OAuth
externalUsername="", # Will be set after OAuth
status=ConnectionStatus.PENDING # Start with PENDING status
)
if existing_connection:
# Update existing connection
existing_connection.status = ConnectionStatus.PENDING
existing_connection.lastChecked = datetime.now()
existing_connection.externalId = "" # Reset for new OAuth flow
existing_connection.externalUsername = "" # Reset for new OAuth flow
# Convert connection to dict and ensure datetime fields are serialized
connection_dict = existing_connection.to_dict()
for field in ['connectedAt', 'lastChecked', 'expiresAt']:
if field in connection_dict and connection_dict[field] is not None:
if isinstance(connection_dict[field], datetime):
connection_dict[field] = connection_dict[field].isoformat()
elif isinstance(connection_dict[field], (int, float)):
connection_dict[field] = datetime.fromtimestamp(connection_dict[field]).isoformat()
# Update connection record directly
interface.db.recordModify("connections", existing_connection.id, connection_dict)
return existing_connection
else:
# Create new connection with PENDING status
connection = interface.addUserConnection(
userId=currentUser.id,
authority=authority,
externalId="", # Will be set after OAuth
externalUsername="", # Will be set after OAuth
status=ConnectionStatus.PENDING # Start with PENDING status
)
# Convert connection to dict and ensure datetime fields are serialized
connection_dict = connection.to_dict()
for field in ['connectedAt', 'lastChecked', 'expiresAt']:
if field in connection_dict and connection_dict[field] is not None:
if isinstance(connection_dict[field], datetime):
connection_dict[field] = connection_dict[field].isoformat()
elif isinstance(connection_dict[field], (int, float)):
connection_dict[field] = datetime.fromtimestamp(connection_dict[field]).isoformat()
# Save connection record
interface.db.recordModify("connections", connection.id, connection_dict)
return connection
# Convert connection to dict and ensure datetime fields are serialized
connection_dict = connection.to_dict()
for field in ['connectedAt', 'lastChecked', 'expiresAt']:
if field in connection_dict and connection_dict[field] is not None:
if isinstance(connection_dict[field], datetime):
connection_dict[field] = connection_dict[field].isoformat()
elif isinstance(connection_dict[field], (int, float)):
connection_dict[field] = datetime.fromtimestamp(connection_dict[field]).isoformat()
# Save connection record
interface.db.recordModify("connections", connection.id, connection_dict)
return connection
except HTTPException:
raise
@ -147,6 +118,76 @@ async def create_connection(
detail=f"Failed to create connection: {str(e)}"
)
@router.put("/{connectionId}", response_model=UserConnection)
@limiter.limit("10/minute")
async def update_connection(
request: Request,
connectionId: str = Path(..., description="The ID of the connection to update"),
connection_data: Dict[str, Any] = Body(...),
currentUser: User = Depends(getCurrentUser)
) -> UserConnection:
"""Update an existing connection"""
try:
interface = getInterface(currentUser)
# Find the connection
connection = None
if currentUser.privilege in ['admin', 'sysadmin']:
# Admins can update any connection
users = interface.getAllUsers()
for user in users:
connections = interface.getUserConnections(user.id)
for conn in connections:
if conn.id == connectionId:
connection = conn
break
if connection:
break
else:
# Regular users can only update their own connections
connections = interface.getUserConnections(currentUser.id)
for conn in connections:
if conn.id == connectionId:
connection = conn
break
if not connection:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND,
detail="Connection not found"
)
# Update connection fields
for field, value in connection_data.items():
if hasattr(connection, field):
setattr(connection, field, value)
# Update lastChecked timestamp
connection.lastChecked = datetime.now()
# Convert connection to dict and ensure datetime fields are serialized
connection_dict = connection.to_dict()
for field in ['connectedAt', 'lastChecked', 'expiresAt']:
if field in connection_dict and connection_dict[field] is not None:
if isinstance(connection_dict[field], datetime):
connection_dict[field] = connection_dict[field].isoformat()
elif isinstance(connection_dict[field], (int, float)):
connection_dict[field] = datetime.fromtimestamp(connection_dict[field]).isoformat()
# Update connection record
interface.db.recordModify("connections", connectionId, connection_dict)
return connection
except HTTPException:
raise
except Exception as e:
logger.error(f"Error updating connection: {str(e)}")
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail=f"Failed to update connection: {str(e)}"
)
@router.post("/{connectionId}/connect")
@limiter.limit("10/minute")
async def connect_service(

163
modules/routes/routeSecurityGoogle.py
View file

@ -8,10 +8,8 @@ import logging
import json
from typing import Dict, Any, Optional
from datetime import datetime, timedelta
from google.oauth2.credentials import Credentials
from google_auth_oauthlib.flow import Flow
from google.auth.transport.requests import Request as GoogleRequest
from googleapiclient.discovery import build
from requests_oauthlib import OAuth2Session
import httpx
from modules.shared.configuration import APP_CONFIG
from modules.interfaces.interfaceAppObjects import getInterface, getRootInterface
@ -42,9 +40,25 @@ REDIRECT_URI = APP_CONFIG.get("Service_GOOGLE_REDIRECT_URI")
SCOPES = [
"https://www.googleapis.com/auth/gmail.readonly",
"https://www.googleapis.com/auth/userinfo.profile",
"https://www.googleapis.com/auth/userinfo.email"
"https://www.googleapis.com/auth/userinfo.email",
"openid"
]
@router.get("/config")
async def get_config():
"""Debug endpoint to check Google OAuth configuration"""
return {
"client_id": CLIENT_ID,
"client_secret": "***" if CLIENT_SECRET else None,
"redirect_uri": REDIRECT_URI,
"scopes": SCOPES,
"config_loaded": bool(CLIENT_ID and CLIENT_SECRET and REDIRECT_URI),
"config_source": {
"client_id_from": "config.ini" if CLIENT_ID and "354925410565" in CLIENT_ID else "env file",
"redirect_uri_from": "config.ini" if REDIRECT_URI and "gateway-int.poweron-center.net" in REDIRECT_URI else "env file"
}
}
@router.get("/login")
@limiter.limit("5/minute")
async def login(
@ -54,19 +68,30 @@ async def login(
) -> RedirectResponse:
"""Initiate Google login"""
try:
# Create OAuth flow
flow = Flow.from_client_config(
{
"web": {
"client_id": CLIENT_ID,
"client_secret": CLIENT_SECRET,
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"redirect_uris": [REDIRECT_URI]
}
},
scopes=SCOPES
)
# Debug: Log configuration values
logger.info(f"Google OAuth Configuration - CLIENT_ID: {CLIENT_ID}, REDIRECT_URI: {REDIRECT_URI}")
# Validate required configuration
if not CLIENT_ID:
logger.error("Google OAuth CLIENT_ID is not configured")
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail="Google OAuth CLIENT_ID is not configured"
)
if not CLIENT_SECRET:
logger.error("Google OAuth CLIENT_SECRET is not configured")
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail="Google OAuth CLIENT_SECRET is not configured"
)
if not REDIRECT_URI:
logger.error("Google OAuth REDIRECT_URI is not configured")
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail="Google OAuth REDIRECT_URI is not configured"
)
# Generate auth URL with state - use state as is if it's already JSON, otherwise create new state
try:
@ -80,14 +105,25 @@ async def login(
"connectionId": connectionId
})
# Generate auth URL with state
auth_url, _ = flow.authorization_url(
logger.info(f"Using state parameter: {state_param}")
# Use OAuth2Session directly - it works reliably
oauth = OAuth2Session(
client_id=CLIENT_ID,
redirect_uri=REDIRECT_URI,
scope=SCOPES
)
auth_url, state = oauth.authorization_url(
"https://accounts.google.com/o/oauth2/auth",
access_type="offline",
include_granted_scopes="true",
state=state_param,
prompt="select_account" # Force account selection screen
prompt="select_account"
)
logger.info(f"Generated Google OAuth URL using OAuth2Session: {auth_url}")
return RedirectResponse(auth_url)
except Exception as e:
@ -109,27 +145,54 @@ async def auth_callback(code: str, state: str, request: Request) -> HTMLResponse
logger.info(f"Processing Google auth callback: state_type={state_type}, connection_id={connection_id}, user_id={user_id}")
# Create OAuth flow
flow = Flow.from_client_config(
{
"web": {
"client_id": CLIENT_ID,
"client_secret": CLIENT_SECRET,
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"redirect_uris": [REDIRECT_URI]
}
},
scopes=SCOPES
# Use OAuth2Session directly for token exchange
oauth = OAuth2Session(
client_id=CLIENT_ID,
redirect_uri=REDIRECT_URI
)
# Exchange code for credentials
flow.fetch_token(code=code)
credentials = flow.credentials
# Get token using OAuth2Session
token_data = oauth.fetch_token(
"https://oauth2.googleapis.com/token",
client_secret=CLIENT_SECRET,
code=code,
include_client_id=True
)
# Get user info
user_info_response = flow.oauth2session.get("https://www.googleapis.com/oauth2/v2/userinfo")
user_info = user_info_response.json()
token_response = {
"access_token": token_data.get("access_token"),
"refresh_token": token_data.get("refresh_token", ""),
"token_type": token_data.get("token_type", "bearer"),
"expires_in": token_data.get("expires_in", 0)
}
logger.info("Successfully got token using OAuth2Session")
if not token_response.get("access_token"):
logger.error("Token acquisition failed: No access token received")
return HTMLResponse(
content="<html><body><h1>Authentication Failed</h1><p>Could not acquire token.</p></body></html>",
status_code=400
)
# Get user info using the access token
headers = {
'Authorization': f"Bearer {token_response['access_token']}",
'Content-Type': 'application/json'
}
async with httpx.AsyncClient() as client:
user_info_response = await client.get(
"https://www.googleapis.com/oauth2/v2/userinfo",
headers=headers
)
if user_info_response.status_code != 200:
logger.error(f"Failed to get user info: {user_info_response.text}")
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail="Failed to get user info from Google"
)
user_info = user_info_response.json()
logger.info(f"Got user info from Google: {user_info.get('email')}")
if state_type == "login":
# Handle login flow
@ -152,10 +215,10 @@ async def auth_callback(code: str, state: str, request: Request) -> HTMLResponse
token = Token(
userId=user.id, # Use local user's ID
authority=AuthAuthority.GOOGLE,
tokenAccess=credentials.token,
tokenRefresh=credentials.refresh_token,
tokenType=credentials.token_type,
expiresAt=credentials.expiry.timestamp() if credentials.expiry else None,
tokenAccess=token_response["access_token"],
tokenRefresh=token_response.get("refresh_token", ""),
tokenType=token_response.get("token_type", "bearer"),
expiresAt=datetime.now().timestamp() + token_response.get("expires_in", 0),
createdAt=datetime.now()
)
@ -173,7 +236,7 @@ async def auth_callback(code: str, state: str, request: Request) -> HTMLResponse
if (window.opener) {{
window.opener.postMessage({{
type: 'google_auth_success',
access_token: {json.dumps(credentials.token)},
access_token: {json.dumps(token_response["access_token"])},
token_data: {json.dumps(token.to_dict())}
}}, '*');
}}
@ -261,7 +324,7 @@ async def auth_callback(code: str, state: str, request: Request) -> HTMLResponse
# Update connection with external service details
connection.status = ConnectionStatus.ACTIVE
connection.lastChecked = datetime.now()
connection.expiresAt = credentials.expiry if credentials.expiry else None
connection.expiresAt = datetime.now() + timedelta(seconds=token_response.get("expires_in", 0))
connection.externalId = user_info.get("id")
connection.externalUsername = user_info.get("email")
connection.externalEmail = user_info.get("email")
@ -273,10 +336,10 @@ async def auth_callback(code: str, state: str, request: Request) -> HTMLResponse
token = Token(
userId=user.id, # Use local user's ID
authority=AuthAuthority.GOOGLE,
tokenAccess=credentials.token,
tokenRefresh=credentials.refresh_token,
tokenType=credentials.token_type,
expiresAt=credentials.expiry.timestamp() if credentials.expiry else None,
tokenAccess=token_response["access_token"],
tokenRefresh=token_response.get("refresh_token", ""),
tokenType=token_response.get("token_type", "bearer"),
expiresAt=datetime.now().timestamp() + token_response.get("expires_in", 0),
createdAt=datetime.now()
)
interface.saveToken(token)
@ -296,7 +359,7 @@ async def auth_callback(code: str, state: str, request: Request) -> HTMLResponse
status: 'connected',
type: 'google',
lastChecked: '{datetime.now().isoformat()}',
expiresAt: '{credentials.expiry.isoformat() if credentials.expiry else None}'
expiresAt: '{(datetime.now() + timedelta(seconds=token_response.get("expires_in", 0))).isoformat()}'
}}
}}, '*');
// Wait for message to be sent before closing

2
modules/shared/configuration.py
View file

@ -80,7 +80,7 @@ class Configuration:
def _loadEnv(self):
"""Load environment variables from .env file"""
# Find .env file in the gateway directory
envPath = Path(__file__).parent.parent.parent / 'env_dev.env'
envPath = Path(__file__).parent.parent.parent / '.env'
if not envPath.exists():
logger.warning(f"Environment file not found at {envPath.absolute()}")
return

11
modules/workflow/managerWorkflow.py
View file

@ -8,15 +8,11 @@ from modules.interfaces.interfaceAppObjects import User
from modules.interfaces.interfaceChatModel import (UserInputRequest, ChatMessage, ChatWorkflow, TaskItem, TaskStatus)
from modules.interfaces.interfaceChatObjects import ChatObjects
from modules.chat.managerChat import ChatManager
from modules.chat.managerChat import ChatManager, WorkflowStoppedException
from modules.interfaces.interfaceChatModel import WorkflowResult
logger = logging.getLogger(__name__)
class WorkflowStoppedException(Exception):
"""Exception raised when workflow is stopped by user"""
pass
class WorkflowManager:
"""Manager for workflow processing and coordination"""
@ -25,11 +21,6 @@ class WorkflowManager:
self.chatManager = ChatManager(currentUser, chatInterface)
self.currentUser = currentUser
def _checkWorkflowStopped(self, workflow: ChatWorkflow) -> None:
"""Check if workflow has been stopped"""
if workflow.status == "stopped":
raise WorkflowStoppedException("Workflow was stopped by user")
async def workflowProcess(self, userInput: UserInputRequest, workflow: ChatWorkflow) -> None:
"""Process a workflow with user input using unified workflow phases"""
try: