wiki/z-archive/appdoc/doc_progress_reporting_generation_proposal.md

Progress Reporting for Document Generation - Analysis & Proposal

Current State Analysis

Existing Progress Reporting System

ProgressLogger (gateway/modules/shared/progressLogger.py):

• Centralized progress logging with hierarchical support
• Methods: startOperation(), updateOperation(), finishOperation()
• Supports parent-child relationships via parentOperationId
• Creates ChatLog entries with progress (0.0-1.0), status, and hierarchical structure

ChatLog Model (gateway/modules/datamodels/datamodelChat.py):

• Fields: progress (0.0-1.0), status, parentId, operationId
• Hierarchical display support via parentId (references parent's operationId)

Current Generation Progress Reporting

Chapter Structure Generation (subStructureGeneration.py):

• ✅ Has progress logging: startOperation() and finishOperation()
• ❌ No progress updates during generation
• ❌ No chapter-level granularity

Structure Filling (subStructureFilling.py):

• ✅ Overall operation: startOperation() and finishOperation() for "Chapter Content Generation"
• ✅ Individual sections: startOperation() and finishOperation() for each section
• ❌ Missing: Chapter-level progress operations
• ❌ Missing: Section progress updates (only start/finish, no intermediate progress)
• ❌ Missing: Overall progress calculation showing "Chapter X/Y, Section Z/W"

Content Generation (mainServiceGeneration.py):

• ✅ Has contentProgressCallback that reports section-level progress
• ✅ Maps section progress to overall progress (30-90%)
• ❌ But only at section level, not chapter level
• ❌ No hierarchical display of chapters and sections

Gap Analysis

What Users Currently See:

[Progress] Chapter Content Generation - Filling (0%)
[Progress] Section Generation - Section (0%)
[Progress] Section Generation - Section (0%)
[Progress] Section Generation - Section (0%)
...
[Progress] Chapter Content Generation - Filling (100%)

What Users Should See:

[Progress] Chapter Content Generation - Filling (0%)
  [Progress] Chapter 1/5: Introduction (0%)
    [Progress] Section 1/3: Overview (0%)
    [Progress] Section 1/3: Overview (50%) - Generating content...
    [Progress] Section 1/3: Overview (100%)
    [Progress] Section 2/3: Background (0%)
    ...
  [Progress] Chapter 1/5: Introduction (100%)
  [Progress] Chapter 2/5: Analysis (0%)
    ...

Proposed Integration

1. Chapter-Level Progress Operations

Location: subStructureFilling.py → _fillChapterSections()

Changes:

• Create chapter-level operation IDs
• Start chapter operation before processing sections
• Update chapter progress as sections complete
• Finish chapter operation when all sections done

Implementation:

# Before processing chapters
totalChapters = sum(len(doc.get("chapters", [])) for doc in chapterStructure.get("documents", []))
chapterIndex = 0

for doc in chapterStructure.get("documents", []):
    for chapter in doc.get("chapters", []):
        chapterIndex += 1
        chapterId = chapter.get("id", "unknown")
        chapterTitle = chapter.get("title", "Untitled")
        
        # Start chapter operation
        chapterOperationId = f"{fillOperationId}_chapter_{chapterId}"
        self.services.chat.progressLogStart(
            chapterOperationId,
            "Chapter Generation",
            f"Chapter {chapterIndex}/{totalChapters}",
            f"{chapterTitle}",
            parentOperationId=fillOperationId
        )
        
        # Process sections within chapter
        sections = chapter.get("sections", [])
        totalSections = len(sections)
        
        for sectionIndex, section in enumerate(sections):
            # ... existing section processing ...
            
            # Update chapter progress after each section
            chapterProgress = (sectionIndex + 1) / totalSections if totalSections > 0 else 1.0
            self.services.chat.progressLogUpdate(
                chapterOperationId,
                chapterProgress,
                f"Section {sectionIndex + 1}/{totalSections} completed"
            )
        
        # Finish chapter operation
        self.services.chat.progressLogFinish(chapterOperationId, True)

2. Section Progress Updates

Location: subStructureFilling.py → _fillChapterSections()

Changes:

• Add progress updates during section processing
• Report progress at key stages:
  • 0.2: Building prompt
  • 0.4: Calling AI
  • 0.6: Processing response
  • 0.8: Validating content
  • 1.0: Complete

Implementation:

# Start section operation (existing)
sectionOperationId = f"{fillOperationId}_section_{sectionId}"
self.services.chat.progressLogStart(
    sectionOperationId,
    "Section Generation",
    "Section",
    f"Generating section {sectionId}",
    parentOperationId=chapterOperationId  # Parent is chapter, not fillOperationId
)

try:
    # Update: Building prompt
    self.services.chat.progressLogUpdate(sectionOperationId, 0.2, "Building generation prompt")
    
    generationPrompt = self._buildSectionGenerationPrompt(...)
    
    # Update: Calling AI
    self.services.chat.progressLogUpdate(sectionOperationId, 0.4, "Calling AI for content generation")
    
    aiResponse = await self.aiService.callAi(...)
    
    # Update: Processing response
    self.services.chat.progressLogUpdate(sectionOperationId, 0.6, "Processing AI response")
    
    # Parse and validate
    generatedElements = json.loads(...)
    
    # Update: Validating content
    self.services.chat.progressLogUpdate(sectionOperationId, 0.8, "Validating generated content")
    
    elements.extend(generatedElements)
    
    # Finish section (existing)
    self.services.chat.progressLogFinish(sectionOperationId, True)
    
except Exception as e:
    self.services.chat.progressLogFinish(sectionOperationId, False)
    # ... error handling ...

3. Overall Progress Calculation

Location: subStructureFilling.py → _fillChapterSections()

Changes:

• Calculate overall progress based on chapters and sections
• Update parent operation (fillOperationId) with overall progress

Implementation:

# Calculate overall progress
def calculateOverallProgress(chapterIndex, totalChapters, sectionIndex, totalSections):
    """Calculate overall progress: 0.0 to 1.0"""
    if totalChapters == 0:
        return 1.0
    
    # Progress from completed chapters
    completedChaptersProgress = chapterIndex / totalChapters
    
    # Progress from current chapter
    currentChapterProgress = (sectionIndex / totalSections) / totalChapters if totalSections > 0 else 0
    
    return completedChaptersProgress + currentChapterProgress

# Update overall progress after each section
overallProgress = calculateOverallProgress(
    chapterIndex - 1,  # -1 because we're processing current chapter
    totalChapters,
    sectionIndex,
    totalSections
)
self.services.chat.progressLogUpdate(
    fillOperationId,
    overallProgress,
    f"Chapter {chapterIndex}/{totalChapters}, Section {sectionIndex + 1}/{totalSections}"
)

4. Chapter Structure Generation Progress

Location: subStructureFilling.py → _generateChapterSectionsStructure()

Changes:

• Add progress reporting for chapter structure generation
• Report progress per chapter

Implementation:

# Count total chapters
totalChapters = sum(len(doc.get("chapters", [])) for doc in chapterStructure.get("documents", []))
chapterIndex = 0

for doc in chapterStructure.get("documents", []):
    for chapter in doc.get("chapters", []):
        chapterIndex += 1
        chapterId = chapter.get("id", "unknown")
        chapterTitle = chapter.get("title", "Untitled")
        
        # Update progress
        progress = chapterIndex / totalChapters if totalChapters > 0 else 1.0
        self.services.chat.progressLogUpdate(
            parentOperationId,  # Use parent operation (structure generation)
            progress,
            f"Generating sections for Chapter {chapterIndex}/{totalChapters}: {chapterTitle}"
        )
        
        # ... existing chapter structure generation ...

Implementation Plan

Phase 1: Chapter-Level Progress (High Priority)

1. Add chapter operation tracking in _fillChapterSections()
2. Create chapter-level operations with proper parent hierarchy
3. Update chapter progress as sections complete

Phase 2: Section Progress Updates (High Priority)

1. Add progress updates during section processing
2. Report progress at key stages (prompt building, AI call, response processing)
3. Update section parent to be chapter operation (not fill operation)

Phase 3: Overall Progress Calculation (Medium Priority)

1. Implement overall progress calculation function
2. Update parent operation with overall progress
3. Include chapter/section counts in status messages

Phase 4: Chapter Structure Generation Progress (Low Priority)

1. Add progress updates during chapter structure generation
2. Report progress per chapter

Benefits

1. User Visibility: Users can see exactly which chapter and section is being processed
2. Better UX: Hierarchical progress display shows document structure
3. Debugging: Easier to identify where generation is stuck
4. Performance Monitoring: Can track time per chapter/section
5. Consistency: Uses existing ProgressLogger infrastructure

Example Progress Display

Before:

[0%] Chapter Content Generation - Filling
[0%] Section Generation - Section
[100%] Section Generation - Section
[0%] Section Generation - Section
[100%] Section Generation - Section
...
[100%] Chapter Content Generation - Filling

After:

[0%] Chapter Content Generation - Filling
  [0%] Chapter 1/5: Introduction
    [0%] Section 1/3: Overview
    [20%] Section 1/3: Overview - Building prompt
    [40%] Section 1/3: Overview - Calling AI
    [60%] Section 1/3: Overview - Processing response
    [80%] Section 1/3: Overview - Validating content
    [100%] Section 1/3: Overview
    [33%] Chapter 1/5: Introduction - Section 1/3 completed
    [0%] Section 2/3: Background
    ...
    [100%] Chapter 1/5: Introduction
  [20%] Chapter Content Generation - Chapter 1/5 completed
  [0%] Chapter 2/5: Analysis
    ...

Files to Modify

1. gateway/modules/services/serviceAi/subStructureFilling.py

   • _fillChapterSections(): Add chapter and section progress
   • _generateChapterSectionsStructure(): Add chapter structure progress

2. No changes needed to:

   • progressLogger.py (already supports hierarchy)
   • datamodelChat.py (already supports parentId)
   • mainServiceChat.py (wrapper methods already exist)

Testing Considerations

1. Test with documents containing multiple chapters
2. Test with chapters containing multiple sections
3. Verify hierarchical display in frontend
4. Test error handling (failed sections should not break progress)
5. Test with single chapter/section documents