ValueOn AG e331c7fbce performance tests with json repair and cut handling

2026-01-06 17:18:51 +01:00

13 KiB

Raw Blame History

AI Call Iteration Flow - JSON Merging System

This document describes the iteration flow for handling large JSON responses from AI that may be truncated and need to be merged across multiple iterations.

Overview

When an AI response is too large, it may be truncated (cut) at an arbitrary point. The iteration system:

Detects incomplete JSON
Requests continuation from the AI
Merges the continuation with the existing JSON
Repeats until complete or max failures reached

Key Variables

Variable	Type	Purpose
`jsonBase`	`str \| None`	The merged JSON string (CUT version for overlap matching)
`candidateJson`	`str`	Temporary holder for merged result until validated
`lastValidCompletePart`	`str \| None`	Fallback - last successfully parsed CLOSED JSON
`lastOverlapContext`	`str`	Context for retry/continuation prompts
`lastHierarchyContextForPrompt`	`str`	Context for retry/continuation prompts
`mergeFailCount`	`int`	Global counter (max 3 failures)

Key Distinction: hierarchyContext vs completePart

Field	Description	Use Case
`hierarchyContext`	CUT JSON - truncated at cut point	Used as `jsonBase` for merging with next AI fragment
`completePart`	CLOSED JSON - all structures properly closed	Used for validation, parsing, and fallback

Why this matters:

The next AI fragment starts with an overlap that matches the CUT point
If we used completePart (closed), the overlap detection would FAIL
We must use hierarchyContext (cut) so overlap matching works correctly

Flow Steps

Step 1: BUILD PROMPT

Location: subAiCallLooping.py lines 163-212
Function: buildContinuationContext() from modules/shared/jsonUtils.py

First iteration: Use original prompt
Continuation: buildContinuationContext(allSections, lastRawResponse, ...)
- Internally calls getContexts(lastRawResponse) to get overlap/hierarchy
- Builds continuation prompt with overlapContext + hierarchyContextForPrompt

Step 2: CALL AI

Location: subAiCallLooping.py lines 214-299
Function: self.aiService.callAi(request)

Returns response.content as result
Store raw response: lastRawResponse = result

Step 4: MERGE

Location: subAiCallLooping.py lines 338-396
Function: JsonResponseHandler.mergeJsonStringsWithOverlap() from modules/services/serviceAi/subJsonResponseHandling.py

IF first iteration (jsonBase is None):
    → candidateJson = result
ELSE:
    → mergedJsonString, hasOverlap = mergeJsonStringsWithOverlap(jsonBase, result)
    
    IF hasOverlap = False (MERGE FAILED):
        → mergeFailCount++
        → If mergeFailCount >= 3: return lastValidCompletePart (fallback)
        → Else: continue (retry with unchanged jsonBase)
    ELSE:
        → candidateJson = mergedJsonString (don't update jsonBase yet!)

TRY DIRECT PARSE of candidateJson:
    IF parse succeeds:
        → jsonBase = candidateJson (commit)
        → FINISHED! Return normalized result
    ELSE:
        → Proceed to Step 5

Step 5: GET CONTEXTS

Location: subAiCallLooping.py lines 420-427
Function: getContexts() from modules/shared/jsonContinuation.py

contexts = getContexts(candidateJson)

Returns JsonContinuationContexts:

overlapContext: "" if JSON is complete (no cut point)
hierarchyContext: CUT JSON (for merging with next fragment)
hierarchyContextForPrompt: CUT JSON with budget limits (for prompts)
completePart: CLOSED JSON (repaired if needed)
jsonParsingSuccess: True if completePart is valid JSON

Enhancement: If original JSON is already complete → overlapContext = ""
This signals "JSON is complete, no more continuation needed"

Step 6: DECIDE

Location: subAiCallLooping.py lines 429-528

Case A: `jsonParsingSuccess=true` AND `overlapContext=""`

→ FINISHED

JSON is complete (no cut point)
jsonBase = contexts.completePart (use CLOSED version for final result)
Return completePart as result

Case B: `jsonParsingSuccess=true` AND `overlapContext!=""`

→ CONTINUE to next iteration

JSON parseable but has cut point
jsonBase = contexts.hierarchyContext ← CUT version for next merge!
lastValidCompletePart = contexts.completePart ← CLOSED version for fallback
Store contexts for next prompt
mergeFailCount = 0 (reset on success)
lastRawResponse = jsonBase
Continue to next iteration

Case C: `jsonParsingSuccess=false`

→ RETRY with same prompt

Do NOT update jsonBase (keep previous valid state)
mergeFailCount++
If mergeFailCount >= 3: return lastValidCompletePart (fallback)
Else: continue (retry with unchanged jsonBase/lastRawResponse)

Flow Diagram

     ┌───────────────────────────────────────────────────────────────┐
     │                     ITERATION START                           │
     └───────────────────────────┬───────────────────────────────────┘
                                 │
     ┌───────────────────────────▼───────────────────────────────────┐
     │  STEP 1: BUILD PROMPT                                         │
     │  - First: original prompt                                     │
     │  - Next: buildContinuationContext(lastRawResponse)            │
     └───────────────────────────┬───────────────────────────────────┘
                                 │
     ┌───────────────────────────▼───────────────────────────────────┐
     │  STEP 2: CALL AI → result                                     │
     └───────────────────────────┬───────────────────────────────────┘
                                 │
     ┌───────────────────────────▼───────────────────────────────────┐
     │  STEP 4: MERGE jsonBase + result → candidateJson              │
     └───────────────────────────┬───────────────────────────────────┘
                                 │
                    ┌────────────▼────────────┐
                    │      Merge OK?          │
                    └────────────┬────────────┘
                                 │
           ┌─────────────────────┼─────────────────────┐
           │ NO                  │ YES                 │
           ▼                     ▼                     │
    ┌──────────────┐    ┌──────────────────┐          │
    │ fails++      │    │ TRY DIRECT PARSE │          │
    │ if >=3:      │    │ of candidateJson │          │
    │   RETURN     │    └────────┬─────────┘          │
    │   fallback   │             │                    │
    │ else: RETRY  │    ┌────────▼─────────┐          │
    │   (continue) │    │   Parse OK?      │          │
    └──────────────┘    └────────┬─────────┘          │
                                 │                    │
           ┌─────────────────────┼─────────────────────┐
           │ YES                 │ NO                  │
           ▼                     ▼                     │
    ┌──────────────┐    ┌──────────────────────────────┐
    │  FINISHED ✓  │    │  STEP 5: getContexts()       │
    │  Return      │    │  → jsonParsingSuccess        │
    │  normalized  │    │  → overlapContext            │
    │  result      │    └────────────┬─────────────────┘
    └──────────────┘                 │
                        ┌────────────▼────────────────────┐
                        │         STEP 6: DECIDE          │
                        └────────────┬────────────────────┘
                                     │
        ┌────────────────────────────┼────────────────────────────┐
        │                            │                            │
        ▼                            ▼                            ▼
┌───────────────────┐  ┌───────────────────────┐  ┌───────────────────┐
│ success=true      │  │ success=true          │  │ success=false     │
│ overlap=""        │  │ overlap!=""           │  │                   │
│ ─────────────     │  │ ─────────────────     │  │ ─────────────     │
│ FINISHED ✓        │  │ CONTINUE              │  │ RETRY             │
│                   │  │                       │  │                   │
│ jsonBase =        │  │ jsonBase =            │  │ jsonBase unchanged│
│   completePart    │  │   hierarchyContext    │  │ fails++           │
│   (CLOSED)        │  │   (CUT for merge!)    │  │                   │
│                   │  │                       │  │ if >=3: fallback  │
│ Return result     │  │ fallback =            │  │ else: retry       │
│                   │  │   completePart        │  │                   │
│                   │  │   (CLOSED)            │  │                   │
│                   │  │                       │  │                   │
│                   │  │ Next iteration →      │  │                   │
└───────────────────┘  └───────────────────────┘  └───────────────────┘

Files Involved

File	Purpose
`modules/services/serviceAi/subAiCallLooping.py`	Main iteration loop
`modules/shared/jsonContinuation.py`	`getContexts()` - context extraction & repair
`modules/shared/jsonUtils.py`	`buildContinuationContext()` - prompt building
`modules/services/serviceAi/subJsonResponseHandling.py`	`mergeJsonStringsWithOverlap()`
`modules/services/serviceAi/subJsonMerger.py`	`ModularJsonMerger` - actual merge logic
`modules/datamodels/datamodelAi.py`	`JsonContinuationContexts` model

Error Handling

Merge Failures

Max 3 consecutive failures allowed
On failure: retry with unchanged jsonBase (previous valid state)
After 3 failures: return lastValidCompletePart as fallback

Parse Failures

If getContexts() cannot produce valid JSON: increment fail counter
Retry with same prompt (don't update jsonBase)
After 3 failures: return lastValidCompletePart as fallback

Fallback Strategy

lastValidCompletePart stores the last successfully parsed CLOSED JSON
Always available as fallback when things go wrong
Ensures we return valid JSON even after multiple failures

13 KiB Raw Blame History