119 lines
5.5 KiB
Markdown
119 lines
5.5 KiB
Markdown
## Normalization Service Integration (Design + Refactor)
|
|
|
|
### Goal
|
|
Introduce a deterministic Normalization stage between extraction merge and format-specific rendering to guarantee a single canonical, aligned table for downstream outputs (CSV/HTML/Excel/XML/JSON objects) without language-specific code.
|
|
|
|
### Scope
|
|
- Applies to all workflows that use per-chunk extraction and merged JSON.
|
|
- Lives after merged JSON assembly and before Generation/Rendering.
|
|
- Single-path flow (no fallbacks), testable end-to-end.
|
|
- Cache mapping for the current workflow run; prompts remain generic.
|
|
|
|
### Canonical Output Contract
|
|
- One consolidated table section in merged JSON:
|
|
- headers: ["Date","Merchant","CreditCardNumber","TotalAmount","Currency","VATRate"]
|
|
- rows: normalized and aligned values
|
|
- This canonical table is the only input the format engines require.
|
|
|
|
### High-Level Flow
|
|
1) Extraction per chunk → merged JSON (unchanged)
|
|
2) Normalization Service:
|
|
- Analyze structures: discover candidate headers/paths and value samples (no AI)
|
|
- Get mapping from AI (bounded, deterministic mapping JSON only)
|
|
- Apply mapping deterministically in code (flatten + rename + value normalize)
|
|
- Validate canonical table; fail if zero rows
|
|
- Emit debug artifacts for traceability
|
|
3) Generation/Rendering consumes canonical table
|
|
|
|
### Module and Integration Points
|
|
- Add `modules/services/serviceNormalization/mainServiceNormalization.py` with public API:
|
|
- `discoverStructures(mergedJson) -> StructureInventory`
|
|
- `requestHeaderMapping(inventory, canonicalSpec, cache) -> MappingSpec`
|
|
- `applyMapping(mergedJson, mappingSpec) -> CanonicalMergedJson`
|
|
- `validateCanonical(canonicalJson) -> ValidationReport`
|
|
- `persistArtifacts(artifacts, whenDebug=True)`
|
|
|
|
- Call site: `SubDocumentProcessing.processDocumentsPerChunkJson(...)`
|
|
- After `_mergeChunkResultsJson(...)`
|
|
- Before `SubDocumentGeneration`/`GenerationService` rendering
|
|
|
|
### Detailed Steps
|
|
#### 1) Discover Structures (No AI)
|
|
- Input: merged JSON with sections (tables, paragraphs, headings). For JSON/XML sources, also consider objects.
|
|
- Output: `StructureInventory` containing:
|
|
- `tableHeaders`: set of distinct header labels (deduped across sections)
|
|
- `headerSamples`: small value samples per header (e.g., up to 5 distinct examples)
|
|
- `objectPaths`: candidate field paths for JSON/XML object sources with value samples (optional)
|
|
|
|
Notes:
|
|
- Keep RAM processing; streaming not required for this step.
|
|
- No language rules in code.
|
|
|
|
#### 2) AI Mapping (Headers/Paths → Canonical)
|
|
- Prompt inputs (generic):
|
|
- Canonical schema (names + short definitions + constraints)
|
|
- Discovered header inventory (and object paths, if any) + minimal value samples
|
|
- Response (strict JSON):
|
|
- `mappings`: { "Datum": "Date", "Händler": "Merchant", ... }
|
|
- Optional `objectPathMappings` for JSON/XML flattening
|
|
- `normalizationPolicy` with specific rules per field (e.g., decimalSeparator, dateFormatCandidates, currencyPlacement)
|
|
|
|
Constraints:
|
|
- One-to-one or null mappings only (no invented fields)
|
|
- No free text; reject if schema invalid
|
|
- Cache mapping for the current workflow run (in-memory + test-chat/ai/mapping.json when debug)
|
|
|
|
#### 3) Apply Mapping (Deterministic Code)
|
|
- Flatten JSON/XML objects per `objectPathMappings` (when provided) into rows.
|
|
- Transform table sections to the canonical headers per `mappings`.
|
|
- Normalize values per `normalizationPolicy`:
|
|
- Decimals: comma → dot
|
|
- Currency: split symbol/code into `Currency`, keep numeric in `TotalAmount`
|
|
- Date: parse with candidate formats → ISO or target format
|
|
- CreditCardNumber: extract last 4 if masked
|
|
- Merge all rows into a single table with canonical headers.
|
|
|
|
#### 4) Validate
|
|
- Field-level validators (parseable date, numeric amounts, currency in allowed set).
|
|
- If zero rows after normalization: fail with a clear error and persist artifacts.
|
|
|
|
### Artifacts (Debug Mode)
|
|
- `*_header_inventory.json` / `*_path_inventory.json`
|
|
- `*_mapping.json` (AI output)
|
|
- `*_normalization_report.json` (counts, per-field conversions)
|
|
- `*_canonical_merged.json` (final single table)
|
|
|
|
### Error Handling
|
|
- Single-path only (no fallback). If mapping invalid or yields zero rows:
|
|
- Raise an error with pointers to artifacts for diagnosis
|
|
- The workflow surface shows a concise failure reason
|
|
|
|
### Testing Strategy
|
|
- Unit: mapping schema validation; deterministic value normalization; row merge.
|
|
- Integration: feed synthetic merged JSON with heterogeneous headers; assert canonical table shape and non-empty rows.
|
|
- Golden files: store sample inventories, mappings, canonical outputs under test resources.
|
|
|
|
### Non-Goals
|
|
- Do not encode language heuristics in code.
|
|
- Do not normalize via AI over full data rows (AI only returns mapping + policies; code applies them).
|
|
|
|
### Performance Considerations
|
|
- RAM-based processing acceptable for current document sizes.
|
|
- Mapping prompt uses only inventories and samples (small payload), not full data.
|
|
|
|
### Security & Compliance
|
|
- Debug artifacts gated by existing debug flag.
|
|
- Mapping cache limited to current workflow run unless explicitly persisted.
|
|
|
|
### Rollout Plan
|
|
1) Implement NormalizationService with feature flag (default on in dev/test).
|
|
2) Wire into `SubDocumentProcessing.processDocumentsPerChunkJson`.
|
|
3) Add debug artifact writing.
|
|
4) Create unit/integration tests and golden fixtures.
|
|
5) Validate CSV/HTML outputs are populated for heterogeneous sources.
|
|
|
|
### Open Extensions
|
|
- Multi-table merging strategies (per VAT rate, per document) before final consolidation.
|
|
- Domain-specific canonical schemas selectable by task type.
|
|
|
|
|