Re-indexing the same file always triggered a full embedding run —
ingestion.skipped.duplicate never fired. Two independent causes:
1. _computeIngestionHash included contentObjectId in its payload, but
extractors generate fresh uuid4() per run, making the hash a
per-run nonce. Now hashed over (contentType, data) in extractor
order — stable across re-extractions, sensitive to content,
ordering, and type changes.
2. _autoIndexFile upserted the fresh pre-scan FileContentIndex before
requestIngestion's duplicate check, wiping structure._ingestion
and status=indexed from the prior run. The pre-upsert now merges
the existing _ingestion metadata and preserves the indexed status.
Verified end-to-end: second PATCH /scope on an already-indexed file
logs and returns in ~2s
with zero embedding API calls.
Adds test_ingestion_hash_stability.py (5 cases).
The default MergeStrategy concatenates every extracted text part into a
single ContentPart, collapsing a 500-page PDF into one chunk with a
blurred average embedding — RAG retrieval was effectively broken.
- ExtractionOptions.mergeStrategy is now Optional[MergeStrategy]; passing
None preserves per-part granularity. Default factory kept for
backward compatibility.
- routeDataFiles._autoIndexFile, _workspaceTools.readFile, and
_documentTools.describeImage explicitly pass mergeStrategy=None.
- Agent tools no longer carry redundant extraction + requestIngestion
fallback paths: the unified ingestion lane owns all corpus writes,
and readFile/describeImage are pure consumers of the knowledge store.
- Unit test asserts runExtraction(mergeStrategy=None) keeps every part.
API and persisted records use PowerOnModel system fields:
- sysCreatedAt, sysCreatedBy, sysModifiedAt, sysModifiedBy
Removed legacy JSON/DB field names:
- _createdAt, _createdBy, _modifiedAt, _modifiedBy
Frontend (frontend_nyla) and gateway call sites were updated accordingly.
Database:
- Bootstrap runs idempotent backfill (_migrateSystemFieldColumns) from old
underscore columns and selected business duplicates into sys* where sys* IS NULL.
- Re-run app bootstrap against each PostgreSQL database after deploy.
- Optional: DROP INDEX IF EXISTS "idx_invitation_createdby" if an old index remains;
new index: idx_invitation_syscreatedby on Invitation(sysCreatedBy).
Tests:
- RBAC integration tests aligned with current GROUP mandate filter and UserMandate-based
UserConnection GROUP clause; buildRbacWhereClause(..., mandateId=...) must be passed
explicitly (same as production request context).
