wiki/z-archive/implementation/implementation_ui_table_pagination.md

UI Table Pagination Implementation Concept

Overview

This document defines the architecture and implementation strategy for server-side pagination in the UI table rendering system. All pagination logic is implemented in the backend - the frontend only receives paginated data and metadata needed for rendering pagination controls.

Core Principles

1. Backend-Driven Pagination: All data slicing, sorting, and filtering happens on the server
2. Stateless Requests: Each UI action (page change, sort, filter) triggers a new API request with complete pagination state
3. Explicit Structure: Clear, well-defined data structures for requests and responses
4. No Fallbacks: Clean implementation without backward compatibility workarounds
5. Extensibility: Designed to support future filtering capabilities

Architecture

Request Flow

Data Loading:

UI Action (initial load, next page, sort, filter)
    ↓
formGeneric.js constructs PaginationRequest
    ↓
API call with pagination parameters
    ↓
Backend applies pagination, sorting, filtering
    ↓
Response with PaginatedResponse (items + pagination metadata)
    ↓
formGeneric.js renders table and pagination controls

Data Modification (Add/Update/Delete):

UI Action (add, update, delete)
    ↓
API call for CUD operation
    ↓
Backend executes operation
    ↓
Success response
    ↓
formGeneric.js triggers data refresh with current pagination state
    ↓
API call with same pagination parameters as before
    ↓
Response with fresh PaginatedResponse
    ↓
formGeneric.js re-renders table with updated data

Response Flow

Backend returns:
{
    "items": [...],              // Page of data items
    "pagination": {              // Pagination metadata (or null if not paginated)
        "currentPage": 1,
        "pageSize": 10,
        "totalItems": 100,
        "totalPages": 10,
        "sort": [...],           // Current sort configuration
        "filters": {...}          // Current filters (for future use)
    }
}

Data Models

PaginationRequest (Query Parameters)

class PaginationRequest(BaseModel):
    """
    Pagination request parameters sent from frontend to backend.
    All fields are optional. If pagination=None, no pagination is applied.
    """
    pagination: Optional[PaginationParams] = None

class PaginationParams(BaseModel):
    """
    Complete pagination state including page, sorting, and filters.
    """
    page: int = Field(ge=1, description="Current page number (1-based)")
    pageSize: int = Field(ge=1, le=1000, description="Number of items per page")
    sort: List[SortField] = Field(default_factory=list, description="List of sort fields in priority order")
    filters: Optional[Dict[str, Any]] = Field(default=None, description="Filter criteria (structure TBD for future implementation)")


### PaginatedResponse

```python
class PaginatedResponse(BaseModel):
    """
    Response containing paginated data and metadata.
    """
    items: List[Any] = Field(..., description="Array of items for current page")
    pagination: Optional[PaginationMetadata] = Field(..., description="Pagination metadata (None if pagination not applied)")

class PaginationMetadata(BaseModel):
    """
    Pagination metadata returned to frontend for rendering controls.
    Contains all information needed to render pagination UI and handle user interactions.
    """
    currentPage: int = Field(..., description="Current page number (1-based)")
    pageSize: int = Field(..., description="Number of items per page")
    totalItems: int = Field(..., description="Total number of items across all pages (after filters)")
    totalPages: int = Field(..., description="Total number of pages (calculated from totalItems / pageSize)")
    sort: List[SortField] = Field(..., description="Current sort configuration applied")
    filters: Optional[Dict[str, Any]] = Field(default=None, description="Current filters applied (for future use)")

Backend Implementation

Route Structure

Before (no pagination):

@router.get("", response_model=List[Prompt])
async def get_prompts(...) -> List[Prompt]:
    prompts = managementInterface.getAllPrompts()
    return prompts

After (with pagination):

@router.get("", response_model=PaginatedResponse[Prompt])
async def get_prompts(
    request: Request,
    pagination: Optional[str] = Query(None, description="JSON-encoded PaginationParams object"),
    currentUser: User = Depends(getCurrentUser)
) -> PaginatedResponse[Prompt]:
    """
    Get prompts with optional pagination, sorting, and filtering.
    
    Query Parameters:
    - pagination: JSON-encoded PaginationParams object, or None for no pagination
    
    Examples:
    - GET /api/prompts (no pagination - returns all items)
    - GET /api/prompts?pagination={"page":1,"pageSize":10,"sort":[]}
    - GET /api/prompts?pagination={"page":2,"pageSize":20,"sort":[{"field":"name","direction":"asc"}]}
    """
    import json
    from modules.shared.paginationModels import PaginationParams
    
    # Parse pagination parameter
    paginationParams = None
    if pagination:
        try:
            paginationDict = json.loads(pagination)
            paginationParams = PaginationParams(**paginationDict) if paginationDict else None
        except (json.JSONDecodeError, ValueError) as e:
            raise HTTPException(
                status_code=400,
                detail=f"Invalid pagination parameter: {str(e)}"
            )
    
    managementInterface = interfaceDbComponentObjects.getInterface(currentUser)
    
    # Call interface method with optional pagination
    result = managementInterface.getAllPrompts(pagination=paginationParams)
    
    # If pagination was requested, result is PaginatedResult
    # If no pagination, result is List[Prompt]
    if paginationParams:
        return PaginatedResponse(
            items=result.items,
            pagination=PaginationMetadata(
                currentPage=paginationParams.page,
                pageSize=paginationParams.pageSize,
                totalItems=result.totalItems,
                totalPages=result.totalPages,
                sort=paginationParams.sort,
                filters=paginationParams.filters
            )
        )
    else:
        return PaginatedResponse(
            items=result,
            pagination=None
        )

Interface Method Signature (Generic Approach)

All interface methods that return lists should support optional pagination:

# In interfaceDbComponentObjects.py, interfaceDbAppObjects.py, interfaceDbChatObjects.py

class ComponentObjects:
    def getAllPrompts(self, pagination: Optional[PaginationParams] = None) -> Union[List[Prompt], PaginatedResult]:
        """
        Returns prompts based on user access level.
        Supports optional pagination, sorting, and filtering.
        
        Args:
            pagination: Optional pagination parameters. If None, returns all items.
        
        Returns:
            If pagination is None: List[Prompt]
            If pagination is provided: PaginatedResult with items and metadata
        """
        # Get all records from database
        allPrompts = self.db.getRecordset(Prompt)
        filteredPrompts = self._uam(Prompt, allPrompts)
        
        # If no pagination requested, return all items
        if pagination is None:
            return [Prompt(**prompt) for prompt in filteredPrompts]
        
        # Apply filtering (if filters provided)
        if pagination.filters:
            filteredPrompts = self._applyFilters(filteredPrompts, pagination.filters)
        
        # Apply sorting (in order of sortFields)
        if pagination.sort:
            filteredPrompts = self._applySorting(filteredPrompts, pagination.sort)
        
        # Count total items after filters
        totalItems = len(filteredPrompts)
        totalPages = math.ceil(totalItems / pagination.pageSize) if totalItems > 0 else 0
        
        # Apply pagination (skip/limit)
        startIdx = (pagination.page - 1) * pagination.pageSize
        endIdx = startIdx + pagination.pageSize
        pagedPrompts = filteredPrompts[startIdx:endIdx]
        
        # Convert to model objects
        items = [Prompt(**prompt) for prompt in pagedPrompts]
        
        return PaginatedResult(
            items=items,
            totalItems=totalItems,
            totalPages=totalPages
        )
    
    def _applyFilters(self, records: List[Dict[str, Any]], filters: Dict[str, Any]) -> List[Dict[str, Any]]:
        """Apply filter criteria to records (implementation for future filtering)."""
        # TODO: Implement filtering logic
        return records
    
    def _applySorting(self, records: List[Dict[str, Any]], sortFields: List[SortField]) -> List[Dict[str, Any]]:
        """Apply multi-level sorting to records."""
        def sortKey(record):
            key_values = []
            for sortField in sortFields:
                value = record.get(sortField.field)
                # Handle None values
                if value is None:
                    key_values.append(("", sortField.direction == "desc"))
                else:
                    key_values.append((value, sortField.direction == "desc"))
            return key_values
        
        # Sort records using all sort fields in priority order
        sortedRecords = sorted(records, key=sortKey)
        
        # For descending fields, reverse those groups
        # (This is a simplified approach - more complex sorting can be implemented)
        for sortField in reversed(sortFields):
            if sortField.direction == "desc":
                # This needs more sophisticated implementation for multi-field desc sorting
                pass
        
        return sortedRecords

All other getAll* methods follow the same pattern:

• getAllFiles(pagination: Optional[PaginationParams] = None)
• getAllUsers(pagination: Optional[PaginationParams] = None)
• getAllMandates(pagination: Optional[PaginationParams] = None)
• getWorkflows(pagination: Optional[PaginationParams] = None)
• getMessages(workflowId: str, pagination: Optional[PaginationParams] = None) (for nested resources)
• etc.

Implementation across all three interfaces:

• interfaceDbComponentObjects.py: getAllPrompts, getAllFiles
• interfaceDbAppObjects.py: getAllMandates, getUsersByMandate, getAllUsers (if exists)
• interfaceDbChatObjects.py: getWorkflows, getMessages, getLogs, etc.

PaginatedResult (Internal)

class PaginatedResult(BaseModel):
    """
    Internal result structure from interface layer.
    Used when pagination is requested.
    """
    items: List[Any]
    totalItems: int
    totalPages: int  # Calculated as: math.ceil(totalItems / pageSize)

Note: Interface methods return Union[List[Model], PaginatedResult]:

• When pagination=None: Return List[Model] (backward compatible)
• When pagination is provided: Return PaginatedResult

Frontend Implementation

Pagination State Object

// In formGeneric.js module state
const paginationState = {
    currentPage: 1,
    pageSize: 10,  // Default configured per view
    totalItems: null,
    totalPages: null,
    sort: [],  // Array of {field: string, direction: "asc"|"desc"}
    filters: null  // Object (for future use)
};

API Call Structure

// In apiCalls.js
getPrompts: async function(pagination = null) {
    let url = '/api/prompts';
    if (pagination) {
        // Encode pagination object as JSON query parameter
        const paginationJson = JSON.stringify(pagination);
        url += `?pagination=${encodeURIComponent(paginationJson)}`;
    }
    return await privateApi.get(url);
}

formGeneric.js Integration

Initial Load:

async function loadData(entityType) {
    const module = moduleRegistry.get(entityType);
    const { config, state } = module;
    
    // Get default page size from config
    const defaultPageSize = config.pagination?.defaultPageSize || 10;
    
    // Construct pagination request
    const paginationRequest = {
        page: 1,
        pageSize: defaultPageSize,
        sort: [],
        filters: null
    };
    
    // Call API
    const response = await config.apiEndpoint.get(paginationRequest);
    
    // Update state
    state.items = response.items;
    if (response.pagination) {
        state.currentPage = response.pagination.currentPage;
        state.pageSize = response.pagination.pageSize;
        state.totalItems = response.pagination.totalItems;
        state.totalPages = response.pagination.totalPages;
        state.sort = response.pagination.sort;
        state.filters = response.pagination.filters;
    } else {
        // No pagination - all items loaded
        state.currentPage = 1;
        state.pageSize = response.items.length;
        state.totalItems = response.items.length;
        state.totalPages = 1;
        state.sort = [];
        state.filters = null;
    }
    
    // Render
    await renderItems(entityType);
}

Page Change:

async function goToPage(entityType, pageNumber) {
    const module = moduleRegistry.get(entityType);
    const { config, state } = module;
    
    // Construct pagination request with new page
    const paginationRequest = {
        page: pageNumber,
        pageSize: state.pageSize,
        sort: state.sort,
        filters: state.filters
    };
    
    // Call API
    const response = await config.apiEndpoint.get(paginationRequest);
    
    // Update state
    state.items = response.items;
    state.currentPage = response.pagination.currentPage;
    // ... update other pagination state
    
    // Render
    await renderItems(entityType);
}

After Create/Update/Delete (Refresh Data):

// After successful create/update/delete operation
async function refreshAfterModification(entityType) {
    const module = moduleRegistry.get(entityType);
    const { config, state } = module;
    
    // Construct pagination request with current state
    const paginationRequest = {
        page: state.currentPage,
        pageSize: state.pageSize,
        sort: state.sort,
        filters: state.filters
    };
    
    // Call API to get fresh data
    const response = await config.apiEndpoint.get(paginationRequest);
    
    // Update state with fresh data
    state.items = response.items;
    if (response.pagination) {
        state.currentPage = response.pagination.currentPage;
        state.totalItems = response.pagination.totalItems;
        state.totalPages = response.pagination.totalPages;
        state.sort = response.pagination.sort;
        state.filters = response.pagination.filters;
    }
    
    // Render updated table
    await renderItems(entityType);
}

// Example: After creating an item
async function createItem(entityType, itemData) {
    try {
        // Create item via API
        const newItem = await config.apiEndpoint.create(itemData);
        
        // Refresh data after successful creation
        await refreshAfterModification(entityType);
        
        // Call onItemCreated event
        if (config.events?.onItemCreated) {
            config.events.onItemCreated(newItem);
        }
    } catch (error) {
        ui.log.error(`Error creating ${entityType}:`, error);
        showError(`Error creating ${entityType}`, error.message);
    }
}

// Example: After updating an item
async function updateItem(entityType, itemId, updateData) {
    try {
        // Update item via API
        const updatedItem = await config.apiEndpoint.update(itemId, updateData);
        
        // Refresh data after successful update
        await refreshAfterModification(entityType);
        
        // Call onItemUpdated event
        if (config.events?.onItemUpdated) {
            config.events.onItemUpdated(updatedItem);
        }
    } catch (error) {
        ui.log.error(`Error updating ${entityType}:`, error);
        showError(`Error updating ${entityType}`, error.message);
    }
}

// Example: After deleting an item
async function deleteItem(entityType, itemId) {
    try {
        // Delete item via API
        await config.apiEndpoint.delete(itemId);
        
        // Refresh data after successful deletion
        await refreshAfterModification(entityType);
        
        // Call onItemDeleted event
        if (config.events?.onItemDeleted) {
            config.events.onItemDeleted(itemId);
        }
    } catch (error) {
        ui.log.error(`Error deleting ${entityType}:`, error);
        showError(`Error deleting ${entityType}`, error.message);
    }
}

Sort Change:

async function applySort(entityType, fieldName, direction) {
    const module = moduleRegistry.get(entityType);
    const { config, state } = module;
    
    // Update sort array (replace existing sort for this field, or add new)
    const newSort = [...state.sort];
    const existingIndex = newSort.findIndex(s => s.field === fieldName);
    
    if (existingIndex >= 0) {
        // Update existing sort
        newSort[existingIndex].direction = direction;
    } else {
        // Add new sort field
        newSort.push({ field: fieldName, direction: direction });
    }
    
    // Reset to page 1 when sorting changes
    const paginationRequest = {
        page: 1,
        pageSize: state.pageSize,
        sort: newSort,
        filters: state.filters
    };
    
    // Call API
    const response = await config.apiEndpoint.get(paginationRequest);
    
    // Update state
    state.items = response.items;
    state.currentPage = 1;
    state.sort = newSort;
    // ... update other pagination state
    
    // Render
    await renderItems(entityType);
}

Page Size Change:

async function changePageSize(entityType, newPageSize) {
    const module = moduleRegistry.get(entityType);
    const { config, state } = module;
    
    // Reset to page 1 when page size changes
    const paginationRequest = {
        page: 1,
        pageSize: newPageSize,
        sort: state.sort,
        filters: state.filters
    };
    
    // Call API
    const response = await config.apiEndpoint.get(paginationRequest);
    
    // Update state
    state.items = response.items;
    state.currentPage = 1;
    state.pageSize = newPageSize;
    // ... update other pagination state
    
    // Render
    await renderItems(entityType);
}

Module Configuration

Per-View Default Page Size:

// In formPrompts.js (or other module files)
formGeneric.initModule(globalStateObj, {
    entityType: 'prompt',
    apiEndpoint: {
        get: api.getPrompts,  // Must accept pagination parameter
        create: api.createPrompt,
        update: api.updatePrompt,
        delete: api.deletePrompt
    },
    pagination: {
        defaultPageSize: 20  // Configure default for this view
    },
    // ... other config
});

UI Rendering Requirements

Pagination Controls

The frontend must render pagination controls using only the information from PaginationMetadata:

1. Page Numbers: Render buttons for pages 1 through totalPages
2. Current Page Indicator: Highlight currentPage
3. Previous/Next Buttons:
   • Previous disabled if currentPage === 1
   • Next disabled if currentPage === totalPages
4. Page Size Selector: Dropdown to change pageSize (triggers new API call)
5. Item Count Display: Show "Showing X to Y of Z items"

Sort Indicators

• Display sort indicators in table headers based on sort array
• Show direction (↑ for ASC, ↓ for DESC)
• Show priority order for multi-level sorting (1, 2, 3...)
• Clicking a column header toggles/changes its sort

Empty State

• If totalItems === 0: Show "No items found" message
• If current page has no items but totalItems > 0: Redirect to page 1 (should not happen with proper backend implementation)

Validation Rules

Backend Validation

1. Page Number: Must be >= 1 and <= totalPages
2. Page Size: Must be >= 1 and <= 1000 (configurable max)
3. Sort Fields: Must be valid field names for the entity type
4. Sort Direction: Must be "asc" or "desc"

Frontend Validation

1. Page Navigation: Disable buttons when at boundaries (page 1 or last page)
2. Page Size: Only allow predefined values (e.g., 10, 20, 50, 100)
3. Sort Fields: Only allow sorting on fields that exist in field definitions

Error Handling

Backend Errors

• Invalid Page: Return 400 Bad Request with error message
• Invalid Sort Field: Return 400 Bad Request with list of valid fields
• Invalid Page Size: Return 400 Bad Request with allowed range

Frontend Errors

• API Error: Display error message, keep current pagination state
• Network Error: Show retry option, maintain pagination state
• Invalid Response: Fallback to error state, log error for debugging

Migration Strategy

Phase 1: Backend Data Models

1. Create PaginationParams, SortField models in modules/shared/paginationModels.py
2. Create PaginatedResponse, PaginationMetadata models
3. Create PaginatedResult internal model for interface layer

Phase 2: Backend Interface Updates

1. interfaceDbComponentObjects.py:

   • Update getAllPrompts(pagination: Optional[PaginationParams] = None)
   • Update getAllFiles(pagination: Optional[PaginationParams] = None)
   • Add _applyFilters() and _applySorting() helper methods

2. interfaceDbAppObjects.py:

   • Update getAllMandates(pagination: Optional[PaginationParams] = None)
   • Update getUsersByMandate(mandateId: str, pagination: Optional[PaginationParams] = None)
   • Add _applyFilters() and _applySorting() helper methods

3. interfaceDbChatObjects.py:

   • Update getWorkflows(pagination: Optional[PaginationParams] = None)
   • Update getMessages(workflowId: str, pagination: Optional[PaginationParams] = None)
   • Update getLogs(workflowId: str, pagination: Optional[PaginationParams] = None)
   • Add _applyFilters() and _applySorting() helper methods

Phase 3: Backend Route Updates

1. Update all list endpoints to accept pagination query parameter
2. Parse pagination JSON parameter
3. Call interface methods with pagination parameter
4. Return PaginatedResponse with items and metadata

Phase 4: Frontend Implementation

1. Update apiCalls.js functions to accept pagination parameter (as JSON string in query)
2. Update formGeneric.js to construct and send pagination requests
3. Update formGeneric.js to handle paginated responses
4. Update pagination controls to trigger API calls
5. Update sort controls to trigger API calls
6. Add refreshAfterModification() function for CUD operations
7. Update create/update/delete handlers to refresh data after success
8. Add default page size configuration to each module

Phase 5: Testing

1. Test pagination with various page sizes
2. Test sorting (single and multi-level)
3. Test filtering (when implemented)
4. Test CUD operations with data refresh
5. Test edge cases (empty results, single page, invalid page numbers, etc.)
6. Test error handling

Future Enhancements

Filtering (Prepared Structure)

The filters field is prepared for future filtering implementation:

// Future filter structure (to be defined)
const filters = {
    fieldName: {
        operator: "equals" | "contains" | "greaterThan" | "lessThan" | ...,
        value: "..."
    },
    // Multiple filters combined with AND/OR logic
};

Server-Side Filtering

When filtering is implemented:

1. Filters are included in PaginationRequest
2. Backend applies filters before counting totalItems
3. PaginationMetadata.filters contains applied filters
4. Frontend displays active filters and allows removal

Summary Checklist

Backend Must Provide:

• ✅ Paginated data (page of items only)
• ✅ Current page number
• ✅ Page size
• ✅ Total items count (after filters)
• ✅ Total pages count
• ✅ Current sort configuration
• ✅ Current filter configuration (for future)

Frontend Must Handle:

• ✅ Construct pagination requests
• ✅ Send pagination requests to API
• ✅ Receive and parse paginated responses
• ✅ Render pagination controls
• ✅ Handle page navigation
• ✅ Handle sort changes
• ✅ Handle page size changes
• ✅ Display empty states
• ✅ Handle errors

Missing Information Check:

For Rendering:

• ✅ Items to display (from response.items)
• ✅ Current page (from response.pagination.currentPage)
• ✅ Total pages (from response.pagination.totalPages)
• ✅ Total items (from response.pagination.totalItems)
• ✅ Page size (from response.pagination.pageSize)
• ✅ Sort state (from response.pagination.sort)
• ✅ Filter state (from response.pagination.filters)

Nothing missing - all required information is provided in the response structure.

Implementation Notes

1. No Client-Side Pagination: All data slicing happens on the server
2. No Fallbacks: Clean implementation - if pagination is requested, it must work
3. Explicit State: Every UI action results in an explicit API call with full pagination state
4. Default Configuration: Each view module configures its default page size
5. Extensibility: Structure supports future filtering without breaking changes