gateway/docs/code-documentation/date-time-handling.md

# Date and Time Handling

This document describes how dates and times are handled throughout the gateway project, including storage formats, transformations, and retrieval mechanisms.

## Overview

The gateway uses **Unix timestamps** (floats representing seconds since epoch) as the standard format for all date and time values. This ensures consistency across different database backends and eliminates timezone-related issues by always working in UTC.

## Core Components

### Time Utilities Module

The primary module for date/time handling is `modules/shared/timeUtils.py`, which provides:

#### `getUtcTimestamp() -> float`

Returns the current UTC timestamp as a float (seconds since Unix epoch with millisecond precision).

**Implementation:**
- Uses Python's `time.time()` which returns seconds since epoch as a float
- Provides millisecond precision
- Always returns UTC time

#### `getUtcNow() -> datetime`

Returns the current UTC time as a `datetime` object with timezone information.

#### `parseTimestamp(value: Any, default: Optional[float] = None) -> Optional[float]`

**Critical function for database retrieval** - Parses timestamp values from various formats and converts them to float.

This function handles the transformation/decoding of timestamps when reading from the database, especially important for PostgreSQL connectors that may return numeric fields as strings in some environments (e.g., Azure PostgreSQL).

**Supported input formats:**
- `float`: Returns as-is
- `int`: Converts to float
- `str`: Attempts to parse as numeric string (e.g., "1234567890.123")
- `None`: Returns default value (or None if no default provided)

**Usage:** Call `parseTimestamp()` with the database value and an optional default. For example: `parseTimestamp(db_value, default=getUtcTimestamp())`.

**Why this is needed:**
- PostgreSQL connectors (especially Azure PostgreSQL) may return numeric fields as strings
- Ensures consistent float type regardless of database return format
- Provides safe fallback with default values

#### `createExpirationTimestamp(expiresInSeconds: int) -> float`

Creates an expiration timestamp by adding seconds to the current UTC timestamp. Call with the number of seconds until expiration, for example: `createExpirationTimestamp(3600)` for 1 hour from now.

## Database Storage

### Storage Format

All timestamps are stored in the database as **floats** (Unix timestamps in seconds):

- **PostgreSQL**: Stored as `NUMERIC` or `DOUBLE PRECISION` type
- **JSON Database**: Stored as JSON number (float) in record files

### Saving Timestamps

#### PostgreSQL Connector (`connectorDbPostgre.py`)

When saving records, timestamp fields (`_createdAt`, `_modifiedAt`) are handled specially. The connector checks if the value is a string and attempts to convert it to float before storing.

**Process:**
1. Timestamp values are converted to float if they're strings
2. Stored directly as numeric values in PostgreSQL
3. No encryption or encoding - stored as plain numeric values

#### JSON Connector (`connectorDbJson.py`)

Timestamps are stored directly as JSON numbers. The connector calls `getUtcTimestamp()` to get the current time and sets `_createdAt` on creation and `_modifiedAt` on every save operation.

**Process:**
1. Timestamps are generated as floats using `getUtcTimestamp()`
2. Stored as JSON numbers in record files
3. No transformation needed - JSON natively supports float numbers

### Retrieving Timestamps

#### PostgreSQL Connector

When reading from PostgreSQL, timestamps may be returned in different formats:

- **Expected**: Float values
- **Actual (Azure PostgreSQL)**: Sometimes returned as strings
- **Solution**: Use `parseTimestamp()` to normalize

**Example from code:** In `interfaceDbChatObjects.py`, timestamps are parsed using `parseTimestamp(msg.get("publishedAt"), default=getUtcTimestamp())`.

#### JSON Connector

Timestamps are read directly from JSON and typically remain as floats, but `parseTimestamp()` is still used for safety to ensure consistent float type regardless of JSON parsing quirks. Call `parseTimestamp(record.get("timestamp"), default=getUtcTimestamp())` when reading timestamp fields.

## Data Model Definitions

Timestamps are defined in Pydantic models using the `float` type with `getUtcTimestamp` as the default factory. For example, in `ChatLog` model, the `timestamp` field uses `Field(default_factory=getUtcTimestamp, ...)`.

**Common timestamp fields:**
- `_createdAt`: When the record was created
- `_modifiedAt`: When the record was last modified
- `publishedAt`: When a message was published
- `timestamp`: Generic timestamp field
- `expiresAt`: When something expires
- `connectedAt`: When a connection was established
- `lastChecked`: When something was last verified
- `creationDate`: When something was created

## Transformation Flow

### Saving to Database

```
Application Code
    ↓
getUtcTimestamp() → float (e.g., 1234567890.123)
    ↓
Database Connector
    ↓
PostgreSQL: Store as NUMERIC/DOUBLE PRECISION
JSON: Store as JSON number
```

### Reading from Database

```
Database
    ↓
PostgreSQL: May return as float, int, or string
JSON: Returns as float (or int if no decimals)
    ↓
parseTimestamp() → Normalizes to float
    ↓
Application Code (always receives float)
```

## Key Points

1. **No Encryption**: Timestamps are **not encrypted**. They are stored as plain numeric values (floats) in the database. The "encrypted format" you may have seen refers to the numeric Unix timestamp format, which is not human-readable but is not encrypted.

2. **Type Transformation**: The main transformation happens when **reading** from the database:
   - Database may return timestamps as strings (especially Azure PostgreSQL)
   - `parseTimestamp()` converts strings/int/float → float
   - This ensures consistent float type in application code

3. **UTC Only**: All timestamps are in UTC. No timezone conversions happen at the database level.

4. **Millisecond Precision**: Timestamps use float type, providing millisecond precision (e.g., `1234567890.123`).

5. **Default Values**: When timestamps are missing or invalid, `parseTimestamp()` can provide safe defaults using the `default` parameter.

## Usage Examples

### Creating a Record with Timestamps

When creating records, call `getUtcTimestamp()` to set `_createdAt` and `_modifiedAt` fields. These are typically auto-set by the database connectors.

### Reading and Using Timestamps

After retrieving a record from the database, use `parseTimestamp(record.get("publishedAt"), default=getUtcTimestamp())` to safely parse the timestamp value before using it for filtering or comparison.

### Filtering by Timestamp

When filtering messages or records by timestamp, iterate through the results and use `parseTimestamp(msg.get("publishedAt"), default=getUtcTimestamp())` to parse each timestamp, then compare against your threshold value.

### Sorting by Timestamp

When sorting logs or records by timestamp, use `parseTimestamp()` in the sort key function, for example: `logs.sort(key=lambda x: parseTimestamp(x.get("timestamp"), default=0))`.

## Common Patterns

### Automatic Timestamp Management

Database connectors automatically manage `_createdAt` and `_modifiedAt`:

- `_createdAt`: Set only on record creation (if not already present)
- `_modifiedAt`: Updated on every save operation

### Timestamp Filtering

When filtering records by timestamp, always use `parseTimestamp()` to handle all database return types. Call `parseTimestamp(record.get("timestamp"), default=0)` before comparison. Do not directly compare timestamp values as they may be strings from the database.

### Expiration Timestamps

For expiration logic, use `createExpirationTimestamp(3600)` to create an expiration timestamp, then compare it with the current time using `getUtcTimestamp()`. Parse any stored expiration timestamps using `parseTimestamp()` before comparison.

## Database-Specific Notes

### PostgreSQL (Azure)

- May return numeric fields as strings
- Always use `parseTimestamp()` when reading timestamp fields
- Stored as `NUMERIC` or `DOUBLE PRECISION` type

### JSON Database

- Timestamps stored as JSON numbers
- Usually returned as floats, but `parseTimestamp()` provides safety
- No special handling needed beyond normalization

## Troubleshooting

### Issue: Timestamp is a string instead of float

**Solution**: Use `parseTimestamp()` to convert: `parseTimestamp(string_timestamp, default=getUtcTimestamp())`.

### Issue: Timestamp is None

**Solution**: Provide a default value: `parseTimestamp(record.get("timestamp"), default=getUtcTimestamp())`.

### Issue: Timestamp comparison fails

**Cause**: Comparing string to float
**Solution**: Always parse first using `parseTimestamp(value, default=0)` before any comparison operations.

## Summary

The date/time handling system:

1. **Stores** timestamps as Unix timestamps (floats) in UTC
2. **Transforms** database return values (string/int/float) → float using `parseTimestamp()`
3. **Provides** utilities for timestamp generation, parsing, and expiration
4. **Ensures** consistency across different database backends
5. **Handles** edge cases (None values, string returns, invalid formats)

The transformation you mentioned is the **type normalization** that happens when reading from the database, not encryption. Timestamps are stored as plain numeric values and converted to consistent float types for application use.

---

## Frontend Requirements and Implementation

### API Response Format

The backend API returns timestamps as **Unix timestamps (floats)** in all JSON responses. Timestamps are always in UTC and represented as numbers (not strings).

**Example API Response:** The API returns timestamps as numeric values in JSON, for example: `_createdAt: 1704067200.123`, `_modifiedAt: 1704067300.456`, `publishedAt: 1704067250.789`, `expiresAt: 1704153600.0`.

**Important Notes:**
- Timestamps are **always numbers** (float) in JSON responses
- All timestamps are in **UTC** (no timezone information in the value)
- Timestamps have **millisecond precision** (decimal places)
- Timestamp fields may be `null` or `undefined` if not set

### Frontend Requirements

#### 1. Timestamp Decoding Module

The frontend **must** implement a core timestamp utility module that handles:

- **Decoding/parsing** timestamp values from API responses
- **Type normalization** (handles number, string, null, undefined)
- **Conversion** to JavaScript Date objects
- **Formatting** for display (relative time, absolute date/time)
- **Timezone handling** (convert UTC to user's local timezone)

#### 2. Required Functionality

The frontend timestamp module must provide:

1. **Parse/Decode Function**: Convert API timestamp (number/string/null) → JavaScript Date
2. **Format Functions**: Convert Date → human-readable strings
3. **Relative Time**: "2 hours ago", "in 5 minutes", etc.
4. **Absolute Time**: "2024-01-01 12:30:45 UTC" or localized format
5. **Comparison Utilities**: Compare timestamps, check expiration, etc.
6. **Safe Defaults**: Handle null/undefined/invalid values gracefully

#### 3. Display Requirements

Based on frontend documentation requirements, timestamps should be displayed as:

- **List Views**: Relative time ("2 hours ago") for recent items, absolute date for older items
- **Detail Views**: Absolute date/time with timezone information
- **Filters**: Date/time pickers that convert to/from Unix timestamps
- **Sorting**: Sort by timestamp value (numeric comparison)

### Core Frontend Timestamp Module

The frontend should implement a core timestamp utility module in `src/utils/timestampUtils.ts` (or `.js`) that provides the following functions:

#### Required Functions

- **`parseTimestamp(value, defaultValue)`**: Parse a timestamp value from the API and convert to JavaScript Date. Handles number, string, null, and undefined inputs. Validates timestamp range (1970-2100) and converts seconds to milliseconds for JavaScript Date constructor.

- **`getUtcTimestamp()`**: Get current UTC timestamp as Unix timestamp (float). Returns `Date.now() / 1000`.

- **`formatRelativeTime(timestamp, options)`**: Format timestamp as relative time string. Returns strings like "just now", "2 minutes ago", "3 hours ago", "2 days ago", or "in 5 minutes" for future timestamps. Options include `includeSeconds`, `futurePrefix`, and `pastSuffix`.

- **`formatAbsoluteTime(timestamp, options)`**: Format timestamp as absolute date/time string. Options include `includeTime`, `includeSeconds`, `includeTimezone`, and `format` ('iso', 'local', or 'utc').

- **`formatTimestamp(timestamp, options)`**: Smart formatting that uses relative time for recent items (default < 24 hours) and absolute time for older items. Options include `relativeThreshold`, `showRelative`, and `showAbsolute`.

- **`isExpired(expiresAt)`**: Check if a timestamp has expired (is in the past). Returns true if expired, false otherwise.

- **`isFuture(timestamp)`**: Check if a timestamp is in the future. Returns true if in future, false otherwise.

- **`compareTimestamps(timestamp1, timestamp2)`**: Compare two timestamps. Returns -1 if timestamp1 < timestamp2, 0 if equal, 1 if timestamp1 > timestamp2.

- **`dateToTimestamp(date)`**: Convert JavaScript Date to Unix timestamp (float). Defaults to current date if not provided.

- **`toISOString(timestamp)`**: Convert Unix timestamp to ISO 8601 string.

#### Usage Examples

**Basic Usage:** Import functions from the timestamp utils module. Use `parseTimestamp(apiData._createdAt)` to parse timestamps from API responses. Use `formatTimestamp(apiData._createdAt)` for smart display formatting (returns relative time for recent items, absolute time for older). Use `formatRelativeTime(apiData._createdAt)` for relative time strings like "2 hours ago". Use `formatAbsoluteTime(apiData._createdAt, { format: 'utc' })` for absolute time strings.

**In React Components:** Import `parseTimestamp`, `formatTimestamp`, and `isExpired` from the timestamp utils module. Call `parseTimestamp(workflow._createdAt)` to parse timestamps. Use `formatTimestamp(workflow._createdAt)` to display formatted time. Use `isExpired(expiresAt)` to check if a token has expired.

**Sorting:** Import `compareTimestamps` and use it in sort functions: `workflows.sort((a, b) => compareTimestamps(a._createdAt, b._createdAt))`.

**Filtering:** Import `isExpired` and use it to filter arrays: `tokens.filter(token => !isExpired(token.expiresAt))`.

### Frontend Integration Guidelines

#### 1. Always Use the Timestamp Module

**✅ DO:** Import and use `parseTimestamp()` from the timestamp utils module. Call `parseTimestamp(apiData._createdAt)` to safely parse timestamps.

**❌ DON'T:** Do not assume timestamps are always numbers and directly create Date objects like `new Date(apiData._createdAt * 1000)` as this may fail if the value is a string or null.

#### 2. Handle Null/Undefined Values

**✅ DO:** Always provide a default value when calling `parseTimestamp()`, for example: `parseTimestamp(apiData.expiresAt, new Date())`. Check if the returned date is valid before using it.

**❌ DON'T:** Do not assume timestamps always exist. Do not directly create Date objects without checking for null values first.

#### 3. Use Appropriate Formatting

- **List Views**: Use `formatTimestamp()` for smart relative/absolute formatting
- **Detail Views**: Use `formatAbsoluteTime()` with timezone information
- **Filters**: Convert Date picker values to Unix timestamps using `dateToTimestamp()`

#### 4. Timezone Considerations

- All backend timestamps are in **UTC**
- Frontend should convert to **user's local timezone** for display
- Use `formatAbsoluteTime()` with `format: 'local'` for user-friendly display
- Always show timezone information in detail views

### Testing the Timestamp Module

**Test Cases to Cover:**
1. Parse number timestamps (float and int)
2. Parse string timestamps
3. Handle null/undefined values
4. Handle invalid values (NaN, out of range)
5. Format relative time (past and future)
6. Format absolute time (UTC and local)
7. Compare timestamps
8. Check expiration
9. Convert Date to timestamp

**Example Test:** Import functions from the timestamp utils module and test that `parseTimestamp()` correctly handles number timestamps, string timestamps, and null values. Test that `formatRelativeTime()` correctly formats timestamps as relative time strings. Test that `isExpired()` correctly identifies expired timestamps.

### Summary: Frontend Timestamp Handling

1. **Always decode** API timestamps using `parseTimestamp()` - handles number/string/null
2. **Format appropriately** - relative for recent, absolute for older items
3. **Handle timezones** - convert UTC to user's local timezone for display
4. **Provide defaults** - handle null/undefined gracefully
5. **Use consistently** - use the core module throughout the project
6. **Test thoroughly** - cover all input formats and edge cases

The core timestamp module ensures consistent, safe handling of timestamps across the entire frontend application, matching the backend's `parseTimestamp()` functionality for type normalization and error handling.