PowerOn/gateway
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
gateway/docs/code-documentation/date-time-handling.md

17 KiB
Raw Blame History

Date and Time Handling

This document describes how dates and times are handled throughout the gateway project, including storage formats, transformations, and retrieval mechanisms.

Overview

The gateway uses Unix timestamps (floats representing seconds since epoch) as the standard format for all date and time values. This ensures consistency across different database backends and eliminates timezone-related issues by always working in UTC.

Core Components

Time Utilities Module

The primary module for date/time handling is modules/shared/timeUtils.py, which provides:

getUtcTimestamp() -> float

Returns the current UTC timestamp as a float (seconds since Unix epoch with millisecond precision).

Implementation:

  • Uses Python's time.time() which returns seconds since epoch as a float
  • Provides millisecond precision
  • Always returns UTC time

getUtcNow() -> datetime

Returns the current UTC time as a datetime object with timezone information.

parseTimestamp(value: Any, default: Optional[float] = None) -> Optional[float]

Critical function for database retrieval - Parses timestamp values from various formats and converts them to float.

This function handles the transformation/decoding of timestamps when reading from the database, especially important for PostgreSQL connectors that may return numeric fields as strings in some environments (e.g., Azure PostgreSQL).

Supported input formats:

  • float: Returns as-is
  • int: Converts to float
  • str: Attempts to parse as numeric string (e.g., "1234567890.123")
  • None: Returns default value (or None if no default provided)

Usage: Call parseTimestamp() with the database value and an optional default. For example: parseTimestamp(db_value, default=getUtcTimestamp()).

Why this is needed:

  • PostgreSQL connectors (especially Azure PostgreSQL) may return numeric fields as strings
  • Ensures consistent float type regardless of database return format
  • Provides safe fallback with default values

createExpirationTimestamp(expiresInSeconds: int) -> float

Creates an expiration timestamp by adding seconds to the current UTC timestamp. Call with the number of seconds until expiration, for example: createExpirationTimestamp(3600) for 1 hour from now.

Database Storage

Storage Format

All timestamps are stored in the database as floats (Unix timestamps in seconds):

  • PostgreSQL: Stored as NUMERIC or DOUBLE PRECISION type
  • JSON Database: Stored as JSON number (float) in record files

Saving Timestamps

PostgreSQL Connector (connectorDbPostgre.py)

When saving records, timestamp fields (_createdAt, _modifiedAt) are handled specially. The connector checks if the value is a string and attempts to convert it to float before storing.

Process:

  1. Timestamp values are converted to float if they're strings
  2. Stored directly as numeric values in PostgreSQL
  3. No encryption or encoding - stored as plain numeric values

JSON Connector (connectorDbJson.py)

Timestamps are stored directly as JSON numbers. The connector calls getUtcTimestamp() to get the current time and sets _createdAt on creation and _modifiedAt on every save operation.

Process:

  1. Timestamps are generated as floats using getUtcTimestamp()
  2. Stored as JSON numbers in record files
  3. No transformation needed - JSON natively supports float numbers

Retrieving Timestamps

PostgreSQL Connector

When reading from PostgreSQL, timestamps may be returned in different formats:

  • Expected: Float values
  • Actual (Azure PostgreSQL): Sometimes returned as strings
  • Solution: Use parseTimestamp() to normalize

Example from code: In interfaceDbChatObjects.py, timestamps are parsed using parseTimestamp(msg.get("publishedAt"), default=getUtcTimestamp()).

JSON Connector

Timestamps are read directly from JSON and typically remain as floats, but parseTimestamp() is still used for safety to ensure consistent float type regardless of JSON parsing quirks. Call parseTimestamp(record.get("timestamp"), default=getUtcTimestamp()) when reading timestamp fields.

Data Model Definitions

Timestamps are defined in Pydantic models using the float type with getUtcTimestamp as the default factory. For example, in ChatLog model, the timestamp field uses Field(default_factory=getUtcTimestamp, ...).

Common timestamp fields:

  • _createdAt: When the record was created
  • _modifiedAt: When the record was last modified
  • publishedAt: When a message was published
  • timestamp: Generic timestamp field
  • expiresAt: When something expires
  • connectedAt: When a connection was established
  • lastChecked: When something was last verified
  • creationDate: When something was created

Transformation Flow

Saving to Database

Application Code
    ↓
getUtcTimestamp() → float (e.g., 1234567890.123)
    ↓
Database Connector
    ↓
PostgreSQL: Store as NUMERIC/DOUBLE PRECISION
JSON: Store as JSON number

Reading from Database

Database
    ↓
PostgreSQL: May return as float, int, or string
JSON: Returns as float (or int if no decimals)
    ↓
parseTimestamp() → Normalizes to float
    ↓
Application Code (always receives float)

Key Points

  1. No Encryption: Timestamps are not encrypted. They are stored as plain numeric values (floats) in the database. The "encrypted format" you may have seen refers to the numeric Unix timestamp format, which is not human-readable but is not encrypted.

  2. Type Transformation: The main transformation happens when reading from the database:

    • Database may return timestamps as strings (especially Azure PostgreSQL)
    • parseTimestamp() converts strings/int/float → float
    • This ensures consistent float type in application code

  3. UTC Only: All timestamps are in UTC. No timezone conversions happen at the database level.

  4. Millisecond Precision: Timestamps use float type, providing millisecond precision (e.g., 1234567890.123).

  5. Default Values: When timestamps are missing or invalid, parseTimestamp() can provide safe defaults using the default parameter.

Usage Examples

Creating a Record with Timestamps

When creating records, call getUtcTimestamp() to set _createdAt and _modifiedAt fields. These are typically auto-set by the database connectors.

Reading and Using Timestamps

After retrieving a record from the database, use parseTimestamp(record.get("publishedAt"), default=getUtcTimestamp()) to safely parse the timestamp value before using it for filtering or comparison.

Filtering by Timestamp

When filtering messages or records by timestamp, iterate through the results and use parseTimestamp(msg.get("publishedAt"), default=getUtcTimestamp()) to parse each timestamp, then compare against your threshold value.

Sorting by Timestamp

When sorting logs or records by timestamp, use parseTimestamp() in the sort key function, for example: logs.sort(key=lambda x: parseTimestamp(x.get("timestamp"), default=0)).

Common Patterns

Automatic Timestamp Management

Database connectors automatically manage _createdAt and _modifiedAt:

  • _createdAt: Set only on record creation (if not already present)
  • _modifiedAt: Updated on every save operation

Timestamp Filtering

When filtering records by timestamp, always use parseTimestamp() to handle all database return types. Call parseTimestamp(record.get("timestamp"), default=0) before comparison. Do not directly compare timestamp values as they may be strings from the database.

Expiration Timestamps

For expiration logic, use createExpirationTimestamp(3600) to create an expiration timestamp, then compare it with the current time using getUtcTimestamp(). Parse any stored expiration timestamps using parseTimestamp() before comparison.

Database-Specific Notes

PostgreSQL (Azure)

  • May return numeric fields as strings
  • Always use parseTimestamp() when reading timestamp fields
  • Stored as NUMERIC or DOUBLE PRECISION type

JSON Database

  • Timestamps stored as JSON numbers
  • Usually returned as floats, but parseTimestamp() provides safety
  • No special handling needed beyond normalization

Troubleshooting

Issue: Timestamp is a string instead of float

Solution: Use parseTimestamp() to convert: parseTimestamp(string_timestamp, default=getUtcTimestamp()).

Issue: Timestamp is None

Solution: Provide a default value: parseTimestamp(record.get("timestamp"), default=getUtcTimestamp()).

Issue: Timestamp comparison fails

Cause: Comparing string to float Solution: Always parse first using parseTimestamp(value, default=0) before any comparison operations.

Summary

The date/time handling system:

  1. Stores timestamps as Unix timestamps (floats) in UTC
  2. Transforms database return values (string/int/float) → float using parseTimestamp()
  3. Provides utilities for timestamp generation, parsing, and expiration
  4. Ensures consistency across different database backends
  5. Handles edge cases (None values, string returns, invalid formats)

The transformation you mentioned is the type normalization that happens when reading from the database, not encryption. Timestamps are stored as plain numeric values and converted to consistent float types for application use.

Frontend Requirements and Implementation

API Response Format

The backend API returns timestamps as Unix timestamps (floats) in all JSON responses. Timestamps are always in UTC and represented as numbers (not strings).

Example API Response: The API returns timestamps as numeric values in JSON, for example: _createdAt: 1704067200.123, _modifiedAt: 1704067300.456, publishedAt: 1704067250.789, expiresAt: 1704153600.0.

Important Notes:

  • Timestamps are always numbers (float) in JSON responses
  • All timestamps are in UTC (no timezone information in the value)
  • Timestamps have millisecond precision (decimal places)
  • Timestamp fields may be null or undefined if not set

Frontend Requirements

1. Timestamp Decoding Module

The frontend must implement a core timestamp utility module that handles:

  • Decoding/parsing timestamp values from API responses
  • Type normalization (handles number, string, null, undefined)
  • Conversion to JavaScript Date objects
  • Formatting for display (relative time, absolute date/time)
  • Timezone handling (convert UTC to user's local timezone)

2. Required Functionality

The frontend timestamp module must provide:

  1. Parse/Decode Function: Convert API timestamp (number/string/null) → JavaScript Date
  2. Format Functions: Convert Date → human-readable strings
  3. Relative Time: "2 hours ago", "in 5 minutes", etc.
  4. Absolute Time: "2024-01-01 12:30:45 UTC" or localized format
  5. Comparison Utilities: Compare timestamps, check expiration, etc.
  6. Safe Defaults: Handle null/undefined/invalid values gracefully

3. Display Requirements

Based on frontend documentation requirements, timestamps should be displayed as:

  • List Views: Relative time ("2 hours ago") for recent items, absolute date for older items
  • Detail Views: Absolute date/time with timezone information
  • Filters: Date/time pickers that convert to/from Unix timestamps
  • Sorting: Sort by timestamp value (numeric comparison)

Core Frontend Timestamp Module

The frontend should implement a core timestamp utility module in src/utils/timestampUtils.ts (or .js) that provides the following functions:

Required Functions

  • parseTimestamp(value, defaultValue): Parse a timestamp value from the API and convert to JavaScript Date. Handles number, string, null, and undefined inputs. Validates timestamp range (1970-2100) and converts seconds to milliseconds for JavaScript Date constructor.

  • getUtcTimestamp(): Get current UTC timestamp as Unix timestamp (float). Returns Date.now() / 1000.

  • formatRelativeTime(timestamp, options): Format timestamp as relative time string. Returns strings like "just now", "2 minutes ago", "3 hours ago", "2 days ago", or "in 5 minutes" for future timestamps. Options include includeSeconds, futurePrefix, and pastSuffix.

  • formatAbsoluteTime(timestamp, options): Format timestamp as absolute date/time string. Options include includeTime, includeSeconds, includeTimezone, and format ('iso', 'local', or 'utc').

  • formatTimestamp(timestamp, options): Smart formatting that uses relative time for recent items (default < 24 hours) and absolute time for older items. Options include relativeThreshold, showRelative, and showAbsolute.

  • isExpired(expiresAt): Check if a timestamp has expired (is in the past). Returns true if expired, false otherwise.

  • isFuture(timestamp): Check if a timestamp is in the future. Returns true if in future, false otherwise.

  • compareTimestamps(timestamp1, timestamp2): Compare two timestamps. Returns -1 if timestamp1 < timestamp2, 0 if equal, 1 if timestamp1 > timestamp2.

  • dateToTimestamp(date): Convert JavaScript Date to Unix timestamp (float). Defaults to current date if not provided.

  • toISOString(timestamp): Convert Unix timestamp to ISO 8601 string.

Usage Examples

Basic Usage: Import functions from the timestamp utils module. Use parseTimestamp(apiData._createdAt) to parse timestamps from API responses. Use formatTimestamp(apiData._createdAt) for smart display formatting (returns relative time for recent items, absolute time for older). Use formatRelativeTime(apiData._createdAt) for relative time strings like "2 hours ago". Use formatAbsoluteTime(apiData._createdAt, { format: 'utc' }) for absolute time strings.

In React Components: Import parseTimestamp, formatTimestamp, and isExpired from the timestamp utils module. Call parseTimestamp(workflow._createdAt) to parse timestamps. Use formatTimestamp(workflow._createdAt) to display formatted time. Use isExpired(expiresAt) to check if a token has expired.

Sorting: Import compareTimestamps and use it in sort functions: workflows.sort((a, b) => compareTimestamps(a._createdAt, b._createdAt)).

Filtering: Import isExpired and use it to filter arrays: tokens.filter(token => !isExpired(token.expiresAt)).

Frontend Integration Guidelines

1. Always Use the Timestamp Module

✅ DO: Import and use parseTimestamp() from the timestamp utils module. Call parseTimestamp(apiData._createdAt) to safely parse timestamps.

❌ DON'T: Do not assume timestamps are always numbers and directly create Date objects like new Date(apiData._createdAt * 1000) as this may fail if the value is a string or null.

2. Handle Null/Undefined Values

✅ DO: Always provide a default value when calling parseTimestamp(), for example: parseTimestamp(apiData.expiresAt, new Date()). Check if the returned date is valid before using it.

❌ DON'T: Do not assume timestamps always exist. Do not directly create Date objects without checking for null values first.

3. Use Appropriate Formatting

  • List Views: Use formatTimestamp() for smart relative/absolute formatting
  • Detail Views: Use formatAbsoluteTime() with timezone information
  • Filters: Convert Date picker values to Unix timestamps using dateToTimestamp()

4. Timezone Considerations

  • All backend timestamps are in UTC
  • Frontend should convert to user's local timezone for display
  • Use formatAbsoluteTime() with format: 'local' for user-friendly display
  • Always show timezone information in detail views

Testing the Timestamp Module

Test Cases to Cover:

  1. Parse number timestamps (float and int)
  2. Parse string timestamps
  3. Handle null/undefined values
  4. Handle invalid values (NaN, out of range)
  5. Format relative time (past and future)
  6. Format absolute time (UTC and local)
  7. Compare timestamps
  8. Check expiration
  9. Convert Date to timestamp

Example Test: Import functions from the timestamp utils module and test that parseTimestamp() correctly handles number timestamps, string timestamps, and null values. Test that formatRelativeTime() correctly formats timestamps as relative time strings. Test that isExpired() correctly identifies expired timestamps.

Summary: Frontend Timestamp Handling

  1. Always decode API timestamps using parseTimestamp() - handles number/string/null
  2. Format appropriately - relative for recent, absolute for older items
  3. Handle timezones - convert UTC to user's local timezone for display
  4. Provide defaults - handle null/undefined gracefully
  5. Use consistently - use the core module throughout the project
  6. Test thoroughly - cover all input formats and edge cases

The core timestamp module ensures consistent, safe handling of timestamps across the entire frontend application, matching the backend's parseTimestamp() functionality for type normalization and error handling.