# Security Component Documentation
Comprehensive documentation for the Security component of the Gateway application, detailing authentication, authorization, token management, CSRF protection, and integration with other system components.
## Table of Contents
1. [Overview](#overview)
2. [High-Level Architecture](#high-level-architecture)
3. [Request Flow](#request-flow)
4. [Component Overview](#component-overview)
5. [Detailed Component Explanations](#detailed-component-explanations)
6. [How Components Work Together](#how-components-work-together)
7. [Security Patterns & Best Practices](#security-patterns--best-practices)
8. [Configuration](#configuration)
9. [Security Considerations](#security-considerations)
10. [Troubleshooting](#troubleshooting)
> **Note**: For API endpoint documentation and usage examples, see [Security API Documentation](./security-api.md).
## Overview
### What is the Security Component?
The Security component is a comprehensive authentication and authorization system that protects the Gateway application. It ensures that only authenticated and authorized users can access protected resources, while providing a seamless user experience through automatic token management and refresh mechanisms.
### Why Does It Exist?
Modern web applications need robust security to protect user data and system resources. The Security component provides:
- **Authentication**: Verifying who a user is (login process)
- **Authorization**: Determining what a user can do (permissions)
- **Session Management**: Maintaining user sessions securely
- **Token Management**: Handling JWT tokens for stateless authentication
- **OAuth Integration**: Supporting third-party authentication (Microsoft, Google)
- **Attack Prevention**: Protecting against common web vulnerabilities (CSRF, XSS)
### What Problems Does It Solve?
1. **User Authentication**: Users need to prove their identity to access the system
2. **Session Security**: Sessions must be maintained securely without exposing sensitive data
3. **Token Expiration**: OAuth tokens expire and need automatic refresh to avoid user disruption
4. **Multi-Tenancy**: Users belong to different mandates (organizations) and must be scoped correctly
5. **Cross-Site Attacks**: Protection against CSRF and XSS attacks
6. **Rate Limiting**: Preventing brute force attacks and system abuse
### Key Features
- **Multi-Authority Authentication**: Supports three authentication methods:
- **Local Authentication**: Username/password stored in the database
- **Microsoft OAuth**: Single Sign-On (SSO) via Microsoft Azure AD
- **Google OAuth**: Single Sign-On (SSO) via Google accounts
- **JWT Token Management**:
- Creates secure JSON Web Tokens (JWT) for authentication
- Manages both access tokens (short-lived) and refresh tokens (long-lived)
- Validates tokens on every request
- Supports token revocation for LOCAL authentication
- **Cookie-Based Authentication**:
- Uses secure httpOnly cookies to store tokens (prevents JavaScript access)
- Falls back to Authorization headers for API clients
- Automatically configures security settings based on environment (HTTPS vs HTTP)
- **Automatic Token Refresh**:
- Background refresh of expired OAuth tokens
- Proactive refresh before expiration
- Non-blocking operation (doesn't slow down user requests)
- **CSRF Protection**:
- Validates CSRF tokens for state-changing operations (POST, PUT, DELETE, PATCH)
- Exempts login and OAuth callback endpoints
- Prevents cross-site request forgery attacks
- **Rate Limiting**:
- Built-in rate limiting for authentication endpoints
- Prevents brute force attacks
- Configurable limits per endpoint
- **Database-Backed Token Validation**:
- LOCAL tokens are tracked in the database
- Supports token revocation
- Validates token status on every request
## High-Level Architecture
### The Big Picture
The Security component acts as a protective layer around the entire Gateway application. Every HTTP request passes through security middleware before reaching your application code. Think of it as a security checkpoint at the entrance of a building - everyone must pass through it, and only authorized people are allowed in.
```mermaid
graph TB
subgraph "Client"
Browser[Web Browser
or API Client]
end
subgraph "Security Component - Request Processing"
CSRF[CSRF Protection
Validates CSRF tokens]
TokenRefresh[Token Refresh
Refreshes expired tokens]
Auth[Authentication
Validates JWT tokens]
end
subgraph "Application"
Routes[API Routes
Your application code]
end
Browser -->|HTTP Request| CSRF
CSRF -->|Valid CSRF| TokenRefresh
TokenRefresh -->|Valid Token| Auth
Auth -->|Authenticated User| Routes
Routes -->|Response| Browser
style CSRF fill:#fce4ec,stroke:#880e4f,stroke-width:2px
style TokenRefresh fill:#f3e5f5,stroke:#4a148c,stroke-width:2px
style Auth fill:#e1f5ff,stroke:#01579b,stroke-width:2px
```
### Component Structure
The Security component consists of six main modules, each with a specific responsibility:
```mermaid
graph TB
subgraph "Security Component"
Auth[auth.py
Authentication & User Context
Validates tokens and extracts user info]
JWT[jwtService.py
JWT Creation & Cookie Management
Creates tokens and manages cookies]
TokenMgr[tokenManager.py
OAuth Token Refresh
Refreshes Microsoft/Google tokens]
TokenRefreshSvc[tokenRefreshService.py
Token Refresh Orchestration
Coordinates token refresh operations]
TokenRefreshMw[tokenRefreshMiddleware.py
Automatic Token Refresh
Middleware that triggers refresh]
CSRF[csrf.py
CSRF Protection
Validates CSRF tokens]
end
subgraph "External Dependencies"
Config[Configuration
APP_CONFIG]
DB[(Database
PostgreSQL)]
Interfaces[Interfaces
Data Access Layer]
Routes[Routes
API Endpoints]
end
subgraph "External OAuth Providers"
MSFT[Microsoft OAuth]
Google[Google OAuth]
end
Auth --> JWT
Auth --> Interfaces
Auth --> Config
Auth --> Routes
JWT --> Config
TokenMgr --> Config
TokenMgr --> MSFT
TokenMgr --> Google
TokenMgr --> Interfaces
TokenRefreshSvc --> TokenMgr
TokenRefreshSvc --> Interfaces
TokenRefreshSvc --> DB
TokenRefreshMw --> TokenRefreshSvc
CSRF --> Routes
Routes --> Auth
style Auth fill:#e1f5ff,stroke:#01579b,stroke-width:3px
style JWT fill:#e8f5e9,stroke:#1b5e20,stroke-width:2px
style TokenMgr fill:#fff3e0,stroke:#e65100,stroke-width:2px
style TokenRefreshSvc fill:#fff3e0,stroke:#e65100,stroke-width:2px
style TokenRefreshMw fill:#f3e5f5,stroke:#4a148c,stroke-width:2px
style CSRF fill:#fce4ec,stroke:#880e4f,stroke-width:2px
```
### How It All Fits Together
1. **Middleware Layer** (runs first): CSRF protection and token refresh middleware intercept requests
2. **Authentication Layer** (runs when needed): Validates JWT tokens and extracts user information
3. **Route Layer** (runs last): Your application code receives authenticated requests with user context
## Request Flow
### What Happens When a User Makes a Request?
Let's trace through what happens when a user makes an API request:
#### Step 1: Request Arrives
A user's browser or API client sends an HTTP request to the Gateway API. This request might be:
- A GET request to fetch data
- A POST request to create something
- A PUT request to update something
- A DELETE request to remove something
#### Step 2: CSRF Protection (if state-changing)
If the request is a state-changing operation (POST, PUT, DELETE, PATCH), the CSRF middleware checks for a CSRF token in the `X-CSRF-Token` header. This prevents malicious websites from making requests on behalf of the user.
**Why?** Imagine you're logged into the Gateway application. A malicious website could try to trick your browser into making a request to the Gateway API (like deleting your data). CSRF protection prevents this by requiring a token that only the legitimate Gateway application knows.
#### Step 3: Token Refresh (if needed)
The token refresh middleware checks if the user has any expired OAuth tokens (Microsoft or Google). If so, it automatically refreshes them in the background without blocking the request.
**Why?** OAuth tokens expire after a certain time (usually 1 hour). Instead of waiting for the token to expire and then failing, the system proactively refreshes tokens before they expire. This happens silently in the background so users never notice.
#### Step 4: Authentication
The authentication layer extracts the JWT token from either:
- An httpOnly cookie (for web browsers)
- An Authorization header (for API clients)
It then validates the token:
- Checks the token format (is it a valid JWT?)
- Verifies the signature (was it signed with our secret key?)
- Checks expiration (has it expired?)
- Validates the user exists and is enabled
- For LOCAL tokens, checks the database to ensure the token hasn't been revoked
#### Step 5: User Context Extraction
If authentication succeeds, the system extracts user information from the token:
- Username
- User ID
- Mandate ID (which organization the user belongs to)
- Authentication authority (LOCAL, MSFT, or GOOGLE)
#### Step 6: Route Handler Execution
Finally, the request reaches your route handler with a fully authenticated user object. Your code can trust that:
- The user is who they claim to be
- The user has permission to make this request (based on your route's authentication requirements)
- The user's context (mandate, etc.) is correct
### Visual Request Flow
```mermaid
sequenceDiagram
participant Client as Client/Browser
participant CSRF as CSRF Middleware
participant TokenMw as Token Refresh Middleware
participant Auth as Authentication Layer
participant Route as Route Handler
Client->>CSRF: HTTP Request
(POST /api/data)
alt State-Changing Request
CSRF->>CSRF: Check X-CSRF-Token header
alt Invalid CSRF Token
CSRF-->>Client: 403 Forbidden
else Valid CSRF Token
CSRF->>TokenMw: Continue
end
else Read-Only Request
CSRF->>TokenMw: Continue
end
TokenMw->>TokenMw: Check for expired OAuth tokens
alt Tokens Need Refresh
TokenMw->>TokenMw: Refresh tokens (background)
end
TokenMw->>Auth: Continue
Auth->>Auth: Extract JWT from cookie/header
Auth->>Auth: Validate token signature
Auth->>Auth: Check token expiration
Auth->>Auth: Lookup user in database
Auth->>Auth: Validate user status
alt Authentication Failed
Auth-->>Client: 401 Unauthorized
else Authentication Succeeded
Auth->>Route: Request + User Object
Route->>Route: Process request with user context
Route-->>Client: Response
end
```
## Component Overview
Before diving into the details of each component, let's understand what each one does at a high level:
### 1. auth.py - The Authentication Core
**Role**: The heart of authentication. Validates tokens and extracts user information.
**Think of it as**: A security guard who checks IDs at the door. They verify your token (ID) is valid, check if you're allowed in, and tell the system who you are.
### 2. jwtService.py - Token Factory
**Role**: Creates JWT tokens and manages HTTP cookies.
**Think of it as**: A ticket office that issues tickets (tokens) and manages how they're stored (cookies).
### 3. tokenManager.py - OAuth Token Handler
**Role**: Refreshes OAuth tokens from Microsoft and Google.
**Think of it as**: A renewal office that extends your Microsoft/Google access passes before they expire.
### 4. tokenRefreshService.py - Token Refresh Coordinator
**Role**: Orchestrates token refresh operations, handles rate limiting, and tracks refresh attempts.
**Think of it as**: A manager who coordinates when and how tokens should be refreshed, ensuring we don't refresh too frequently.
### 5. tokenRefreshMiddleware.py - Automatic Refresh Trigger
**Role**: FastAPI middleware that automatically triggers token refresh when users make requests.
**Think of it as**: An automatic system that checks your tokens in the background and refreshes them when needed, without you having to think about it.
### 6. csrf.py - CSRF Protection
**Role**: Validates CSRF tokens to prevent cross-site request forgery attacks.
**Think of it as**: A bouncer who checks that requests are coming from legitimate sources, not malicious websites.
## Detailed Component Explanations
### 1. auth.py - Authentication & User Context
#### What Does It Do?
The `auth.py` module is responsible for authenticating users on every request. It's the first line of defense that determines whether a request should be allowed to proceed.
#### Key Components Explained
**CookieAuth Class**
This is a custom implementation of FastAPI's HTTPBearer security scheme. It's smart enough to check two places for authentication tokens:
1. **httpOnly Cookies** (preferred for web browsers): Cookies are automatically sent by the browser, making them convenient for web applications. The `httpOnly` flag prevents JavaScript from accessing them, which protects against XSS attacks.
2. **Authorization Header** (for API clients): Programmatic API clients (like mobile apps or scripts) send tokens in the `Authorization: Bearer ` header.
**Why both?** Web browsers work best with cookies (they're automatically included), while API clients prefer headers (they have more control). This dual approach supports both use cases.
**Example Flow:**
```python
# When a request comes in:
1. Check cookie: request.cookies.get('auth_token')
2. If not found, check header: request.headers.get("Authorization")
3. Extract token from whichever source has it
4. Return token for validation
```
**_getUserBase Function**
This is the core authentication function that performs all the security checks. Let's break down what it does step by step:
**Step 1: Token Format Validation**
```python
# Checks if token has the correct JWT structure: header.payload.signature
if token.count(".") != 2:
raise credentialsException # Invalid format
```
**Why?** JWTs have a specific format. If the token doesn't have exactly two dots, it's not a valid JWT and we reject it immediately.
**Step 2: Token Signature Verification**
```python
# Decodes and verifies the token was signed with our secret key
payload = jwt.decode(token, SECRET_KEY, algorithms=[ALGORITHM])
```
**Why?** This ensures the token was actually issued by our server. If someone tries to forge a token, they won't have our secret key, so the signature won't match and the token will be rejected.
**Step 3: Extract User Information**
```python
username = payload.get("sub") # Subject (username)
mandateId = payload.get("mandateId") # Which organization
userId = payload.get("userId") # User ID
authority = payload.get("authenticationAuthority") # LOCAL, MSFT, or GOOGLE
tokenId = payload.get("jti") # Token ID for tracking
```
**Why?** The token contains all the information we need to identify the user. We extract it so we can verify it matches what's in the database.
**Step 4: User Lookup**
```python
user = appInterface.getUserByUsername(username)
if user is None:
raise credentialsException # User doesn't exist
```
**Why?** The token might be valid, but the user might have been deleted or the username might have changed. We always verify against the database.
**Step 5: User Status Check**
```python
if not user.enabled:
raise HTTPException(status_code=403, detail="User is disabled")
```
**Why?** Even if authentication succeeds, disabled users shouldn't be able to access the system. This provides a way to temporarily disable accounts without deleting them.
**Step 6: Context Validation**
```python
if str(user.mandateId) != str(mandateId) or str(user.id) != str(userId):
raise HTTPException(status_code=401, detail="User context has changed")
```
**Why?** If a user's mandate or ID changes (maybe they were moved to a different organization), their old tokens become invalid. This forces them to log in again with the new context.
**Step 7: Database Token Validation (for LOCAL tokens)**
```python
# For LOCAL tokens, check if token exists in database and is active
if authority == AuthAuthority.LOCAL:
active_token = appInterface.findActiveTokenById(tokenId, userId, ...)
if not active_token:
raise credentialsException # Token was revoked
```
**Why?** LOCAL tokens are stored in the database so we can revoke them. If a user logs out or their token is revoked, we check the database to ensure the token is still valid.
**getCurrentUser Function**
This is a simple wrapper around `_getUserBase` that provides a clean interface for route handlers. Routes use it like this:
```python
@router.get("/protected")
async def protected_endpoint(currentUser: User = Depends(getCurrentUser)):
# currentUser is guaranteed to be authenticated and enabled
return {"message": f"Hello, {currentUser.username}!"}
```
**Why a wrapper?** It provides a clear, simple interface for routes. Routes don't need to know about the internal `_getUserBase` function - they just use `getCurrentUser` and trust that it works.
#### Security Checks Performed
The authentication process performs multiple layers of security checks:
1. **JWT Format Validation**: Ensures the token has the correct structure
2. **Signature Verification**: Verifies the token was signed with our secret key
3. **Expiration Check**: Ensures the token hasn't expired
4. **User Existence**: Verifies the user still exists in the database
5. **User Status**: Checks if the user is enabled
6. **Context Validation**: Ensures token context matches user record
7. **Token Revocation**: For LOCAL tokens, checks database for revocation status
#### Dependencies
- `jwtService.py`: Uses JWT decoding functions (via the `jose` library)
- `interfaceDbAppObjects`: Accesses the database to look up users and validate tokens
- `datamodelUam.User`: User data model
- `datamodelSecurity.Token`: Token data model
- `slowapi.Limiter`: Rate limiting utility (exported for use in routes)
---
### 2. jwtService.py - JWT Creation & Cookie Management
#### What Does It Do?
The `jwtService.py` module is responsible for creating JWT tokens and managing how they're stored in HTTP cookies. It's the "token factory" that issues authentication tokens.
#### Key Functions Explained
**createAccessToken**
Creates a short-lived JWT access token that users include with every request.
```python
def createAccessToken(data: dict, expiresDelta: Optional[timedelta] = None) -> Tuple[str, datetime]:
# Adds a unique token ID (jti) if not present
if "jti" not in toEncode:
toEncode["jti"] = str(uuid.uuid4())
# Sets expiration time
expire = getUtcNow() + (expiresDelta or timedelta(minutes=ACCESS_TOKEN_EXPIRE_MINUTES))
toEncode.update({"exp": expire})
# Signs and encodes the token
encodedJwt = jwt.encode(toEncode, SECRET_KEY, algorithm=ALGORITHM)
return encodedJwt, expire
```
**What it does:**
1. Ensures every token has a unique ID (JTI) for tracking
2. Sets an expiration time (default: 60 minutes)
3. Signs the token with our secret key
4. Returns both the token string and expiration time
**Why short-lived?** If a token is stolen, it will expire quickly, limiting the damage. Access tokens are meant to be used frequently and replaced often.
**createRefreshToken**
Creates a long-lived refresh token used to obtain new access tokens.
```python
def createRefreshToken(data: dict) -> Tuple[str, datetime]:
toEncode = data.copy()
if "jti" not in toEncode:
toEncode["jti"] = str(uuid.uuid4())
toEncode["type"] = "refresh" # Marks it as a refresh token
expire = getUtcNow() + timedelta(days=REFRESH_TOKEN_EXPIRE_DAYS) # Default: 7 days
toEncode.update({"exp": expire})
encodedJwt = jwt.encode(toEncode, SECRET_KEY, algorithm=ALGORITHM)
return encodedJwt, expire
```
**What it does:**
1. Similar to access token creation
2. Marks the token as type "refresh"
3. Sets a longer expiration (default: 7 days)
4. Returns the token and expiration
**Why long-lived?** Refresh tokens are used less frequently (only when access tokens expire). They allow users to stay logged in for extended periods without re-entering credentials.
**setAccessTokenCookie**
Stores the access token in an httpOnly cookie.
```python
def setAccessTokenCookie(response: Response, token: str, expiresDelta: Optional[timedelta] = None):
maxAge = expiresDelta.total_seconds() if expiresDelta else ACCESS_TOKEN_EXPIRE_MINUTES * 60
response.set_cookie(
key="auth_token",
value=token,
httponly=True, # JavaScript can't access it
secure=USE_SECURE_COOKIES, # Only sent over HTTPS in production
samesite="strict", # Prevents CSRF attacks
path="/", # Available to entire application
max_age=maxAge
)
```
**Security Settings Explained:**
- **httponly=True**: Prevents JavaScript from accessing the cookie. This protects against XSS attacks where malicious scripts try to steal tokens.
- **secure=USE_SECURE_COOKIES**: In production (HTTPS), cookies are only sent over encrypted connections. In development (HTTP), this is disabled.
- **samesite="strict"**: Cookies are only sent with requests from the same site. This prevents CSRF attacks where malicious sites try to make requests with your cookies.
- **path="/"**: The cookie is available to all paths in the application.
**setRefreshTokenCookie**
Similar to `setAccessTokenCookie`, but for refresh tokens with longer expiration.
**clearAccessTokenCookie / clearRefreshTokenCookie**
These functions remove cookies when users log out. They use a dual-method approach for maximum browser compatibility:
```python
def clearAccessTokenCookie(response: Response):
# Method 1: Raw Set-Cookie header with expiration in the past
response.headers.append(
"Set-Cookie",
f"auth_token=deleted; Path=/; Max-Age=0; Expires=Thu, 01 Jan 1970 00:00:00 GMT; HttpOnly; SameSite=Strict"
)
# Method 2: FastAPI's built-in method
response.delete_cookie(key="auth_token", path="/")
```
**Why two methods?** Different browsers handle cookie deletion differently. Using both methods ensures the cookie is deleted regardless of browser quirks.
#### Configuration
The module uses these configuration values:
- `APP_JWT_KEY_SECRET`: The secret key used to sign tokens. **Must be kept secret!**
- `Auth_ALGORITHM`: JWT signing algorithm (default: HS256)
- `APP_TOKEN_EXPIRY`: How long access tokens are valid (default: 60 minutes)
- `APP_REFRESH_TOKEN_EXPIRE_DAYS`: How long refresh tokens are valid (default: 7 days)
- `APP_API_URL`: Used to determine if cookies should be secure (HTTPS) or not (HTTP)
---
### 3. tokenManager.py - OAuth Token Refresh
#### What Does It Do?
The `tokenManager.py` module handles refreshing OAuth tokens from Microsoft and Google. When users authenticate via OAuth, they receive tokens that expire after a certain time. This module refreshes those tokens before they expire, so users don't get interrupted.
#### Understanding OAuth Token Refresh
When a user logs in with Microsoft or Google:
1. They're redirected to Microsoft/Google's login page
2. After successful login, Microsoft/Google gives us:
- An **access token** (short-lived, ~1 hour)
- A **refresh token** (long-lived, can be used to get new access tokens)
When the access token expires, we use the refresh token to get a new access token without requiring the user to log in again.
#### Key Methods Explained
**refreshMicrosoftToken**
Refreshes a Microsoft OAuth token.
```python
def refreshMicrosoftToken(self, refreshToken: str, userId: str, oldToken: Token) -> Optional[Token]:
# Microsoft token refresh endpoint
tokenUrl = f"https://login.microsoftonline.com/{self.msft_tenant_id}/oauth2/v2.0/token"
# Prepare refresh request
data = {
"client_id": self.msft_client_id,
"client_secret": self.msft_client_secret,
"grant_type": "refresh_token",
"refresh_token": refreshToken,
"scope": "Mail.ReadWrite Mail.Send Mail.ReadWrite.Shared User.Read"
}
# Make request to Microsoft
response = client.post(tokenUrl, data=data)
if response.status_code == 200:
tokenData = response.json()
# Create new token object with refreshed data
newToken = Token(
userId=userId,
authority=AuthAuthority.MSFT,
connectionId=oldToken.connectionId, # Preserve connection ID
tokenAccess=tokenData["access_token"],
tokenRefresh=tokenData.get("refresh_token", refreshToken), # Keep old if new not provided
expiresAt=createExpirationTimestamp(tokenData.get("expires_in", 3600)),
)
return newToken
```
**What it does:**
1. Sends a request to Microsoft's token endpoint with the refresh token
2. Microsoft validates the refresh token and issues a new access token
3. Creates a new Token object with the refreshed data
4. Preserves the connection ID (so we know which Microsoft connection this is for)
**Why preserve connection ID?** Users can have multiple OAuth connections (e.g., multiple Microsoft accounts). The connection ID identifies which specific connection this token belongs to.
**refreshGoogleToken**
Similar to `refreshMicrosoftToken`, but for Google's OAuth endpoint:
```python
def refreshGoogleToken(self, refreshToken: str, userId: str, oldToken: Token) -> Optional[Token]:
tokenUrl = "https://oauth2.googleapis.com/token"
# ... similar process to Microsoft
```
**refreshToken**
A generic method that routes to the appropriate provider-specific refresh method:
```python
def refreshToken(self, oldToken: Token) -> Optional[Token]:
# Cooldown check: don't refresh if refreshed recently
if secondsSinceLastRefresh < 10 * 60: # 10 minutes
return oldToken # Return existing token
# Route to appropriate provider
if oldToken.authority == AuthAuthority.MSFT:
return self.refreshMicrosoftToken(...)
elif oldToken.authority == AuthAuthority.GOOGLE:
return self.refreshGoogleToken(...)
```
**Cooldown Mechanism:** Prevents excessive refresh attempts. If a token was refreshed less than 10 minutes ago, we skip refreshing it again. This prevents hitting OAuth provider rate limits.
**ensureFreshToken**
Proactively refreshes a token if it's about to expire:
```python
def ensureFreshToken(self, token: Token, *, secondsBeforeExpiry: int = 30 * 60) -> Optional[Token]:
nowTs = getUtcTimestamp()
expiresAt = token.expiresAt or 0
# If token expires within threshold (default: 30 minutes), refresh it
if expiresAt < (nowTs + secondsBeforeExpiry):
refreshed = self.refreshToken(token)
if refreshed and saveCallback:
saveCallback(refreshed) # Persist the refreshed token
return refreshed
return token # Token is still fresh
```
**Why proactive?** Instead of waiting for tokens to expire and then failing, we refresh them 30 minutes before expiration. This ensures tokens are always fresh when needed.
#### Features
- **Cooldown Mechanism**: 10-minute minimum between refresh attempts prevents rate limit exhaustion
- **Automatic Persistence**: Uses callback mechanism to save refreshed tokens to database
- **Proactive Refresh**: Refreshes tokens 30 minutes before expiration
- **Error Handling**: Gracefully handles OAuth provider failures
- **Connection Preservation**: Maintains connection IDs during refresh
#### Dependencies
- `httpx`: HTTP client for making requests to OAuth providers
- `interfaceDbAppObjects`: For token persistence (via callback)
- `datamodelSecurity.Token`: Token data model
- `datamodelUam.AuthAuthority`: Authority enum (LOCAL, MSFT, GOOGLE)
---
### 4. tokenRefreshService.py - Token Refresh Orchestration
#### What Does It Do?
The `tokenRefreshService.py` module is a high-level service that coordinates token refresh operations. It handles multiple connections, rate limiting, and tracks refresh attempts. Think of it as the "manager" that decides when and how tokens should be refreshed.
#### Key Methods Explained
**refresh_expired_tokens**
Refreshes all expired OAuth tokens for a user.
```python
async def refresh_expired_tokens(self, user_id: str) -> Dict[str, Any]:
# Get all connections for the user
connections = root_interface.getUserConnections(user_id)
refreshed_count = 0
failed_count = 0
rate_limited_count = 0
for connection in connections:
# Only refresh expired OAuth connections
if connection.tokenStatus == 'expired' and connection.authority in [AuthAuthority.GOOGLE, AuthAuthority.MSFT]:
# Check rate limiting
if self._is_rate_limited(connection.id):
rate_limited_count += 1
continue
# Record attempt
self._record_refresh_attempt(connection.id)
# Refresh based on authority
if connection.authority == AuthAuthority.GOOGLE:
success = await self._refresh_google_token(root_interface, connection)
elif connection.authority == AuthAuthority.MSFT:
success = await self._refresh_microsoft_token(root_interface, connection)
if success:
refreshed_count += 1
else:
failed_count += 1
return {
"refreshed": refreshed_count,
"failed": failed_count,
"rate_limited": rate_limited_count
}
```
**What it does:**
1. Gets all OAuth connections for the user
2. Filters to only expired connections
3. Checks rate limits for each connection
4. Attempts to refresh each expired token
5. Tracks success/failure/rate-limited counts
6. Returns summary of results
**Why batch processing?** Users might have multiple OAuth connections (e.g., both Microsoft and Google). This method refreshes all of them in one operation.
**proactive_refresh**
Proactively refreshes tokens that are about to expire (within 5 minutes).
```python
async def proactive_refresh(self, user_id: str) -> Dict[str, Any]:
connections = root_interface.getUserConnections(user_id)
current_time = getUtcTimestamp()
five_minutes = 5 * 60
for connection in connections:
# Only refresh active tokens that expire soon
if (connection.tokenStatus == 'active' and
connection.tokenExpiresAt and
connection.authority in [AuthAuthority.GOOGLE, AuthAuthority.MSFT]):
time_until_expiry = connection.tokenExpiresAt - current_time
if 0 < time_until_expiry <= five_minutes:
# Refresh this token
# ... (similar to refresh_expired_tokens)
```
**What it does:**
1. Gets all active OAuth connections
2. Checks which ones expire within 5 minutes
3. Refreshes those tokens proactively
4. Returns summary of results
**Why 5 minutes?** This is a safety margin. If a token expires in 5 minutes, we refresh it now to ensure it's still valid when the user needs it.
**Rate Limiting**
The service implements per-connection rate limiting:
```python
def _is_rate_limited(self, connection_id: str) -> bool:
now = getUtcTimestamp()
if connection_id not in self.rate_limit_map:
return False
# Remove attempts older than 1 hour
recent_attempts = [
attempt_time for attempt_time in self.rate_limit_map[connection_id]
if now - attempt_time < (self.refresh_window_minutes * 60)
]
self.rate_limit_map[connection_id] = recent_attempts
return len(recent_attempts) >= self.max_attempts_per_hour # Default: 3
```
**What it does:**
1. Tracks refresh attempts per connection
2. Uses a 1-hour sliding window
3. Limits to 3 attempts per hour per connection
4. Prevents OAuth provider rate limit exhaustion
**Why rate limiting?** OAuth providers (Microsoft, Google) have rate limits. If we refresh too frequently, we'll hit those limits and all refresh attempts will fail. Rate limiting prevents this.
#### Features
- **Rate Limiting**: Maximum 3 refresh attempts per hour per connection
- **Batch Processing**: Handles multiple connections in one operation
- **Status Tracking**: Tracks refreshed, failed, and rate-limited counts
- **Audit Logging**: Logs security events for compliance
- **Silent Operation**: Doesn't block requests (runs asynchronously)
#### Dependencies
- `tokenManager.TokenManager`: Token refresh logic
- `interfaceDbAppObjects`: Database access for connections and tokens
- `auditLogger`: Security event logging
---
### 5. tokenRefreshMiddleware.py - Automatic Token Refresh
#### What Does It Do?
The `tokenRefreshMiddleware.py` module provides FastAPI middleware that automatically triggers token refresh when users make requests. It runs silently in the background, so users never notice when their tokens are being refreshed.
#### Understanding Middleware
Middleware in FastAPI is code that runs before your route handlers. It can:
- Inspect requests
- Modify requests
- Perform background tasks
- Block requests (return errors)
In our case, the middleware intercepts requests, checks if tokens need refreshing, and triggers refresh operations in the background.
#### Key Classes Explained
**TokenRefreshMiddleware**
Refreshes expired tokens when specific endpoints are accessed.
```python
class TokenRefreshMiddleware(BaseHTTPMiddleware):
def __init__(self, app, enabled: bool = True):
super().__init__(app)
self.enabled = enabled
self.refresh_endpoints = {
'/api/connections',
'/api/files',
'/api/chat',
'/api/msft',
'/api/google'
}
async def dispatch(self, request: Request, call_next: Callable) -> Response:
if not self.enabled:
return await call_next(request)
# Check if this endpoint might need token refresh
if not self._should_check_tokens(request):
return await call_next(request)
# Extract user ID
user_id = self._extract_user_id(request)
if not user_id:
return await call_next(request)
# Trigger background refresh (non-blocking)
asyncio.create_task(self._silent_refresh_tokens(user_id))
# Continue with request
return await call_next(request)
```
**What it does:**
1. Checks if the request is to a monitored endpoint (one that might use OAuth tokens)
2. Extracts the user ID from the request
3. Triggers token refresh in the background (doesn't wait for it)
4. Continues processing the request immediately
**Why only specific endpoints?** Not all endpoints use OAuth tokens. We only check endpoints that are likely to need OAuth tokens (like `/api/msft` or `/api/google`).
**Why non-blocking?** Token refresh can take a few seconds. If we waited for it, every request would be slow. By running it in the background, requests complete immediately while tokens refresh silently.
**ProactiveTokenRefreshMiddleware**
Proactively refreshes tokens before they expire.
```python
class ProactiveTokenRefreshMiddleware(BaseHTTPMiddleware):
def __init__(self, app, enabled: bool = True, check_interval_minutes: int = 5):
super().__init__(app)
self.enabled = enabled
self.check_interval_minutes = check_interval_minutes
self.last_check = {} # Track last check time per user
async def dispatch(self, request: Request, call_next: Callable) -> Response:
user_id = self._extract_user_id(request)
if not user_id:
return await call_next(request)
# Check if we need to do proactive refresh (every 5 minutes)
if self._should_check_proactive_refresh(user_id):
asyncio.create_task(self._proactive_refresh_tokens(user_id))
self.last_check[user_id] = getUtcTimestamp()
return await call_next(request)
```
**What it does:**
1. Extracts user ID from request
2. Checks if it's been 5 minutes since last proactive check
3. If so, triggers proactive refresh in background
4. Updates last check time
5. Continues with request
**Why 5-minute interval?** We don't want to check on every request (that would be wasteful). Checking every 5 minutes is frequent enough to catch tokens before they expire, but not so frequent that it impacts performance.
#### Monitored Endpoints
The middleware only checks these endpoints:
- `/api/connections` - Managing OAuth connections
- `/api/files` - File operations that might use OAuth
- `/api/chat` - Chat features that might use OAuth
- `/api/msft` - Microsoft-specific operations
- `/api/google` - Google-specific operations
**Why these?** These endpoints are most likely to use OAuth tokens. Checking all endpoints would be wasteful.
#### Features
- **Asynchronous Background Refresh**: Doesn't block requests
- **Endpoint-Specific Triggering**: Only checks relevant endpoints
- **User ID Extraction**: Automatically extracts user ID from request context
- **Configurable Intervals**: Default 5 minutes for proactive refresh
- **Silent Operation**: Errors don't affect request processing
#### Dependencies
- `tokenRefreshService`: Refresh orchestration service
---
### 6. csrf.py - CSRF Protection
#### What Does It Do?
The `csrf.py` module provides CSRF (Cross-Site Request Forgery) protection middleware. It validates CSRF tokens for state-changing operations to prevent malicious websites from making requests on behalf of authenticated users.
#### Understanding CSRF Attacks
Imagine this scenario:
1. You're logged into the Gateway application
2. You visit a malicious website
3. The malicious website contains code that makes a request to the Gateway API (e.g., delete your account)
4. Your browser automatically includes your authentication cookies with the request
5. The Gateway API sees your valid authentication and executes the malicious request
**CSRF protection prevents this** by requiring a special token (CSRF token) that only the legitimate Gateway application knows. Malicious websites can't get this token, so their requests are rejected.
#### Key Features Explained
**Protected Methods**
Only state-changing HTTP methods are protected:
- **POST**: Creating new resources
- **PUT**: Updating existing resources
- **DELETE**: Deleting resources
- **PATCH**: Partial updates
**Why only these?** GET requests don't change data, so they're safe. Only requests that modify data need CSRF protection.
**Exempt Paths**
These paths are exempt from CSRF protection:
- `/api/local/login` - Login endpoint (users aren't authenticated yet)
- `/api/local/register` - Registration endpoint (users aren't authenticated yet)
- `/api/msft/login` - Microsoft OAuth login
- `/api/google/login` - Google OAuth login
- `/api/msft/callback` - Microsoft OAuth callback
- `/api/google/callback` - Google OAuth callback
**Why exempt?** These endpoints either don't require authentication (login/register) or are called by OAuth providers (callbacks). CSRF protection isn't needed or would interfere with OAuth flows.
**Token Validation**
The middleware checks for a CSRF token in the `X-CSRF-Token` header:
```python
async def dispatch(self, request: Request, call_next):
# Skip CSRF check for exempt paths
if request.url.path in self.exempt_paths:
return await call_next(request)
# Skip CSRF check for non-state-changing methods
if request.method not in self.protected_methods:
return await call_next(request)
# Skip OPTIONS requests (CORS preflight)
if request.method == "OPTIONS":
return await call_next(request)
# Get CSRF token from header
csrf_token = request.headers.get("X-CSRF-Token")
if not csrf_token:
return JSONResponse(status_code=403, content={"detail": "CSRF token missing"})
# Validate token format
if not self._is_valid_csrf_token(csrf_token):
return JSONResponse(status_code=403, content={"detail": "Invalid CSRF token format"})
return await call_next(request)
```
**Token Format Validation**
The middleware validates that the CSRF token has a valid format:
```python
def _is_valid_csrf_token(self, token: str) -> bool:
if not token or not isinstance(token, str):
return False
# Length validation (16-64 characters)
if len(token) < 16 or len(token) > 64:
return False
# Must be a valid hex string
try:
int(token, 16)
return True
except ValueError:
return False
```
**Why format validation?** This is a basic check to ensure the token looks valid. More sophisticated validation (like checking against a session) could be added, but format validation prevents obviously invalid tokens.
#### How It Works
1. **Request Arrives**: User makes a POST/PUT/DELETE/PATCH request
2. **Path Check**: Is this an exempt path? If yes, skip CSRF check
3. **Method Check**: Is this a protected method? If no, skip CSRF check
4. **Token Extraction**: Get CSRF token from `X-CSRF-Token` header
5. **Token Validation**: Check if token exists and has valid format
6. **Request Processing**: If valid, continue. If invalid, return 403 Forbidden
#### Features
- **State-Changing Methods Only**: Only protects POST, PUT, DELETE, PATCH
- **Exempt Paths**: Login, registration, and OAuth callbacks are exempt
- **Token Format Validation**: Basic validation prevents malformed tokens
- **Header-Based**: Uses standard `X-CSRF-Token` header
- **CORS-Aware**: Skips OPTIONS requests (CORS preflight)
---
## How Components Work Together
### The Complete Picture
Now that we understand each component individually, let's see how they work together to provide comprehensive security.
### Request Processing Flow
Here's what happens when a user makes a request:
```mermaid
sequenceDiagram
participant Client
participant CSRF as CSRF Middleware
participant TokenMw as Token Refresh Middleware
participant Auth as Authentication (auth.py)
participant JWT as JWT Service
participant Route as Route Handler
Client->>CSRF: HTTP Request
(POST /api/data)
alt State-Changing Request
CSRF->>CSRF: Validate X-CSRF-Token
alt Invalid Token
CSRF-->>Client: 403 Forbidden
else Valid Token
CSRF->>TokenMw: Continue
end
else Read-Only Request
CSRF->>TokenMw: Continue
end
TokenMw->>TokenMw: Check if endpoint needs refresh
alt Needs Refresh Check
TokenMw->>TokenMw: Extract User ID
TokenMw->>TokenMw: Trigger background refresh
(async, non-blocking)
end
TokenMw->>Auth: Continue
Auth->>Auth: Extract token from cookie/header
Auth->>JWT: Decode token (via jose library)
JWT-->>Auth: Token payload
Auth->>Auth: Validate signature, expiration
Auth->>Auth: Lookup user in database
Auth->>Auth: Validate user status & context
alt Authentication Failed
Auth-->>Client: 401 Unauthorized
else Authentication Succeeded
Auth->>Route: Request + User Object
Route->>Route: Process request
Route-->>Client: Response
end
```
### Token Refresh Flow
When tokens need to be refreshed:
```mermaid
sequenceDiagram
participant Request as API Request
participant Middleware as TokenRefreshMiddleware
participant Service as TokenRefreshService
participant TokenMgr as TokenManager
participant Provider as OAuth Provider
participant Interface as Interface Layer
participant DB as Database
Request->>Middleware: HTTP Request
(to /api/connections)
Middleware->>Middleware: Extract User ID
Middleware->>Middleware: Check Endpoint
(should refresh?)
Middleware->>Service: refresh_expired_tokens(userId)
(async, background)
Service->>Interface: getUserConnections(userId)
Interface->>DB: Query Connections
DB-->>Interface: Connection Records
Interface-->>Service: Connections List
loop For Each Expired Connection
Service->>Service: Check Rate Limit
alt Rate Limited
Service->>Service: Skip Connection
else Not Rate Limited
Service->>Interface: getConnectionToken(connectionId)
Interface->>DB: Query Token
DB-->>Interface: Token Record
Interface-->>Service: Token Object
Service->>TokenMgr: refreshToken(token)
alt Microsoft Token
TokenMgr->>Provider: POST /token
(refresh_token grant)
Provider-->>TokenMgr: New Access Token
else Google Token
TokenMgr->>Provider: POST /token
(refresh_token grant)
Provider-->>TokenMgr: New Access Token
end
TokenMgr->>TokenMgr: Create New Token Object
TokenMgr-->>Service: Refreshed Token
Service->>Interface: saveConnectionToken(token)
Interface->>DB: Update Token Record
Service->>Interface: Update Connection Status
Interface->>DB: Update Connection
end
end
Service-->>Middleware: Refresh Results
(refreshed, failed, rate_limited)
Note over Request,DB: Request continues normally
(non-blocking)
Request->>Request: Process Request
Request-->>Request: Return Response
```
### Middleware Stack Order
The middleware is registered in `app.py` in this order (execution order is reverse):
1. **CORS Middleware** (FastAPI built-in) - Handles cross-origin requests
2. **CSRF Middleware** - Validates CSRF tokens
3. **TokenRefreshMiddleware** - Refreshes expired tokens
4. **ProactiveTokenRefreshMiddleware** - Proactively refreshes tokens
**Execution Flow:**
```
Request → ProactiveTokenRefreshMiddleware → TokenRefreshMiddleware → CSRFMiddleware → CORS → Route Handler
```
Each middleware processes the request in sequence. If any middleware rejects the request (e.g., CSRF validation fails), the request stops and an error is returned.
### Component Dependencies
**auth.py** depends on:
- `jwtService.py` - For JWT decoding (via jose library)
- `interfaceDbAppObjects` - For user and token database access
- `datamodelUam.User` - User model
- `datamodelSecurity.Token` - Token model
- `slowapi.Limiter` - Rate limiting utility (exported for use in routes)
**tokenManager.py** depends on:
- `httpx` - HTTP client for OAuth API calls
- `interfaceDbAppObjects` - For token persistence (via callback)
- `datamodelSecurity.Token` - Token model
**tokenRefreshService.py** depends on:
- `tokenManager.TokenManager` - Token refresh logic
- `interfaceDbAppObjects` - For connection and token access
- `auditLogger` - Security event logging
**tokenRefreshMiddleware.py** depends on:
- `tokenRefreshService` - Refresh orchestration service
### Interface Layer Integration
Security components interact with the interface layer for:
1. **User Lookup**: `interface.getUserByUsername(username)` - Retrieves user by username
2. **Token Management**:
- `interface.findActiveTokenById()` - Validates LOCAL tokens against database
- `interface.saveConnectionToken()` - Persists refreshed OAuth tokens
3. **Connection Management**:
- `interface.getUserConnections()` - Gets all connections for a user
- `interface.getConnectionToken()` - Retrieves token for a connection
**Abstraction Benefits:**
- Security components don't need database connection details
- User context (mandateId) is automatically handled by interfaces
- Consistent data access patterns across the application
- Easier testing with mock interfaces
## Security Patterns & Best Practices
### 1. Defense in Depth
Multiple layers of security validation ensure that if one layer fails, others still protect the system:
1. **Middleware Layer**: CSRF protection, token refresh
2. **Authentication Layer**: JWT validation, user verification
3. **Database Layer**: Token status tracking, user status checks
4. **Route Layer**: Explicit authentication dependencies
**Why?** No single security measure is perfect. Multiple layers provide redundancy and make attacks much harder.
### 2. Token Security
- **httpOnly Cookies**: Prevents XSS attacks from accessing tokens
- **Secure Flag**: Automatically enabled for HTTPS environments
- **SameSite=Strict**: Prevents CSRF attacks
- **JWT Expiration**: Short-lived access tokens (configurable, default 60 minutes)
- **Refresh Tokens**: Longer-lived tokens stored securely (default 7 days)
- **Token Revocation**: Database-backed revocation for LOCAL tokens
**Why?** Tokens are sensitive. These measures ensure they're stored and transmitted securely.
### 3. OAuth Token Management
- **Automatic Refresh**: Background refresh prevents user disruption
- **Rate Limiting**: Prevents OAuth provider exhaustion (max 3 attempts per hour)
- **Cooldown Period**: Prevents excessive refresh attempts (10-minute minimum)
- **Proactive Refresh**: Refreshes before expiration (30-minute threshold)
- **Error Handling**: Graceful degradation on refresh failures
**Why?** OAuth tokens expire frequently. Automatic refresh ensures users don't get interrupted, while rate limiting prevents hitting provider limits.
### 4. User Context Validation
- **Mandate Scoping**: Ensures user operates within correct mandate
- **User ID Validation**: Verifies token user ID matches database
- **Status Checks**: Validates user is enabled and active
- **Context Mismatch Detection**: Forces re-authentication on context changes
**Why?** Users can belong to multiple organizations (mandates). Context validation ensures they can only access data for their current mandate.
### 5. CSRF Protection
- **State-Changing Methods**: Only protects POST, PUT, DELETE, PATCH
- **Exempt Paths**: Login and OAuth endpoints exempted
- **Token Format Validation**: Basic validation prevents malformed tokens
- **Header-Based**: Uses X-CSRF-Token header (standard pattern)
**Why?** CSRF attacks are common. This protection prevents malicious websites from making requests on behalf of authenticated users.
### 6. Error Handling
- **Consistent Error Responses**: Standard HTTP status codes
- **Security-Aware Logging**: Logs security events without exposing sensitive data
- **Graceful Degradation**: Token refresh failures don't block requests
- **Audit Logging**: Security events logged for compliance
**Why?** Proper error handling prevents information leakage and ensures the system degrades gracefully when things go wrong.
## Configuration
### Required Environment Variables
```ini
# JWT Configuration
APP_JWT_KEY_SECRET=
Auth_ALGORITHM=HS256
APP_TOKEN_EXPIRY=60 # minutes
APP_REFRESH_TOKEN_EXPIRY=7 # days
# API Configuration
APP_API_URL=https://api.example.com # Determines secure cookie flag
# OAuth Configuration (for token refresh)
Service_MSFT_CLIENT_ID=
Service_MSFT_CLIENT_SECRET=
Service_MSFT_TENANT_ID=
Service_GOOGLE_CLIENT_ID=
Service_GOOGLE_CLIENT_SECRET=
```
### Middleware Configuration
In `app.py`:
```python
# CSRF protection
app.add_middleware(CSRFMiddleware)
# Token refresh middleware
app.add_middleware(TokenRefreshMiddleware, enabled=True)
# Proactive token refresh
app.add_middleware(
ProactiveTokenRefreshMiddleware,
enabled=True,
check_interval_minutes=5
)
```
## Security Considerations
### Best Practices Implemented
1. **Never Log Tokens**: Tokens are never logged in plaintext
2. **Secure Cookie Configuration**: httpOnly, secure (HTTPS), SameSite=Strict
3. **Token Expiration**: Short-lived access tokens reduce attack window
4. **Database Validation**: LOCAL tokens validated against database for revocation
5. **Rate Limiting**: Prevents brute force and DoS attacks
6. **Context Validation**: Prevents token reuse across mandates/users
7. **Error Messages**: Generic error messages don't leak information
### Potential Improvements
1. **CSRF Token Storage**: Currently validates format only; could add session-based validation
2. **Token Rotation**: Could implement refresh token rotation for enhanced security
3. **IP Validation**: Could add IP address validation for token usage
4. **Device Fingerprinting**: Could track devices for additional security
5. **MFA Support**: Could add multi-factor authentication support
## Troubleshooting
### Common Issues
1. **401 Unauthorized**:
- Check token expiration
- Verify user status (enabled?)
- Check for context mismatch (mandateId/userId changes)
- For LOCAL tokens, verify token exists in database and is active
2. **403 Forbidden**:
- Check CSRF token header (`X-CSRF-Token`)
- Verify user enabled status
- Check if path is exempt from CSRF protection
3. **Token Refresh Failures**:
- Check OAuth provider credentials
- Verify rate limits haven't been exceeded
- Check OAuth provider status
- Review logs for specific error messages
4. **Cookie Not Set**:
- Verify HTTPS in production (cookies require secure flag)
- Check SameSite settings
- Verify cookie path settings
### Debugging
Enable debug logging:
```python
import logging
logging.getLogger("modules.security").setLevel(logging.DEBUG)
```
Check audit logs for security events:
```python
# Security events are logged via audit_logger.logSecurityEvent()
```
## Related Documentation
- [Security API Documentation](./security-api.md) - API endpoints and route usage examples
- [Architecture Overview](./architecture-overview.md) - Overall system architecture
- [Data Models](../datamodels/) - User, Token, and Connection models
- [Interfaces](../interfaces/) - Data access layer documentation