gateway/modules/features/chatbot

Chatbot Feature Documentation

Overview

The chatbot feature provides an intelligent conversational interface that processes user queries, executes database searches, performs web research, and generates contextual responses. The implementation leverages LangGraph to orchestrate complex multi-step workflows while seamlessly integrating with the existing infrastructure including AI Center, database systems, and event streaming.

Architecture

The chatbot feature follows a modular architecture centered around LangGraph's state graph pattern. The system processes user messages through a structured workflow that can dynamically invoke tools, query databases, search the web, and generate responses based on context.

Core Components

Workflow Management: Each conversation is managed as a workflow with a unique identifier. Workflows track the conversation state, message history, and processing status. New conversations create fresh workflows, while existing conversations resume their workflows with incremented round numbers.

LangGraph State Graph: The heart of the chatbot is a LangGraph state graph that orchestrates the conversation flow. The graph maintains conversation state through a checkpointer system and routes between agent processing and tool execution nodes based on the model's decisions.

Event Streaming: Real-time updates are delivered to clients through an event-driven streaming system. Status updates, messages, logs, and completion events are emitted asynchronously and queued for delivery to connected clients.

LangGraph Implementation

State Management

LangGraph manages conversation state through a state graph that tracks messages in the conversation. The state is persisted using a custom checkpointer that bridges LangGraph's checkpoint system with the existing database infrastructure. This allows conversations to be resumed, state to be maintained across sessions, and message history to be preserved.

Graph Structure

The workflow graph consists of two primary nodes:

Agent Node: Processes user messages and conversation history using the AI model. The agent analyzes the input, determines what actions are needed, and decides whether to generate a response directly or invoke tools. The agent has access to the full conversation history, which is automatically trimmed to fit within the model's context window while preserving the most recent and relevant messages.

Tools Node: Executes tools when the agent determines they are needed. Tools can query databases, search the web, or send status updates. After tool execution, the workflow returns to the agent node to process the tool results and generate an appropriate response.

Conditional Routing

The graph uses conditional edges to determine workflow progression. After the agent processes a message, the system checks whether the agent requested tool calls. If tools were requested, the workflow routes to the tools node. If no tools are needed, the workflow completes and returns the final response to the user.

Message Window Management

To handle long conversations that exceed model context limits, the system implements intelligent message windowing. Messages are trimmed from the beginning while preserving system prompts and ensuring the conversation ends on a human or tool message. This maintains context continuity while respecting token limits.

Integration with Existing Infrastructure

AI Center Integration

The chatbot integrates with the AI Center through a custom bridge that implements LangChain's chat model interface. This bridge allows LangGraph to use AI Center's model selection, routing, and calling infrastructure while maintaining compatibility with LangChain's expected interfaces.

Model Selection: When processing messages, the bridge converts LangChain message formats to AI Center's expected format and uses the model selector to choose the appropriate AI model based on operation type, processing mode, and available models. The selection respects role-based access control and considers model capabilities.

Tool Calling Support: The bridge handles tool calling by detecting when models support function calling and converting tool definitions between LangChain and AI Center formats. For OpenAI-compatible models, the bridge directly calls the API with tool definitions. For other models, it relies on connector-specific implementations.

Operation Types: The chatbot uses AI Center's operation type system to select models appropriate for different tasks. Database queries use data analysis operation types, while web searches use web search operation types, ensuring optimal model selection for each task.

Database Integration

Message Storage: All conversation messages are stored in the existing chat database through the database interface. The custom checkpointer converts between LangGraph's message format and the database's message format, ensuring seamless persistence. Messages are stored with metadata including workflow identifiers, round numbers, sequence numbers, and timestamps.

Workflow Persistence: Workflow state is maintained in the database, allowing conversations to be resumed across sessions. The system tracks workflow status, current round numbers, and activity timestamps. When resuming a conversation, the workflow round number is incremented to maintain conversation continuity.

Document Management: User-uploaded files are tracked as document references within workflows. The system creates document records that link files to specific messages and rounds, enabling the chatbot to reference and process uploaded documents in its responses.

Tool Integration

SQL Query Tool: The chatbot includes a tool that executes SQL queries against the preprocessor database. This tool uses the existing database connector infrastructure, ensuring proper connection management, query execution, and result formatting. The tool returns formatted results that the agent can use to answer user questions about products, inventory, prices, and other database-stored information.

Web Search Tool: Web research capabilities are provided through a Tavily search tool that integrates with AI Center's Tavily connector. The tool uses AI Center's model registry and selector to find and use Tavily models, ensuring consistent integration with the existing AI infrastructure. Search results include full content from multiple sources, allowing comprehensive research.

Streaming Status Tool: A special tool allows the agent to send status updates during processing. These updates are captured by the event streaming system and delivered to clients in real-time, providing users with visibility into what the chatbot is doing.

Event Streaming System

The chatbot uses an event-driven streaming architecture to deliver real-time updates to clients. An event manager maintains queues for each workflow, allowing multiple clients to receive updates for the same conversation.

Event Types: The system emits several types of events including chat data events (messages and logs), completion events, and error events. Each event includes metadata about its type, timestamp, and associated workflow.

Queue Management: Event queues are created when workflows start and cleaned up after conversations complete. The cleanup system ensures resources are properly released while allowing sufficient time for clients to receive all events.

Event Bridging: LangGraph's native event streaming is bridged to the custom event system. Status updates from tool calls are captured and converted to the appropriate event format. Final responses are extracted from LangGraph's output and emitted as message events.

Configuration System

The chatbot supports multiple configuration profiles loaded from JSON files. Each configuration specifies:

System Prompts: Customizable instructions that define the chatbot's behavior, personality, and capabilities. Prompts can include placeholders for dynamic content like dates.

Database Schema: Information about available database tables and structures, enabling the agent to generate appropriate queries.

Tool Configuration: Settings for which tools are enabled and how they should behave. This includes SQL query settings, web search parameters, and streaming options.

Model Configuration: Operation types and processing modes that determine which AI models are selected for different tasks.

Conversation Flow

Initial Request Processing

When a user submits a message, the system first creates or loads the workflow. For new conversations, a conversation name is generated using AI based on the user's initial prompt. The user's message is stored in the database and an event is emitted to notify connected clients.

Background Processing

Message processing occurs asynchronously in the background, allowing the API to return immediately while processing continues. The system creates a LangGraph chatbot instance configured with the appropriate model, memory checkpointer, and tools.

Tool Execution

When the agent determines that tools are needed, it requests tool calls. The tools node executes the requested tools, which may involve database queries, web searches, or status updates. Tool results are added to the conversation state and returned to the agent for processing.

Response Generation

After tool execution or when no tools are needed, the agent generates a final response based on the conversation history and any tool results. The response is stored in the database through the checkpointer system and emitted as an event to connected clients.

Completion

Once processing completes, a completion event is emitted and the workflow status is updated. The event queue remains available for a grace period to ensure all clients receive the final events before cleanup.

Error Handling

The system includes comprehensive error handling at multiple levels. Workflow errors are caught and stored as error messages in the database. Error events are emitted to notify clients of failures. The system gracefully handles cases where workflows are stopped by users, preventing unnecessary error messages from being stored.

Memory and Context Management

The custom checkpointer bridges LangGraph's checkpoint system with the database, ensuring conversation history is preserved. The system intelligently filters messages when storing checkpoints, skipping intermediate tool call requests and only storing final user and assistant messages. This prevents duplicate storage while maintaining complete conversation context.

Multi-Language Support

The system supports multiple languages through configuration. Conversation names are generated in the user's preferred language, and the AI models can process and respond in various languages based on the system prompt and user input.

Scalability Considerations

The asynchronous processing model allows the system to handle multiple concurrent conversations efficiently. Each workflow operates independently with its own event queue and processing task. The database checkpointer ensures state persistence without blocking processing, and the event streaming system efficiently manages multiple client connections per workflow.