Simplify AI agent orchestration with Lakebase Postgres

Traditionally, the auditing process has been characterized by its labor-intensive nature, often requiring extensive document reviews and meticulous information extraction. In a bid to streamline this process, CLA (CliftonLarsonAllen LLP), a prominent professional services firm with an expanding global footprint, collaborated with the Databricks Forward Deployed Engineering team to create an innovative auditing solution. This partnership culminated in the development of a document processing application that significantly reduces extraction time from hours to mere minutes, all while maintaining high-quality standards. The application is entirely constructed on the Databricks platform, leveraging technologies such as Lakebase Postgres, Databricks Apps, Lakeflow Jobs, MLflow, and Unity Catalog Volumes. This article delves into a crucial component of this system: the Lakebase-powered orchestration layer.

Orchestration Challenges for Agentic Workloads

Document parsing is a prevalent and high-volume agentic workload, as organizations across various sectors often need to convert substantial quantities of contracts, invoices, financial filings, and other documents into structured data. Executing this at scale presents five distinct challenges inherent to distributed systems:

• Unpredictable per-task latency: Processing times can vary dramatically, with a two-page invoice taking seconds while a two-hundred-page contract may require several minutes, complicating task duration predictions.
• Rate-limit-aware throttling: LLM and vision model endpoints impose limits on the number of requests and tokens processed within a specific timeframe. Sending numerous tasks simultaneously can overwhelm these limits, resulting in throttling and repeated retries. The orchestrator must proactively manage in-flight work to avoid these issues.
• Workload prioritization: Urgent submissions should not be delayed by large bulk batches. Implementing per-task priority ensures that higher-priority work is processed first.
• Cost attribution per task: Financial teams require the ability to attribute expenditures to specific tasks, customers, and agents, detailing AI token usage and compute consumption.
• Real-time progress visibility: Users uploading numerous documents need access to live progress updates.

Many organizations resort to combining several specialized systems for orchestration and observability, each introducing its own infrastructure and operational complexities. For long-running, independent agentic tasks, this overhead often outweighs the actual scheduling complexity. The Databricks-native solution we developed effectively addresses all these requirements, with Lakebase serving as the foundational element.

Solution Architecture

The complete application stack is built exclusively on Databricks services:

• Web Application (Databricks Apps): A FastAPI-based user interface allows users to upload PDFs (stored in Unity Catalog Volumes) and submit parsing requests, which are recorded directly in the Lakebase task table.
• Lakebase: An autoscaling Postgres database that maintains the orchestrator’s relational state across various tables, including tasks (documents to be parsed, with status and structured results) and task_attempts (tracking execution attempts and associated metadata). Lakebase acts as the single source of truth for the orchestrator’s state.
• Orchestrator (Databricks Apps): A long-running worker daemon and operator dashboard that dequeues tasks from Lakebase, dispatches them to the AI Agents layer, and writes results back. The dashboard provides real-time status updates by reading from the same tables.
• AI Agents (Lakeflow Jobs): Lakeflow Jobs perform the parsing tasks, reading PDFs from Unity Catalog Volumes, processing them through Intelligent Document Processing and vision/LLM calls, storing parsed outputs in Lakebase, and acknowledging the orchestrator via webhook. MLflow Tracing captures execution details, including model calls, token usage, latency, and cost metadata.

Data flows seamlessly between these components: the Web App writes PDFs to Unity Catalog Volumes and parsing requests to Lakebase; the Orchestrator dequeues tasks and dispatches Databricks Jobs to the AI Agents layer; and the AI Agents process documents, returning results to Lakebase while updating the Orchestrator on their status. This architecture eliminates the need for external message brokers, separate schedulers, or dedicated caching layers.

Task Queue Implementation

The task queue is underpinned by two Postgres tables in Lakebase. The tasks table records each logical unit of work, including the task’s status, lease information, agent assignment, and results. The task_attempts table captures execution attempts, including the Databricks Job run ID and cost metadata. This parent-child relationship supports retries and preserves observability for cost attribution and debugging.

Concurrent Priority Aware Dequeuing

A basic dequeue query might select the next available task using WHERE status = ‘enqueued’ and LIMIT batch_size. However, this approach is inadequate for concurrent workers without row locking, which could lead to multiple workers selecting the same task. Implementing FOR UPDATE SKIP LOCKED ensures concurrency safety, allowing each worker to lock the row it selects while others skip it. Additionally, an ORDER BY priority DESC, created_at clause guarantees that higher-priority tasks are processed first, maintaining FIFO order within each priority level.

Crash Recovery via Lease-Based Locking

Workers may terminate unexpectedly due to various conditions. To prevent tasks from being indefinitely marked as processing, an expiring lease is recorded at dequeue time. A periodic sweeper re-enqueues any task whose lease has expired, allowing for automatic recovery of tasks held by terminated workers within minutes.

Rate-Limit-Aware Throttling

LLM and vision model endpoints typically enforce two distinct quotas: requests per second and tokens per minute. The orchestrator supports three throttling modes, configurable per agent:

• Concurrency cap: A MAX_CONCURRENT_TASKS parameter limits the number of concurrently dispatched tasks, ensuring accurate counts across worker restarts and deployments.
• Token budget: A MAX_TPM parameter restricts the projected token rate across in-flight tasks, allowing new tasks to be dequeued only if they fit within the budget.
• Combined cap: When both MAX_CONCURRENT_TASKS and MAX_TPM are configured, the orchestrator applies the tighter constraint.

Throttling decisions are made at dequeue time, ensuring that tasks unable to fit within the current quota remain in the queue for reconsideration in the next cycle.

Idempotent Webhook Callbacks

Upon task completion, the AI Agents layer posts a callback to the orchestrator. The callback handler is designed to be idempotent, accepting both PROCESSING and ENQUEUED states, thus preventing double-billing or duplicate processing.

These four patterns collectively yield a task queue that is robust against concurrency issues, resilient to crashes, aware of rate limits, and idempotent under retries. Real-time visibility into the system is provided through an integrated dashboard.

Real-Time Operator Dashboard

As numerous documents are processed, operators require a clear view of agent performance, task status, and workload costs. The orchestrator integrates this functionality into a single dashboard accessible through the same Databricks App that runs the worker daemon.

Dashboard Capabilities

The operator-facing dashboard presents a suite of operational metrics that characterize the running system, with filtering options for date range, task status, and agent:

• Task totals by status: Real-time counts of tasks in various states (enqueued, processing, completed, failed, cancelled).
• Input and output tokens: Token counts aggregated from MLflow Traces.
• LLM cost: Estimates from MLflow Traces, available shortly after each model call.
• Compute cost: Serverless Jobs compute costs attributed to the orchestrator’s task runs.
• Median response time: Calculated across completed tasks to avoid distortion from outliers.
• Confidence: Per-document confidence scores from the AI Agents layer, displayed alongside task results.

Implementation

State changes in the tasks table trigger Postgres LISTEN/NOTIFY events. The backend maintains a single LISTEN connection, distributing events over Server-Sent Events (SSE) to connected dashboard clients. Browsers establish an EventSource connection to receive live updates within approximately one second of state changes. Polling serves as a fallback at a ten-second interval, ensuring the dashboard remains current even in cases of dropped connections.

The dashboard’s data is sourced from three different latency characteristics: Postgres (instant), MLflow’s trace API (sub-second), and system billing tables (potentially tens of seconds). Fast queries feed every refresh cycle, while slower queries execute only upon user action, returning optimistically until results are available.

Per-Application Cost Attribution

Databricks system billing tables are account-scoped, meaning every job and model call contributes to the same system.billing.usage rows. To isolate application-level costs, the orchestrator tracks which Databricks Job runs it submitted, filtering billing queries accordingly. This architecture allows for easy monitoring and reporting of costs, enabling operators to analyze expenditures based on date range, task status, or agent.

Lakebase as the Orchestration Backbone

Postgres-as-queue patterns are well-established within the data engineering community, and Lakebase enhances these patterns with operational characteristics that make them viable for production architectures on Databricks:

• Autoscaling compute: Lakebase adjusts Postgres compute units based on workload, optimizing costs.
• OAuth-rotated authentication: Utilizing short-lived OAuth tokens for connection authentication eliminates the need for static credentials.
• Unity Catalog integration: Lakebase shares identity and governance with the rest of Databricks, simplifying permissions management.
• Branching and snapshots: Cloning production task tables for debugging is natively supported.

These capabilities mitigate the operational burdens that often lead teams to adopt managed message brokers in lieu of self-hosted Postgres for task queueing.