Vision

The auditing process has traditionally been labor-intensive, requiring extensive document reviews. CLA (CliftonLarsonAllen LLP) partnered with the Databricks Forward Deployed Engineering team to develop a document processing application that reduces extraction time from hours to minutes while maintaining quality. The application is built on the Databricks platform, utilizing technologies like Lakebase Postgres, Databricks Apps, Lakeflow Jobs, MLflow, and Unity Catalog Volumes. The application addresses challenges in document parsing, including unpredictable per-task latency, rate-limit-aware throttling, workload prioritization, cost attribution per task, and real-time progress visibility. It eliminates the need for multiple specialized systems by integrating components such as a web application for user interaction, Lakebase for state management, an orchestrator for task management, and AI agents for document processing. The task queue is supported by two Postgres tables in Lakebase, tracking task status and execution attempts. Concurrency safety is ensured through a locking mechanism, and crash recovery is facilitated by lease-based locking. The orchestrator implements rate-limit-aware throttling and idempotent webhook callbacks to manage task processing effectively. An operator dashboard provides real-time metrics on task status, agent performance, and workload costs, with live updates through Postgres LISTEN/NOTIFY events. Cost attribution is managed at the application level, allowing for detailed monitoring of expenditures. Lakebase enhances the orchestration process with features like autoscaling compute, OAuth-rotated authentication, Unity Catalog integration, and support for branching and snapshots.