Google Cloud has introduced a significant enhancement to AlloyDB, enabling the integration of AlloyDB AI Functions with Gemini models to refine search indexing for logographic languages such as Chinese, Japanese, and Korean. This advancement specifically addresses the challenges associated with full-text and hybrid search workloads within managed PostgreSQL environments.
Addressing Database Search Challenges
The update tackles a persistent issue in database search functionality. Traditional PostgreSQL text parsing methods typically rely on the assumption that words are separated by spaces. This poses particular difficulties for languages that are written in a continuous script, where punctuation serves as the primary means of delineation.
In such scenarios, a database might interpret an entire sentence as a single lexeme, hindering users from retrieving specific phrases or nouns unless they input the complete sentence verbatim. With the new capabilities of AlloyDB, developers can now invoke Gemini models directly from SQL through in-database AI Functions, facilitating word segmentation and stop-word removal prior to the creation of full-text indexes. This model-driven approach is presented as a more efficient alternative to traditional third-party PostgreSQL extensions and external preprocessing services.
In-database Flow
The proposed architecture maintains raw text, segmented text, search vectors, and embeddings within a single table. In Google Cloud’s implementation, generated columns ensure that any updates to the segmented text automatically refresh both the full-text search vector and the vector embedding in AlloyDB.
For extensive document collections, Google Cloud has outlined a batching process utilizing a PL/pgSQL stored procedure. Instead of processing one model request per row, this procedure aggregates rows into arrays, submits a batched request to Gemini, unpacks the returned outputs using GENERATE_SERIES, updates the target records, and commits each batch promptly. This approach is designed to mitigate row-locking issues and reduce rollback risks associated with lengthy update operations, while also addressing potential alignment discrepancies that may arise from parallel cursor streams.
An illustrative example from the documentation features the Chinese sentence “你们研究所有十个图书馆”. Without segmentation, PostgreSQL would index this entire string as a single entity. However, after preprocessing with Gemini, the sentence is reformatted with spaces between meaningful words, allowing terms like “研究所” and “图书馆” to be effectively searchable.
Search Options
Google Cloud has also detailed how text search configurations should be tailored to the specific dataset. For databases containing solely Chinese text, developers might opt for PostgreSQL’s simple configuration, which minimally alters the segmented output aside from converting it to lowercase.
Conversely, for bilingual datasets that incorporate both Chinese and English elements, such as product names or technical terms, the english configuration may prove more advantageous. This setup allows for the normalization of English words and the removal of common English stop words while preserving Chinese characters as exact lexemes.
This preprocessing logic can also be applied during query execution. A user search can be directed to Gemini from SQL, with specific instructions for text segmentation and the elimination of low-value grammatical words before the database translates the output into a text search query.
RUM and Vectors
In terms of ranking, Google Cloud highlighted AlloyDB’s compatibility with RUM indexes on search vectors. Unlike standard GIN indexes, RUM indexes store lexeme positions directly, enabling the database to compute relevance and word distance within the index.
In the provided example, a query transforms a segmented search string into a PostgreSQL tsquery, filters matching rows, and organizes them by distance. Google Cloud asserts that this method can yield relevant matches in mere milliseconds.
The company also described a hybrid search methodology that merges full-text search with vector search. In this framework, a ScaNN index is established on an embedding column generated from the multilingual Gemini embedding model, and the outcomes from both text and vector retrieval are integrated through Reciprocal Rank Fusion.
The SQL pattern employs distinct common table expressions for vector and text retrieval, subsequently merging the ranked lists with a full outer join and calculating a final score based on each document’s ranking across both result sets. This strategy aims to highlight both precise keyword matches and semantically related documents.
The overarching theme is that increased AI processing capabilities can now be executed within the database environment, rather than relying on external application layers. This development is likely to resonate with organizations seeking robust multilingual search and vector retrieval solutions without the burden of managing separate search infrastructures or transferring data outside of transactional systems.
Developers can now construct bilingual and hybrid search workflows in AlloyDB using standard SQL and stored procedures, eliminating the need for external microservices, as emphasized by Google Cloud.
