struggles

Google Cloud has enhanced AlloyDB by integrating AlloyDB AI Functions with Gemini models to improve search indexing for logographic languages like Chinese, Japanese, and Korean. This update addresses challenges in database search functionality, particularly for languages written in continuous scripts, where traditional PostgreSQL text parsing struggles. Developers can now invoke Gemini models directly from SQL to facilitate word segmentation and stop-word removal before creating full-text indexes, offering a more efficient alternative to traditional methods. The architecture maintains raw text, segmented text, search vectors, and embeddings in a single table, with generated columns ensuring automatic updates to search vectors and embeddings. A batching process using a PL/pgSQL stored procedure allows for efficient processing of extensive document collections by aggregating rows into arrays for model requests, thus reducing row-locking issues. Google Cloud provides tailored text search configurations for different datasets, suggesting a simple configuration for solely Chinese text and an English configuration for bilingual datasets. This preprocessing logic can also be applied during query execution, allowing user searches to be directed to Gemini for text segmentation and stop-word elimination. AlloyDB supports RUM indexes on search vectors, which store lexeme positions for computing relevance and word distance, enabling rapid query responses. Additionally, a hybrid search methodology combines full-text and vector search using a ScaNN index on embeddings, merging results through Reciprocal Rank Fusion. This development allows for robust multilingual search and vector retrieval solutions within the database environment, eliminating the need for external microservices.