natural language processing

Organizations are using generative AI and machine learning tools to extract value from unstructured data, employing specialized machine learning models to convert data into vectors for efficient searching and retrieval. Amazon Aurora PostgreSQL-Compatible Edition, with the pgvector extension, allows developers to create a vector store for generative AI applications. Multi-tenant architectures in software-as-a-service (SaaS) applications ensure tenant data isolation to maintain security and privacy. In a multi-tenant scenario, home survey requests are processed, and survey reports are stored in Amazon S3. Embedding models convert these documents into vector embeddings, which are stored in a vector store. There are two approaches to building a vector-based application: a self-managed approach, where developers handle data processing and queries, and a fully-managed option using Amazon Bedrock Knowledge Bases. The self-managed approach involves coding to process data, generate vector embeddings, and store them in Aurora. The architecture includes data ingestion via AWS Lambda, embedding conversion, SQL queries for data retrieval, and response enhancement using a large language model (LLM). Key terminologies include vector embeddings, embedding models, chunking, pgvector, HNSW index, and IVFFlat index. The process of creating a vector store involves configuring the Aurora PostgreSQL database, enabling the pgvector extension, and establishing a schema for the vector store. The vector table includes fields for UUID, vector embeddings, text chunks, metadata, and tenant identification. Data ingestion requires coding to convert data into embeddings and insert them into the database. Retrieval of vector data is crucial for enhancing prompts sent to LLMs, with various distance functions supported for identifying similar vectors. Multi-tenancy is enforced using row-level security in PostgreSQL, ensuring tenant data isolation. The process includes updating query functions to support tenant isolation and setting the current tenant ID for queries.