CSV file

A common task in enterprise systems involves loading large volumes of data into PostgreSQL, which can be slow if using a loop in Java that calls the save() method for each record. Initial tests show that a basic insert operation takes over 10 minutes. Hibernate retains every object in the session-level cache until the transaction concludes, leading to high memory consumption. Enabling batching with the jdbc.batch_size setting improves performance by about 23%, but high memory usage persists. Application-side caching of IDs is faster than database-side caching, achieving a 30% performance gain. Using saveAll() results in minimal improvement as it still calls save() in a loop. Enabling reWriteBatchedInserts in the JDBC driver allows for multi-row inserts, reducing operation time to 4 minutes and 37 seconds. Clearing the session-level cache with EntityManager further optimizes memory usage, resulting in an 18-second time gain. Setting order_inserts=true enhances batching efficiency. With all optimizations, insertion time is reduced to 4 minutes and 19 seconds. A custom data insertion layer can be created to generate INSERT statements, with reflection being a straightforward but not optimal method. Prepared statements improve performance and security, with an efficient batch size of 5,000 records. PostgreSQL's COPY command facilitates faster bulk data transfers than multi-row inserts. Integrating COPY with Spring transactions enhances performance, and parallelizing the insertion process with a ThreadPoolExecutor significantly improves speed and memory usage.