data visualization

The dataset utilized consists of police log entries from the Cambridge Police Department (CPD), which includes the date and time of incidents, type of incident, location, and a detailed description. The project follows a structured ETL process that involves extracting data via the Socrata Open Data API, validating the data for integrity, transforming it for optimal storage, and loading it into a PostgreSQL database. The extraction is performed using a Python client for the API, and validation checks ensure the presence of expected columns and the integrity of the data. The transformation process includes removing duplicates and splitting the datetime column into separate components. The data is then loaded into PostgreSQL, where a table is created to store the incidents. The entire ETL process is automated using Prefect, allowing for daily execution. Finally, the data is visualized using Metabase, which connects to the PostgreSQL database to create dashboards that display crime trends over time.