NewApp Digest
search bot

environment variables

I Built My Second ETL Pipeline. This Time, I Started Thinking Like a Data Engineer
Tech Optimizer
July 21, 2026

I Built My Second ETL Pipeline. This Time, I Started Thinking Like a Data Engineer

The author transitioned from a data analyst role to a data engineer, creating a 12-month self-study roadmap focused on learning by building projects. The first project was a GitHub ETL pipeline, which evolved from a simple Python script to a more complex system using SQLite and GitHub Actions for automation. The author realized that the challenges of data engineering lay in system design and orchestration rather than just writing ETL logic. For the second project, the author built an automated RSS ingestion pipeline to explore engineering decisions in creating a reliable data pipeline. The project emphasized the importance of separating application responsibilities from orchestration tasks, leading to the use of Docker for consistent execution environments and Kestra for orchestration. The author validated each component of the project incrementally: first the Python ETL, then PostgreSQL, followed by Docker, and finally Kestra. This approach ensured that each layer functioned correctly before adding complexity. The final architecture included distinct responsibilities for each component: Python for data processing, PostgreSQL for data storage, Docker for execution, and Kestra for orchestration. The author learned that effective engineering involves considering system reliability, error handling, and observability, shifting the focus from merely writing code to designing entire systems. The experience transformed the author's mindset, emphasizing the importance of incremental building and the separation of execution and orchestration responsibilities.
Introduction to COM usage by Windows threats
Winsage
June 25, 2026

Introduction to COM usage by Windows threats

Component Object Model (COM) is a technology in Windows that enables object activation, inter-process communication, and automation across different programming languages. Malware exploits COM interfaces for activities such as lateral movement, execution, downloading, exfiltration, persistence, evasion, system discovery, and automation of Windows and Office functionalities. Reverse engineering COM-heavy binaries involves navigating GUIDs and indirect vtable calls to understand malware mechanics. Research at the AVAR 2025 conference and CARO 2026 workshop discusses methodologies for analyzing COM binaries and case studies of malware families that utilize COM. COM is an application binary interface (ABI) model that allows software components to be reused and enables interaction between different programming languages through interfaces defined at the binary level. Distributed COM (DCOM) allows clients to activate COM objects on remote systems. COM classes are identified by unique class identifiers (CLSIDs), and interfaces by interface identifiers (IIDs). The Windows registry stores COM registration data, with classes and interfaces located under specific keys. Malware often acts as a COM client, utilizing the COM runtime to instantiate classes and request interfaces. ProgIDs provide human-readable registry entries for COM classes. The CoCreateInstance function helps create class objects by resolving CLSID registrations. All COM interfaces derive from IUnknown, which manages object lifetimes and interface querying. COM has its own security model, and identifying classes and interfaces used by malware is crucial for threat researchers. Tools like ComView and OleView.NET assist in inspecting COM registrations. The analysis workflow includes identifying activation API calls, extracting CLSID and IID values, consulting registry definitions, and mapping vtable calls. Qakbot, a banking trojan, exemplifies the use of COM in malware, with its architecture enabling malicious activities like credential theft. Dynamic analysis tools can log COM-related calls in real-time to trace execution flow. Notable malware families that utilize COM include Gh0stRAT, which uses Task Scheduler COM interfaces, and the Attor platform, which employs BITS for file transfers. WarmCookie demonstrates the use of COM for persistence through Task Scheduler. Understanding COM's role in malware is essential for cybersecurity professionals.
A Brief History Of Unix Commands On Windows: CoreUtils (Again)
Winsage
June 16, 2026

A Brief History Of Unix Commands On Windows: CoreUtils (Again)

The interaction between Unix/Linux and Windows has historically been marked by significant differences in their architectures and philosophies. Unix uses a fork() function for process management, while Windows employs CreateProcess(), complicating the implementation of Unix-like tools on Windows. Early solutions to bridge this gap included the MKS Toolkit, which provided Unix-like commands for Windows, and UWIN from AT&T Bell Labs, which aimed to create a Unix interface layer on Windows. Cygwin offered a compatibility DLL to run Unix software on Windows, but required rebuilding from source. Microsoft's initiatives included POSIX, Interix, and later Services for UNIX. The introduction of the Windows Subsystem for Linux (WSL) allowed users to run a Linux userland directly on Windows, with WSL 2 incorporating a real Linux kernel. Recently, Microsoft released Coreutils for Windows, providing native builds of Unix-style tools to enhance cross-platform consistency.
Query billion-scale vectors with SQL: Integrating Amazon S3 Vectors and Aurora PostgreSQL
Tech Optimizer
May 10, 2026

Query billion-scale vectors with SQL: Integrating Amazon S3 Vectors and Aurora PostgreSQL

For managing relational data within Amazon Aurora PostgreSQL and enhancing capabilities with similarity search over large embedding collections, an integration with Amazon S3 Vectors is available. This integration utilizes the pgvector extension for low-latency similarity searches on vectors stored in the database. Amazon S3 Vectors provides cost-effective storage for extensive datasets, scaling to hundreds of millions or billions of embeddings. The integration, facilitated through AWS Lambda, allows users to query S3 Vectors directly from Aurora PostgreSQL using standard SQL, enabling the combination of vector similarity results with relational filters in a single query. The integration offers benefits such as basic key-value metadata support in S3 Vectors for simple filtering, while Aurora PostgreSQL is optimal for complex SQL filters, multi-table joins, and access-control policies. The architecture separates concerns, with Lambda handling API integration and Aurora managing relational data. Security is maintained through IAM role separation and network security measures. Data consistency considerations arise due to the distribution of data across Aurora PostgreSQL and S3 Vectors, necessitating explicit synchronization processes to manage potential stale results. The integration is suitable for use cases that can tolerate eventual consistency, such as recommendations and content discovery. Performance expectations include Lambda invocation latencies of 100-500 milliseconds, while Aurora pgvector typically provides single-digit millisecond response times. S3 Vectors achieves sub-second performance for cold queries and less than 100 milliseconds for warm queries. From a cost perspective, S3 Vectors is more economical than Aurora, making it suitable for high-volume vector data that requires archiving and queryability. The integration architecture consists of three components: Aurora PostgreSQL, AWS Lambda for API translation, and S3 Vectors for executing similarity searches. A typical query involves several stages, including SQL query invocation, function processing, API translation, similarity search, and result processing. Prerequisites for this integration include familiarity with Aurora PostgreSQL, AWS Lambda, vector databases, and SQL. Required AWS resources include an Aurora PostgreSQL cluster, a VPC with internet access, and appropriate permissions for IAM roles and Lambda functions. To deploy the integration, users must gather configuration values from their Aurora cluster, deploy a CloudFormation stack, associate the IAM role, update the Lambda function code, and install the necessary PostgreSQL schema. Sample data can be uploaded to the S3 Vectors index, allowing for testing of vector operations and combined queries. When combining relational and vector queries, developers should be aware of the trade-offs involved in pre-filtering data by metadata, which can improve performance but may reduce recall. Troubleshooting may involve addressing connectivity issues, IAM role misconfigurations, and performance optimizations. To clean up resources after testing, users should remove the PostgreSQL schema, disassociate the Lambda role from the Aurora cluster, and delete the CloudFormation stack to ensure all resources are removed.
How to Set Up PostgreSQL on Windows Easily in 2026
Tech Optimizer
May 4, 2026

How to Set Up PostgreSQL on Windows Easily in 2026

The installation process for PostgreSQL can be challenging for new users due to several factors. A common issue is the requirement to set a strong password for the default 'postgres' user, which users often forget, leading to a complicated reset process. Another challenge is port conflicts, as PostgreSQL uses port 5432 by default; if another application is using this port, it complicates installation. Additionally, users must configure PostgreSQL to be included in the system PATH for command-line operations, a step that is frequently overlooked, causing frustration when executing commands.
Search