I decided to transition from a data analyst role to that of a data engineer, a journey that initially felt daunting due to the vast array of concepts to grasp—data warehouses, orchestration tools, distributed processing, streaming systems, cloud platforms, and infrastructure. The list was seemingly endless.
Rather than attempting to learn everything simultaneously, I opted for a more structured approach. I devised a 12-month self-study roadmap centered around a straightforward principle: learn by building.
Instead of hopping from tutorial to tutorial, I focused on creating a series of small projects. Each project introduced a handful of new concepts while reinforcing previous knowledge. My aim was not to construct the most sophisticated applications but to cultivate an engineering mindset by addressing one problem at a time.
The first project on this path was a GitHub ETL pipeline. It started as a simple Python script that retrieved repository data and exported it to a CSV file. As I progressed, I enhanced the project by replacing CSV files with SQLite, ensuring the pipeline was idempotent to avoid duplicate data, and ultimately automating it with GitHub Actions.
Upon completion, I realized a crucial insight that had eluded me at the outset: constructing the ETL logic was the easier part. The more challenging questions emerged when I shifted my perspective from a one-time script execution to envisioning a system that needed to operate autonomously.
- How should it be scheduled?
- What happens if it fails midway?
- Where should the retry logic reside?
- How do you package the application to ensure it runs consistently across environments?
These inquiries imparted far more knowledge about data engineering than merely parsing JSON or crafting SQL queries ever could.
My initial project presented another challenge. While GitHub Actions effectively automated a small ETL pipeline, I was eager to understand the nuances of using a workflow orchestrator specifically designed for data engineering. I sought to learn how engineers delineate orchestration from execution, how containerized workloads fit into this framework, and what a production-ready pipeline looks like, even on a modest scale.
Thus, for my second project, I chose to build an automated RSS ingestion pipeline.
At first glance, it appears to be a straightforward application that retrieves articles from an RSS feed, parses them into structured objects, and stores them in PostgreSQL. However, the objective was never to create an RSS reader; rather, it was to delve into the engineering decisions that elevate a Python script into a dependable data pipeline.
Why Build Another ETL Pipeline?
After completing my first ETL project, I contemplated branching out into entirely different areas—perhaps a data warehouse project, Apache Spark, or an API with a more intricate transformation layer. Instead, I opted to build… another ETL pipeline.
At first, this may seem like a regression. After all, I had already constructed an extraction pipeline, made it idempotent, and scheduled it using GitHub Actions. Why repeat the exercise?
Because my goal was not to learn a new dataset, but to adopt a new way of thinking.
A key lesson from my first project lingered in my mind: writing the ETL logic was not the challenging aspect; it was everything surrounding it.
- How should the pipeline be executed?
- How should it recover from failures?
- How do you package it to ensure consistent operation on any machine?
- Where does scheduling belong?
And perhaps the most significant question of all:
Where should the application’s responsibilities end, and where should the orchestration layer’s responsibilities begin?
These questions transcend the specifics of RSS feeds or GitHub repositories; they are fundamental engineering inquiries that I realized I could explore with virtually any data source. This is why I selected an RSS feed—not for its excitement, but for its intentional simplicity.
The extraction logic required only a few lines of Python, allowing me to concentrate less on business logic and more on architectural considerations.
For similar reasons, I decided to move away from GitHub Actions for this project. While GitHub Actions served as a useful introduction to scheduling, demonstrating how to automate a workflow and providing my first experience of running an ETL pipeline without manual intervention, it was not specifically designed for orchestrating data workflows.
This led me to Kestra.
Rather than asking, “How do I run this Python script every hour?”, I began to pose a different set of questions:
- How should retries be configured?
- How are workflow executions tracked?
- How should environment variables be passed into a container?
- What does a failed execution look like?
- How do you separate the application from the infrastructure that runs it?
These questions were precisely what I aimed to explore.
By selecting a simple ETL pipeline, I could focus on the engineering decisions without being sidetracked by complex business logic.
In retrospect, I believe this was the right choice. The project’s interest lies not in its ability to process RSS feeds, but in how building it compelled me to consider reliability, repeatability, and orchestration in ways my initial project did not.
The First Architectural Decision: Docker Before Kestra
With the project defined, my instinct was to dive straight into Kestra. After all, orchestration was one of the primary motivations for choosing this project. Why not start there?
Instead, I made a decision that ultimately saved me considerable frustration later: I ignored Kestra entirely.
This might seem counterintuitive, but I wanted to avoid introducing multiple moving parts before confirming that the core application functioned correctly.
Thus, I built the project in layers. I began by writing the Python ETL, intentionally keeping its scope narrow. Its sole responsibility was to fetch the RSS feed, parse each entry into an Article object, and save the results to PostgreSQL. Nothing more.
feed = feedparser.parse(RSS_URL)
articles = parse_feed(feed)
save_articles(articles)This was the extent of the ETL’s functionality. I deliberately kept the application small because the focus of this project was not the transformation logic but rather everything that surrounded it.
Once that was reliably operational, I shifted my focus to the database. To ensure repeated executions were safe, I made the inserts idempotent using PostgreSQL’s ON CONFLICT DO NOTHING. This way, running the pipeline multiple times wouldn’t create duplicate rows.
INSERT INTO articles (...)
VALUES (...)
ON CONFLICT (id) DO NOTHING;This single line rendered the pipeline safe for repeated execution. Whether Kestra ran the workflow once or a hundred times, PostgreSQL would manage the prevention of duplicate records.
Only after the ETL and database were working in tandem did I introduce Docker.
This decision transformed my perspective on the project.
Initially, Docker appeared to be just another tool to learn. By the end of the project, it had evolved into something far more significant: the unit of execution.
FROM python:3.13-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
CMD ["python", "fetch_rss.py"]By packaging the ETL in this manner, the application became self-contained. Instead of asking Kestra to comprehend my Python project, I could simply instruct it to execute a container that already knew how to run the pipeline.
This shift in thinking was subtle yet transformative; it altered the relationship between my application and the orchestration layer.
Once the ETL was encapsulated in a container, it became irrelevant whether it ran on my laptop, within Kestra, or on another machine entirely. The runtime environment remained constant.
This consistency provided me with something I had previously lacked: confidence.
Before introducing Kestra, I could manually run the container and verify that it fetched the RSS feed, connected to PostgreSQL, and persisted the expected records. If an issue arose, I knew the problem wasn’t obscured by another layer of orchestration.
This validation step proved invaluable later on. During development, I encountered challenges related to networking, container configuration, and workflow execution. Because the Docker image had already been independently validated, I could quickly eliminate the ETL itself as the source of the issue and concentrate on the orchestration layer.
Reflecting on this, I believe it was one of the most significant lessons from the project. It’s tempting to connect every component as swiftly as possible and hope for the best. A more effective strategy is to validate each layer before introducing the next one.
In this project, the order of validation was as follows:
- Validate the Python ETL.
- Validate PostgreSQL persistence.
- Validate the Docker image.
- Finally, let Kestra orchestrate a container that I already trusted.
Each layer built upon the previous one. By the time Kestra entered the equation, I was not trying to debug Python, PostgreSQL, Docker, and orchestration simultaneously; I was focused on solving a single problem.
This incremental approach made the entire project feel much more manageable, and it’s a workflow I intend to adopt for future data engineering endeavors.
The Assumption That Turned Out to Be Wrong
With a functioning Docker image, I was under the impression that the hard part was behind me.
- I had a Python ETL that worked.
- I had PostgreSQL running in Docker.
- I had confirmed that the container could fetch RSS articles and save them to the database.
Now, all that remained was for Kestra to execute it.
Or so I believed.
My initial assumption was straightforward: Kestra would point to my Python files, execute the script, and everything would function as it had from the command line.
It didn’t.
I quickly learned that there was a crucial distinction I had not fully grasped: Kestra is an orchestrator. It is not responsible for building Python environments or managing application dependencies; its role is to determine when and how workloads should run.
This realization shifted the project’s direction. Instead of treating Kestra as another platform for executing Python code, I began to view it as the layer responsible for orchestrating a workload that already existed—my Docker image.
Once I embraced this mental shift, the architecture became much cleaner.
- The ETL evolved into a self-contained application.
- Docker became the deployment artifact.
- Kestra assumed the role of orchestrator.
Each layer now had a clear responsibility, and none needed to understand the internal workings of the others.
Interestingly, reaching this point was not entirely straightforward. My initial instinct was to find a way for Kestra to execute the Python project directly. While this approach seemed simpler, the more I explored it, the more I realized I was asking the orchestrator to take on responsibilities that the application should inherently provide.
Once I accepted Docker as the execution artifact, the workflow became significantly simpler.
Some approaches appeared promising at first but introduced unnecessary complexity, while others worked but did not align with how Kestra encourages the execution of containerized workloads.
Ultimately, I settled on a workflow that felt surprisingly straightforward. Instead of attempting to teach Kestra how to run Python, I allowed it to perform its primary function: launching a container.
tasks:
- id: run_etl
type: io.kestra.plugin.scripts.shell.Commands
containerImage: rss-pipeline-etl:latest
taskRunner:
type: io.kestra.plugin.scripts.runner.docker.Docker
commands:
- python /app/fetch_rss.pyIn reviewing the workflow now, what stands out is not the volume of YAML it contains, but rather how little Kestra needs to know about my application. Its sole responsibility is to launch a container that already understands how to execute the ETL.
Inside that container, my application is fully aware of its tasks.
This minor architectural adjustment resolved not only the immediate execution issue but also reinforced a recurring theme in my learning journey: effective engineering often involves simplifying rather than complicating.
- Python should not concern itself with orchestration.
- Kestra should not manage Python dependencies.
- Docker should remain agnostic to RSS feeds.
Each component addresses a distinct problem. Once I stopped expecting one tool to solve every issue, the entire system became easier to comprehend.
In hindsight, this was likely the most significant mindset shift throughout the project. I did not merely learn how to utilize Kestra; I grasped the true essence of orchestration.
Once a Pipeline Runs Automatically, Everything Changes
Up until this point, I had been executing the pipeline manually. If an error occurred, I was present to read the message, make adjustments, and try again.
That safety net vanishes the moment a pipeline begins to operate autonomously.
One of the first configurations I established in Kestra was a simple hourly schedule.
triggers:
- id: hourly_schedule
type: io.kestra.plugin.core.trigger.Schedule
cron: "0 * * * *"On paper, this was merely a cron expression. In practice, it signified a substantial shift.
The pipeline no longer relied on me to remember to execute it. Every hour, Kestra would initiate a new execution, launch the ETL container, and process the latest articles from the RSS feed.
This immediately prompted another question: What occurs if one of those executions fails?
During development, I intentionally introduced failures to explore this question. I directed the pipeline at an invalid database host and observed the outcome.
The first execution failed, as anticipated. More importantly, it did not stop there. Because the workflow was configured with retries, Kestra automatically attempted the execution again after a brief delay.
retry:
type: constant
maxAttempts: 3
interval: PT30SThis was one of my favorite moments in the project. Once I rectified the configuration, the workflow completed successfully without necessitating any modifications to the application itself.
This was a highlight for me, not due to the complexity of retries, but because it illuminated another separation of responsibilities. The ETL should not determine whether it deserves another chance; that is an orchestration concern.
By relocating retry logic into Kestra, the Python application remained focused on its singular responsibility: processing the feed and persisting the results.
The orchestration layer managed resilience.
Scheduling introduced another challenge I had encountered in my first ETL project. Repeated executions imply repeated attempts to write data. If the same RSS article appears in multiple hourly runs, the pipeline should not insert it twice.
Fortunately, I had already devised a solution for a similar issue. The database layer was designed to be idempotent using PostgreSQL’s ON CONFLICT DO NOTHING.
INSERT INTO articles (...)
VALUES (...)
ON CONFLICT (id) DO NOTHING;This ensured that every execution could safely attempt to insert the same records without creating duplicates.
The combination of scheduling, retries, and idempotent writes rendered the pipeline significantly more forgiving. If a run failed, Kestra could retry it. If a successful retry encountered data that had already been written, PostgreSQL would simply ignore the duplicates.
Neither layer needed to be aware of the other’s operations; each handled its own responsibilities.
The final enhancement was visibility. Early in development, my logs reflected the typical output of a project still in debugging mode. I printed entire RSS objects to the console to confirm the parser was functioning correctly.
While effective, this approach was not visually appealing. As the pipeline stabilized, those debug statements became less useful. I replaced them with logs that conveyed the execution status rather than dumping raw data.
Before
print(feed.entries[0])After
print("=== RSS PIPELINE START ===")
first = feed.entries[0]
print("First entry preview:")
print(f"Title: {first.get('title')}")
print(f"Link: {first.get('link')}")
print(f"Feed title: {feed.feed.title}")
print(f"Fetched articles: {len(articles)}")
print(f"Saved {len(articles)} articles to the database.")
print("=== RSS PIPELINE END ===")Each run now narrates a clear story.
=== RSS PIPELINE START ===
First entry preview:
Title: Christian Ledermann: Migrate From mypy To ty And pyrefly
Link: https://dev.to/...
Feed title: Planet Python
Fetched articles: 25
Saved 25 articles to the database.
=== RSS PIPELINE END ===These modifications did not enhance the application’s intelligence; they improved its comprehensibility. This distinction is crucial.
Effective observability is not about generating more logs; it’s about producing the right logs.
By the project’s conclusion, I discovered something intriguing: the Python code had not expanded significantly. Most of the effort had been invested in enhancing its reliability.
- Scheduling ensured it ran autonomously.
- Retries enabled recovery from transient failures.
- Idempotency safeguarded the database against duplicate writes.
- Logging improved the clarity of each execution.
Individually, none of these changes were particularly complex. Collectively, they transformed a simple Python script into a system that functioned much more like a production-ready application.
The Final Architecture
By the end of the project, the architecture had evolved into something that felt surprisingly straightforward.
Looking at the final architecture, it’s easy to assume the project was always destined to reach this point.
In reality, each layer was added only after the previous one had been validated.
- The ETL was developed first.
- Then PostgreSQL was integrated.
- Next, Docker was introduced.
- Finally, Kestra was implemented.
The order of these steps was significant. Since every component had been tested independently, I never found myself debugging Python, Docker, PostgreSQL, and Kestra simultaneously. Each decision reduced the number of unknowns rather than increasing them.
Moreover, every component ended up with a distinct responsibility.
- Python handles fetching, parsing, and persisting RSS articles.
- PostgreSQL manages data storage and prevents duplicates.
- Docker provides a consistent execution environment.
- Kestra determines when the workload should run and how to handle failures.
None of these components attempt to perform each other’s tasks. Ironically, this clarity made the final system feel much simpler than I had anticipated.
What This Project Changed About the Way I Think
When I began my journey into data engineering, I presumed the most challenging aspect would be writing ETL code, as most beginner tutorials emphasize this skill.
- You learn how to call an API.
- You transform the data.
- You save it somewhere.
Repeat.
While these skills are valuable, they represent only a fraction of the broader picture. This project taught me that the real engineering begins after the script functions correctly.
Once a pipeline is expected to run hourly, withstand transient failures, avoid duplicate data, and generate logs that clarify what transpired, the questions become far more engaging.
You transition from focusing on individual functions to contemplating entire systems.
A significant mindset shift for me was recognizing the distinction between execution and orchestration. Initially, these concepts felt nearly interchangeable; now, they seem entirely separate.
The Python application should concentrate on business logic, while the orchestrator should focus on when, where, and how that application operates.
Maintaining these responsibilities separately simplified the entire project.
This experience also transformed my perception of Docker. Prior to this project, I viewed Docker primarily as a means to package applications. Now, I regard it as a deployment artifact.
Once the ETL was encapsulated in a container and validated independently, I could cease worrying about whether it would behave differently within Kestra. This confidence emerged as one of the most significant advantages of containerizing the application.
Perhaps the most crucial lesson, however, had little to do with Kestra or Docker. It was the importance of incremental building.
Every major decision adhered to the same pattern:
- Construct the smallest functional component.
- Validate it.
- Only then introduce the next layer.
This methodology made the project feel considerably less overwhelming than attempting to connect everything from the outset. In retrospect, I believe this is a lesson I will carry into all future projects, regardless of the technology involved.
Looking Ahead
This RSS pipeline represents only the second project in my data engineering learning journey. Compared to my first ETL pipeline, the code itself is not dramatically more complex. What has changed is my approach to problem-solving.
Instead of asking, “How do I write this script?”, I found myself contemplating questions such as:
- Where should this responsibility reside?
- What happens if it fails?
- Can I run it repeatedly without worrying about duplicate data?
- Can I trust it to run autonomously?
These inquiries prompted me to think less like someone merely writing Python code and more like a system designer. I suspect this is the true value of undertaking projects.
Every project introduces a new tool, but the most impactful projects gradually reshape your thinking. This one certainly did.
I anticipate that my next project will challenge a completely different set of assumptions, and I look forward to discovering what they may be.
This is part of my ongoing series documenting my transition from systems analyst to data engineer. If you’ve been following along, thank you.
