Technical support manifests in various forms, from simple troubleshooting advice like “try rebooting” to complex engineering challenges that demand a lifetime of dedication. Within the realm of Postgres Professional, the question arises: is it the technology or the people that hold greater significance? To explore this, we spoke with Kamil Karimov, a Senior Technical Support Engineer at Postgres Professional.
Technical support found me first
After spending several years in a different field, I found myself at a standstill, yearning for growth and a deeper connection with PostgreSQL. My initial intention was not to enter tech support; rather, I sought any opportunity related to this powerful DBMS. Like many, I envisioned support roles as merely responding to queries with brief instructions like “click here” or “reboot that.” However, an opening for a Junior Support Engineer at Postgres Professional caught my eye. The role promised engagement with PostgreSQL and Postgres Pro, encompassing responsibilities such as consulting, troubleshooting DBMS issues, and executing technical tasks. It was an enticing prospect, especially since I had yet to gain hands-on experience with PostgreSQL.
What a technical support engineer actually does
Our work predominantly involves genuine engineering. Clients approach us with a spectrum of issues, ranging from installation queries to critical situations where applications crash, potentially indicating data corruption. The initial step in our process is always to gather diagnostics, allowing us to pinpoint the problem and identify the responsible system.
The tools we employ vary based on the situation. For instance, when performance issues arise, we might utilize GDB or perf. In some cases, system views within Postgres can provide valuable insights. Among the arsenal of tools at Postgres Professional, we frequently leverage pgpro_stats in conjunction with pgpro_pwr. These tools enable us to generate workload reports during incidents and compare them against baseline performance metrics.
Deep product knowledge is crucial; often, it is sufficient to narrow down the problem area. Once we identify the issue, we strive to replicate it by creating a test environment that mirrors the conditions leading to the error. If we suspect a bug or identify potential improvements, we collaborate with developers. Our work resembles that of a database engineer or DBA, albeit with a stronger emphasis on troubleshooting. We manage requests through a dedicated support portal, which proves far more efficient than traditional email communication, where important details can easily be overlooked.
The 6 TB case
One notable incident involved a client whose standard checkpoint process ceased functioning. Over a span of four hours, their pg_wal directory ballooned to an astonishing 6 TB, consuming all available disk space. Upon investigation, we observed that checkpointing had initially operated normally before abruptly halting. Utilizing our crash_info tool, we captured a backtrace and identified unexpected calls within the checkpoint process.
Diving deeper into the code, we discovered that a specific code path activates when the parameter archive_timeout is enabled. This was a non-intuitive scenario, as we had previously assumed that archive_timeout would not engage when the primary parameter archive_mode was disabled. By disabling the parameter, we resolved the issue, turning what could have been a frustrating experience into an engaging bug hunt.
Who comes to us
Most of our clients possess a robust technical background, which significantly aids our interactions. For those less familiar with the technology, we provide detailed explanations and simple examples, which typically suffice. An ideal support request includes:
- a comprehensive problem description;
- a timeline of events leading up to the issue;
- all necessary diagnostics, preferably accompanied by a reproducible case.
However, the reality is often different. Occasionally, students reach out for assistance with coursework or theses, and we gently redirect them, emphasizing the value of solving problems independently. At times, clients request our intervention in disputes between developers and DBAs. For instance, when an application developer blames a malfunctioning VACUUM process while the DBA insists it operates normally, we are more than willing to mediate.
Certain phrases trigger immediate concern. For example, “We found something on the internet and tried it, but it didn’t help,” typically indicates that the situation has worsened.
What you need to know to work in Postgres Professional support
Both hard and soft skills are essential in this role. On the technical front, a support engineer should:
- possess a thorough understanding of Postgres Pro, including its architecture, features, quirks, and the broader ecosystem;
- be well-versed in operating systems, capable of administering and troubleshooting them;
- have a grasp of networking and a general understanding of server infrastructure;
- be able to program or at least read code.
On the communication side, it is vital to recognize that clients often approach us in a state of stress. Some have the luxury of time to resolve issues calmly, while others require immediate assistance. A support engineer must cultivate strong communication skills, ensuring that clients feel valued and assured that we are doing everything possible to assist them.
Technical support is as much about people as it is about technology. In fact, the entire IT landscape is evolving in this direction. Even roles that do not involve direct customer interaction still require collaboration with various teams, making it essential to engage beyond mere technical focus.
Growing inside and outside tech support
Opportunities for growth within the support team can be boundless, depending on the company. Each new release introduces fresh features, cases, and bugs that encourage exploration of PostgreSQL’s codebase and the honing of skills. Overall, technical support serves as an excellent launchpad into the IT field, exposing individuals to diverse tasks and areas of expertise. Ultimately, this journey may lead one back to a passion for support itself. For those intrigued by this field, I recommend Brendan Gregg’s book Systems Performance, which addresses scenarios like “nothing works, please help.” Additionally, engaging with conference talks and consuming PostgreSQL-related literature can be immensely beneficial.
The future of technical support is…
As features and abstraction layers continue to expand, we can expect to encounter increasingly complex cases. This trend has become evident over the past three years. Conversely, for simpler routine tasks, AI is poised to take on a larger role. Personally, I have begun utilizing AI as a more intelligent search engine and for generating DDL.
Words of wisdom from an experienced engineer
- If a P1 ticket arrives first thing in the morning, brace yourself for an eventful day.
- Fridays are notorious for ticket influx: clients often spend the week grappling with issues, and whatever remains unresolved is handed off to us on Friday.
- Avoid making evening plans on Fridays; clients seem to sense this and will inevitably send urgent requests.
- Remember to take breaks—run errands, go for a walk, or hit the gym. Continuous work without rest leads to diminished productivity.
- If you are new to support, embrace every task that comes your way; it’s the quickest route to skill enhancement.
