Margus Heinmaa · Senior Platform Engineer, DevOps / SRE · CKA
Platform engineering · Kubernetes · Adobe Commerce at scale

What I build,
and what breaks.

I run a multi-tenant Kubernetes platform hosting enterprise e-commerce storefronts across Europe, and I build the systems the team uses to operate it. Seven pieces of platform work, and four production outages taken from first signal to root cause.

Schematic · the shape of a faceted-crawl night req/s to origin customer-facing 5xx
Illustrative, not a capture: three bursts two to three hours apart on an unmoved baseline, and the effect once an edge rule lands.

On confidentiality. I have removed client names, domains, cluster and namespace identifiers, ticket references, vendor extensions and repository paths. Sectors and dates are described only broadly. What is left is the technical detail: what broke, how I found it, and what I changed.

The platform

What I operate

A managed hosting platform for high-traffic Adobe Commerce storefronts on Google Cloud. I own observability, incident response, edge security, and capacity and cost management across it.

10+
GKE clusters in production
1500+
Servers and workloads under configuration management
Dozens
Enterprise storefronts, multi-country
12+
Years at the same agency, dev lead to platform
20+
Years in web infrastructure overall
CKA
Certified Kubernetes Administrator
Platform work

Seven things I have worked on

Internal systems, the delivery pipeline, the secrets path, and the tooling the team uses during an incident. Some of these I proposed and built; two are design and review work, which I have marked as such.

Platform inventory

Rebuilding the inventory service I originally wrote

We had client data in one system, people and teams in another, and projects, servers, clusters and namespaces spread across several more. Answering something like which team works with which client meant opening five tools and assembling the answer by hand, so most people guessed from memory instead. I proposed a single service that pulls from each of those systems and joins them, and I wrote it. It has been the platform's authoritative inventory for years since.

  • The individual facts already existed in about a dozen upstream systems: cloud, DNS and CDN, uptime and on-call, HR, source control, package registries, CRM and mail. Nothing joined them up: which client runs in which namespace on which cluster, which certificate covers which domain, who has access to what.
  • I wrote the first version single-handed in PHP. I am now rewriting it in Python and Django, because a service only one person can maintain is a risk however well it runs, and most of the department does not work in PHP. I picked Python because that is what the team works in.
  • The rewrite drops write-back. The original wrote to a good number of the services it read from, including group membership, user provisioning, monitoring checks and DNS; the replacement only writes through a small set of explicit operational actions.
  • Things I hand-rolled the first time round, a route-derived ACL, a custom revision and audit trait, and configuration held in the database, now come from framework libraries instead. I triaged all 77 models one at a time into ported, replaced, deliberately dropped, backlog, or flagged as an open decision the team still owns, so we can decide what to switch off with evidence.
  • The second version adds GraphQL alongside REST, full-text search across the estate, and an MCP server, so the agent tooling further down this page can query inventory directly instead of scraping for it.
Developer experience

Internal developer portal on Backstage

I built and then handed over an internal portal covering the software catalog, technical documentation across the platform's repositories, and the org chart. It runs on Backstage's current backend and frontend systems and deploys through GitOps.

  • No catalog entity is authored in the UI. Everything is generated from source control, so nothing is lost when it regenerates and the portal cannot drift from the repositories.
  • An in-house plugin connects it to the inventory service. The docs toolchain is one shared, version-pinned image so every repository builds its documentation the same way, which does mean a single blast radius whenever that image moves.
  • The handover led with the component's two worst problems: an unbacked single-replica database, and a dependency chain that makes the portal a blocker on retiring the original API.
CI/CD

Auditing 767 Jenkins job definitions before migrating them

Our CI estate had 767 hand-maintained job definitions across five directories. Moving them onto a generator is easy to do badly, because a job that quietly stops triggering does not raise an error. I audited all 767 first.

  • I extracted the complete call vocabulary of every legacy definition and diffed it against what the generator emits. That found 18 pull-request test jobs that would have stopped building silently, because the generator unconditionally emitted a build strategy that excludes change requests.
  • 135 files carried branch-exclusion rules everyone assumed were dead. I scripted an overlap check across all 712 include/exclude pairs and confirmed none of them could match a branch its includes admitted, which turned an assumption into a documented deletion.
  • The acceptance criterion I hold migrations to, and the one I wrote into the source-control migration tickets as well: re-running the generator produces no configuration diff for any existing job.
Secrets · design

Moving build secrets to direct Vault reads

Build secrets reached pipelines through four hops: Vault, an operator, a Kubernetes secret, a credential, then the build. Every hop left another copy of the secret that could drift out of sync.

  • I designed the move to direct Vault reads at build time, authenticated by Kubernetes ServiceAccount rather than a static token. Access becomes per-service and per-build, scoped by policy, auditable in Vault, with no duplicate copy in the cluster and no resync lag when a secret rotates.
  • It ships as a vertical slice: prove the pattern end to end on the smallest scope, fan out, then retire the machinery that frees up. Unconverted pipelines keep working throughout.
  • I also own the Vault and Consul upgrades. Consul is Vault's storage backend and its raft data cannot be downgraded, so the runbook is built around that: staged rollout, snapshot restore as the only way back, and the order written down and reviewed before anything is touched.
Infrastructure as code · review

Reviewing the Terraform CI migration

I reviewed the design for moving our Terraform estate off a managed service and onto our own CI pipelines. A like-for-like port would have carried over permissions that were already too broad. Two changes I pushed for:

  • Approval gates instead of automatic apply. Auto-apply on merge is fine for development. For production, and for the workspace that manages authentication itself, the apply should pause on a required-reviewer gate, with credentials scoped per environment so a production apply cannot run with development credentials.
  • Harden the state bucket like the secrets store it is. Terraform state records resource attributes that are secrets whatever the provider calls them, so "versioning is enabled" is not a hardening story. I asked for customer-managed encryption keys, IAM-only access with no ACLs, least privilege scoped per prefix, lifecycle pruning of old versions, and the most sensitive workspaces in their own bucket rather than pooled with everything else.
Observability

Auditing the monitoring dashboards

I own the monitoring stack: metrics, long-term storage, logs, dashboards and alert routing. I audited the fleet-wide component health dashboard and found that a good part of it had stopped working without anyone noticing.

  • All five log panels were returning nothing. Their label selectors had been correct when written, but log collection had since moved behind a sidecar that multiplexes streams and distinguishes them by filename. The panels did not error; they returned an empty chart.
  • The same audit found search-cluster panels working for namespaces running one exporter and blank for those running another with a different metric schema, plus a Kubernetes event type with zero hits fleet-wide over twelve hours, either dead or a silent guard and nobody knew which.
  • Alongside the stack I run a monitoring rotation that turns raw alert volume into a prioritised cluster health report, because the volume on its own does not tell you which cluster to look at first.
Agent tooling

Automating the first hour of an investigation

Each of the outages below started with the same hour of work: pull the pods, the events, the CPU throttling, the cache hit ratios and the top talkers at the edge, then assemble them into something I can reason about. That part is mechanical, so I built it into tooling.

  • Ten or so custom agent skills covering cluster situation reports, namespace incident analysis, resource right-sizing and node diagnostics, wired to Kubernetes, the monitoring stack and cloud APIs through MCP. Cluster health reviews went from hours to roughly fifteen minutes.
  • Everything is read-only by default, and discovery is separate from mutation. Anything touching shared state runs as two invocations with a reviewable manifest in between, so a person approves the specific change, not the general intent.

Also on the record

  • On-premise virtualization. Designed and built the company's own virtualization cluster with VPN connectivity and monitoring, then trained the team to run it.
  • Source-control migration. Enabling a move between hosting providers across the CI estate, gated on the same no-configuration-diff test.
  • Helm chart library. Author and maintainer of the in-house chart library the platform's workloads ship on.
  • Team leadership. Three years leading development teams, up to three teams and around 25 engineers, before moving into platform engineering.
  • Consulting. Advising on commerce infrastructure in client-owned environments on other clouds.
  • Training. Internal material and sessions covering the CI platform and the network access process.
Outages

Four incidents

Each one written the way I worked it: what happened, what I found, what I changed. One was closed permanently; the others were stopped at the edge or on capacity, with the structural remediation specified but not all of it shipped.

Cache defeat

A crawler that bypassed the cache

What happened

Three externally confirmed downtime alerts in one night, roughly two to three hours apart, with the web pods cascading into restart loops each time. Overall traffic to the site looked completely normal, so nothing in the business metrics suggested a traffic event.

Investigation

One URL pattern accounted for 70–95% of all requests in each alert window: a single listing endpoint that accepts a multi-value filter parameter. A bot was walking combinations of filter values. Each combination is a distinct cache key, so the CDN and Varnish missed on every request and every one became a full application render against MySQL and the search cluster.

The traffic used rotating, realistic browser User-Agents spread across consumer and residential-proxy networks, indistinguishable from real customer traffic. Per-IP rate limiting would not have caught it and there was no ASN or geography to key on.

What turned load into downtime The web tier ran a small, statically sized PHP-FPM pool per pod. A burst wedged the whole pool for over a minute, and the liveness probe was itself a PHP request through that same pool. The probe timed out, Kubernetes restarted the pods, and the tier lost capacity while it was already saturated.

By the recurrence the cache hit ratio had fallen from around 45% to 29%, with millions of objects evicted under LRU pressure. The databases were idle and healthy throughout, which ruled out the backend and moved the investigation to the edge.

Fix

Mitigation had to be at the edge, because no origin-side lever helps when the cache never gets a hit. A WAF rule on that path, matching on filter-parameter cardinality above a threshold, was the right shape.

A rule for that path already existed and had never once matched this crawler. It was scoped to a single country the traffic was not coming from. Broadening it produced 935 challenge actions in ten minutes, dropped origin-bound hits on the path by 55–60%, and brought every web pod back healthy.

Auditing the rest of the rules on that path then found a second one sitting in log-only mode, matching thousands of requests and passing every one to origin. Closing that, widening the worker pool and moving the liveness probe off the application endpoint were the follow-ups. The first rule had been in place for a long time without matching any of this traffic, so I now check a rule's match count before treating it as coverage.

935
Challenges in the first 10 min
−55–60%
Origin hits on the abused path
18 / 18
Web pods recovered healthy
~45% → 29%
Cache hit ratio, baseline to recurrence
Scheduled load

The cache warmer was causing the outages

What happened

This environment restarted its web fleet on a near-daily cadence for eleven days before producing a hard, customer-facing outage. Because it happened almost daily, I looked at scheduled work before traffic.

Cause

A third-party cache-warmer extension ran on cron and flooded the PHP-FPM pool with cache-miss page renders, which is what it is designed to do. The problem was what it ran into.

The web pods were losing roughly 76% of scheduler periods to their own CPU limit. The horizontal autoscaler was already pinned at its maximum replica count and reporting it. Redis was 3 GB and 97% full, thrashing under LRU eviction, so warmed entries did not survive and each run re-warmed what the previous run had lost. Varnish hit ratio during the incident was around 6%. The liveness probe was again a real application request, so under CPU starvation the fleet cascade-restarted itself.

Three ceilings A CPU limit, an autoscaler maximum and a cache size. None of them alone was dramatic enough to page anyone, and all three interacting produced a daily outage.
What I changed

I specified a set of changes, because any single one would have moved the ceiling instead of removing it: double the web CPU limit, raise the autoscaler maximum, double the Redis allocation so warmed entries survive, move the liveness probe off the application endpoint, and re-tune the warmer's schedule and concurrency so it stops competing with live traffic. Not all of it has shipped.

Autoscaling

The autoscaler could not reach its own maximum

Recurrence

This environment had already had a faceted-crawl outage of the same family as the first case, and the obvious remediation, a much larger PHP-FPM worker pool, had been applied and verified. When it happened again three weeks later, the larger worker pool was already in place, so worker starvation was no longer the likely bottleneck.

Cause

Web CPU stepped from 0.4 to 12 cores. Every web pod sat pinned at its CPU limit with 84–90% throttling, the liveness probe timed out, and the restart loop began. The tier needed to scale out, and the autoscaler was configured to allow eight replicas.

It had never reached eight. The namespace memory ResourceQuota was smaller than the autoscaler's own ceiling: maximum replicas multiplied by the per-pod memory request came to more than the quota allowed for the whole namespace. The autoscaler topped out around four pods, and every pod beyond that failed to be created at all, with a quota rejection that surfaced nowhere near the alert.

The same cause, twice That quota exhaustion had also wedged an unrelated image deployment mid-rollout for several days, failing its progress deadline. Two symptoms days apart, one cause, and neither had been reported as an incident.

An earlier change had raised the quota's memory limits and left requests alone. Requests were the binding constraint, sitting at 97% consumed, so the ceiling people thought they had raised was not the one blocking them.

Fix

I reconciled the quota with the autoscaler so the configured maximum is actually reachable, raising requests.memory along with limits, then right-sized the per-pod memory request downwards so eight replicas fit comfortably, not exactly. The wedged rollout unblocked as soon as the change synced.

One other finding worth carrying forward: the trigger that time was a different crawl shape from the first incident, and the edge rule written for that first incident could not match it. The rule matched on query parameters and this crawl used none, so it passed straight through.

84–90% → 29–50%
Scheduler periods throttled, after the fix
4 → 8
Replicas the autoscaler could reach
Immediate
Recovery of the deploy stuck for days
Node loss

One node failed and nothing recovered

What happened

A complete site-down outage shortly after 01:00 local time. A single worker node had stopped posting status and gone unreachable, which is a routine cloud event the platform should absorb without anyone noticing.

Investigation

That node hosted two single-replica StatefulSets, the environment's MySQL database and its Redis cache. Kubernetes marks the pods for deletion but cannot confirm termination on a node it cannot reach, so they sat in Terminating, and a StatefulSet will not recreate a pod until the old object is gone. Both behaviours are correct on their own. Together they meant the pods would not come back without manual intervention.

Everything downstream then failed its health checks for want of a database: the web tier first, then the storefront rendering layer. The autoscaler read those idle pods at about 1% CPU, concluded the environment was over-provisioned, and scaled the web tier down.

The database's read-write-once disk also stayed attached to the dead node until the cloud provider force-detached it, which risks a multi-attach failure on reschedule.

What I changed

Immediately: force-deleted the stuck stateful pods so the scheduler could place them on healthy nodes, and cleared the stale volume attachment where the disk was still bound to the dead node. The web and storefront tiers recovered on their own once the database and cache were back.

Specified afterwards: a replicated database topology so node loss stops being a total-outage event, and an alert on node readiness. No alert existed on node readiness, so the outage was found by the client.

Lesson
I added node-readiness alerting because the client found this before we did, and flagged the single-replica database and cache as a single point of failure for the whole environment.
Recurring causes

What showed up more than once

The triggers were all different: two crawlers, a scheduled job, a hardware failure. The underlying causes repeated. These are the checks I run now.

01

Check what the liveness probe actually runs

A probe that issues a real application request times out when the application is busy, so Kubernetes restarts pods that are only slow. That turned load into downtime in three of the four incidents here. I check what a liveness probe actually executes before looking anywhere else.

Seen in: cache defeat · scheduled load · autoscaling
02

Check which of two limits binds first

An autoscaler maximum above what the namespace quota can admit; a cache store configured larger than its container's memory limit. Each passes review on its own. I now read the two together and check which one binds first.

Seen in: autoscaling · scheduled load
03

Check the cache hit ratio before the request rate

Every load-driven incident here was a flood of uniquely keyed requests. Once each request has its own cache key, rate limits, extra replicas and bigger pods do very little, because the cache never gets a hit. I check the cache hit ratio early to tell the two apart. Two of the four cases here were still closed on capacity, not at the edge.

Seen in: cache defeat · autoscaling · scheduled load
04

Check that redundancy has ever been exercised

The autoscaler allowed eight replicas, but the namespace quota only had room for about four, and nobody had checked because it had never needed eight. For autoscalers, verify the namespace quota can admit the configured maximum. For stateful workloads, verify there is more than one replica.

Seen in: node loss · autoscaling
05

Check that a rule is matching, not just present

The edge rule that would have prevented the first incident already existed, scoped to a country the traffic never came from. A second rule on the same path was matching thousands of requests in log-only mode. Both appeared in the rule list as coverage. Before closing an incident, I pull match counts and actions for every rule on the path.

Seen in: cache defeat
06

Check why nobody was paged

The most severe of the four was found by the client, with no alert firing. Treat a client-reported outage with no alert as a monitoring action in its own right, not a footnote to the root cause.

Seen in: node loss
Tools

What I work in

Twenty years in web infrastructure, twelve at the same agency and the last eight on this platform, so I can read the application's behaviour as well as the cluster's.

Platform & infrastructure

Multi-cluster GKE on Google Cloud, cloud infrastructure in Terraform, fleet configuration through Puppet and Ansible. Workloads ship via Helm and ArgoCD on a GitOps model, against an in-house chart library and Jenkins pipelines.

KubernetesGKETerraformHelmArgoCDPuppetAnsible

Observability

Metrics, long-term storage, logs, dashboards and alert routing, plus the monitoring rotation that turns alert volume into a prioritised cluster health report.

PrometheusThanosGrafanaLokiAlertmanager

Edge & security

A multi-tenant CDN edge in front of client storefronts: per-brand custom hostnames, WAF rules, rate limiting and bot mitigation. This is where a cache-miss flood has to be stopped when the trigger is external.

Cloudflare SaaSWAFBot mitigationTLSVault

Languages

Python and Go for tooling and services, Bash for the fleet, PHP and TypeScript from a long stretch of application development.

PythonGoBashPHPTypeScriptGroovy

Data & messaging

The stateful half of the platform: managed and in-cluster relational databases, caches, search clusters and message brokers, including the failure modes above.

MySQL / MariaDBCloudSQLPostgreSQLRedis / ValkeyOpenSearchRabbitMQ

Application layer

What actually runs on the platform. Years building and leading teams on this stack before moving into platform engineering.

Adobe CommercePHP-FPMVarnishNginxMySQLRedisOpenSearch