The Data Incident Postmortem: A Template for Data Engineering Teams

The gap between data teams that improve and data teams that experience the same incidents repeatedly is usually one thing: postmortems. A postmortem that captures root cause, timeline, and prevention actions creates organizational memory. Without it, the same pipeline breaks for the same reason six months later, the same triage process unfolds, and the same hours are lost. The template below is structured to capture what matters without becoming a bureaucratic burden.

Why Most Data Incident Postmortems Fail

Data engineering teams that try to adopt postmortems often abandon them after a few months for one of three reasons. First, the template is too long and too generic — borrowed from software engineering incident response frameworks that ask about SLO breaches and customer-facing impact in ways that do not map to internal data pipeline incidents. Second, there is no clear owner of the postmortem document — it gets assigned to "the team" which means it gets assigned to no one. Third, the action items from previous postmortems are not tracked, so the process has no visible impact on the incident rate and loses credibility.

A postmortem that gets used is short, specific to data incidents, has a named owner, and has a visible output: action items that are tracked to completion and a measurable change in incident rate over time.

The Decube Data Incident Postmortem Template

This template is designed to be completed within 24 hours of incident resolution. The named owner is the engineer who led the incident response. It should take 30-45 minutes to complete for a typical pipeline incident.

Section 1: Incident Summary (5 minutes)

• Incident date and time (start, detection, resolution)
• Duration: time from incident start to data restored
• Detection: how was the incident first identified? (Monitoring alert, stakeholder complaint, engineer noticed, other)
• Affected assets: which tables, pipelines, dashboards were impacted?
• Business impact: which teams, decisions, or reports were affected?
• Severity: P1 (executive dashboard affected), P2 (operational report affected), P3 (internal data issue, no stakeholder impact)

Section 2: Root Cause (15 minutes)

This is the most important section and the one most commonly done poorly. Root cause should be specific and verifiable — not "data quality issue" or "pipeline failure" but the precise change or condition that caused the incident. Use the five-whys method: ask why the visible symptom occurred, then why that occurred, until you reach a root cause that, if addressed, would prevent recurrence.

• Symptom: what was observed? (e.g., "revenue dashboard showed 0 for yesterday")
• Why 1: what caused the symptom? (e.g., "the orders_daily dbt model failed")
• Why 2: what caused that? (e.g., "column order_type was renamed to order_category in the source CRM")
• Why 3: what caused that? (e.g., "product team deployed a schema migration without notifying data engineering")
• Why 4: what enabled that? (e.g., "there is no process requiring product team to notify data engineering of schema changes")
• Root cause: absence of a schema change notification process between product engineering and data engineering

Note: the root cause in this example is organizational, not technical. Many data incidents have organizational root causes that cannot be addressed with monitoring tooling alone. Honest root cause analysis surfaces these rather than stopping at the technical proximate cause.

Section 3: Timeline (10 minutes)

Reconstruct the timeline with timestamps. Include: when the source change occurred, when the pipeline last ran successfully, when the pipeline first failed, when the failure was detected, when investigation started, key investigation steps and what they revealed, when the fix was applied, when data was verified as correct, when stakeholders were notified. A precise timeline identifies where time was lost and which investigation steps were most and least efficient. Teams that track timelines across incidents typically discover a consistent bottleneck that, once addressed, significantly reduces mean time to resolution.

Section 4: What Worked Well

Postmortems focus on failure, which creates a bias toward negative findings. Explicitly capturing what worked well — the monitoring that detected the issue, the runbook that guided the investigation, the ownership record that got the right person paged — reinforces effective practices and prevents them from being abandoned in the next round of process changes.

Section 5: What Did Not Work Well

Be specific. "Communication was slow" is not actionable. "The Slack alert went to the #data-alerts channel that 14 people are in but no one is explicitly responsible for, so it sat unacknowledged for 47 minutes" is actionable. Specific failure descriptions enable specific action items.

Section 6: Action Items

Each action item should have a named owner and a due date. Action items without owners do not get done. Action items without due dates get deprioritized indefinitely. Categories of typical action items from data incident postmortems:

• Monitoring gaps: "Add freshness monitor for orders_daily with 4-hour alert threshold" — Owner: [engineer], Due: [date]
• Ownership gaps: "Assign named owner to orders_daily pipeline and configure PagerDuty routing" — Owner: [manager], Due: [date]
• Process gaps: "Create schema change communication protocol with product engineering team and include in their deployment checklist" — Owner: [data lead + product eng manager], Due: [date]
• Documentation gaps: "Write runbook for orders pipeline failures" — Owner: [engineer], Due: [date]
• Tooling gaps: "Evaluate schema change detection tools and select one for implementation" — Owner: [data lead], Due: [date]

Tracking Postmortem Effectiveness Over Time

A postmortem process that produces no measurable improvement is not a postmortem process — it is documentation theater. The measure of effectiveness is simple: is the rate of similar incidents declining over time? Every postmortem should be tagged with its root cause category (schema change, connectivity failure, logic error, etc.) and tracked in a shared log. Quarterly review of the log reveals which categories are declining (indicating prevention actions are working) and which are recurring (indicating prevention actions are insufficient or not being implemented).

The category-level view is also useful for prioritizing observability investment. If 40% of incidents in the past quarter were triggered by schema changes, that is a clear signal to invest in automated schema change detection before other observability improvements.

How Decube Supports Postmortem Quality

Decube's incident timeline view generates an automatic timeline of relevant events in the 24 hours before an incident is reported — schema changes detected, freshness violations, volume anomalies, lineage changes — that provides a starting point for the postmortem timeline without requiring manual reconstruction from memory and log files. This reduces the time required to complete Section 3 from 20-30 minutes to 5-10 minutes for incidents that involved Decube-monitored assets.