Reliability lab2026Public source

n8n Reliability Lab

A workflow lab for validation, idempotency, retry, approval, and recovery.

I designed the reliability model, PostgreSQL ledger, workflow boundaries, deterministic fault fixtures, acceptance scenarios, and operator-recovery path.

Role: Workflow reliability designer and builder
Capability proof: Shows how a small automation can become auditable and recoverable through idempotency, persisted attempts, bounded retry, approval, dead letters, and replay.
Technology: n8n · JavaScript · PostgreSQL · Docker

Recording pendingn8n reliability walkthrough

A prepared recording flow for bounded retry, dead-letter handling, and operator recovery in a localhost lab.

Data: Synthetic leads, events, credentials, and services
Runtime: Local Docker: n8n, PostgreSQL, and WireMock on loopback only
External side effects: None outside the lab; WireMock is a deterministic fixture, not a real CRM
Latest verification: No verified public demo artifact yet

Coverage

500 → 429 → 201 with a final delivered state
503 × 3 → dead letter → operator replay → 201 with recovery recorded
Persisted attempts, bounded retry, and protected recovery flow

Limitations

The public recording, captions, and metadata review are not complete
The current recording script does not independently show a pure happy path or duplicate submission
The localhost lab will not be exposed as an online system

Context

A webhook demo is easy when every downstream call succeeds. The more useful question is what remains true after duplicate delivery, a retryable failure, an approval wait, an exhausted retry budget, or an operator replay.

The n8n Reliability Lab is a small localhost system that accepts synthetic leads, persists state before side effects, gates selected events on human approval, records bounded delivery attempts, opens dead letters, and safely reuses the same delivery path for recovery.

Problem

n8n execution memory cannot safely coordinate overlapping submissions, process restarts, and operator actions by itself. The lab therefore makes PostgreSQL the ledger for event identity, state transitions, approval decisions, attempts, dead letters, and replay ownership.

The delivery guarantee is deliberately described as effectively-once, not exactly-once. The durable event ID becomes the downstream Idempotency-Key, but a crash can still happen after an external service accepts a request and before the attempt is recorded. An idempotency-aware receiver can collapse that replay; the lab does not pretend the crash window disappeared.

Acceptance scenarios cover invalid input, duplicate submission, approval, transient 500 → 429 → 201, exhausted 503 × 3, and protected recovery. A 90-second walkthrough script is prepared for the retry and recovery sequences, but no public recording or verified Watch link is published yet. n8n, PostgreSQL, and WireMock remain on localhost, and WireMock is a deterministic fault fixture rather than a real CRM.

What this case proves

Persistence and idempotency boundaries matter more than the number of workflow nodes.
Failure should become a queryable state with a bounded recovery path.
Honest reliability language is narrower—and more credible—than an impossible cross-system guarantee.

Constraints

Constraints that shape the system

n8n executions can restart or overlap, so workflow memory cannot be the source of truth for delivery or approval state.
PostgreSQL and an external HTTP service cannot provide mathematical exactly-once delivery as one transaction.
Recovery must be safe when an operator repeats a replay request or when the original failure had an uncertain side effect.

Key decisions

Choices, alternatives, and trade-offs

Persist before side effects

Context: If a workflow calls the CRM before recording intent and attempt state, a restart can turn one event into an untraceable duplicate.
Choice: Use PostgreSQL functions and row-level claims to persist intake, decisions, attempts, dead letters, and replay ownership.
Alternatives: Keep state only in n8n execution data; Use a simple retry loop without a durable ledger
Trade-off: The database contract is more explicit, but recovery no longer depends on reconstructing workflow memory.

Inspect supporting evidence

State an effectively-once boundary

Context: A crash can occur after the fixture accepts a request but before PostgreSQL records the response.
Choice: Reuse the durable event ID as the downstream Idempotency-Key and document the remaining crash window.
Alternatives: Claim exactly-once across systems; Disable retries and accept more manual recovery
Trade-off: The guarantee is deliberately narrower, but it is implementable and reviewable.

Inspect supporting evidence

Architecture and failure behavior

Responsibility, input, output, and failure

01
Validate and claim
Normalize input and atomically claim event identity.
Input
Webhook payload
Output
Rejected, awaiting approval, or processing state
Failure
Invalid or conflicting input stops before downstream delivery.
02
PostgreSQL ledger
Own state transitions, attempts, decisions, and replay claims.
Input
Workflow commands
Output
Versioned business and audit records
Failure
Constraints reject illegal transitions and concurrent ownership.
03
Approval boundary
Require a protected operator decision for high-risk events.
Input
Pending request and operator action
Output
Approved or rejected durable state
Failure
Opposing or repeated decisions become conflicts or no-ops.
04
Bounded delivery
Record each HTTP attempt and classify retryability.
Input
Claimed event and deterministic fixture behavior
Output
Delivered or dead-letter state
Failure
Retries stop at the bound; non-retryable faults stop immediately.
05
Operator recovery
Claim one open dead letter and replay the shared delivery path.
Input
Protected replay request
Output
Recovered delivery, no-op, or not-found result
Failure
Repeated replay after recovery creates no new CRM call.

Failure modes

How failure is detected, bounded, and escalated

Trigger	Detection and response	Side effects and human boundary	Assurance
The webhook payload is structurally invalid.	Validation fails before event finalization. Persist or return a rejected result with bounded detail.	No CRM request is made. The sender corrects the payload contract.	Verified
The same event is submitted again.	Atomic event-id claim returns the existing record. Return a duplicate/no-op result.	The successful downstream call count remains one. A conflicting payload requires review instead of overwrite.	Verified
The fixture responds 500, then 429, then 201.	Each response is recorded and classified as retryable or successful. Retry within the bound and finish delivered on 201.	The same Idempotency-Key is reused for every attempt. None while the bounded sequence recovers.	Verified
Three retryable 503 responses exhaust delivery.	The persisted attempt count reaches the configured maximum. Open one dead letter and emit one local alert fixture call.	No fourth automatic attempt occurs. A protected operator replay can claim the dead letter once.	Verified
A high-value or high-risk event requires approval.	Classification moves the event to awaiting_approval. Delivery waits for an authenticated operator decision.	No downstream call occurs before approval. Reject is terminal; an opposing later decision is a conflict.	Verified

Project evidence

Evidence and its verification scope

Verification

Deterministic failure acceptance scenariosVerified scope

Invalid payloads, duplicate events, transient failures, exhausted retries, and approval gates are exercised as explicit scenarios.

Local workflow lab and fixtures.n8n Reliability Lab README — workflow evidence

Architecture

Effectively-once boundaryVerified scope

The repository describes idempotent PostgreSQL-to-HTTP delivery without claiming mathematical exactly-once behavior across independent systems.

Documented local system boundary.n8n Reliability Lab README — reliability guarantees

Demo boundary

Evidence from a running local n8n instanceVerified scope

Workflow close-ups are captured from the running local instance rather than recreated as an unverified diagram.

Local demonstration only.n8n Reliability Lab README — workflow evidence

Honest status

Implemented, not claimed, and still required

Running status: Reliability lab
Data: All people, companies, events, credentials, and remote systems are synthetic.
External integrations: n8n, PostgreSQL, and WireMock run on localhost; WireMock is a deterministic fault fixture, not a CRM.
Revenue claim: This lab does not claim production usage, customers, or revenue.
Source: Public source
Before production: Managed persistence, secret management, operator identity, observability, network policy, and real downstream idempotency retention remain required.

Lab boundary: This is a local reliability laboratory. “Production-oriented” describes the failure patterns and controls, not a claim of current production use.

Reflection and next priorities

What worked

Putting concurrency rules in PostgreSQL made duplicate delivery, approval conflict, and replay ownership testable across workflow restarts.

What I would change

I would add a first-class observability view for dead-letter age and replay ownership before expanding the number of workflow examples.

Next priorities

Run the complete Docker acceptance flow and preserve a dated verification record.
Record the bounded-retry and recovery walkthrough with captions and a transcript.
Add a managed staging profile before adapting the pattern to a real workflow.

Not worth doing yet: More n8n nodes or provider examples are lower value than proving one recovery path with real operational ownership.

Hiring

Discuss a role using this evidence

Discuss a suitable role

Project

Discuss a focused adaptation

View source Discuss this project