Technology
How American Express contains payment failures without sharing in-flight state
ByteByteGo's interview with American Express engineer Ben Cane explains a payment architecture organized around independent failure domains. Each cell contains the services and data needed for a real-time transaction; a deliberately thin router handles boundaries. The central tradeoff is to repeat short, recoverable work rather than couple healthy cells to a failing cell's partial state. Recovery still depends on payment-specific rules, including irreversible steps and duplicate suppression.
Editorial summary · Publisher article
What matters
- Reference data is distributed ahead of requests; selected state-sensitive transactions are routed to the data they need.
- Cells avoid blocking cross-cell calls in the transaction path, while replication and global observability remain asynchronous.
- Restarting work is allowed only within a defined recovery window; downstream idempotency handles applicable retry cases.
- Gradual traffic restoration limits the risk of moving everything back to a newly recovered cell.
Failure domains, not merely microservices
A cell is a self-sufficient processing stack, not just a service or server. Its boundary includes databases, DNS and supporting infrastructure. Cells can be removed from rotation independently. Reference data and telemetry still move between boundaries, but a payment must not wait on another cell during its critical processing path. This limits the reach of an outage, at the cost of duplicating infrastructure and managing more independent units.
Route to changing data
Immutable and slowly changing reference information is pushed into cells before requests arrive, avoiding cold-cache lookups to a central authority. Frequently changing transaction state cannot be made perfectly current everywhere through asynchronous replication. For use cases that require it, deterministic routing sends related requests to the appropriate data. Other payment types can use priority-based routing to healthy cells; the article does not present one routing mode as correct for every transaction.
Keep the global router deliberately limited
The router parses enough of a message to route it but avoids accumulating business logic and synchronous data dependencies. Configuration updates and logging are asynchronous, and multiple instances operate across regions. When configuration cannot be refreshed, the router retains known values. Under sustained pressure it can drop general diagnostic logs rather than block payment processing. Those availability choices carry observability costs; they do not imply that required financial records are expendable.
Restart only where business semantics permit
When a service fails, the orchestrator can return the original request to the router for processing in another healthy cell. It discards partial internal work rather than asking the replacement cell to resume from failing infrastructure. Short payment workflows make that repetition tolerable. But recovery is bounded by a point of no return, such as an external issuer interaction. The design deliberately places irreversible work late where possible; it does not authorize blindly replaying completed side effects.
Recovery has costs as well as safeguards
Stable transaction identifiers support duplicate suppression in relevant downstream systems. Percentage-based failback gradually tests recovering capacity. Strong-consistency requirements can force a transaction to be rejected when necessary data cannot be validated. Global dashboards lag local activity, and duplicated services increase operational overhead. The transferable lesson is to choose explicit failure and recovery semantics, not assume that a cell label alone provides consistency or exactly-once execution.
Context & caveats
- This is an attributed architecture interview, not an independent reliability audit or a universal payment-system recipe.
- The article's sponsor segment is excluded. Dropped diagnostic logs are not the same as lost transactional records.
Why this made the edition
Interactive editorial review of the supplied source; selected for substantive reader value with limitations retained.
Source material