Triage
ResolvedDetected 30d agoPRB-2026-05-13-001

Failure rate increase on shop-checkout

Agent's plan

The shop-checkout service is experiencing a significant increase in failure rate (from 0.1% to 18.4%) and elevated response times, primarily due to a TypeError related to reading an undefined property 'currency' in the calculateTotals function during POST /checkout requests. This issue started immediately after a recent deployment of shop-checkout from version v4.1.9 to v4.2.0.

Blast radius

The shop-checkout service is directly affected, leading to failed checkout operations for users. The problem is contained to this service.

Top hypothesis

Recent deployment of v4.2.0 introduced a bug

90%

The problem started immediately after the deployment of shop-checkout v4.2.0 (finding 3, finding 0). The specific error message 'TypeError: Cannot read properties of undefined (reading 'currency')' (finding 2) strongly suggests a code-level issue introduced in the new version, leading to the observed failure rate increase (finding 1).

Proposed actions

  1. rollback · Rollback shop-checkout service to version v4.1.9

    This action directly addresses the leading hypothesis that the v4.2.0 deployment introduced the bug. Rolling back to the last known stable version (v4.1.9) is the most direct and reversible way to mitigate the high failure rate and restore service functionality quickly.

    target: shop-checkout

    low riskreversible

Postmortem

What happened

The online store's checkout service experienced a significant increase in failed transactions, preventing customers from completing their purchases. This problem began shortly after a new software version was deployed to the shop-checkout service. We quickly identified the new release as the cause and reverted the service to its previous stable version, which restored normal operations.

Timeline

  • 2026-05-15T08:46:14.519556Z Detected failure rate increase on shop-checkout.
  • 2026-05-15T08:46:14.965729Z Investigation started.
  • 2026-05-15T08:46:16.943915Z Problem P-251301: Failure rate increase on shop-checkout identified.
  • 2026-05-15T08:46:16.943929Z Failure rate on shop-checkout jumped from 0.1% to 18.4%.
  • 2026-05-15T08:46:16.943943Z Top error: TypeError: Cannot read properties of undefined (reading 'currency') in calculateTotals.
  • 2026-05-15T08:46:16.943962Z Most recent deployment: shop-checkout v4.2.0 at 2026-05-15T08:27:50.642698+00:00.
  • 2026-05-15T08:46:30.063688Z Plan to rollback shop-checkout to v4.1.9 proposed.
  • 2026-05-15T08:46:34.771151Z Plan approved by operator.
  • 2026-05-15T08:46:35.096922Z Rollback started for shop-checkout to v4.1.9.
  • 2026-05-15T08:46:35.230614Z Rollback completed.
  • 2026-05-15T08:46:35.274198Z Metrics returned to baseline.

Root cause

The root cause was a bug introduced in the shop-checkout service version v4.2.0. This deployment, which occurred at 2026-05-15T08:27:50.642698+00:00 (finding 3), caused a TypeError: Cannot read properties of undefined (reading 'currency') in the calculateTotals function during checkout requests (finding 2). This error led directly to the observed increase in failure rates.

Resolution

The issue was resolved by rolling back the shop-checkout service to its previous stable version, v4.1.9. This action was approved by the operator. We confirmed recovery when the service's failure rate returned to baseline, dropping from 18.4% to 0.12%.

Action items

  • Investigate and Fix v4.2.0 Bug: Create a Jira ticket to analyze the TypeError: Cannot read properties of undefined (reading 'currency') in shop-checkout v4.2.0 and implement a code fix.
  • Automated Pre-Deployment Tests: Create a Jira ticket to implement automated tests for the shop-checkout service that run before deployment to catch similar regressions.
  • Canary Release Strategy: Create a Jira ticket to review and update the shop-checkout deployment pipeline to incorporate a canary release strategy for new versions.