fix(queue-amqp): prevent infinite poison-pill loop on message error by supporting requeue: false and DLQ routing #20

Open
opened 2026-10-27 14:51:28 +00:00 by crapougnax · 0 comments
Owner

Migrated from GitHub https://github.com/Quatrain/Core/issues/53 (opened by @crapougnax on 2026-09-21).

🎯 Goal

Prevent infinite redelivery loops ("poison pill") in @quatrain/queue-amqp by enabling configurable negative acknowledgment (requeue: false) and proper Dead Letter Queue (DLQ) routing when a message handler fails.

📋 Context & Problem Statement

In packages/queue-amqp/src/AmqpQueueAdapter.ts (lines 129–138):

} catch (err) {
   this._logger.error(
      `messageHandler function failed with message: ${
         (err as Error).message
      }`
   )
   // Negative acknowledge the message so it can be requeued or handled by DLQ
   channel.nack(msg)
   process.exit(1)
}
  1. Default requeue: true in amqplib:
    The method signature is channel.nack(message, allUpTo?, requeue?). Omitting requeue defaults to true.
  2. DLQ Failure:
    The inline comment states "Negative acknowledge the message so it can be requeued or handled by DLQ". However, RabbitMQ never dead-letters a message when requeue: true; it only routes messages to a Dead Letter Exchange (DLX) when negatively acknowledged with requeue: false (or when TTL/length limits expire).
  3. Infinite Crash Loop (Poison Pill):
    Calling channel.nack(msg) with requeue: true followed by process.exit(1) causes RabbitMQ to put the failed message back at the head of the queue immediately. When the container restarts in Kubernetes or another worker replica consumes the queue, the exact same deterministically broken message is picked up again, causing infinite CrashLoopBackOff and cluster resource exhaustion.

🛠️ Proposed Solution & Architecture

  1. Configurable Requeue Parameter:
    Add a requeueOnError parameter in QueueListenOptions / AmqpQueueAdapter:
    export interface AmqpListenOptions {
       concurrency?: number
       gpu?: boolean
       requeueOnError?: boolean // Defaults to false or configurable via options / env
    }
    
  2. Explicit channel.nack(msg, false, requeue):
    Update AmqpQueueAdapter.ts:
    const shouldRequeue = params?.requeueOnError ?? false
    channel.nack(msg, false, shouldRequeue)
    
    When requeue is false, RabbitMQ will either drop the poison message or route it to the configured Dead Letter Exchange (DLX/DLQ) if dead-lettering is enabled on the queue (x-dead-letter-exchange).
  3. Configurable Exit on Error:
    Rather than unconditionally executing process.exit(1) for every message failure, allow configuring whether the worker process should terminate on handler error or continue processing subsequent messages once the failed message has been safely dead-lettered / nacked.

🧪 Acceptance Criteria

  • channel.nack in AmqpQueueAdapter respects a configurable requeue flag (requeue: false supported).
  • Messages with deterministic handler failures can be routed to RabbitMQ Dead Letter Queues (DLQ) without requeuing.
  • Unit tests cover listen() behavior on message handler failure with both requeue: false and requeue: true.
> Migrated from GitHub https://github.com/Quatrain/Core/issues/53 (opened by @crapougnax on 2026-09-21). ### 🎯 Goal Prevent infinite redelivery loops ("poison pill") in `@quatrain/queue-amqp` by enabling configurable negative acknowledgment (`requeue: false`) and proper Dead Letter Queue (DLQ) routing when a message handler fails. ### 📋 Context & Problem Statement In `packages/queue-amqp/src/AmqpQueueAdapter.ts` (lines 129–138): ```typescript } catch (err) { this._logger.error( `messageHandler function failed with message: ${ (err as Error).message }` ) // Negative acknowledge the message so it can be requeued or handled by DLQ channel.nack(msg) process.exit(1) } ``` 1. **Default `requeue: true` in `amqplib`**: The method signature is `channel.nack(message, allUpTo?, requeue?)`. Omitting `requeue` defaults to `true`. 2. **DLQ Failure**: The inline comment states *"Negative acknowledge the message so it can be requeued or handled by DLQ"*. However, RabbitMQ **never** dead-letters a message when `requeue: true`; it only routes messages to a Dead Letter Exchange (DLX) when negatively acknowledged with `requeue: false` (or when TTL/length limits expire). 3. **Infinite Crash Loop (Poison Pill)**: Calling `channel.nack(msg)` with `requeue: true` followed by `process.exit(1)` causes RabbitMQ to put the failed message back at the head of the queue immediately. When the container restarts in Kubernetes or another worker replica consumes the queue, the exact same deterministically broken message is picked up again, causing infinite CrashLoopBackOff and cluster resource exhaustion. ### 🛠️ Proposed Solution & Architecture 1. **Configurable Requeue Parameter**: Add a `requeueOnError` parameter in `QueueListenOptions` / `AmqpQueueAdapter`: ```typescript export interface AmqpListenOptions { concurrency?: number gpu?: boolean requeueOnError?: boolean // Defaults to false or configurable via options / env } ``` 2. **Explicit `channel.nack(msg, false, requeue)`**: Update `AmqpQueueAdapter.ts`: ```typescript const shouldRequeue = params?.requeueOnError ?? false channel.nack(msg, false, shouldRequeue) ``` When `requeue` is `false`, RabbitMQ will either drop the poison message or route it to the configured Dead Letter Exchange (DLX/DLQ) if dead-lettering is enabled on the queue (`x-dead-letter-exchange`). 3. **Configurable Exit on Error**: Rather than unconditionally executing `process.exit(1)` for every message failure, allow configuring whether the worker process should terminate on handler error or continue processing subsequent messages once the failed message has been safely dead-lettered / nacked. ### 🧪 Acceptance Criteria - [ ] `channel.nack` in `AmqpQueueAdapter` respects a configurable `requeue` flag (`requeue: false` supported). - [ ] Messages with deterministic handler failures can be routed to RabbitMQ Dead Letter Queues (DLQ) without requeuing. - [ ] Unit tests cover `listen()` behavior on message handler failure with both `requeue: false` and `requeue: true`.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
quatrain/core#20
No description provided.