---
title: "Dead-Letter Queue: Stop Failed Agent Tasks from Looping"
description: "Send failed agent tasks to a dead-letter queue, cap retries with backoff, and replay or escalate each one before a poison pill crashes your workers."
canonical: "https://adamtheautomator.com/dead-letter-queue-agent-tasks/"
---

# Dead-Letter Queue: Stop Failed Agent Tasks from Looping

> Send failed agent tasks to a dead-letter queue, cap retries with backoff, and replay or escalate each one before a poison pill crashes your workers.

Source: https://adamtheautomator.com/dead-letter-queue-agent-tasks/

---

ATA Learning

Tap to hide

[

ATA Learning

](/)

*   [Home](/)
*   [Tutorials](/tutorials/)
*   [Instructors](/author/)
*   [Advertising](/advertising/)
*   [Recommended Resources](/resources/)
*   [About Adam](/about-adam/)

Search for:  

*   [](https://twitter.com/adbertram)
*   [](https://github.com/Adam-the-Automator)
*   [](https://www.linkedin.com/company/adam-the-automator-llc)
*   [](/feed/)

![Dead-Letter Queue: Stop Failed Agent Tasks from Looping](https://adamtheautomator.com/wp-content/uploads/publisher/4165d9c85b2b816db4e7cf3aba91f765/240388c80ed62bdb9723757dee9abad2771d764e952d0b7a30a2d01f1d036000.png)

# Dead-Letter Queue: Stop Failed Agent Tasks from Looping

[![](https://secure.gravatar.com/avatar/d0b9d42e21e5622713f8b693aa5c0f9244d5f7dd200ed29b8398f52dee5de337?s=192&d=mm&r=g)Adam Bertram](https://adamtheautomator.com/author/adam-bertram/)16 November 202610 min. read

Categories: [AI Agents](/category/ai-agents/)

Tags:[AI](/tag/ai/)[Automation](/tag/automation/)[Redis](/tag/redis/)[Python](/tag/python/)

Table of Contents

*   [Prerequisites](#prerequisites)
*   [Understanding Agent-Specific Failure Modes](#understanding-agent-specific-failure-modes)
*   [Setting Up a Dead-Letter Queue in BullMQ](#setting-up-a-dead-letter-queue-in-bullmq)
*   [Configuring Failed Job Retention](#configuring-failed-job-retention)
*   [Cap Stalled-Job Redeliveries](#cap-stalled-job-redeliveries)
*   [Configuring Celery’s Late Acknowledgment and Retry Limits](#configuring-celerys-late-acknowledgment-and-retry-limits)
*   [Set Late Acknowledgment and Time Limits](#set-late-acknowledgment-and-time-limits)
*   [Capture Failed Tasks in a Dead-Letter List](#capture-failed-tasks-in-a-dead-letter-list)
*   [Implementing Exponential Backoff for Transient Failures](#implementing-exponential-backoff-for-transient-failures)
*   [Add Jittered Retries to the Celery Task](#add-jittered-retries-to-the-celery-task)
*   [Force a Rate Limit to Watch the Backoff](#force-a-rate-limit-to-watch-the-backoff)
*   [Preserving Context for Safe Replay](#preserving-context-for-safe-replay)
*   [Capturing Task Metadata](#capturing-task-metadata)
*   [Inspect a Failed Job Before Replaying It](#inspect-a-failed-job-before-replaying-it)
*   [Building a Recovery Workflow with Human Review](#building-a-recovery-workflow-with-human-review)
*   [Make Low-Confidence Results Fail Explicitly](#make-low-confidence-results-fail-explicitly)
*   [Classifying Failure Types](#classifying-failure-types)
*   [Trigger Each Failure Class on Purpose](#trigger-each-failure-class-on-purpose)
*   [Monitoring DLQ Metrics and Alerting on Patterns](#monitoring-dlq-metrics-and-alerting-on-patterns)
*   [Metrics to Track](#metrics-to-track)
*   [Setting Up Alerts](#setting-up-alerts)
*   [Run a Poison-Pill Drill Against Your Production Image](#run-a-poison-pill-drill-against-your-production-image)

Don’t count on retry settings alone to protect your agent workers. Depending on how your queue acknowledges work, a task that triggers an out-of-memory kill either vanishes with its worker or bounces from worker to worker until you step in by hand. When your long-running AI pipeline processes hundreds of jobs an hour and one task kills its worker, you need to know which task it was, what it was trying to do and whether replaying it will crash the next worker too.

In this tutorial, you’ll build a dead-letter queue (DLQ) for an agent pipeline in BullMQ or Celery. The DLQ gives you one place to inspect every task that exhausted retries, crashed its worker or returned a low-confidence result your pipeline can’t use.

## Prerequisites

If you want to follow along hands-on, you’ll need:

*   **Node.js 18+ or Python 3.10+** for the [BullMQ](https://docs.bullmq.io/) or [Celery](https://docs.celeryq.dev/) path, with its packages installed through `npm install bullmq ioredis uuid` or `pip install -U "celery[redis]" flower`.
    
*   **[Redis](https://redis.io/) 6+** as the broker for both paths, plus `redis-cli` for inspecting queue keys.
    
*   **An agent task you can make fail on demand**, or the drill tasks this tutorial builds for a rate limit, bad input and a worker crash.
    

## Understanding Agent-Specific Failure Modes

Agent tasks leave partial state scattered across logs, API call history, and the message broker when they crash mid-execution. Standard web requests retry safely after timeouts because they’re stateless, but your agent’s in-flight work has already written to multiple systems before the failure.

[OpenLegion’s guide to AI agent reliability](https://www.openlegion.ai/en/learn/ai-agent-reliability) sorts agent pipeline failures into four classes that standard queues don’t handle well:

| Failure Class | Behavior | Retry Safety |
| --- | --- | --- |
| Transient API failure | HTTP 429 rate limit or 503 timeout | Safe to retry with backoff |
| Confidence collapse | Agent refuses task or returns low-confidence null | Permanent; human review needed |
| Resource exhaustion | Out-of-memory kill or CPU timeout | Dangerous; may loop if requeued |
| Logic error | Exception in tool invocation or malformed output | Permanent; inspect and fix |

The [Celery documentation](https://docs.celeryq.dev/en/stable/userguide/configuration.html) states: “If the worker is terminated during the execution of a task, and the task has not been acknowledged yet, the task will be redelivered to another worker.” This default behavior creates a loop when the task itself causes the worker termination.

* * *

_**Warning: Without a dead-letter queue, a poison pill keeps taking down workers until someone spots the pattern in the crash logs. Cap redeliveries before you let a crashed task go back on the queue.**_

* * *

## Setting Up a Dead-Letter Queue in BullMQ

BullMQ moves jobs that exhaust their retries into a [failed set you can inspect and retry from](https://docs.bullmq.io/guide/retrying-failing-jobs).

### Configuring Failed Job Retention

Create a queue with retry limits and DLQ-aware failure handling:

```javascript
import { Queue, Worker } from 'bullmq';

const agentQueue = new Queue('agent-tasks', {
  connection: { host: 'localhost', port: 6379 },
  defaultJobOptions: {
    attempts: 3,
    backoff: {
      type: 'exponential',
      delay: 2000,
      jitter: 0.5
    },
    removeOnComplete: { count: 100 },
    removeOnFail: false  // Keep failed jobs for DLQ inspection
  }
});
```

Here’s what each option does:

*   `attempts: 3` caps how many times BullMQ runs a job that throws before it gives up.
    
*   `backoff` with `type: 'exponential'` and `delay: 2000` waits 2 seconds before the first retry and doubles the wait each time after, giving a throttled API time to recover. `jitter: 0.5` randomizes each delay so jobs that failed together don’t retry together.
    
*   `removeOnComplete: { count: 100 }` keeps only the 100 most recent completed jobs so successful work doesn’t pile up in Redis.
    
*   `removeOnFail: false` keeps every failed job in Redis for inspection and replay instead of deleting it.
    

### Cap Stalled-Job Redeliveries

`attempts` only counts failures your processor throws. When a task kills the worker process outright (an out-of-memory kill, for example), the worker stops renewing the job’s lock and BullMQ marks the job as [stalled](https://docs.bullmq.io/guide/workers/stalled-jobs), then moves it back to `wait` for the next worker. The worker’s `maxStalledCount` option caps that cycle: once a job stalls more times than the limit (default `1`), BullMQ moves it to the `failed` set instead of handing it to another worker. Set it explicitly where you create the worker, with `processAgentTask` as your job handler function. `1` matches the default, but writing it down puts the cap in front of code review if someone later raises it to quiet a flaky worker:

```javascript
const worker = new Worker('agent-tasks', processAgentTask, {
  connection: { host: 'localhost', port: 6379 },
  maxStalledCount: 1
});
```

Check that the queue reached Redis:

```bash
redis-cli --scan --pattern "bull:agent-tasks:*"
```

The output includes `bull:agent-tasks:meta`, pictured below. That key only proves the queue connected: `attempts`, `backoff` and `removeOnFail` live in your `Queue` instance and get copied onto each job as you add it, so the failure drills later in this tutorial prove they work. Once a job fails, the dead-letter set `bull:agent-tasks:failed` appears too. Use `--scan` rather than `KEYS` on a shared Redis server, because `KEYS` blocks every other client until it finishes walking the keyspace.

![Redis queue keys](https://adamtheautomator.com/wp-content/uploads/publisher/b05bf8e681eff39a59d838440da42227c680e8ab92db4e053a6c2b164ff90ee4.jpg)

## Configuring Celery’s Late Acknowledgment and Retry Limits

Celery’s default acknowledgment model removes tasks from the broker the moment a worker receives them, before execution begins. [Late acknowledgment](https://docs.celeryq.dev/en/stable/userguide/configuration.html#task-acks-late) keeps the task in the broker until the worker confirms success, which makes crashed tasks survive for redelivery but introduces poison pill risk.

### Set Late Acknowledgment and Time Limits

Add these settings to your Celery app:

```python
from celery import Celery

app = Celery('agent_pipeline', broker='redis://localhost:6379/0')

app.conf.update(
    task_acks_late=True,
    task_reject_on_worker_lost=False,
    worker_prefetch_multiplier=1,
    task_time_limit=1800,
    task_soft_time_limit=1740
)
```

| Setting | Purpose | Risk if Wrong |
| --- | --- | --- |
| `task_acks_late` | Keeps task in broker until worker confirms success | Task lost on crash if False |
| `task_reject_on_worker_lost` | Requeues tasks when worker dies | Infinite loop if a task kills the worker |
| `worker_prefetch_multiplier` | Limits how many tasks a worker holds | Worker hogs tasks it never completes |
| `task_time_limit` | Hard kill after 30 minutes | Task runs forever and blocks the worker |
| `task_soft_time_limit` | Python exception at 29 minutes | No graceful cleanup before hard kill |

Retry limits live on each task, not in `app.conf`: the `max_retries` and `default_retry_delay` arguments on the task decorator in the next section set them.

[Celery 5.7](https://docs.celeryq.dev/en/main/history/whatsnew-5.7.html) added [`task_retry_on_worker_lost`](https://docs.celeryq.dev/en/main/userguide/configuration.html#task-retry-on-worker-lost), which retries a task that kills its worker up to the task’s retry limit and then marks it failed. This configuration keeps `task_reject_on_worker_lost=False` instead, so Celery acknowledges a task that kills its worker process and marks it failed on the first crash rather than putting it back on the queue. That stops the loop, but the message leaves the broker, so the dead-letter handler below copies every failed task’s arguments into a Redis list before they’re gone.

Save this as `agent_pipeline.py`, then start a worker from the same folder so `-A agent_pipeline` can import it:

```bash
celery -A agent_pipeline worker --loglevel=INFO
```

Confirm the settings loaded:

```bash
celery -A agent_pipeline inspect conf | grep task_acks_late
```

The output shows `"task_acks_late": true,` in JSON format. If it shows `false`, restart the worker so it reloads `agent_pipeline.py`.

### Capture Failed Tasks in a Dead-Letter List

Celery sends the [`task_failure` signal](https://docs.celeryq.dev/en/stable/userguide/signals.html#task-failure) whenever a task fails, including when its worker process dies mid-task and the task isn’t requeued, and the signal carries the task’s original `args` and `kwargs`. Add a handler to `agent_pipeline.py` so every failure lands in a Redis list you can replay from, then restart the worker:

```python
import json
import socket
from datetime import datetime, timezone

import redis
from celery.signals import task_failure

dlq = redis.Redis(host='localhost', port=6379, db=1)

@task_failure.connect
def send_to_dead_letter(sender=None, task_id=None, exception=None, args=None, kwargs=None, **extra):
    dlq.rpush('celery:dead-letter', json.dumps({
        'task_name': sender.name,
        'task_id': task_id,
        'args': args,
        'kwargs': kwargs,
        'error': repr(exception),
        'worker_hostname': socket.gethostname(),
        'failed_at': datetime.now(timezone.utc).isoformat(),
    }, default=str))
```

The handler writes to Redis database 1, so dead-letter entries stay apart from the broker’s queues in database 0. It runs in the worker’s main process when a pool child dies, which covers a single task getting killed for memory. If the whole worker or its container dies, nothing is left to run the handler, and with `task_acks_late=True` the broker redelivers the unacknowledged message to another worker once the visibility timeout expires.

List what landed:

```bash
redis-cli -n 1 LRANGE celery:dead-letter 0 -1
```

Each entry is one JSON object holding the task’s arguments and its error. Before you replay a `WorkerLostError` entry, fix the input or the resource limit, because that error means the task took its worker process down.

## Implementing Exponential Backoff for Transient Failures

[Exponential backoff](https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/) with jitter spaces out retries to avoid hammering rate-limited APIs into quota exhaustion. It prevents synchronized thundering-herd failures when multiple tasks retry simultaneously after the same upstream outage.

### Add Jittered Retries to the Celery Task

For BullMQ, the `jitter: 0.5` option in the `backoff` configuration randomizes each exponential delay. For Celery, implement it in the task decorator:

```python
import random

# Define your own rate limit exception or import from your LLM library
class RateLimitError(Exception):
    pass

@app.task(bind=True, max_retries=3, default_retry_delay=60)
def process_agent_task(self, task_id, context):
    try:
        # Your agent logic here (call_llm_api is your function)
        result = call_llm_api(context['prompt'])
        return result
    except RateLimitError as exc:
        # Exponential backoff with jitter for rate limits
        retry_delay = (2 ** self.request.retries) * 60
        jitter = random.uniform(0, retry_delay * 0.1)
        raise self.retry(exc=exc, countdown=retry_delay + jitter)
    except Exception as exc:
        # Don't retry logic errors
        raise
```

The backoff formula `2 ** self.request.retries` produces delays of 60, 120, and 240 seconds for retries 1, 2, and 3. Adding 10% jitter prevents multiple tasks from retrying at exactly the same timestamp. The distinction between `RateLimitError` and generic `Exception` prevents retrying permanent failures.

### Force a Rate Limit to Watch the Backoff

To watch the backoff without waiting for a real 429, temporarily make `call_llm_api` raise the error, restart the worker, then queue one task:

```python
def call_llm_api(prompt):
    raise RateLimitError('429 Too Many Requests')
```

```bash
celery -A agent_pipeline call agent_pipeline.process_agent_task --args='["task_1", {"prompt": "test"}]'
```

Check the worker log for retry timestamps. The gap between retries grows from about 60 to 120 to 240 seconds, plus up to 10% jitter on each, as the retry log below shows.

![Retry exponential backoff](https://adamtheautomator.com/wp-content/uploads/publisher/9f7478d6a7295ff1a80a9e878f4894cd01bca6cce62b07dbc043eca138582144.jpg)

After the third retry the task fails for good, which also gives your dead-letter list its first entry. Put your real `call_llm_api` back before you move on.

* * *

_**Pro Tip: Add a circuit breaker that pauses task submission after five consecutive rate-limit failures. Continuing to queue tasks into a rate-limited API wastes both queue capacity and API quota.**_

* * *

## Preserving Context for Safe Replay

The [BullMQ job data model](https://docs.bullmq.io/guide/jobs) lets you attach arbitrary metadata to each job so failed tasks carry enough context to judge a replay. By default you get an error message and stack trace, but not the input that caused the failure or the partial state the agent produced before crashing.

### Capturing Task Metadata

Store these fields with every queued task:

| Field | Why It Matters |
| --- | --- |
| `task_id` | Ties the job to the agent’s own logs and API calls |
| `original_payload` | The input that triggered the failure |
| `partial_output` | What the agent produced before failing |
| `error_trace` | Full stack trace and exception type |
| `worker_hostname` | Which machine ran the task |
| `enqueued_at` | When the task first entered the queue |
| `failed_at` | When the final retry exhausted |

For BullMQ, attach this as job data:

```javascript
import { v4 as uuidv4 } from 'uuid';

await agentQueue.add('process-document', {
  task_id: uuidv4(),
  original_payload: { documentId: '12345', action: 'summarize' },
  partial_output: null,
  enqueued_at: new Date().toISOString()
}, {
  jobId: `doc-12345-${Date.now()}`
});
```

The custom `jobId` names the job after the document it processes, so the failed set shows which document each failure came from.

BullMQ records `attemptsMade`, `failedReason`, `stacktrace` and `finishedOn` on every job. The agent’s partial output and the worker’s hostname only exist while the job runs, so write them from inside `processAgentTask`. Add `import os from 'node:os';` to the top of your worker file, then call this after each step that produces output, where `draft` holds whatever the agent has produced so far:

```javascript
await job.updateData({ ...job.data, partial_output: draft, worker_hostname: os.hostname() });
```

For Celery, the `send_to_dead_letter` handler you added earlier stores the task arguments, error, worker hostname and failure time with every failed task.

### Inspect a Failed Job Before Replaying It

When the task lands in the DLQ, fetch the full context before deciding whether to replay:

```bash
# BullMQ: list failed job IDs, then read one job's hash
redis-cli ZRANGE "bull:agent-tasks:failed" 0 -1
redis-cli HGETALL "bull:agent-tasks:<job-id>"  # Raw output shows atm/ats, not attemptsMade

# Celery: read the dead-letter list
redis-cli -n 1 LRANGE celery:dead-letter 0 -1
```

Read the failure reason first. If a BullMQ job’s `failedReason` is `job stalled more than allowable limit`, or a Celery entry’s `error` holds `WorkerLostError`, the task took its worker down, and replaying it unchanged will take down another one. For any other failure, compare the job’s `data` (or the entry’s `args`) against what the agent expected before replaying it.

## Building a Recovery Workflow with Human Review

Route tasks that exhausted retries to human review instead of replaying them automatically or silently discarding them. These failures likely signal logic errors, malformed input, or resource constraints that need code changes before the task can succeed. The [Retry pattern in Microsoft’s Azure Architecture Center](https://learn.microsoft.com/en-us/azure/architecture/patterns/retry) covers when retries help and when they burn resources.

### Make Low-Confidence Results Fail Explicitly

Make confidence collapses throw explicitly so they reach your dead-letter queue. Inside `processAgentTask`, after the agent call assigns `result`, fail loudly when confidence drops below your threshold (adjust the field name if your agent reports confidence differently):

```javascript
if (result === null || result.confidence < 0.5) {
  throw new Error('ConfidenceCollapse: agent returned no usable answer');
}
```

In `process_agent_task`, the same check goes inside the `try` block, right after `call_llm_api` returns:

```python
if result is None or result.get('confidence', 0) < 0.5:
    raise ValueError('ConfidenceCollapse: agent returned no usable answer')
```

Because `ValueError` isn’t a `RateLimitError`, the generic `except` re-raises it without a retry, so the task fails on the first attempt and `send_to_dead_letter` records it.

### Classifying Failure Types

Create a recovery classification function:

```javascript
async function classifyFailure(job) {
  const error = job.failedReason;
  
  if (error.includes('ECONNREFUSED') || error.includes('429')) {
    return 'transient';  // Safe to retry with backoff
  }
  if (error.includes('ConfidenceCollapse')) {
    return 'confidence_collapse';  // Human review needed
  }
  if (error.includes('stalled more than allowable limit') || error.includes('Out of memory') || error.includes('SIGKILL')) {
    return 'poison_pill';  // Do not retry; escalate immediately
  }
  if (error.includes('ValidationError') || error.includes('Malformed')) {
    return 'bad_input';  // Fix the input, then replay
  }
  return 'unknown';  // Route to human triage
}
```

BullMQ records a job that exceeds `maxStalledCount` with the failure reason `job stalled more than allowable limit`, so `classifyFailure()` matches that string to catch a BullMQ poison pill.

The buckets follow the failure classes table: `poison_pill` is resource exhaustion, and `bad_input` covers logic errors caused by malformed input or tool output.

Create a second queue that holds tasks waiting for a person:

```javascript
const humanReviewQueue = new Queue('human-review', {
  connection: { host: 'localhost', port: 6379 }
});
```

Implement the action handlers:

```javascript
// Replay after fixing input
async function replayTask(jobId, fixedPayload) {
  const failedJob = await agentQueue.getJob(jobId);
  await agentQueue.add(failedJob.name, fixedPayload, {
    jobId: `${jobId}-replay-${Date.now()}`
  });
  await failedJob.remove();
}

// Escalate to human review
async function escalateTask(jobId) {
  const failedJob = await agentQueue.getJob(jobId);
  await humanReviewQueue.add('review-failed-task', {
    original_job: failedJob.data,
    failed_reason: failedJob.failedReason,
    retry_count: failedJob.attemptsMade
  });
  await failedJob.remove();
}
```

Wire the classifier to the handlers with a triage pass over the failed set. Run it on a schedule, or by hand when the DLQ alert fires:

```javascript
const failedJobs = await agentQueue.getFailed();
for (const job of failedJobs) {
  const bucket = await classifyFailure(job);
  console.log(job.id, bucket, job.failedReason);
  if (bucket === 'transient') {
    await job.retry('failed', { resetAttemptsMade: true });
  } else if (bucket !== 'bad_input') {
    await escalateTask(job.id);
  }
}
```

Transient failures go back to the queue for another round of backoff. Poison pills, confidence collapses, and unknown failures move to `human-review` with their payload and failure reason. `bad_input` jobs stay in the failed set until someone fixes the payload and calls `replayTask()` with it.

### Trigger Each Failure Class on Purpose

To confirm `classifyFailure()` sorts real failures into the right buckets, add these checks to the top of `processAgentTask(job)`:

```javascript
if (job.name === 'drill-rate-limit') {
  throw new Error('429 Too Many Requests');
}
if (job.name === 'drill-bad-input') {
  throw new Error('ValidationError: payload is missing documentId');
}
if (job.name === 'drill-poison-pill') {
  process.kill(process.pid, 'SIGKILL');
}
```

Queue one of each:

```javascript
await agentQueue.add('drill-rate-limit', {});
await agentQueue.add('drill-bad-input', {});
await agentQueue.add('drill-poison-pill', {});
```

The first two throw, so BullMQ retries each one up to `attempts` and then moves it to the failed set. The third kills the worker process the same way an out-of-memory kill would. Restart the worker each time it dies; once the job stalls more often than `maxStalledCount` allows, BullMQ fails it with the reason `job stalled more than allowable limit`. Run the triage pass and confirm the three jobs come back as `transient`, `bad_input`, and `poison_pill`, in the output below.

![DLQ triage results](https://adamtheautomator.com/wp-content/uploads/publisher/48083ee7930f5e96790e2fc9db9ce66389cdeb7157906e53ac2bdb5ed3965ec4.jpg)

For Celery, add a drill task to `agent_pipeline.py`, restart the worker, and call it:

```python
import os
import signal

@app.task
def drill_poison_pill():
    os.kill(os.getpid(), signal.SIGKILL)
```

```bash
celery -A agent_pipeline call agent_pipeline.drill_poison_pill
```

The pool child dies, Celery marks the task failed with `WorkerLostError`, and the `celery:dead-letter` list gains an entry.

## Monitoring DLQ Metrics and Alerting on Patterns

Your dead-letter queue only works if you know when it fills. Track key metrics and set alerts so you catch backlogs before they become invisible failures you discover weeks later.

### Metrics to Track

These metrics expose your DLQ health:

```javascript
// Metrics to expose to your monitoring system
const dlqMetrics = {
  total_failed_jobs: await agentQueue.getFailedCount(),
  failure_rate_last_hour: calculateHourlyFailureRate(),  // Your function
  poison_pill_count: await countByClassification('poison_pill'),  // Your function
  avg_time_to_recovery: calculateAvgRecoveryTime(),  // Your function
  replay_success_rate: calculateReplaySuccessRate()  // Your function
};
```

Start with these illustrative thresholds and tune them to your own failure baseline:

| Metric | Threshold | Action |
| --- | --- | --- |
| Failure rate | \>5% of total jobs in 1 hour | Investigate error pattern |
| Poison pill count | \>3 in 10 minutes | Pause task submission |
| DLQ size | \>100 failed jobs | Review recovery workflow |
| Replay success rate | <50% | Inputs need better validation |

### Setting Up Alerts

Turn the DLQ size row into an alert first, because it needs nothing beyond the queue. This check runs in any long-lived Node process that holds `agentQueue`:

```javascript
setInterval(async () => {
  const failed = await agentQueue.getFailedCount();
  if (failed > 100) {
    console.error(`DLQ alert: ${failed} failed jobs in agent-tasks`);
  }
}, 60_000);
```

Swap the `console.error` for a call to your paging webhook, and keep `100` in step with the DLQ size row above. On the Celery path, `redis-cli -n 1 LLEN celery:dead-letter` returns the same count from the dead-letter list.

For Celery, use [Flower](https://flower.readthedocs.io/) to visualize queue health:

```bash
FLOWER_UNAUTHENTICATED_API=true celery -A agent_pipeline flower --port=5555
```

Query Flower’s [tasks API](https://flower.readthedocs.io/en/latest/api.html) for tasks in Celery’s `FAILURE` state:

```bash
curl "http://localhost:5555/api/tasks?state=FAILURE"
```

The response lists every task Flower has seen fail, so you can feed it to a script or an alert instead of clicking through the web UI. For a shared Flower instance, protect it with `--basic-auth=user:password` in place of the env var, and use `curl -u user:password "http://localhost:5555/api/tasks?state=FAILURE"` to query the API.

For production, persist metrics to [Prometheus](https://prometheus.io/docs/introduction/overview/) or your existing monitoring stack.

* * *

_**Reality Check: When the DLQ fills faster than you can triage it, sort the backlog by bucket first. If most of the backlog is `transient`, look for a provider outage. Mostly `bad_input` means validation belongs before task submission, where it rejects the payload before an agent ever runs.**_

* * *

## Run a Poison-Pill Drill Against Your Production Image

Before you ship this, run the poison-pill drill against the same worker image and memory limits you use in production. In BullMQ the job should reach the `failed` set with the reason `job stalled more than allowable limit` once it exceeds [`maxStalledCount`](https://docs.bullmq.io/guide/workers/stalled-jobs); in Celery it should appear in the `celery:dead-letter` list with its arguments intact. If it lands anywhere else, or nowhere, fix that first.

Then watch the classifier’s `unknown` bucket for the first week and write a `classifyFailure()` rule for each new failure class that lands there.

Share this article

[Share on X](https://twitter.com/intent/tweet?url=https%3A%2F%2Fadamtheautomator.com%2Fdead-letter-queue-agent-tasks%2F&text=Dead-Letter%20Queue%3A%20Stop%20Failed%20Agent%20Tasks%20from%20Looping)[Share on Facebook](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fadamtheautomator.com%2Fdead-letter-queue-agent-tasks%2F)[Share on LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fadamtheautomator.com%2Fdead-letter-queue-agent-tasks%2F)

## Related Posts

![](https://adamtheautomator.com/wp-content/uploads/publisher/4065d9c85b2b814f9e26db58285a5c6d/2fe9e6fcd3f137f3e25263ae52f2ee818bb97d59b094ce182fdb414b1031ff1f.jpg)

### [Build a Production-Ready Helpdesk Triage Agent with Mastra](/build-helpdesk-triage-agent-with-mastra/)

Build a TypeScript helpdesk triage agent with Mastra: a Zod-schema classifier proposes a category, then code routes each ticket to a queue and SLA.

![](https://adamtheautomator.com/wp-content/uploads/publisher/4065d9c85b2b812bba2df8cb767fbe34/9ae655c39d56002e696384ad1bf2fb8f518c75940da65a91ac2668477e001dd3.webp)

### [PowerShell MCP Server: Automate Windows Admin Tasks](/powershell-mcp-server-automate-windows-admin-tasks/)

Build a PowerShell MCP server with PSMCP, register it in Claude Desktop, and expose four read-only Windows tools the model can call safely.

![](https://adamtheautomator.com/wp-content/uploads/publisher/3f25d9c85b2b81c4811ed7ed9b290871/d4ebe25fb13196c5dffbe9c4668d57fe178690406552541f4e2f3828e040cc6f.webp)

### [Notion Custom Agents: Automate Your Weekly Project Report](/notion-custom-agents-weekly-report/)

Build a Notion Custom Agent that writes a dated weekly project report from your task database, then test it against an empty week and a repeat run.

## Categories

*   [IT Ops](/category/it-ops/)
*   [Cloud](/category/cloud/)
*   [DevOps](/category/devops/)
*   [Home Ops](/category/home-ops/)
*   [Information Security](/category/infosec/)
*   [Software Development](/category/software-development/)

## Site

*   [Home](/)
*   [Tutorials](/tutorials/)
*   [Instructors](/author/)
*   [Advertising](/advertising/)
*   [Recommended Resources](/resources/)
*   [About Adam](/about-adam/)

Copyright 2026© ATA Learning | [Privacy Policy](/privacy/)
