Async Queue Pattern in JavaScript

Build resilient, scalable asynchronous task processing systems—from basic in-memory queues to advanced distributed patterns—using Node.js. This article covers the design reasoning behind queue architectures, concurrency control mechanisms, and resilience patterns for production systems.

Abstract

Async queues solve a fundamental tension: how to maximize throughput while respecting resource constraints. The core mental model:

1
Producer → Queue (buffer) → Consumer(s)
2
           ↑                    ↓
3
     Backpressure ←──── Concurrency Control

Key design decisions:

Resilience fundamentals:

• Idempotency: Design consumers to handle duplicate delivery safely
• Backpressure: Bound queue size to prevent memory exhaustion
• Exponential backoff + jitter: delay = min(cap, base × 2^attempt) + random() prevents thundering herd
• Dead Letter Queue (DLQ): Isolate poison messages that exceed retry attempts

Part 1: The Foundation of Asynchronous Execution

1.1 The Event Loop and In-Process Concurrency

Node.js uses a single-threaded, event-driven architecture. This model excels at I/O-bound operations but blocks the main thread during CPU-intensive work. Understanding the event loop phases is essential for designing queue processors that remain responsive.

Event Loop Phases (as of Node.js 20+):

The event loop executes in six phases: timers → pending callbacks → idle/prepare → poll → check → close callbacks. Each phase has a FIFO queue of callbacks.

• Microtask processing: After each phase completes, Node.js drains the nextTick queue first, then the Promise microtask queue. Both complete before the next phase begins.
• Priority order: process.nextTick() > Promise microtasks > macrotasks (setTimeout, setImmediate)

Node.js 20 change (libuv 1.45.0): Timers now run only after the poll phase, not before and after as in earlier versions. This affects callback ordering for code that depends on precise timer/I/O interleaving.

Why this matters for queues: A queue processor that performs synchronous CPU work starves the event loop, preventing lock renewals (in distributed queues) and incoming I/O from being handled. Design processors to yield control regularly via await or setImmediate().

1.2 In-Memory Task Queues: Controlling Local Concurrency

In-memory queues throttle async operations within a single process—useful for rate limiting API calls or controlling database connection usage.

Library comparison (as of January 2025):

p-queue provides priority scheduling, per-operation timeouts, and sliding-window rate limiting. fastq uses object pooling via reusify for reduced GC pressure—6x higher adoption for performance-critical scenarios.

Naive implementation (for understanding the pattern):

1 collapsed line
1
type Task<T = void> = () => Promise<T>
2

3
export class AsyncTaskQueue {
4
  private queue = new TaskQueue<Task>()
5
  private activeCount = 0
6
  private concurrencyLimit: number
7

8
  constructor(concurrencyLimit: number) {
9
    this.concurrencyLimit = concurrencyLimit
10
  }
11

12
  addTask<T>(promiseFactory: Task<T>): Promise<T> {
13
    const { promise, resolve, reject } = Promise.withResolvers<T>()
14
    const task: Task = async () => {
15
      try {
16
        const result = await promiseFactory()
17
        resolve(result)
18
      } catch (error) {
19
        reject(error)
20
      }
21
    }
22
    this.queue.enqueue(task)
23
    this.processQueue()
24
    return promise
25
  }
26

27
  private async processQueue(): Promise<void> {
28
    if (this.activeCount >= this.concurrencyLimit || this.queue.isEmpty()) {
29
      return
30
    }
31

32
    const task = this.queue.dequeue()
33
    if (task) {
34
      this.activeCount++
35
      try {
36
        await task()
37
      } finally {
38
        this.activeCount--
39
        this.processQueue()
40
      }
41
    }
42
  }
43
}
44

51 collapsed lines
45
// Queue implementation using a linked list
46
class TaskNode<T> {
47
  value: T
48
  next: TaskNode<T> | null
49
  constructor(value: T) {
50
    this.value = value
51
    this.next = null
52
  }
53
}
54

55
export class TaskQueue<T> {
56
  head: TaskNode<T> | null
57
  tail: TaskNode<T> | null
58
  size: number
59
  constructor() {
60
    this.head = null
61
    this.tail = null
62
    this.size = 0
63
  }
64

65
  // Enqueue: Add an element to the end of the queue
66
  enqueue(value: T) {
67
    const newNode = new TaskNode(value)
68
    if (this.tail) {
69
      this.tail.next = newNode
70
    }
71
    this.tail = newNode
72
    if (!this.head) {
73
      this.head = newNode
74
    }
75
    this.size++
76
  }
77

78
  // Dequeue: Remove an element from the front of the queue
79
  dequeue(): T {
80
    if (!this.head) {
81
      throw new Error("Queue is empty")
82
    }
83
    const value = this.head.value
84
    this.head = this.head.next
85
    if (!this.head) {
86
      this.tail = null
87
    }
88
    this.size--
89
    return value
90
  }
91

92
  isEmpty() {
93
    return this.size === 0
94
  }
95
}
96

17 collapsed lines
97
// Example usage
98
const queue = new AsyncTaskQueue(3)
99

100
const createTask = (id: number, delay: number) => () =>
101
  new Promise<void>((resolve) => {
102
    console.log(`Task ${id} started`)
103
    setTimeout(() => {
104
      console.log(`Task ${id} completed`)
105
      resolve()
106
    }, delay)
107
  })
108

109
queue.addTask(createTask(1, 1000))
110
queue.addTask(createTask(2, 500))
111
queue.addTask(createTask(3, 1500))
112
queue.addTask(createTask(4, 200))
113
queue.addTask(createTask(5, 300))

Note: This uses Promise.withResolvers() (Baseline 2024).

Critical limitations of in-memory queues:

Backpressure pattern with p-queue:

2 collapsed lines
1
import PQueue from "p-queue"
2

3
const queue = new PQueue({ concurrency: 10 })
4

5
async function addWithBackpressure<T>(task: () => Promise<T>): Promise<T> {
6
  // Block producer when queue exceeds threshold
7
  if (queue.size > 100) {
8
    await queue.onSizeLessThan(50) // Wait until queue drains
9
  }
10
  return queue.add(task)
11
}

Part 2: Distributed Async Task Queues

For persistence, horizontal scaling, and cross-process coordination, tasks must be managed by a distributed queue system.

2.1 Distributed Architecture Components

Component responsibilities:

• Producers: Enqueue jobs with payload, priority, delay, and retry configuration
• Message Broker: Persistent store (Redis for BullMQ) providing at-least-once delivery
• Workers (Competing Consumers): Dequeue and process jobs; multiple workers increase throughput
• Dead Letter Queue: Captures jobs exceeding retry attempts for manual inspection

Why Redis for BullMQ? Redis provides atomic operations (MULTI/EXEC), sorted sets for delayed jobs, and pub/sub for worker coordination—all with sub-millisecond latency.

2.2 Node.js Distributed Queue Libraries

2.3 BullMQ Deep Dive

BullMQ (v5.67+) is the production standard for Node.js distributed queues. Key design decisions:

Stalled job detection: Workers acquire a lock when processing a job. If the lock expires (default: 30s) without renewal, BullMQ marks the job as stalled and either requeues it or moves it to failed. This handles worker crashes but also triggers if CPU-bound work starves the event loop.

5 collapsed lines
1
import { Queue } from "bullmq"
2

3
const connection = { host: "localhost", port: 6379 }
4
const emailQueue = new Queue("email-processing", { connection })
5

6
async function queueEmailJob(userId: number, template: string) {
7
  await emailQueue.add(
8
    "send-email",
9
    { userId, template },
10
    {
11
      attempts: 5,
12
      backoff: { type: "exponential", delay: 1000 },
13
      removeOnComplete: { count: 1000 }, // Keep last 1000 completed jobs
14
      removeOnFail: { age: 7 * 24 * 3600 }, // Keep failed jobs for 7 days
15
    },
16
  )
17
}

Worker with concurrency tuning:

2 collapsed lines
1
import { Worker } from "bullmq"
2

3
const emailWorker = new Worker(
4
  "email-processing",
5
  async (job) => {
6
    const { userId, template } = job.data
7
    // I/O-bound: email API call
8
    await sendEmail(userId, template)
9
  },
10
  {
11
    connection: { host: "localhost", port: 6379 },
12
    concurrency: 100, // High concurrency for I/O-bound work
13
    lockDuration: 30000, // 30s lock, must complete or renew
14
  },
15
)

Concurrency tuning guidance:

• I/O-bound jobs (API calls, DB queries): concurrency 100–300
• CPU-bound jobs: Use sandboxed processors (separate process) with low concurrency
• Mixed: Start low, measure, increase—monitor for stalled jobs

Sandboxed processors for CPU-intensive work:

1
// Main process
2
const worker = new Worker("cpu-intensive", "./processor.js", {
3
  connection,
4
  useWorkerThreads: true, // Node.js Worker Threads (BullMQ 3.13+)
5
})
6

7
// processor.js (separate file)
8
export default async function (job) {
9
  // CPU work here doesn't block main process
10
  return heavyComputation(job.data)
11
}

Job flows for dependencies (parent waits for all children):

2 collapsed lines
1
import { FlowProducer } from "bullmq"
2

3
const flowProducer = new FlowProducer({ connection })
4

5
await flowProducer.add({
6
  name: "process-order",
7
  queueName: "orders",
8
  data: { orderId: "123" },
9
  children: [
10
    { name: "validate", queueName: "validation", data: { orderId: "123" } },
11
    { name: "reserve", queueName: "inventory", data: { orderId: "123" } },
12
    { name: "charge", queueName: "payments", data: { orderId: "123" } },
13
  ],
14
})
15
// Parent job waits in 'waiting-children' state until all children complete

Rate limiting (global across all workers):

1
const worker = new Worker("api-calls", processor, {
2
  connection,
3
  limiter: {
4
    max: 10, // 10 jobs
5
    duration: 1000, // per second
6
  },
7
})

Part 3: Engineering for Failure

3.1 Retries with Exponential Backoff and Jitter

Naive immediate retries cause thundering herd—all failed jobs retry simultaneously, overwhelming the recovering service.

Formula: delay = min(cap, base × 2^attempt) + random(0, delay × jitterFactor)

BullMQ implementation:

1
await queue.add("api-call", payload, {
2
  attempts: 5,
3
  backoff: {
4
    type: "exponential",
5
    delay: 1000, // Base: 1s, 2s, 4s, 8s, 16s
6
  },
7
})

Why jitter? Without jitter, jobs that failed at the same time retry at the same time. Jitter spreads retries uniformly, smoothing load on downstream services.

3.2 Dead Letter Queue Pattern

Some jobs are inherently unprocessable: malformed payloads, missing dependencies, or bugs in consumer logic. These “poison messages” must be isolated.

BullMQ automatically moves jobs to “failed” state after exhausting retries. Query failed jobs for manual inspection:

1
const failedJobs = await queue.getFailed(0, 100) // Get first 100 failed jobs
2
for (const job of failedJobs) {
3
  console.log(`Job ${job.id} failed: ${job.failedReason}`)
4
  // Optionally: fix data and retry
5
  await job.retry()
6
}

3.3 Idempotent Consumers

Distributed queues provide at-least-once delivery. Network partitions, worker crashes, or lock expiration can cause duplicate delivery. Consumers must handle this safely.

Idempotency strategies:

3 collapsed lines
1
import { Worker } from "bullmq"
2
import { db } from "./database"
3

4
const worker = new Worker("user-registration", async (job) => {
5
  const { userId, userData } = job.data
6

7
  // Check if already processed using job ID
8
  const existing = await db.processedJobs.findByPk(job.id)
9
  if (existing) {
10
    console.log(`Job ${job.id} already processed, skipping`)
11
    return
12
  }
13

14
  // Atomic: create user + mark job processed
15
  await db.transaction(async (t) => {
16
    await db.users.create(userData, { transaction: t })
17
    await db.processedJobs.create({ jobId: job.id, processedAt: new Date() }, { transaction: t })
18
  })
19
})

Part 4: Advanced Architectural Patterns

4.1 Transactional Outbox Pattern

Problem: How do you atomically update a database AND publish an event? If you publish first and the DB write fails, you’ve sent an invalid event. If you write first and publishing fails, the event is lost.

Solution: Write events to an “outbox” table in the same transaction as business data. A separate relay process polls the outbox and publishes to the message broker.

1
async function createUserWithEvent(userData: UserData) {
2
  return await db.transaction(async (t) => {
3
    const user = await db.users.create(userData, { transaction: t })
4

5
    // Event written atomically with business data
6
    await db.outbox.create(
7
      {
8
        eventType: "USER_CREATED",
9
        aggregateId: user.id,
10
        payload: JSON.stringify({ userId: user.id, ...userData }),
11
        status: "PENDING",
12
      },
13
      { transaction: t },
14
    )
15

16
    return user
17
  })
18
}
19

20
// Separate relay process (runs continuously)
21
async function relayOutboxEvents() {
22
  const pending = await db.outbox.findAll({ where: { status: "PENDING" }, limit: 100 })
23
  for (const event of pending) {
24
    await messageBroker.publish(event.eventType, JSON.parse(event.payload))
25
    await event.update({ status: "SENT" })
26
  }
27
}

Trade-offs:

• Pros: Guarantees consistency between DB and events; survives broker outages
• Cons: Adds latency (polling interval); requires relay process; eventual consistency only

4.2 Saga Pattern for Distributed Transactions

When a business operation spans multiple services (each with its own database), traditional ACID transactions don’t apply. The Saga pattern coordinates distributed operations through a sequence of local transactions with compensating actions for rollback.

Choreography vs. Orchestration:

1
class OrderSagaOrchestrator {
2
  async execute(orderData: OrderData) {
3
    const steps: SagaStep[] = [
4
      { action: () => this.reserveInventory(orderData), compensate: () => this.releaseInventory(orderData) },
5
      { action: () => this.chargePayment(orderData), compensate: () => this.refundPayment(orderData) },
6
      { action: () => this.createShipment(orderData), compensate: () => this.cancelShipment(orderData) },
7
    ]
8

9
    const completed: SagaStep[] = []
10
    try {
11
      for (const step of steps) {
12
        await step.action()
13
        completed.push(step)
14
      }
15
    } catch (error) {
16
      // Compensate in reverse order
17
      for (const step of completed.reverse()) {
18
        await step.compensate()
19
      }
7 collapsed lines
20
      throw error
21
    }
22
  }
23

24
  private async reserveInventory(order: OrderData) {
25
    /* ... */
26
  }
27
  private async releaseInventory(order: OrderData) {
28
    /* ... */
29
  }
30
  // ... other steps
31
}

4.3 Event Sourcing with Message Queues

Event Sourcing stores state changes as an immutable sequence of events rather than current state. Queues distribute these events to consumers that build read models (CQRS).

Why use queues with Event Sourcing?

• Decoupling: Read model consumers don’t need to poll the event store
• Replay: Queue consumers can replay from a point in time (Kafka log compaction)
• Scaling: Multiple consumer groups process events independently

Kafka is commonly used because its durable, replayable log naturally fits the event store pattern. Log compaction retains the last event per key, enabling efficient state reconstruction.

Conclusion

Async queue patterns form the backbone of scalable Node.js systems. The progression from in-memory queues to distributed systems follows increasing requirements:

1. In-memory (p-queue, fastq): Local concurrency control, no durability
2. Distributed (BullMQ): Cross-process coordination, persistence, at-least-once delivery
3. Workflow orchestration (Temporal, Sagas): Complex multi-step operations with compensation

Key design principles apply at every level:

• Bound your queues: Implement backpressure to prevent memory exhaustion
• Design for failure: Exponential backoff, idempotent consumers, dead letter queues
• Monitor everything: Queue depth, processing latency, retry rate, stalled job count
• Match concurrency to workload: High for I/O-bound, low (with sandboxing) for CPU-bound

Appendix

Prerequisites

• Node.js event loop fundamentals (phases, microtask queue)
• JavaScript Promises and async/await
• Redis basics (for BullMQ sections)
• Database transactions (for outbox pattern)

Terminology

• Backpressure: Mechanism to slow producers when consumers can’t keep up
• Competing Consumers: Pattern where multiple workers pull from the same queue
• DLQ (Dead Letter Queue): Queue for messages that fail processing repeatedly
• Idempotency: Property where repeated operations produce the same result
• Jitter: Random delay added to prevent synchronized retries
• Stalled Job: Job whose lock expired without completion (worker crash or CPU starvation)

Summary

• In-memory queues (p-queue v9.x, fastq v1.20.x) control local concurrency with no persistence
• BullMQ v5.x provides Redis-backed distributed queues with retries, rate limiting, flows, and stalled detection
• Exponential backoff with jitter prevents thundering herd on transient failures
• Idempotent consumers are mandatory for at-least-once delivery systems
• Transactional outbox ensures atomic DB writes and event publishing
• Saga pattern coordinates distributed transactions through compensating actions
• Monitor queue depth, latency, retry rate, and stalled job count in production

References

• Node.js Event Loop Documentation - Official guide to event loop phases and microtask processing
• BullMQ Documentation - Complete reference for BullMQ v5.x
• BullMQ Stalled Jobs Guide - Lock-based stalled detection mechanism
• BullMQ Concurrency Guide - Tuning guidance for different workloads
• p-queue npm - Promise-based queue with priority and rate limiting
• fastq GitHub - High-performance in-memory queue by Matteo Collina
• MDN Promise.withResolvers() - Baseline 2024 API
• Saga Pattern - microservices.io - Distributed transaction coordination
• Transactional Outbox - microservices.io - Atomic DB writes and event publishing
• Event Sourcing - Martin Fowler - Foundational pattern explanation