B, LLC Logo

Double Buffering with Cloudflare Queues

I use a double-buffered queue to collect and segment telemetry events into groups bound by UTC hour. My pipeline leverages Cloudflare Workers, Queues, Workflows, and R2 to capture, batch, process, and store telemetry events while maximizing the benefits of Cloudflare's pricing.

Modulo math is used to route the payload to the active queue and again to drain the inactive queue. UTC is used to avoid DST handling.

Producer

The producer evaluates the payload's timestamp to route the message to the active queue:


// Worker

// Convert event timestamp to datetime object
const hour = new Date(payload.timestamp).getUTCHours();

// Route payload to queue
if (hour % 2 === 0) {
    // Even hour (0, 2, 4, 6... 22)
    await env.EVEN_QUEUE.send(payload);
} else {
    // Odd hour (1, 3, 5, 7... 23)
    await env.ODD_QUEUE.send(payload);
}
            

Consumer

The consumer task triggers every hour on the half-hour to drain the inactive queue:


// Workflow

// Hourly at HH:30
if (event.cron === "30 * * * *") {
    ctx.waitUntil(env.TASK_MANAGER.create({
        params: {taskName: "consumer-task"}
    }));
}
            

The consumer task looks back one hour to determine which queue is currently inactive and safe to consume:


// Consumer Task

// Create datetime object for 1 hour ago
const oneHourAgoObj = new Date(Date.now() - 60 * 60 * 1000);
const lastHour = oneHourAgoObj.getUTCHours();

// Select inactive queue for consumption
if (lastHour % 2 === 0) {
	// Select even hours queue by ID
	return env.EVEN_QUEUE_ID;
} else {
	// Select odd hours queue by ID
	return env.ODD_QUEUE_ID;
}
            

Finally, the batch is compressed and written to R2 using a simple partitioning strategy:


// Consumer Task

// Compress payload batch
const gzippedPayload = await compressPayload(macroPayload);

// Construct R2 path segments from datetime object
const year = oneHourAgoObj.getUTCFullYear();
const month = String(oneHourAgoObj.getUTCMonth() + 1).padStart(2, '0');
const day = String(oneHourAgoObj.getUTCDate()).padStart(2, '0');
const hour = String(oneHourAgoObj.getUTCHours()).padStart(2, '0');

// Set R2 path
const path = `/${year}/${month}/${day}/${hour}/dataset.json.gz`;

// Put file in R2
try {
    await env.R2_BUCKET.put(path, gzippedPayload, {
        httpMetadata: { contentType: 'application/json', contentEncoding: 'gzip' }
    });
} catch (error) {
    console.error(`Failed to store payload at ${path}:`, error);
}
            

This pattern transforms chaotic streams of telemetry into a compressed hourly archive. You get a partitioned, serverless data lake optimized for downstream analytics, while keeping your cloud storage bills incredibly low.