Double Buffering with Cloudflare Queues

I use a double-buffered queue pipeline to collect and segment telemetry events into groups bound by UTC hour. This pattern naturally batches data, produces predictable event flow, and minimizes cost.

Modulo math is used to route the payload to the active queue and again to drain the inactive queue. UTC is used to avoid DST handling.

This pipeline leverages Cloudflare Workers, Queues, Workflows, and R2 to capture, batch, process, and store telemetry events while maximizing the benefits of Cloudflare's pricing.

Producer

The producer evaluates the current UTC hour and directs the payload to the active queue:

Consumer

The consumer task triggers every hour on the half-hour to drain the inactive queue:

The consumer task looks back one hour to determine which queue is currently inactive and safe to consume:

Finally, the batch is compressed and written to R2 using a simple partitioning strategy: