Last year, we introduced webhooks to Marble so teams could react to events happening in their workspace. When a post is published, a category is updated, a tag is deleted, or media changes, webhooks make it possible to send that event somewhere else and continue the workflow outside Marble.
The first version was built with QStash from Upstash. Upstash had been kind enough to support Marble, and QStash gave us a very fast path to shipping webhooks without building queue infrastructure ourselves. It was simple to set up, easy to reason about, and exactly the kind of tool that helps a small team move.
That first version worked well for getting webhooks out the door. But over time, the problem changed. We did not only need to send webhooks anymore. We needed to show what happened after Marble tried to send them.
The thing that was missing
In April 2026, customer feedback made the weak spot clear. A webhook system can technically send requests and still feel unreliable if the user cannot see what happened. If someone expected a webhook and did not receive it, we needed to answer basic questions:
Was the event created?
Did Marble attempt a delivery?
Which endpoint did we send it to?
What status code came back?
Did the request time out?
Will Marble retry it?
How many attempts were made?
Our old setup did not give us a good enough answer inside Marble itself. Some webhook emissions from the dashboard were effectively fire-and-forget. If the request did not complete, or if something failed between the dashboard action and the actual delivery, there was not enough durable state for us or the user to inspect later.
That was the product issue. Reliability was not only about whether a network request eventually reached a customer endpoint. It was also about visibility. Users needed delivery logs, retry history, failure reasons, and a way to verify that Marble had done its part.
Why we moved the pipeline
Marble already had parts of its infrastructure on Cloudflare. The public API runs on Cloudflare Workers, and our MCP server does too. Moving webhook processing into a separate jobs worker let us keep the webhook delivery pipeline close to the API while giving it its own queues, retry behavior, and operational surface.
The goal was not to replace QStash because it was bad. QStash helped us ship. The goal was to own the parts of the pipeline that had become product features for Marble:
Persisting workspace events.
Fanning events out to matching webhook endpoints.
Recording every delivery.
Recording every attempt.
Retrying failures.
Showing delivery status in the dashboard.
Once those became requirements, the webhook system needed to be more than an outbound request helper. It needed to be a durable workflow.
How it works now
The new system is built around two Cloudflare Queues and a dedicated jobs worker.
There are two main queues:
marble-events: receives workspace event IDs and turns them into webhook deliveries.
marble-webhook-deliveries: receives delivery IDs and performs the outbound HTTP requests.
The API and dashboard both emit events into the same internal event route. For example, when a post is published from the dashboard, Marble creates a workspace event with the event type, workspace ID, resource type, actor, and payload. That event is then queued for processing.
await db.workspaceEvent.create({
data: {
type: "post_published",
workspaceId,
source: "dashboard",
resourceType: "post",
resourceId: post.id,
actorType: "user",
actorId: user.id,
payload,
},
});
await env.EVENT_QUEUE.send({ eventId: event.id });The event worker loads the event, finds the matching webhook endpoints for that workspace, and creates one delivery row for each endpoint. Each delivery is then sent to the delivery queue.
const delivery = await db.webhookDelivery.upsert({
where: {
eventId_webhookEndpointId: {
eventId: event.id,
webhookEndpointId: webhook.id,
},
},
create: {
eventId: event.id,
workspaceId: event.workspaceId,
webhookEndpointId: webhook.id,
url: webhook.url,
status: "pending",
isTest,
},
update: {},
});
await env.WEBHOOK_DELIVERY_QUEUE.send({
deliveryId: delivery.id,
});The important detail here is the upsert. Fan-out is idempotent by event and endpoint, so if a queue message is retried after a partial failure, Marble does not create duplicate delivery rows for the same event and endpoint.
Delivery attempts and retries
The delivery worker receives a delivery ID, atomically claims the delivery, signs the payload, sends the webhook request, and records the attempt.
Atomic claiming matters because queue systems can retry messages, and distributed workers can overlap. Before sending, Marble updates the delivery from pending or retrying to sending. If another worker already claimed it, the second worker exits without sending a duplicate webhook.
const claim = await db.webhookDelivery.updateMany({
where: {
id: deliveryId,
status: { in: ["pending", "retrying"] },
},
data: {
status: "sending",
attemptCount: { increment: 1 },
lastAttemptAt: new Date(),
},
});
if (claim.count === 0) {
return;
}Each outbound webhook request includes Marble headers that help customers verify and debug deliveries:
x-marble-event: the event name, such aspost.published.x-marble-event-id: the durable workspace event ID.x-marble-delivery-id: the delivery ID shown in Marble.x-marble-timestamp: the time used when signing the payload.x-marble-signature: the HMAC signature customers can verify with their endpoint secret.
Requests have a timeout, and every result is recorded as a delivery attempt. Successful responses mark the delivery as success. Failed responses and network errors are recorded and retried until the delivery reaches its maximum number of attempts.
What users get from this
The main user-facing improvement is not that Marble uses a different queue. The improvement is that webhook delivery is now observable inside Marble.
Instead of wondering whether a webhook disappeared, users can inspect delivery logs and see the status of their webhooks. Marble can show which event was processed, which endpoint was targeted, whether the delivery succeeded, and what happened during each attempt.
That changes support conversations too. If a webhook fails because the customer endpoint returns a 500, times out, or has a bad URL, we can point to the delivery attempt. If Marble failed before sending, that should be visible too. The system becomes easier to reason about for both sides.
Tradeoffs
Owning more of the pipeline means owning more code. The new system gives us better visibility and control, but it also creates new responsibilities:
We need cleanup jobs for old event and delivery logs.
We need better structured logging around workers.
We need a recovery path for events that are persisted but fail before being queued.
We need to keep improving the dashboard experience around delivery logs and replay.
Those are real tradeoffs. But they are tradeoffs we are more comfortable with because webhook delivery logs are now part of the product experience. For Marble, the system is not just infrastructure hidden behind the scenes. It is something users rely on and should be able to inspect.
Where this leaves QStash
QStash was the right tool for our first version. It let us add webhooks quickly. We still think it is a good option if you want to ship asynchronous HTTP delivery without building the machinery yourself.
Our migration was about Marble reaching a point where we needed a workflow shaped around our own product: workspace events, delivery logs, retries, attempts, dashboard visibility, and eventually replay. At that point, owning the worker pipeline made sense.
What comes next
The new webhook pipeline is live, and we are watching it closely. The next pieces are mostly about making the system easier to operate and easier for users to understand:
Richer delivery logs in the dashboard.
Clearer failure messages.
Retention and cleanup for old logs.
Better internal worker logging.
A durable outbox so persisted events cannot be lost if queueing fails.