Webhooks
An outbound webhook is an HTTPS endpoint your organization registers with EvalShift. When something happens in the org — a run finalizes, a baseline is pinned or unpinned — EvalShift POSTs a signed JSON event to that URL. CI is the intended consumer: a run.finalized event lets you post a PR comment, page on a fail verdict, or feed a dashboard without polling the API.
Endpoints are managed at /app/:org/settings/webhooks (open the app) by owners and admins. Outbound webhooks are included on the plans that list them on the pricing page; an org whose plan does not include them keeps its endpoints listed and can disable or delete them, but nothing is delivered.
Set up an endpoint
- +Open Webhooks from the organization menu and press New endpoint.
- +Enter the receiver’s
https://URL, pick the events, and choose whether it hears about every project or one. - +Copy the signing secret. It is shown once; if you lose it, rotate it from the endpoint page.
- +Press Send test. A
webhook.testevent goes through the same queue, signing, retry and logging path as a real one, so a green test means the whole path works. - +Read the Deliveries log on the endpoint page. Every attempt records the HTTP status, the error class and the first KiB of the response.
Events
An endpoint receives an event when the type is in its subscription and its scope is every project or the event’s project. webhook.test ignores the type filter.
| Type | Fired when | data fields |
|---|---|---|
run.finalized | A run's bundle has been uploaded and finalized; its verdict exists. | id, client_run_id, suite_name, source_model, target_model, git_sha, branch, pr_number, verdict, regression_rate, blocking_regression_count, finalized_at, view_url |
baseline.pinned | A run is pinned as a suite's baseline (also when re-pinned to a different run). | project_id, suite_name, run_id, pinned_by_user_id |
baseline.unpinned | A suite's baseline is removed. | project_id, suite_name, previous_run_id |
webhook.test | Someone presses Send test on the endpoint page. Delivered whether or not it is subscribed. | endpoint_id, message |
Every delivery body has the same envelope: id (the event id, shared by every endpoint’s copy and every redelivery), type, api_version, created_at, org, project (null for webhook.test) and the event-specific data. A run.finalized delivery, with whitespace added for reading:
{
"api_version": "2026-09-01",
"created_at": "2026-09-07T22:42:09Z",
"data": {
"blocking_regression_count": 0,
"branch": "feature/swap-model",
"client_run_id": "r_20260907_golden_ab12cd",
"finalized_at": "2026-09-07T22:42:09Z",
"git_sha": "a1b2c3d4e5f60718293a4b5c6d7e8f9012345678",
"id": "aad9f868-62cf-4de6-92db-495cbd6d268e",
"pr_number": 42,
"regression_rate": 0.1,
"source_model": "gemini-2.5-flash",
"suite_name": "golden",
"target_model": "gemini-3.1-flash-lite-preview",
"verdict": "conditional_pass",
"view_url": "https://www.evalshift.dev/app/acme/checkout/runs/aad9f868-62cf-4de6-92db-495cbd6d268e"
},
"id": "9cb7514d-7c5d-441c-b124-09afc731ee67",
"org": {"slug": "acme"},
"project": {"id": "8369bb25-64ca-4b5d-a913-06e99230a5d3", "slug": "checkout"},
"type": "run.finalized"
}| data field | Type | Notes |
|---|---|---|
id | string (UUID) | The server run id — the one every API path and URL uses. |
client_run_id | string | The id the CLI generated, for tracing back to the local run. |
suite_name | string | Suite the run replayed. |
source_model | string | Provider-qualified model identifier, as the CLI reported it. Likewise target_model. |
git_sha | string | 40 lowercase hex characters. |
branch | string | Branch name at push time. |
pr_number | integer | null | GitHub PR number when the run came from an Action; null otherwise. |
verdict | string | pass, conditional_pass, fail or inconclusive. |
regression_rate | number | Fraction in [0, 1] over blocking evaluator records. |
blocking_regression_count | integer | Number of critical/high regressions from blocking evaluators. |
finalized_at | string | RFC 3339, UTC. |
view_url | string | The run page in the web app. |
Headers
| Header | Value |
|---|---|
Content-Type | application/json |
User-Agent | EvalShift-Webhooks/1.0 (+https://www.evalshift.dev) |
X-EvalShift-Event | The event type, e.g. run.finalized. |
X-EvalShift-Event-Id | The envelope id. |
X-EvalShift-Delivery | The delivery id — unique per (event, endpoint, redelivery). Your idempotency key. |
X-EvalShift-Signature | t=<unix seconds>,v1=<hex> — two v1= values during a secret rotation. |
No other headers carry meaning. There is no header naming the org or project; read those from the body after verifying it.
Verify the signature
The scheme is Stripe’s, byte for byte, under a different header name — if you already verify Stripe webhooks, the same code works.
- +Parse
X-EvalShift-Signatureintot(Unix seconds when the request was signed) and one or morev1values (lowercase hex). - +Reject if
|now − t|exceeds your tolerance. 300 seconds is the recommended default; every attempt is signed freshly at send time. - +Compute
HMAC-SHA256(key = secret, message = t + "." + raw_body), hex-encoded. The key is the wholewhsec_…string — do not strip the prefix or base64-decode it. - +Accept if any
v1equals the digest under a constant-time comparison.
raw_body is the request body exactly as received. Most frameworks parse JSON before your handler runs; make sure you get the bytes (FastAPI await request.body(), Flask request.get_data(), Express express.raw({ type: "application/json" })).
Python
import hashlib
import hmac
import time
def verify(secret: str, header: str, body: bytes, tolerance: int = 300) -> bool:
timestamp: int | None = None
candidates: list[str] = []
for part in header.split(","):
key, sep, value = part.strip().partition("=")
if not sep or not value:
continue
if key == "t":
if not value.isdigit():
return False
timestamp = int(value)
elif key == "v1":
candidates.append(value)
if timestamp is None or not candidates:
return False
if abs(time.time() - timestamp) > tolerance:
return False
expected = hmac.new(
secret.encode(), f"{timestamp}.".encode() + body, hashlib.sha256
).hexdigest()
return any(hmac.compare_digest(candidate, expected) for candidate in candidates)Node
const crypto = require("node:crypto");
function verify(secret, header, body, toleranceSeconds = 300) {
const parts = header.split(",").map((p) => p.trim().split("="));
const t = Number(parts.find(([k]) => k === "t")?.[1]);
if (!Number.isInteger(t) || Math.abs(Date.now() / 1000 - t) > toleranceSeconds) return false;
const expected = Buffer.from(
crypto.createHmac("sha256", secret).update(`${t}.`).update(body).digest("hex"),
);
return parts
.filter(([k]) => k === "v1")
.some(([, v]) => {
const candidate = Buffer.from(v ?? "");
return candidate.length === expected.length && crypto.timingSafeEqual(candidate, expected);
});
}
// Express: keep the raw bytes, ack fast, work later.
// app.post("/hooks/evalshift", express.raw({ type: "application/json" }), (req, res) => {
// if (!verify(process.env.EVALSHIFT_WEBHOOK_SECRET, req.get("X-EvalShift-Signature") ?? "", req.body)) {
// return res.status(400).end();
// }
// res.status(204).end();
// queue.push(JSON.parse(req.body.toString("utf8")));
// });Stripe’s SDKs can do steps 1–4 for you (stripe.Webhook.construct_event(body, header, secret) in Python, stripe.webhooks.constructEvent(body, header, secret) in Node) if you pass the X-EvalShift-Signature value where they expect Stripe-Signature.
Test vector
Unit-test a verifier without a live delivery. The timestamp is fixed, so pass a very large tolerance or freeze the clock at t. The body is 281 bytes of UTF-8 with no trailing newline. A verifier fed these four values must return true; flipping any byte of the body, the secret or t must make it return false.
secret: whsec_test_vector_do_not_use_in_production
t: 1788782400 (2026-09-07T12:00:00Z)
body: {"api_version":"2026-09-01","created_at":"2026-09-07T12:00:00Z","data":{"endpoint_id":"2b7f1c9a-6d3e-4e8b-9f0a-1c2d3e4f5a6b","message":"This is a test delivery from EvalShift."},"id":"4e5f6a7b-8c9d-4e0f-a1b2-c3d4e5f6a7b8","org":{"slug":"acme"},"project":null,"type":"webhook.test"}
signature: t=1788782400,v1=1f979779df1b05008f5f3974239f56980c052ebc7892e00e4de621af3efd57f6Retries and timeouts
An attempt succeeds when the receiver returns a 2xx within 10 seconds of the request starting, TLS handshake included. Everything else is a failed attempt — 3xx (redirects are never followed), 4xx, 5xx, a timeout, a TLS or connection error, or a target that fails the address check at send time. That last one ends the whole delivery with blocked_target: no retry can change its answer, so the delivery goes straight to failed. Other failed attempts are retried with exponential backoff, eight attempts in total:
| Attempt | Waits after the previous failure | Elapsed since the first attempt |
|---|---|---|
| 1 | — | 0 |
| 2 | 30 s | 30 s |
| 3 | 1 min | 1 min 30 s |
| 4 | 2 min | 3 min 30 s |
| 5 | 4 min | 7 min 30 s |
| 6 | 8 min | 15 min 30 s |
| 7 | 16 min | 31 min 30 s |
| 8 | 32 min | 63 min 30 s |
Waits are minimums — attempts are picked up by a background worker that polls the queue, so expect each one somewhat later than the table says, never earlier. After the eighth failure the delivery is failed and is not retried again on its own; redeliver it from the endpoint page. Every attempt sends the identical body bytes under the same X-EvalShift-Delivery; only the signature differs, because it carries a fresh t. A 4xx is retried like anything else. Retries are per endpoint: a failing receiver’s backoff never holds back your other endpoints, though a slow attempt (up to the 10-second cap) can delay deliveries queued behind it by that long.
Idempotency
The same event will reach you more than once: on retry, on manual redelivery, and — rarely — when a 2xx you sent did not make it back before the timeout. Two ids let you choose the guarantee you want:
- +
X-EvalShift-Deliveryis unique per (event, endpoint, redelivery). Deduplicating on it suppresses retry duplicates but lets a deliberate redelivery through. The right key for most receivers. - +
X-EvalShift-Event-Id(the body’sid) is the same for every copy of an event, redeliveries included. Deduplicate on it when reprocessing must be a no-op even if an operator replays.
Keep seen ids for at least 24 hours. Return 2xx for a duplicate you chose to ignore — a 4xx would only schedule more retries.
Secret rotation
Rotate secret on the endpoint page returns a new secret once and starts a 24-hour window. During the window every delivery carries two v1= values, one per secret, in no guaranteed order — a verifier that accepts any matching v1 (both samples above do) keeps working whichever secret it holds. Update the receiver inside the window; after 24 hours the old secret stops signing. Rotating again while a window is open retires the older secret immediately. Rotation is recorded in the audit log; the current secret cannot be read back.
Auto-disable
After 5 consecutive deliveries that each exhausted all eight attempts, the endpoint is disabled with disabled_reason: "failing", and the org’s owners and admins are emailed once with the endpoint URL and a link to the org’s webhook settings. A delivery refused before it is sent, because the target now resolves to a non-public address or is otherwise no longer an allowed URL, counts the same as one that exhausted its retries. A DNS failure does not: it retries first, and only counts once its attempts run out. While disabled, new events produce no delivery for it at all; a delivery that was already queued is recorded as skipped with skip_reason: "endpoint_disabled". Any successful delivery in between resets the counter. Re-enable from the endpoint page once the receiver is fixed — that resets the counter too.
Delivery log and redelivery
Every delivery is recorded, whether it succeeded, failed or was skipped. The endpoint page lists them newest first; each delivery opens to the exact payload that was sent and one entry per attempt: HTTP status (absent when there was no response), error class (timeout, connection, unresolvable, blocked_target, redirect, http_4xx, http_5xx, http_error), duration, and the first 1 KiB of the response body. That excerpt is the most useful debugging tool you have — return a short reason when you reject a delivery.
| Status | Meaning |
|---|---|
pending | Queued, or between attempts. |
succeeded | A 2xx came back. |
failed | All attempts failed. Redeliver from the endpoint page. |
skipped | Never sent. skip_reason says why: endpoint_deleted, endpoint_disabled, or not_entitled (the plan had lost webhooks). |
Redeliver sends the same event again as a new delivery: same event id and identical body bytes, a new X-EvalShift-Delivery, a fresh eight-attempt schedule, and a pointer back to the original. A delivery that is still pending cannot be redelivered. History is kept for 30 days.
Network
- +Deliveries come from no fixed IP address. An allowlist by source address will not work. Authenticate deliveries by signature; that is the control.
- +
https://is required. The certificate must be valid for the host and chain to a public root; self-signed certificates fail every attempt. - +Redirects are not followed. A
3xxis a failed attempt. Point the endpoint at the final URL. - +Private and internal targets are rejected, when the URL is saved and again before every send: userinfo or a fragment in the URL; hosts named
localhost,*.localhost,*.localor*.internal; and any host that resolves to a non-public address. A host that merely fails to resolve is retried on the normal backoff. - +Ports other than 443 are allowed as long as the scheme is
https://. Requests carry no cookies and no credentials beyond the signature. The body is not compressed.
API
Everything the endpoint page does is also an API call under /orgs/{org}/webhooks. Creating, changing, rotating, testing, redelivering and deleting require webhook:manage, which only a browser session holds — an API token can never register a URL that receives every future run. Reading endpoints and delivery logs (webhook:read) works with a session or an owner/admin token.
| Method | Path | Purpose |
|---|---|---|
| GET | /orgs/{org}/webhooks | List the org's active endpoints (no secrets). |
| POST | /orgs/{org}/webhooks | Create an endpoint. Returns the secret once. 201. |
| GET | /orgs/{org}/webhooks/{id} | One endpoint. |
| PATCH | /orgs/{org}/webhooks/{id} | Change url, description, event_types, project_id, enabled. |
| DELETE | /orgs/{org}/webhooks/{id} | Soft-delete. 204. |
| POST | …/webhooks/{id}/rotate-secret | Start a 24-hour rotation. Returns the new secret once. 201. |
| POST | …/webhooks/{id}/test | Queue a webhook.test delivery. 202; 409 if disabled. |
| GET | …/webhooks/{id}/deliveries | Delivery log, newest first. ?limit= defaults to 50 (max 200); ?before= takes the created_at of the last row seen. No payloads. |
| GET | …/deliveries/{delivery_id} | One delivery with its payload and every attempt. |
| POST | …/deliveries/{delivery_id}/redeliver | Send the same event again as a new delivery. 202. |
Creating, enabling, testing and redelivering answer 402 when the org’s plan lacks webhooks. At most 10 active endpoints per org; a disabled endpoint still holds its slot. Creating, rotating, testing and redelivering are rate-limited per client address; a 429 carries Retry-After.
Best practices
- +Acknowledge first, process later. Verify the signature, persist the raw event or hand it to a queue, return
2xx. The 10-second budget includes your TLS handshake and cold start. - +Verify before you parse. A body that fails verification is untrusted input; do not deserialize it, log it, or branch on its contents.
- +Dedupe on the delivery id unless you have a reason to prefer the event id.
- +Subscribe narrowly. Pick the events you handle and, if the receiver only cares about one project, scope the endpoint to it. Still ignore rather than reject a type you do not recognize.
- +Send a test after every receiver change, and read the response excerpt in the delivery log when it fails.
- +Rotate the secret when someone who had it leaves, and once a year regardless.
- +Return a reason in the response body when you reject a delivery — you will see the first 1 KiB in the log.
Versioning
api_version is a date, currently 2026-09-01. It names the shape of the envelope and of each event’s data. Additive changes — new event types, new optional fields, new headers — never bump it; parse leniently and ignore what you do not know. Breaking changes bump it, are announced ahead of time, and endpoints are not moved to a new version without an explicit choice. The retry schedule, timeout, auto-disable threshold and retention window are operational parameters, not part of the versioned contract.
