// cloud · settings

Webhooks

signed events for CI and dashboards

An outbound webhook is an HTTPS endpoint your organization registers with EvalShift. When something happens in the org — a run finalizes, a baseline is pinned or unpinned — EvalShift POSTs a signed JSON event to that URL. CI is the intended consumer: a run.finalized event lets you post a PR comment, page on a fail verdict, or feed a dashboard without polling the API.

Endpoints are managed at /app/:org/settings/webhooks (open the app) by owners and admins. Outbound webhooks are included on the plans that list them on the pricing page; an org whose plan does not include them keeps its endpoints listed and can disable or delete them, but nothing is delivered.

Set up an endpoint

  1. +Open Webhooks from the organization menu and press New endpoint.
  2. +Enter the receiver’s https:// URL, pick the events, and choose whether it hears about every project or one.
  3. +Copy the signing secret. It is shown once; if you lose it, rotate it from the endpoint page.
  4. +Press Send test. A webhook.test event goes through the same queue, signing, retry and logging path as a real one, so a green test means the whole path works.
  5. +Read the Deliveries log on the endpoint page. Every attempt records the HTTP status, the error class and the first KiB of the response.
+
the secret is shown once
The create and rotate calls are the only places the secret appears. Store it as you would an API key.

Events

An endpoint receives an event when the type is in its subscription and its scope is every project or the event’s project. webhook.test ignores the type filter.

TypeFired whendata fields
run.finalizedA run's bundle has been uploaded and finalized; its verdict exists.id, client_run_id, suite_name, source_model, target_model, git_sha, branch, pr_number, verdict, regression_rate, blocking_regression_count, finalized_at, view_url
baseline.pinnedA run is pinned as a suite's baseline (also when re-pinned to a different run).project_id, suite_name, run_id, pinned_by_user_id
baseline.unpinnedA suite's baseline is removed.project_id, suite_name, previous_run_id
webhook.testSomeone presses Send test on the endpoint page. Delivered whether or not it is subscribed.endpoint_id, message

Every delivery body has the same envelope: id (the event id, shared by every endpoint’s copy and every redelivery), type, api_version, created_at, org, project (null for webhook.test) and the event-specific data. A run.finalized delivery, with whitespace added for reading:

json
{
  "api_version": "2026-09-01",
  "created_at": "2026-09-07T22:42:09Z",
  "data": {
    "blocking_regression_count": 0,
    "branch": "feature/swap-model",
    "client_run_id": "r_20260907_golden_ab12cd",
    "finalized_at": "2026-09-07T22:42:09Z",
    "git_sha": "a1b2c3d4e5f60718293a4b5c6d7e8f9012345678",
    "id": "aad9f868-62cf-4de6-92db-495cbd6d268e",
    "pr_number": 42,
    "regression_rate": 0.1,
    "source_model": "gemini-2.5-flash",
    "suite_name": "golden",
    "target_model": "gemini-3.1-flash-lite-preview",
    "verdict": "conditional_pass",
    "view_url": "https://www.evalshift.dev/app/acme/checkout/runs/aad9f868-62cf-4de6-92db-495cbd6d268e"
  },
  "id": "9cb7514d-7c5d-441c-b124-09afc731ee67",
  "org": {"slug": "acme"},
  "project": {"id": "8369bb25-64ca-4b5d-a913-06e99230a5d3", "slug": "checkout"},
  "type": "run.finalized"
}
data fieldTypeNotes
idstring (UUID)The server run id — the one every API path and URL uses.
client_run_idstringThe id the CLI generated, for tracing back to the local run.
suite_namestringSuite the run replayed.
source_modelstringProvider-qualified model identifier, as the CLI reported it. Likewise target_model.
git_shastring40 lowercase hex characters.
branchstringBranch name at push time.
pr_numberinteger | nullGitHub PR number when the run came from an Action; null otherwise.
verdictstringpass, conditional_pass, fail or inconclusive.
regression_ratenumberFraction in [0, 1] over blocking evaluator records.
blocking_regression_countintegerNumber of critical/high regressions from blocking evaluators.
finalized_atstringRFC 3339, UTC.
view_urlstringThe run page in the web app.
−
verify the raw bytes
The body is produced once per event as compact JSON — no whitespace, keys sorted at every level — and those exact bytes are what is signed and sent, on every attempt and every redelivery. Verify the bytes you received; never re-serialize the parsed object first.

Headers

HeaderValue
Content-Typeapplication/json
User-AgentEvalShift-Webhooks/1.0 (+https://www.evalshift.dev)
X-EvalShift-EventThe event type, e.g. run.finalized.
X-EvalShift-Event-IdThe envelope id.
X-EvalShift-DeliveryThe delivery id — unique per (event, endpoint, redelivery). Your idempotency key.
X-EvalShift-Signaturet=<unix seconds>,v1=<hex> — two v1= values during a secret rotation.

No other headers carry meaning. There is no header naming the org or project; read those from the body after verifying it.

Verify the signature

The scheme is Stripe’s, byte for byte, under a different header name — if you already verify Stripe webhooks, the same code works.

  1. +Parse X-EvalShift-Signature into t (Unix seconds when the request was signed) and one or more v1 values (lowercase hex).
  2. +Reject if |now − t| exceeds your tolerance. 300 seconds is the recommended default; every attempt is signed freshly at send time.
  3. +Compute HMAC-SHA256(key = secret, message = t + "." + raw_body), hex-encoded. The key is the whole whsec_… string — do not strip the prefix or base64-decode it.
  4. +Accept if any v1 equals the digest under a constant-time comparison.

raw_body is the request body exactly as received. Most frameworks parse JSON before your handler runs; make sure you get the bytes (FastAPI await request.body(), Flask request.get_data(), Express express.raw({ type: "application/json" })).

Python

python
import hashlib
import hmac
import time


def verify(secret: str, header: str, body: bytes, tolerance: int = 300) -> bool:
    timestamp: int | None = None
    candidates: list[str] = []
    for part in header.split(","):
        key, sep, value = part.strip().partition("=")
        if not sep or not value:
            continue
        if key == "t":
            if not value.isdigit():
                return False
            timestamp = int(value)
        elif key == "v1":
            candidates.append(value)
    if timestamp is None or not candidates:
        return False
    if abs(time.time() - timestamp) > tolerance:
        return False
    expected = hmac.new(
        secret.encode(), f"{timestamp}.".encode() + body, hashlib.sha256
    ).hexdigest()
    return any(hmac.compare_digest(candidate, expected) for candidate in candidates)

Node

js
const crypto = require("node:crypto");

function verify(secret, header, body, toleranceSeconds = 300) {
  const parts = header.split(",").map((p) => p.trim().split("="));
  const t = Number(parts.find(([k]) => k === "t")?.[1]);
  if (!Number.isInteger(t) || Math.abs(Date.now() / 1000 - t) > toleranceSeconds) return false;
  const expected = Buffer.from(
    crypto.createHmac("sha256", secret).update(`${t}.`).update(body).digest("hex"),
  );
  return parts
    .filter(([k]) => k === "v1")
    .some(([, v]) => {
      const candidate = Buffer.from(v ?? "");
      return candidate.length === expected.length && crypto.timingSafeEqual(candidate, expected);
    });
}

// Express: keep the raw bytes, ack fast, work later.
// app.post("/hooks/evalshift", express.raw({ type: "application/json" }), (req, res) => {
//   if (!verify(process.env.EVALSHIFT_WEBHOOK_SECRET, req.get("X-EvalShift-Signature") ?? "", req.body)) {
//     return res.status(400).end();
//   }
//   res.status(204).end();
//   queue.push(JSON.parse(req.body.toString("utf8")));
// });

Stripe’s SDKs can do steps 1–4 for you (stripe.Webhook.construct_event(body, header, secret) in Python, stripe.webhooks.constructEvent(body, header, secret) in Node) if you pass the X-EvalShift-Signature value where they expect Stripe-Signature.

Test vector

Unit-test a verifier without a live delivery. The timestamp is fixed, so pass a very large tolerance or freeze the clock at t. The body is 281 bytes of UTF-8 with no trailing newline. A verifier fed these four values must return true; flipping any byte of the body, the secret or t must make it return false.

text
secret:    whsec_test_vector_do_not_use_in_production
t:         1788782400            (2026-09-07T12:00:00Z)
body:      {"api_version":"2026-09-01","created_at":"2026-09-07T12:00:00Z","data":{"endpoint_id":"2b7f1c9a-6d3e-4e8b-9f0a-1c2d3e4f5a6b","message":"This is a test delivery from EvalShift."},"id":"4e5f6a7b-8c9d-4e0f-a1b2-c3d4e5f6a7b8","org":{"slug":"acme"},"project":null,"type":"webhook.test"}
signature: t=1788782400,v1=1f979779df1b05008f5f3974239f56980c052ebc7892e00e4de621af3efd57f6

Retries and timeouts

An attempt succeeds when the receiver returns a 2xx within 10 seconds of the request starting, TLS handshake included. Everything else is a failed attempt — 3xx (redirects are never followed), 4xx, 5xx, a timeout, a TLS or connection error, or a target that fails the address check at send time. That last one ends the whole delivery with blocked_target: no retry can change its answer, so the delivery goes straight to failed. Other failed attempts are retried with exponential backoff, eight attempts in total:

AttemptWaits after the previous failureElapsed since the first attempt
1—0
230 s30 s
31 min1 min 30 s
42 min3 min 30 s
54 min7 min 30 s
68 min15 min 30 s
716 min31 min 30 s
832 min63 min 30 s

Waits are minimums — attempts are picked up by a background worker that polls the queue, so expect each one somewhat later than the table says, never earlier. After the eighth failure the delivery is failed and is not retried again on its own; redeliver it from the endpoint page. Every attempt sends the identical body bytes under the same X-EvalShift-Delivery; only the signature differs, because it carries a fresh t. A 4xx is retried like anything else. Retries are per endpoint: a failing receiver’s backoff never holds back your other endpoints, though a slow attempt (up to the 10-second cap) can delay deliveries queued behind it by that long.

Idempotency

The same event will reach you more than once: on retry, on manual redelivery, and — rarely — when a 2xx you sent did not make it back before the timeout. Two ids let you choose the guarantee you want:

  • +X-EvalShift-Delivery is unique per (event, endpoint, redelivery). Deduplicating on it suppresses retry duplicates but lets a deliberate redelivery through. The right key for most receivers.
  • +X-EvalShift-Event-Id (the body’s id) is the same for every copy of an event, redeliveries included. Deduplicate on it when reprocessing must be a no-op even if an operator replays.

Keep seen ids for at least 24 hours. Return 2xx for a duplicate you chose to ignore — a 4xx would only schedule more retries.

Secret rotation

Rotate secret on the endpoint page returns a new secret once and starts a 24-hour window. During the window every delivery carries two v1= values, one per secret, in no guaranteed order — a verifier that accepts any matching v1 (both samples above do) keeps working whichever secret it holds. Update the receiver inside the window; after 24 hours the old secret stops signing. Rotating again while a window is open retires the older secret immediately. Rotation is recorded in the audit log; the current secret cannot be read back.

Auto-disable

After 5 consecutive deliveries that each exhausted all eight attempts, the endpoint is disabled with disabled_reason: "failing", and the org’s owners and admins are emailed once with the endpoint URL and a link to the org’s webhook settings. A delivery refused before it is sent, because the target now resolves to a non-public address or is otherwise no longer an allowed URL, counts the same as one that exhausted its retries. A DNS failure does not: it retries first, and only counts once its attempts run out. While disabled, new events produce no delivery for it at all; a delivery that was already queued is recorded as skipped with skip_reason: "endpoint_disabled". Any successful delivery in between resets the counter. Re-enable from the endpoint page once the receiver is fixed — that resets the counter too.

Delivery log and redelivery

Every delivery is recorded, whether it succeeded, failed or was skipped. The endpoint page lists them newest first; each delivery opens to the exact payload that was sent and one entry per attempt: HTTP status (absent when there was no response), error class (timeout, connection, unresolvable, blocked_target, redirect, http_4xx, http_5xx, http_error), duration, and the first 1 KiB of the response body. That excerpt is the most useful debugging tool you have — return a short reason when you reject a delivery.

StatusMeaning
pendingQueued, or between attempts.
succeededA 2xx came back.
failedAll attempts failed. Redeliver from the endpoint page.
skippedNever sent. skip_reason says why: endpoint_deleted, endpoint_disabled, or not_entitled (the plan had lost webhooks).

Redeliver sends the same event again as a new delivery: same event id and identical body bytes, a new X-EvalShift-Delivery, a fresh eight-attempt schedule, and a pointer back to the original. A delivery that is still pending cannot be redelivered. History is kept for 30 days.

Network

  • +Deliveries come from no fixed IP address. An allowlist by source address will not work. Authenticate deliveries by signature; that is the control.
  • +https:// is required. The certificate must be valid for the host and chain to a public root; self-signed certificates fail every attempt.
  • +Redirects are not followed. A 3xx is a failed attempt. Point the endpoint at the final URL.
  • +Private and internal targets are rejected, when the URL is saved and again before every send: userinfo or a fragment in the URL; hosts named localhost, *.localhost, *.local or *.internal; and any host that resolves to a non-public address. A host that merely fails to resolve is retried on the normal backoff.
  • +Ports other than 443 are allowed as long as the scheme is https://. Requests carry no cookies and no credentials beyond the signature. The body is not compressed.

API

Everything the endpoint page does is also an API call under /orgs/{org}/webhooks. Creating, changing, rotating, testing, redelivering and deleting require webhook:manage, which only a browser session holds — an API token can never register a URL that receives every future run. Reading endpoints and delivery logs (webhook:read) works with a session or an owner/admin token.

MethodPathPurpose
GET/orgs/{org}/webhooksList the org's active endpoints (no secrets).
POST/orgs/{org}/webhooksCreate an endpoint. Returns the secret once. 201.
GET/orgs/{org}/webhooks/{id}One endpoint.
PATCH/orgs/{org}/webhooks/{id}Change url, description, event_types, project_id, enabled.
DELETE/orgs/{org}/webhooks/{id}Soft-delete. 204.
POST…/webhooks/{id}/rotate-secretStart a 24-hour rotation. Returns the new secret once. 201.
POST…/webhooks/{id}/testQueue a webhook.test delivery. 202; 409 if disabled.
GET…/webhooks/{id}/deliveriesDelivery log, newest first. ?limit= defaults to 50 (max 200); ?before= takes the created_at of the last row seen. No payloads.
GET…/deliveries/{delivery_id}One delivery with its payload and every attempt.
POST…/deliveries/{delivery_id}/redeliverSend the same event again as a new delivery. 202.

Creating, enabling, testing and redelivering answer 402 when the org’s plan lacks webhooks. At most 10 active endpoints per org; a disabled endpoint still holds its slot. Creating, rotating, testing and redelivering are rate-limited per client address; a 429 carries Retry-After.

Best practices

  • +Acknowledge first, process later. Verify the signature, persist the raw event or hand it to a queue, return 2xx. The 10-second budget includes your TLS handshake and cold start.
  • +Verify before you parse. A body that fails verification is untrusted input; do not deserialize it, log it, or branch on its contents.
  • +Dedupe on the delivery id unless you have a reason to prefer the event id.
  • +Subscribe narrowly. Pick the events you handle and, if the receiver only cares about one project, scope the endpoint to it. Still ignore rather than reject a type you do not recognize.
  • +Send a test after every receiver change, and read the response excerpt in the delivery log when it fails.
  • +Rotate the secret when someone who had it leaves, and once a year regardless.
  • +Return a reason in the response body when you reject a delivery — you will see the first 1 KiB in the log.

Versioning

api_version is a date, currently 2026-09-01. It names the shape of the envelope and of each event’s data. Additive changes — new event types, new optional fields, new headers — never bump it; parse leniently and ignore what you do not know. Breaking changes bump it, are announced ahead of time, and endpoints are not moved to a new version without an explicit choice. The retry schedule, timeout, auto-disable threshold and retention window are operational parameters, not part of the versioned contract.