An outbound webhook is an HTTPS endpoint your organization registers with EvalShift. When something happens in the org — a run finalizes, a baseline is pinned or unpinned — EvalShift POSTs a signed JSON event to that URL. CI is the intended consumer: a run.finalized event lets you post a PR comment, page on a fail verdict, or feed a dashboard without polling the API.

Endpoints are managed at /app/:org/settings/webhooks (open the app) by owners and admins. Outbound webhooks are included on the plans that list them on the pricing page; an org whose plan does not include them keeps its endpoints listed and can disable or delete them, but nothing is delivered.

## Set up an endpoint

1. +Open Webhooks from the organization menu and press New endpoint.
2. +Enter the receiver’s https:// URL, pick the events, and choose whether it hears about every project or one.
3. +Copy the signing secret. It is shown once; if you lose it, rotate it from the endpoint page.
4. +Press Send test. A webhook.test event goes through the same queue, signing, retry and logging path as a real one, so a green test means the whole path works.
5. +Read the Deliveries log on the endpoint page. Every attempt records the HTTP status, the error class and the first KiB of the response.

## Events

An endpoint receives an event when the type is in its subscription and its scope is every project or the event’s project. webhook.test ignores the type filter.

Every delivery body has the same envelope: id (the event id, shared by every endpoint’s copy and every redelivery), type, api_version, created_at, org, project (null for webhook.test) and the event-specific data. A run.finalized delivery, with whitespace added for reading:

{
  "api_version": "2026-09-01",
  "created_at": "2026-09-07T22:42:09Z",
  "data": {
    "blocking_regression_count": 0,
    "branch": "feature/swap-model",
    "client_run_id": "r_20260907_golden_ab12cd",
    "finalized_at": "2026-09-07T22:42:09Z",
    "git_sha": "a1b2c3d4e5f60718293a4b5c6d7e8f9012345678",
    "id": "aad9f868-62cf-4de6-92db-495cbd6d268e",
    "pr_number": 42,
    "regression_rate": 0.1,
    "source_model": "gemini-2.5-flash",
    "suite_name": "golden",
    "target_model": "gemini-3.1-flash-lite-preview",
    "verdict": "conditional_pass",
    "view_url": "https://www.evalshift.dev/app/acme/checkout/runs/aad9f868-62cf-4de6-92db-495cbd6d268e"
  },
  "id": "9cb7514d-7c5d-441c-b124-09afc731ee67",
  "org": {"slug": "acme"},
  "project": {"id": "8369bb25-64ca-4b5d-a913-06e99230a5d3", "slug": "checkout"},
  "type": "run.finalized"
}

## Headers

No other headers carry meaning. There is no header naming the org or project; read those from the body after verifying it.

## Verify the signature

The scheme is Stripe’s, byte for byte, under a different header name — if you already verify Stripe webhooks, the same code works.

1. +Parse X-EvalShift-Signature into t (Unix seconds when the request was signed) and one or more v1 values (lowercase hex).
2. +Reject if |now − t| exceeds your tolerance. 300 seconds is the recommended default; every attempt is signed freshly at send time.
3. +Compute HMAC-SHA256(key = secret, message = t + "." + raw_body), hex-encoded. The key is the whole whsec_… string — do not strip the prefix or base64-decode it.
4. +Accept if any v1 equals the digest under a constant-time comparison.

raw_body is the request body exactly as received. Most frameworks parse JSON before your handler runs; make sure you get the bytes (FastAPI await request.body(), Flask request.get_data(), Express express.raw({ type: "application/json" })).

### Python

import hashlib
import hmac
import time


def verify(secret: str, header: str, body: bytes, tolerance: int = 300) -> bool:
    timestamp: int | None = None
    candidates: list[str] = []
    for part in header.split(","):
        key, sep, value = part.strip().partition("=")
        if not sep or not value:
            continue
        if key == "t":
            if not value.isdigit():
                return False
            timestamp = int(value)
        elif key == "v1":
            candidates.append(value)
    if timestamp is None or not candidates:
        return False
    if abs(time.time() - timestamp) > tolerance:
        return False
    expected = hmac.new(
        secret.encode(), f"{timestamp}.".encode() + body, hashlib.sha256
    ).hexdigest()
    return any(hmac.compare_digest(candidate, expected) for candidate in candidates)

### Node

const crypto = require("node:crypto");

function verify(secret, header, body, toleranceSeconds = 300) {
  const parts = header.split(",").map((p) => p.trim().split("="));
  const t = Number(parts.find(([k]) => k === "t")?.[1]);
  if (!Number.isInteger(t) || Math.abs(Date.now() / 1000 - t) > toleranceSeconds) return false;
  const expected = Buffer.from(
    crypto.createHmac("sha256", secret).update(`${t}.`).update(body).digest("hex"),
  );
  return parts
    .filter(([k]) => k === "v1")
    .some(([, v]) => {
      const candidate = Buffer.from(v ?? "");
      return candidate.length === expected.length && crypto.timingSafeEqual(candidate, expected);
    });
}

// Express: keep the raw bytes, ack fast, work later.
// app.post("/hooks/evalshift", express.raw({ type: "application/json" }), (req, res) => {
//   if (!verify(process.env.EVALSHIFT_WEBHOOK_SECRET, req.get("X-EvalShift-Signature") ?? "", req.body)) {
//     return res.status(400).end();
//   }
//   res.status(204).end();
//   queue.push(JSON.parse(req.body.toString("utf8")));
// });

Stripe’s SDKs can do steps 1–4 for you (stripe.Webhook.construct_event(body, header, secret) in Python, stripe.webhooks.constructEvent(body, header, secret) in Node) if you pass the X-EvalShift-Signature value where they expect Stripe-Signature.

### Test vector

Unit-test a verifier without a live delivery. The timestamp is fixed, so pass a very large tolerance or freeze the clock at t. The body is 281 bytes of UTF-8 with no trailing newline. A verifier fed these four values must return true; flipping any byte of the body, the secret or t must make it return false.

secret:    whsec_test_vector_do_not_use_in_production
t:         1788782400            (2026-09-07T12:00:00Z)
body:      {"api_version":"2026-09-01","created_at":"2026-09-07T12:00:00Z","data":{"endpoint_id":"2b7f1c9a-6d3e-4e8b-9f0a-1c2d3e4f5a6b","message":"This is a test delivery from EvalShift."},"id":"4e5f6a7b-8c9d-4e0f-a1b2-c3d4e5f6a7b8","org":{"slug":"acme"},"project":null,"type":"webhook.test"}
signature: t=1788782400,v1=1f979779df1b05008f5f3974239f56980c052ebc7892e00e4de621af3efd57f6

## Retries and timeouts

An attempt succeeds when the receiver returns a 2xx within 10 seconds of the request starting, TLS handshake included. Everything else is a failed attempt — 3xx (redirects are never followed), 4xx, 5xx, a timeout, a TLS or connection error, or a target that fails the address check at send time. That last one ends the whole delivery with blocked_target: no retry can change its answer, so the delivery goes straight to failed. Other failed attempts are retried with exponential backoff, eight attempts in total:

Waits are minimums — the queue is polled every few seconds, so expect each retry a little later than the table says, never earlier. After the eighth failure the delivery is failed and is not retried again on its own; redeliver it from the endpoint page. Every attempt sends the identical body bytes under the same X-EvalShift-Delivery; only the signature differs, because it carries a fresh t. A 4xx is retried like anything else. Retries are per endpoint: a slow receiver does not delay your other endpoints.

## Idempotency

The same event will reach you more than once: on retry, on manual redelivery, and — rarely — when a 2xx you sent did not make it back before the timeout. Two ids let you choose the guarantee you want:

• +X-EvalShift-Delivery is unique per (event, endpoint, redelivery). Deduplicating on it suppresses retry duplicates but lets a deliberate redelivery through. The right key for most receivers.
• +X-EvalShift-Event-Id (the body’s id) is the same for every copy of an event, redeliveries included. Deduplicate on it when reprocessing must be a no-op even if an operator replays.

Keep seen ids for at least 24 hours. Return 2xx for a duplicate you chose to ignore — a 4xx would only schedule more retries.

## Secret rotation

Rotate secret on the endpoint page returns a new secret once and starts a 24-hour window. During the window every delivery carries two v1= values, one per secret, in no guaranteed order — a verifier that accepts any matching v1 (both samples above do) keeps working whichever secret it holds. Update the receiver inside the window; after 24 hours the old secret stops signing. Rotating again while a window is open retires the older secret immediately. Rotation is recorded in the audit log; the current secret cannot be read back.

## Auto-disable

After 5 consecutive deliveries that each exhausted all eight attempts, the endpoint is disabled with disabled_reason: "failing", and the org’s owners and admins are emailed once with the endpoint URL and a link to the org’s webhook settings. A delivery refused before it is sent, because the target now resolves to a non-public address or is otherwise no longer an allowed URL, counts the same as one that exhausted its retries. A DNS failure does not: it retries first, and only counts once its attempts run out. While disabled, new events produce no delivery for it at all; a delivery that was already queued is recorded as skipped with skip_reason: "endpoint_disabled". Any successful delivery in between resets the counter. Re-enable from the endpoint page once the receiver is fixed — that resets the counter too.

## Delivery log and redelivery

Every delivery is recorded, whether it succeeded, failed or was skipped. The endpoint page lists them newest first; each delivery opens to the exact payload that was sent and one entry per attempt: HTTP status (absent when there was no response), error class (timeout, connection, unresolvable, blocked_target, redirect, http_4xx, http_5xx, http_error), duration, and the first 1 KiB of the response body. That excerpt is the most useful debugging tool you have — return a short reason when you reject a delivery.

Redeliver sends the same event again as a new delivery: same event id and identical body bytes, a new X-EvalShift-Delivery, a fresh eight-attempt schedule, and a pointer back to the original. A delivery that is still pending cannot be redelivered. History is kept for 30 days.

## Network

• +Deliveries come from no fixed IP address. An allowlist by source address will not work. Authenticate deliveries by signature; that is the control.
• +https:// is required. The certificate must be valid for the host and chain to a public root; self-signed certificates fail every attempt.
• +Redirects are not followed. A 3xx is a failed attempt. Point the endpoint at the final URL.
• +Private and internal targets are rejected, when the URL is saved and again before every send: userinfo or a fragment in the URL; hosts named localhost, *.localhost, *.local or *.internal; and any host that resolves to a non-public address. A host that merely fails to resolve is retried on the normal backoff.
• +Ports other than 443 are allowed as long as the scheme is https://. Requests carry no cookies and no credentials beyond the signature. The body is not compressed.

## API

Everything the endpoint page does is also an API call under /orgs/{org}/webhooks. Creating, changing, rotating, testing, redelivering and deleting require webhook:manage, which only a browser session holds — an API token can never register a URL that receives every future run. Reading endpoints and delivery logs (webhook:read) works with a session or an owner/admin token.

Creating, enabling, testing and redelivering answer 402 when the org’s plan lacks webhooks. At most 10 active endpoints per org; a disabled endpoint still holds its slot. Mutating routes are rate-limited per org; a 429 carries Retry-After.

## Best practices

• +Acknowledge first, process later. Verify the signature, persist the raw event or hand it to a queue, return 2xx. The 10-second budget includes your TLS handshake and cold start.
• +Verify before you parse. A body that fails verification is untrusted input; do not deserialize it, log it, or branch on its contents.
• +Dedupe on the delivery id unless you have a reason to prefer the event id.
• +Subscribe narrowly. Pick the events you handle and, if the receiver only cares about one project, scope the endpoint to it. Still ignore rather than reject a type you do not recognize.
• +Send a test after every receiver change, and read the response excerpt in the delivery log when it fails.
• +Rotate the secret when someone who had it leaves, and once a year regardless.
• +Return a reason in the response body when you reject a delivery — you will see the first 1 KiB in the log.

## Versioning

api_version is a date, currently 2026-09-01. It names the shape of the envelope and of each event’s data. Additive changes — new event types, new optional fields, new headers — never bump it; parse leniently and ignore what you do not know. Breaking changes bump it, are announced ahead of time, and endpoints are not moved to a new version without an explicit choice. The retry schedule, timeout, auto-disable threshold and retention window are operational parameters, not part of the versioned contract.