This page is the data contract for EvalShift Cloud — what a push sends, field by field, and what never leaves your machine. If you need to clear EvalShift with a security or compliance team, this is the page to hand them. The CLI contains no telemetry: no analytics, no crash reporting, no phone-home of any kind.
The CLI opens exactly two kinds of network connections, both initiated by you:
- +Your model providers (Anthropic, OpenAI, Google — whichever you configure), using your own API keys:
runsends the rendered prompts and conversation histories to both models,evaluatesends outputs to the embedding andllm_judgemodels, andreportsends the worst regressions' inputs and outputs todefaults.insights_modelunless you pass--no-insights. This traffic goes to your providers, never to EvalShift. - +EvalShift Cloud (
api.evalshift.dev, or your--host), only when you runlogin,whoami,push, orall --push. The local commands —doctor,run,evaluate,analyze,report,bundle— send nothing to EvalShift-operated services.
## What push sends, block by block
push uploads one file, run_bundle.json.gz, plus three pieces of request metadata: the bearer token (an Authorization header, sent only to the configured host), the compressed bundle size, and the thresholds from evalshift.yaml when set. login additionally sends a client name that includes your machine's hostname, so you can recognize the session in the dashboard. The bundle itself contains:
| Block | What is inside |
|---|---|
| manifest | Run id, org/project slug, source and target model ids, suite name, git commit SHA, branch name, PR number, the local suite file path as a string (it can reveal directory or user names), two content hashes, the run timestamp, and the CLI version. |
| examples[] | One row per prompt × example: the example's template variables (inputs) verbatim; its expected reference output verbatim; both models' full output text; tool-call traces (tool names and arguments; imported agent traces also carry tool results capped at 16 KB each, retrieval queries and documents, and guardrail verdicts; plus final text and refusal/error messages, capped at 256 KB per side); per-evaluator scores and error strings; per-side cost and latency; tags and slice names. |
| aggregate · analysis · decision · economics | Pass/fail counts, statistical comparisons, the migration verdict, and per-role token/cost/latency rollups. Numbers and verdict labels, not content. |
| methodology_notes | The model ids and the statistical-contract sentences shown in every report. |
| insights | The machine-written run narrative, when one was generated. It is prose about your run and can paraphrase or quote the regressions it summarizes. |
| evaluator_config | Config version; prompt list metadata only — prompt names, file paths, and variable names, with every prompt body replaced by a content_hash; defaults (model ids, concurrency, cache flag, cost ceiling, max_tokens); slice definitions; and the full evaluators block, including each llm_judge criterion_prompt text — keep judge criteria free of secrets. |
| dataset_snapshot | Suite path, example count, slice names, and one examples_hash. No example content. |
## What never leaves your machine
- +Provider API keys and the Cloud token. Neither is ever inside a bundle. Keys go only to their own providers; the token goes only to the configured host as an auth header.
- +Prompt bodies and system prompts. A
manualprompt's content is replaced by acontent_hash; apython_stringprompt's body never enters the config at all — only its file path and variable name do. - +Suite conversation histories (
history, including any embedded system message). The dataset snapshot ships hashes, not examples. - +Tool definitions. Toolsets — names, descriptions, JSON schemas — are not in the bundle; only the calls a model actually made at run time appear, in the traces.
- +Local artefacts:
raw.jsonl(the raw provider requests and responses), the SQLite response cache,.evalshift/captures/,state.json,report.json, andreport.html.
The content hashes that replace this data (dataset_hash, examples_hash, prompts[].content_hash) are SHA-256 digests, so Cloud diffs and baselines still align across runs without the content itself uploading.
inputs, expected, both model outputs, and tool traces upload verbatim. If your suite rows contain customer data, or a model echoes a secret it was given at run time, that content is in the bundle — EvalShift cannot tell the difference. Redact at capture time with the SDK redaction boundary so sensitive values never reach disk, and inspect the bundle before pushing.## Inspect before you push
bundle builds the artefact without uploading, and push --bundle uploads exactly the file you inspected — the bytes are deterministic, so what you read is what is sent:
# build the exact artefact a push would upload — nothing is sent evalshift bundle <run-id> # read every byte of it gunzip -c .evalshift/runs/<run-id>/run_bundle.json.gz | jq . | less # happy with it? push exactly that file evalshift push --bundle .evalshift/runs/<run-id>/run_bundle.json.gz
If a run must not leave the machine, simply never push it — every local artefact, the HTML report included, works without an account. See Cloud setup for the push workflow itself, and the CLI reference for AI tools ↗ for the same contract in machine-readable form.