This page is the data contract for EvalShift Cloud — what a push sends, field by field, and what never leaves your machine. If you need to clear EvalShift with a security or compliance team, this is the page to hand them. The CLI contains no telemetry: no analytics, no crash reporting, no phone-home of any kind.
The CLI opens network connections only when you ask it to, to at most three kinds of destination:
- +Your model providers (whichever you configure — any provider LiteLLM supports), using your own API keys:
runsends the rendered prompts and conversation histories to both models;evaluatesends both outputs to thesemanticembedding model and to eachllm_judgemodel — the judge also gets a multi-turn example's conversation so far — andtool_argumentssends free-text argument values to the embedding model;reportsends the worst regressions' inputs and outputs todefaults.insights_model(defaults.judge_modelwhen unset —gemini-3.1-flash-lite-previewunless you set it, not yourllm_judge's model; skipped when that model has no key) unless you pass--no-insights. On an Anthropic or DeepSeek scaffold the embedding model is OpenAI's or Gemini's. This traffic goes to your providers, never to EvalShift. - +EvalShift Cloud (
api.evalshift.dev, or your--host), only when you runlogin,whoami,push, orcompare --push. The local commands —doctor,run,evaluate,analyze,report,bundle— send nothing to EvalShift-operated services. - +Your object store, only with
captures.storeset:capture sync,capture listandcapture fetchlist and download captures from the bucket you named;doctor— and socompare, which runs it first — lists it once to check it is reachable. All of it uses your own cloud credentials. Nothing is written to it, and nothing from it goes to EvalShift.
## What push sends, block by block
push uploads one file, run_bundle.json.gz, plus request metadata: the bearer token (an Authorization header, sent only to the configured host), and, on the request that creates the run, the compressed bundle size and a copy of the bundle's manifest block. login additionally sends a client name that includes your machine's hostname, so you can recognize the session in the dashboard. In CI, the GitHub Action also sends one preflight request before the suite runs: the project slug, whether the repository is private (from the GitHub context unless you set repo-private), and a parallelism of 1. The bundle itself contains:
| Block | What is inside |
|---|---|
| manifest | Run id, org/project slug, source and target model ids, suite name, git commit SHA, branch name, PR number, the local suite file path as a string (it can reveal directory or user names), two content hashes, the run timestamp, and the CLI version. |
| examples[] | One row per prompt × example: the example's template variables (inputs) verbatim; its expected reference output verbatim; both models' full output text; the replay's own tool-call traces (tool names, arguments and call ids, each tagged with its round; final text and refusal messages; capped at 256 KB per side with a truncated flag — no tool results, no model-call events); per-evaluator scores, error strings and one-line explanations, and for each trace_invariants evaluator the rules each side broke (rule id, type, tool, round, detail; the detail line may quote an argument value); per-side cost and latency; tags and slice names. Agent traces imported with evalshift traces import are not included; the trace-rule verdicts computed on them are (see below). |
| aggregate · analysis · decision · economics | Pass/fail counts, statistical comparisons, the migration verdict with its recommendation sentences, and per-role token/cost/latency rollups — numbers, labels and generated advice, not your content. Since CLI 1.4.0 the recommendations include one line per evaluator skipped for a missing API key, naming its model and env var (never a key). decision.policy is the resolved migration_policy (null when the config has none) and carries max_invariant_violations (default 0). |
| methodology_notes | The model ids and the statistical-contract sentences shown in every report. |
| insights | The machine-written run narrative, when one was generated. It is prose about your run and can paraphrase or quote the regressions it summarizes. |
| evaluator_config | Config version; prompt list metadata only — prompt names, file paths, and variable names, with every prompt body replaced by a content_hash; defaults (model ids, concurrency, cache flag, cost ceiling, max_tokens, samples_per_example); an always-empty slices list, kept only so the config hash stays stable; and the full evaluators block, including each llm_judge criterion_prompt text — keep judge criteria free of secrets. trace_invariants rules and owner upload verbatim; an empty list is omitted from eval_config_hash, so adding rules changes the hash — a PR run will not diff against baselines until main pushes one with the rules. |
| dataset_snapshot | Suite path, example count, slice names, and one examples_hash. No example content. |
## What never leaves your machine
- +Provider API keys and the Cloud token. Neither is ever inside a bundle. Keys go only to their own providers; the token goes only to the configured host as an auth header.
- +Prompt bodies and system prompts. A
manualprompt's content is replaced by acontent_hash; apython_stringprompt's body never enters the config at all — only its file path and variable name do. - +Suite conversation histories (
history, including any embedded system message). The dataset snapshot ships hashes, not examples. - +Tool definitions. Toolsets — names, descriptions, JSON schemas — are not in the bundle; only the calls a model actually made at run time appear, in the traces.
- +Imported agent traces. Timelines attached with
evalshift traces import— tool results, retrievals, guardrail verdicts — stay in the local run directory. They feed theagent_traceevaluator andtrace_invariantsentries withtraces: imported, whose scores do upload — along with the trace-rule violation entries computed on them (rule id, tool name, round, and a detail line that for anargumentsrule quotes the offending value) — but the timelines themselves never enter the bundle. - +Local artefacts:
raw.jsonl(the raw provider requests and responses), the SQLite response cache,.evalshift/captures/(and, withcaptures.storeconfigured, the bucket you named that they are mirrored from — read with your credentials, never uploaded to EvalShift; the bundle is the same whether a capture came from disk or from the bucket),state.json,report.json, andreport.html.
The content hashes that replace this data (dataset_hash, examples_hash, prompts[].content_hash) are SHA-256 digests, so Cloud diffs and baselines still align across runs without the content itself uploading.
inputs, expected, both model outputs, and tool traces upload verbatim. If your suite rows contain customer data, or a model echoes a secret it was given at run time, that content is in the bundle — EvalShift cannot tell the difference. Redact at capture time with the SDK redaction boundary so sensitive values never reach disk, and inspect the bundle before pushing.## Inspect before you push
bundle builds the artefact without uploading, and push --bundle uploads exactly the file you inspected — the bytes are deterministic, so what you read is what is sent:
# build the exact artefact a push would upload — nothing is sent evalshift bundle <run-id> # read every byte of it gunzip -c .evalshift/runs/<run-id>/run_bundle.json.gz | jq . | less # happy with it? push exactly that file evalshift push --bundle .evalshift/runs/<run-id>/run_bundle.json.gz
If a run must not leave the machine, simply never push it — every local artefact, the HTML report included, works without an account. See Cloud setup for the push workflow itself, and the CLI reference for AI tools ↗ for the same contract in machine-readable form.