Local runs stay on your machine by default. doctor, run, evaluate, analyze, and report only call the LLM providers you configured — any provider LiteLLM supports — using your own API keys. Nothing leaves your machine.

EvalShift Cloud is opt-in. When you run evalshift login and then push or all --push, the CLI uploads the immutable run_bundle.json.gz for that run to EvalShift Cloud, bound to your token. You can use EvalShift indefinitely without ever logging in.

The local SQLite cache at ~/.evalshift/cache.db only contains provider responses for your prompts and inputs; it is never uploaded.

Only what push uploads: the immutable run bundle (manifest, aggregate stats, per-example scores, your prompt outputs from both models, the tool-call traces behind them, the run narrative, and the economics rollup) plus server-side metadata it derives (run timeline, diffs against compatible prior runs, PR-comment text, audit events). report.html is not uploaded — it stays in the run directory for local viewing, and the web app renders the run from the bundle.

Provider API keys, your config secrets, and your SQLite cache are not part of the bundle and never leave your machine. Finalized runs are immutable; the only legal mutation is a soft-delete that hides the run from the timeline.

No. The whole local pipeline (doctor → run → evaluate → analyze → report) works exactly the same without an account. Cloud adds run history, web viewing, side-by-side diffs, and the GitHub Action’s PR comment + commit-status gate; skip it if you don’t need those.

From 0.15.0 onward, the CLI is Apache-2.0: free for any use, commercial included, with no share-back requirement and an explicit patent grant. Earlier releases stay under the terms they shipped with — 0.4.0 through 0.14.1 under AGPL-3.0-or-later, 0.3.0 and earlier under MIT. The capture SDK and the GitHub Action are MIT. EvalShift Cloud is a separate proprietary product.

The orchestrator records the error in raw.jsonl (with error="...") and moves on. The run still completes; failed calls are recorded with a neutral 0.5/0.5 score and an error note in the evaluation phase, and rows carrying that note are excluded from the paired statistics rather than counted as a zero delta. Outputs truncated at the max_tokens cap take the same path — scoring cut-off text would manufacture a regression that never happened.

Anything LiteLLM supports. The evalshift_cli.models.registry provides friendly aliases and sane defaults for common models (Claude, GPT, Gemini), but the registry is advisory, not gating. A model id that isn’t in the registry — for example a fresh preview from a vendor playground — gets passed through to LiteLLM with a prefix-inferred provider. LiteLLM is the source of truth at call time.

Yes. evalshift run --resume finds the latest in-progress run for the project, validates that the config + suite haven’t changed since, and continues from where it left off. Already-completed calls (including ones that errored at the LLM layer) are skipped.

A config or suite change between attempts aborts the resume — start a fresh run instead.

• +Set the SQLite cache to be on (it’s the default). A re-run of the exact same configuration is free.
• +Use cheaper models. The model registry assigns sensible defaults but you can drop everything to flash/mini/haiku tier.
• +Skip the LLM judge. Structural and semantic evaluators are much cheaper. Drop the evaluators.llm_judge section to disable the judge entirely.
• +Cap with max_cost_usd in defaults (a future tightening will hard-enforce; currently a soft ceiling).

It means the id you passed isn’t in EvalShift’s curated registry. The id is sent to LiteLLM as-is (with provider prefix inferred from the prefix). If LiteLLM doesn’t know the model either, you’ll get a clean error from the provider when you make the call.

Two common causes:

• +Every delta is identical. When the variance in deltas is near zero, the test is skipped and severity defaults to none.
• +Your sample size is too small. With n < 5 the test is skipped and severity is insufficient (not none).

If you expected a real signal, double-check your evaluator output range — many “all the same” cases are evaluators returning a constant.

You don’t need LangChain to use EvalShift. Tool definitions ride with each suite example: promotion carries the exact toolset the capture recorded (toolset_ref), and hand-authored suites can inline a tools list ({name, description, input_schema}). See Golden suite. A capture keeps three notions apart: the tools offered to the model, the calls the model requested in its response (requested_tool_calls, SDK 0.4.0+), and the calls the app executed. Promotion prefers the requested calls as ground truth when the capture recorded them on every model call, and falls back to the executed ones otherwise.

If you do use LangChain, the capture SDK ships a callback handler that records runs with no per-function instrumentation — see framework adapters. For an agent loop that runs entirely outside EvalShift, export the timeline and score it with evalshift traces import plus the agent_trace evaluator.

Yes, by teacher-forced replay: each turn is one suite example carrying the conversation so far in history, and both models are handed that prefix verbatim before the current turn. Only the current turn’s output is compared, so every turn stays a clean paired measurement. History carries the agent loop too — recorded tool_calls and their tool-role results are replayed, so the candidate sees the context production saw.

What is deliberately not supported is re-driving a whole conversation — letting the candidate generate turn 1 and feeding its own reply into turn 2. That breaks the paired comparison: the two models would no longer be answering the same question. See Golden suite.

They’re hidden on purpose — development aids kept out of the main help output. Both still run if you invoke them by name. validate also prints the CI pin-drift warning after its success line (advisory; the exit code is unchanged).

No. Since CLI 0.14.0 the CLI imports as evalshift_cli and depends on evalshift-sdk, so pip install evalshift installs both into one environment and import evalshift is always the SDK. A production agent that only records captures installs evalshift-sdk alone. evalshift doctor shows which package the import name resolves to and warns when it is shadowed by an older CLI’s leftover files.