Local runs stay on your machine by default. doctor, run, evaluate, analyze, and report only call the LLM providers you configured (Anthropic, OpenAI, Google) using your own API keys. Nothing leaves your machine.

Hosted EvalShift (currently private alpha) is opt-in. When you run evalshift login and then push or all --push, the CLI uploads the immutable run_bundle.json.gz for that run to the hosted backend bound to your token. You can use EvalShift indefinitely without ever logging in.

The local SQLite cache at ~/.evalshift/cache.db only contains provider responses for your prompts and inputs; it is never uploaded.

Only what push uploads: the immutable run bundle (manifest, aggregate stats, per-example scores, your prompt outputs from both models, the rendered HTML report) plus server-side metadata it derives (run timeline, diffs against compatible prior runs, PR-comment text, audit events).

Provider API keys, your config secrets, and your SQLite cache are not part of the bundle and never leave your machine. Finalized runs are immutable; the only legal mutation is a soft-delete that hides the run from the timeline.

No. The whole local pipeline (doctor → run → evaluate → analyze → report) works exactly the same without an account. Hosted adds run history, web viewing, side-by-side diffs, and the GitHub Action’s PR comment + commit-status gate; skip it if you don’t need those.

From 0.4.0 onward, the CLI is AGPL-3.0-or-later. Versions 0.3.0 and earlier remain available under their original MIT terms. Local use, including commercial use, is fine; derivative works and network-hosted services built on the CLI must be released under the same license. Commercial dual-licensing is available on request. The hosted web app is a separate proprietary product.

The orchestrator records the error in raw.jsonl (with error="...") and moves on. The run still completes; failed calls are recorded with a neutral 0.5/0.5 score in the evaluation phase so the analysis can account for them rather than silently dropping examples.

Anything LiteLLM supports. The evalshift.models.registry provides friendly aliases and sane defaults for common models (Claude, GPT, Gemini), but the registry is advisory, not gating. A model id that isn’t in the registry — for example a fresh preview from a vendor playground — gets passed through to LiteLLM with a prefix-inferred provider. LiteLLM is the source of truth at call time.

Yes. evalshift run --resume finds the latest in-progress run for the project, validates that the config + suite haven’t changed since, and continues from where it left off. Already-completed calls (including ones that errored at the LLM layer) are skipped.

A config or suite change between attempts aborts the resume — start a fresh run instead.

• +Set the SQLite cache to be on (it’s the default). A re-run of the exact same configuration is free.
• +Use cheaper models. The model registry assigns sensible defaults but you can drop everything to flash/mini/haiku tier.
• +Skip the LLM judge. Structural and semantic evaluators are much cheaper. Drop the evaluators.llm_judge section to disable the judge entirely.
• +Cap with max_cost_usd in defaults (a future tightening will hard-enforce; currently a soft ceiling).

It means the id you passed isn’t in EvalShift’s curated registry. The id is sent to LiteLLM as-is (with provider prefix inferred from the prefix). If LiteLLM doesn’t know the model either, you’ll get a clean error from the provider when you make the call.

Two common causes:

• +Every delta is identical. When the variance in deltas is near zero, the test is skipped and severity defaults to none.
• +Your sample size is too small. With n < 5 the test is skipped and severity is insufficient (not none).

If you expected a real signal, double-check your evaluator output range — many “all the same” cases are evaluators returning a constant.

You don’t need LangChain to use EvalShift. v0.2 reads tool definitions from a yaml/json file (Anthropic-shape or OpenAI-shape — either works). If your tools are defined as LangChain Tool objects, export them to JSON Schema once and point tools_path at the result.

LangChain AgentExecutor auto-detection (read tools straight from the agent code) is deferred to v0.3.

Not in v0.2. The pipeline scores one assistant turn against another for the same user input. Multi-turn evaluation (where each turn might produce its own tool calls) is on the v0.3 roadmap.

They’re hidden — they’re development aids that we plan to relocate under a hidden --debug group at the v0.1.0 cut. Both still run if you invoke them by name.