Changelog

1.6.2 · 2026-10-27 · max_cost_increase and max_latency_increase are judged per example — the latency budget caps the typical example’s slowdown (the 20%-trimmed geometric mean of paired target/source ratios) and the cost budget the change in paired total cost, counting only examples both models answered without error, so the same run can get a different verdict (breaking) · the two budget rows report a signed observed value and count example pairs, not calls · the report shows the per-example figures beside the run-wide cost and latency changes · semantic and llm_judge no longer score a turn that ended in a tool call · a skipped test’s placeholders no longer print as measurements

1.6.1 · 2026-10-23 · the report’s Top regressions lists every regressed example, one card per example naming each evaluator that dropped it, worst first, up to 50 per prompt · a drop inside an evaluator’s own threshold is no longer a regression: those examples fold into a “within the evaluator’s threshold” line and their rows are not tinted · a card outlines the target’s trace in red only when its tool calls differ from the source’s · the per-example breakdown and Trace rules broken tables show 10 rows and unfold the rest behind a script-free “Show N more” label · more room between the report’s sections

1.6.0 · 2026-10-22 · captures.rounds in evalshift.yaml (first | all) sets how many recorded agent rounds the cases capture sync promotes replay over · bounded capture suites: captures.max_age and captures.max_cases keep each captured suite a capped sample of recent production on defaults.source_model, with suite freshness rows in doctor and compare; promoted case files record captured_at and source_model, which older CLIs reject — upgrade every CLI and the CI pin together · evalshift-sdk 0.7.0’s conversation folders are read beside the flat layout and fetched whole · init scaffolds the two limits · litellm pinned below 1.106 so tool-calling replays work on a fresh install · the narrative no longer reads a tool-calling reply as an empty output or a run’s total cost as a cost per call · a change that missed significance renders as an unconfirmed drop or gain instead of Equivalent

1.5.0 · 2026-10-15 · report and compare end with a one-line request to star the project on GitHub — interactive terminals only, so CI logs, the GitHub Action’s output and piped output never carry it

1.4.3 · 2026-10-13 · failure categories count distinct examples, not scored rows — one example that picked another tool reads “Different tools chosen (1 example)”, not 2, in the narrative and in Top regression causes · the cached run narrative is keyed on the figures it describes, so re-running evaluate or analyze after an evalshift.yaml edit regenerates it on the next report instead of rendering stale prose beside the new verdict

1.4.2 · 2026-10-12 · a blocking semantic evaluator no longer fails a run whose rows all stay within min_similarity — those rows enter the paired test as no change, so re-analysing an older run can lower its semantic severities · a fail caused by comparison severity alone names each comparison in its reason, and the recommendation names the actual cause · tool evaluators (including an inherited trace_invariants) skip pairs offered no tools instead of booking them unmeasured, and evaluate warns when one checked nothing · the run narrative is told which budgets were breached and which held, and a FAIL summary that blames a held budget is rejected and retried

1.4.1 · 2026-10-11 · semantic no longer reads formatting as drift: a code fence wrapping the whole output is dropped and JSON is re-serialized canonically before embedding, so fenced pretty-printed JSON against the same JSON compact scores 1.0 instead of ~0.73

1.4.0 · 2026-10-11 · compare, run and evaluate check every judge and embedding model’s API key before the first call — an advisory evaluator without one is skipped (skipped_evaluators), a blocking one exits 1, so CI that was green can turn red on upgrade (breaking) · init writes semantic active for every provider, borrowing OpenAI or Gemini embeddings for Anthropic and DeepSeek, and init --ci wires that key as a second secret · doctor gains an evaluator keys row; its captures.store row warns instead of failing compare · capture sync and promote read each provider’s own message shape (Claude blocks, OpenAI content: null and developer messages, Responses, LangChain) and skip unreplayable turns; capture sync --repair rebuilds cases an older sync stringified · store messages name the package to pip install, not an extra · a suspended account is named as such

1.3.0 · 2026-10-09 · trace_invariants evaluator: hand-written forbidden / required / order / call_count / arguments rules over tool-call traces, with owners and a “Trace rules broken” report panel · migration_policy.max_invariant_violations budget (default 0), scaffolded by init (older CLIs reject it — bump the action pin) · push uploads each score’s explanation and the trace rules each side broke · captures.store in evalshift.yaml mirrors captures from an S3 / GCS / Azure bucket before capture sync and capture list, plus capture fetch, --since and --offline (extras evalshift[s3|gcs|azure]) · doctor reports the store · resuming a run started by an older CLI fails on the config hash

1.2.2 · 2026-10-05 · push shows the upgrade prompt instead of a traceback when hosted plan limits are hit or the connection drops while auto-creating a project · repositories moved to the evalshift GitHub org; the CI pin check recognizes both owners

1.2.1 · 2026-10-01 · top regressions in the HTML report read as expandable — the worst one in each section starts open, with a “Show details” hint per row

1.2.0 · 2026-10-01 · DeepSeek is a supported provider (model registry, DEEPSEEK_API_KEY, doctor, init --provider deepseek) · tool-calling examples now served from the response cache, one entry per replayed round · top-level slices: removed from evalshift.yaml: it never had any effect, and a config that still sets it fails to load, naming the fix (breaking) · the scaffolded CI key now needs policy:read, not just run:create + run:read · doctor's evalshift.yaml row points at validate for the offending key and fix

1.1.0 · 2026-09-19 · the run bundle carries the migration_policy its verdict was computed under, so the hosted PR gate checks the same budgets (needs a server that accepts decision.policy — upgrade self-hosted servers first) · push warns when a run carries no policy and the gate would report inconclusive · thresholds removed from evalshift.yaml: a config that still sets it fails to load, naming the fix (breaking)

1.0.1 · 2026-09-18 · init --ci scaffolds the eval step with suite-name, so a suite’s own evaluators load in CI · login reuses a working stored token instead of minting a new one

1.0.0 · 2026-09-18 · first stable release — a written compatibility contract names what is public (evalshift.yaml, command names and flags, exit codes, documented artifact fields, the run bundle) and what is not · evalshift all renamed to evalshift compare, with all kept as a permanent working alias · suite selection now prints a ready-to-run command per wired suite

0.15.0 · 2026-09-15 · license switch AGPL-3.0-or-later → Apache-2.0 (0.4.0 through 0.14.1 remain AGPL, 0.3.0 and earlier remain MIT)

0.14.0 · 2026-09-10 · samples_per_example repeats every call and scores the mean · teacher-forced multi-round replay · runs record generation parameters a model cannot honour (fail_on_dropped_params) · tool-choice constraints replayed · judge-family and CI pin-drift warnings in doctor · import package renamed to evalshift_cli (breaking)

0.13.1 · 2026-08-28 · init --ci scaffolds a production-shaped GitHub Actions workflow: suite discovery, one job per suite, a single gate join check

0.13.0 · 2026-08-28 · initial public release

0.12.0 · 2026-08-21 · captured generation config replayed at dispatch (temperature, structured output) · init's default budgets loosened for a first migration · max_tool_argument_drift gains a Wilson interval · docs lead with the capture SDK

0.11.0 · 2026-08-14 · replay history carries the agent loop (tool calls + results) · tool_arguments against: expected · tool_selection mode: expected_set · promotion recovers expected text from the last model call · skipped tool-only turns no longer read as equivalent

0.10.2 · 2026-08-08 · capture sync dedup spans runs · identical slices collapsed before the BH family is built · latency budget labelled as a mean · AI-agent references documented in the repo

0.10.1 · 2026-08-04 · evaluate scores concurrently under defaults.concurrency · embeddings and judge verdicts hit the response cache

0.10.0 · 2026-08-03 · run insights (machine-written narrative, verified figures) · plan-limit upgrade prompt on push · leaner bundle contract — older CLIs are rejected by EvalShift Cloud

0.9.x · 2026-07-23 · GitHub Action reference wired into agent files · doctor row names the failing check

0.8.0 · 2026-07-23 · evalshift tools sync — AST-extract tool definitions from your Python and wire tools_path · field-level push validation errors

0.7.x · 2026-07-09 · run retention + evalshift runs clean · multi-turn suites (history, conversation_id, turn_index) with teacher-forced replay · evalshift --version reads the installed package

0.6.0 · 2026-07-03 · migration policy + migration_decision.json + --policy-gate · defaults.max_tokens and truncation detection (truncated calls excluded from statistics) · failure taxonomy

0.5.0 · 2026-05-18 · Cloud CLI flows: login, whoami, bundle, push, all --push · project + thresholds config · init --ci scaffolds the GitHub Action

0.4.0 · 2026-05-16 · license switch MIT → AGPL-3.0-or-later (0.3.0 and earlier remain MIT)

0.3.0 · 2026-05-12 · evalshift all single-command pipeline · reusable stage cores · orchestrator progress callback

0.2.0 · agent migrations · tool_selection / tool_arguments / tool_trace_structure evaluators

0.1.0 · structural / semantic / llm-judge evaluators · BH-FDR analysis · HTML report

0.7.0 · 2026-10-22 · conversation folders: a capture.agent_session turn with a conversation_id is written to captures/<suite>/conv_<16 hex>/, on disk and in an object store — requires evalshift CLI 1.6.0 or later, so upgrade the CLI first; EVALSHIFT_CAPTURE_LAYOUT=flat keeps the 0.6 layout (breaking) · sampling keeps conversations whole: with EVALSHIFT_SAMPLE_RATE set, a session with a conversation_id is sampled by a hash of that id, so every turn is captured or none is · max_captures and capture_ttl count conversations, evicting whole folders oldest first · the PyPI page links to evalshift.dev and the SDK docs

0.6.0 · 2026-10-11 · EVALSHIFT_SINK is now EVALSHIFT_CAPTURE_STORE, matching the CLI’s captures.store (the old name still works) · with capture on, an unusable store raises SinkConfigurationError at the first decorator, client wrapper, callback handler or configure() call instead of falling back to local disk (breaking) · messages name the package to pip install (boto3, google-cloud-storage, azure-storage-blob azure-identity), not an extra · MissingExtraError renamed MissingStoreDependencyError and evalshift.config.SINK_ENV replaced by CAPTURE_STORE_ENV (breaking)

0.5.0 · 2026-10-09 · ObjectStoreSink: EVALSHIFT_SINK=s3:// | gs:// | az:// ships captures and toolset sidecars to a bucket you own, for hosts whose disk does not outlive them (Fargate, Lambda, pods) · background upload queue, fail-open, one warning on the first failed upload · flush_captures() for shutdown hooks and Lambda · extras evalshift-sdk[s3|gcs|azure]; local disk stays the default

0.4.1 · 2026-10-01 · docs only — DeepSeek and evalshift compare naming, plus a docs-currency audit fix across DOCS.md, llms.txt, llms-full.txt, SCHEMA.md and README.md

0.4.0 · 2026-09-10 · provider client wrappers wrap_openai / wrap_anthropic / wrap_genai · requested_tool_calls on model_call · tool_choice, parallel_tool_calls and strict recorded · trace schema 2.1.0 · pip install evalshift now installs the SDK alongside the CLI

0.3.0 · 2026-08-23 · first release from the public repository · redact= required at every capture entry point (breaking) · toolset capture as content-addressed sidecars · trace schema 2.0.0, 1.x captures refused (breaking)

0.2.0 · 2026-08-07 · first release on PyPI · capture decorators, record_model_call and redaction · multi-turn conversation envelope (trace schema 1.1.0)