[bot] Merge master/9927b287 into rel/dev - #1825
Merged
Merged
Conversation
A run's output was a pass/fail table and a turn count, which is enough to know that something failed and not enough to know why. Reading a failure meant re-running the item by hand against the live agent. `gd-eval report`, and `run --html`, write one self-contained HTML file for a run or several side by side. Per item it shows the whole conversation rather than a count: every tool call with the arguments it was invoked with, every reasoning step, and each step's own wall time in execution order, so the pipeline the agent actually followed is visible. `--redact` drops conversation and response ids and raw reasoning and renames models to Model A/B for output that can leave the building. `timeline_detail` builds the breakdown and the tool calls together from one event list. They are index-joined -- the timeline carries only a name, and the join back to arguments is by index -- so building them apart would let them drift silently. Two wall-clock caps bound a run. httpx's timeout is per-read, so an agent streaming reasoning events resets it on every chunk and a runaway item ran 815s under a 300s client timeout. `--turn-timeout` bounds one turn and `--item-timeout` is a hard ceiling across all of an item's turns, anchored at conversation creation so a multi-turn item cannot spend N times the budget. Both default to uncapped, and `TurnTimeoutError` is not retried: a cap that fires is a verdict, not a transient failure. jira: AIS-48 risk: low Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Hand-authoring eval questions means writing a question and then guessing the
metric, dimension and filter it should produce, which is how a dataset ends up
full of questions the data model cannot answer.
`gd-eval generate` inverts that. It reads the charts a customer already built
via the declarative analytics model, translates each visible insight's buckets,
sorts and filters into an `expected_output.visualization` spec, and only then
asks an LLM to write the analyst question that chart answers. The expected
output is copied out of a live object rather than invented, so every question
is answerable in the real LDM by construction and the LLM only writes English.
Anything inexpressible is skipped with a printed reason rather than
approximated, and a question that contradicts its own spec is a hard error:
ranking words require a real sort, filter words a real filter, a breakdown
clause a non-empty view_by. One rewrite is attempted, then the item is dropped.
Every item is a copy of a real insight; nothing is derived or classified. The
definition came from the platform and is trusted as it stands: no gate compares
its title against its content, no shape is assigned, and no ranked variant is
synthesised from it. What the generator adds is the question and the checks on
that question, because the question is the one thing here the platform did not
write.
The sort a question asks for is now graded, which it previously was not.
`sort_by` was written into every fixture and read by nobody -- the evaluator
loads `expected_output.visualization` into `CreatedVisualization`, which
declared no such field and is configured `extra="ignore"`, so pydantic
discarded it and `strict_pass` covered cross-references, metrics, dimensions,
filters and chart type only. A question saying "sorted by Order id ascending"
-- seven of forty on one real workspace -- asked for something no check saw.
`AacQuery` gains `sort_by`, `check_sorts` compares it, and `strict_pass` counts
it. Entries stay raw dicts for the same reason `filter_by` does: the agent adds
keys this does not read, and a typed model would reject a chart that is
otherwise correct. The comparison uses the shape the agent emits, taken from
recorded runs: `{type: metric_sort, direction, metrics: [alias]}` and `{type:
attribute_sort, direction, by: alias}`, with one build sending both `by` and
`metrics` on a metric sort -- so the entry's own `type` decides which key names
the fields, never whichever key is present. Aliases resolve to uris and date
granularities fold to one spelling, as filters already do, and order is
significant: sorted by region then revenue is not sorted by revenue then
region.
The check is deliberately not symmetric with the filter ones. An empty
`sort_by` records that the fixture has no sort, not that the chart must be
unsorted -- a generated item inherits that emptiness from an insight whose
author sorted in Analytical Designer and saved without the sort sticking. A
spurious filter changes which rows a reader sees and is always wrong, while a
volunteered sort changes only their order, and ascending on a time axis is what
any renderer picks unprompted. So a required sort is enforced and a volunteered
one is free; the cost is that a wrong sort over an unsorted fixture goes
ungraded, the lesser error while `[]` cannot distinguish "unsorted" from
"unrecorded".
A tiebreak the agent appends after the recorded sorts is free as well: "state
descending" is satisfied by "state descending, then city", so the recorded
sorts must lead and match in order, and anything after them is not compared.
Grading a previously ungraded dimension means items that passed while omitting
a sort their fixture records now fail. Re-baseline before comparing a run
against an older one.
The declarative-to-AAC mapping is not ours. `convert()` calls the platform's
own `declarative_visualization_to_aac()` (gooddata-code-convertors, via
gooddata-sdk): both definitions the evaluator compares are platform output, so
the platform's conversion is the right owner, and on one production workspace
it converts every insight where the hand-written mapping had covered 44 of 62.
What stays ours is deciding what the evaluator cannot yet score -- derived
measures, measure-level filters, a repeater's label among its metrics -- each
skipped with a printed reason, and stripping the no-op filters AD saves for an
"All" selection, which would otherwise let a question claim a filter its chart
lacks. A map's `location` bucket is skipped on purpose: it holds a rendering
label, and a question built from it reads as "broken down by City pushpin
latitude". Chart type names are the convertor's, which are also the agent's.
One granularity is patched: the convertor maps `GDC.time.week_us` to `WEEK_US`,
which is not a platform enum; the SDK's own table says `WEEK`.
Two classes of question are unwinnable however well the agent behaves, and both
are reported: a name the model carries more than once (one workspace has six
labels titled "Product Title"), droppable with `--skip-ambiguous`; and a date
granularity's cyclical twin, since MONTH walks consecutive months while
MONTH_OF_YEAR stacks every January. Granularities move to `core/granularity.py`,
shared with scoring, which also folds `attribute/x.month` and `label/x.month`
to one uri -- a date dataset exposes each granularity as an attribute whose
only label carries the same id, and comparing the raw strings failed a chart
that was correct.
Alias resolution and the uri-to-title fallback come from `core/scoring.py`:
one copy, which is the one the evaluator scores against.
The package's `AGENTS.md` gains a section running the whole pipeline --
generate, run, report, models -- with the connection precedence, the snapshot
loop that makes generation iterable offline, and the environment variables each
subcommand reads. `generate` and `report` were not mentioned there at all.
`openai` joins the package's `dev` dependency group, so a plain `uv run` in a
fresh clone has the phrasing step and the LLM judge without naming the extra.
It stays under `optional-dependencies` for anyone installing from PyPI, and
every import site is still guarded or deferred.
Three fixes from the first live run over a customer workspace. A ranking filter
with no `attribute` ranks the full dimension tuple, so the "top N <dimension>"
shorthand in the writer's brief is only true with one dimension; with two it
told the writer a within-group scope the filter lacks, and the agent built what
the question said. `CreatedVisualization.id` is optional: the agent sometimes
omits it, nothing scores on it, and a required field turned a scorable chart
into an errored item.
jira: AIS-48
risk: high
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Hbj7SaGm6ucimov4NeMwqt
convert() returned a plain dict, so every reader downstream reached into it with subscripts and .get(), and a missing or misspelled key surfaced as a KeyError somewhere in the phrasing step rather than at the conversion that produced it. The spec is now a CreatedVisualization from end to end: _to_aac validates the convertor's output into it, and describe, the rule builder, the contradiction checks and build read it by attribute. CreatedVisualization rather than the convertor's own pydantic_models.Visualisation, which is a 23-way union whose members disagree on which buckets exist -- columns and rows only on the table variant -- so every read would need a variant check, and every scalar arrives wrapped in a RootModel. CreatedVisualization is also the shape the scorer and the envelope already use, which is what makes one model enough. AacQueryField gains type and filter_by as declared optionals; they were extras, and the derived-measure checks read them by attribute now. The envelope JSON is unchanged. One skip reason is not: an arithmetic measure carries a list in `using`, which no field of this model can hold, so it is caught as a validation error and reported as a field shape the evaluator cannot compare rather than a derived measure. jira: AIS-48 risk: low Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Hbj7SaGm6ucimov4NeMwqt
feat(gooddata-eval): generate eval datasets from insights, plus a readable HTML report
yenkins-admin
requested review from
hkad98,
lupko and
pcerny
as code owners
September 24, 2026 07:23
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Advanced Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## rel/dev #1825 +/- ##
===========================================
+ Coverage 82.58% 82.88% +0.30%
===========================================
Files 325 328 +3
Lines 20593 21235 +642
===========================================
+ Hits 17006 17600 +594
- Misses 3587 3635 +48 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🚀 Automated PR to perform merge from master into rel/dev with changes up to 9927b28 (created by https://github.com/gooddata/gooddata-python-sdk/actions/runs/35969340088).